"We use our enterprise search engine everywhere throughout the company," says Watts, search and syndication manager at the Austin-based company. "We've had enterprise search since before the dot-com era and outgrown many different tools along the way."
Image Credit: David ClarkThe company uses an engine from Fast Search & Transfer ASA (FAST) in Oslo to index a half-million internal documents and several hundred thousand more on its customer-facing sites. It's also in the process of indexing 100 million records in its data warehouse as part of a business intelligence project. On top of that index lie about 30 interfaces designed for specific applications.
"If you are using a Web search engine like Google or Yahoo or MSN, you have one interface into a variety of different types of documents," Watts says. "But with enterprise search, you can tailor it to your business needs."
One such interface is designed for tech support, an area where National Instruments spares no expense -- it hires engineers for front-line support. There is a special portal interface providing quick access to the information those engineers need to quickly answer customer questions. The search system also serves its customers in 35 languages.
"When we make these technologies -- like a fully searchable product catalog in a new language -- sales trend up dramatically in those regions," says Watts. "FAST enables us to put that kind of e-commerce information in front of the users."
Google and, to some extent, its rival Yahoo Inc. are, of course, the search engines dominating the news. Business sections track Google's stock fluctuations, and the February congressional hearings on its China operations were front-page material.
"Google has created a lot of visibility around search on the consumer side," says Matthew Brown, an analyst at Forrester Research Inc. "Companies are very excited about that and see that search can be a mission-critical application for the enterprise."
But what works well in Internet search doesn't always meet enterprise needs. So although Google does offer enterprise search appliances and just released two new low-cost versions in January, it's not necessarily where the news lies in terms of cutting-edge technology.
"In the basic information-retrieval area, Google is being very disruptive," says Brown. "It has come in at a very compelling price point, but they are not a product leader and are not differentiating themselves in terms of features and functionality."
Brown breaks down search into three main categories. There is the traditional function of simply locating and retrieving documents. Next comes the ability to do a deeper analysis of the data to locate patterns and trends, and to meet specific needs for business intelligence, regulatory compliance, discovery in legal cases or other areas. Finally, there is real-time monitoring and analysis of data, particularly for security and financial applications.
Google is strong in retrieval, but not in the other areas. Instead, companies such as Autonomy Inc., Endeca Technologies Inc. and FAST are leading the way by providing deeper analysis and the ability to integrate with other enterprise applications to improve employee productivity, meet compliance needs or drive business initiatives.
"These search engines have more analytic abilities to discover relationships among documents by picking up common terms," says Rita E. Knox, an analyst at Gartner Inc. "Many enterprises also need augmented capabilities beyond standard search, such as identifying what is a company name, a person's name or a geographical location."
Targeted search
Selecting the right search engine, therefore, relies on identifying business needs and finding the best match.
"Companies tend not to use a single product but a variety of products, based on what they are trying to do," says Knox.
For instance, National Instruments used four search tools before standardizing on FAST. Other organizations also implement multiple applications targeted to particular business needs.
Sutter Health, a nonprofit organization in Sacramento that operates 26 Northern California hospitals, is using Autonomy's software for competitive research. It analyzes newspapers, professional journals, health care industry Web sites and information on the group's internal bulletin boards. Jim Harrison, Sutter's vice president for business intelligence, says Autonomy is automating what had been a resource-intensive and manual process.
"It is significantly improving our ability to respond as a group to questions people are asking us," he says. "It gives us more time to sit down and analyze the data because we aren't sitting around figuring out where we need to file an article or where it was located after we filed it."
World Book Inc. uses search engines to provide content to customers. The Chicago-based encyclopedia publisher uses Endeca's search engine to improve customer service. Since the company is competing with free information services on the Web, it must provide a significantly better customer experience, says Chief Technology Officer Tim Hardy.
"Attracting and retaining subscribers requires that we offer an experience that helps users quickly and easily find desirable information while showcasing the breadth of quality content available," says Hardy. "High-failure search rates, slow performance and an inability to showcase the different types of available content with a homegrown solution were hurting our ability to scale the business and attract new subscribers."
It was also killing IT productivity. The staff spent 80 percent of its time on search maintenance and 20 percent on improvements.
World Book uses Endeca on a cluster of Sun Solaris servers to access content in 10 databases, including encyclopedia articles, images, maps and dictionaries, and then present that content to clients. It performs the indexing off-line and then pushes it to the production environment at least once a week. Hardy says that using Endeca has increased the search speed by a factor of eight to 10 while providing richer results.
"Content spotlighting showcases relevant results for all types of content in a single view," he says. "For example, a search for 'Iraq' returns encyclopedia articles on Iraq, plus maps of the country and region, recent photos, special reports on the war in Iraq and its aftermath, and audio files of President Bush sending troops to Iraq."
The new search engine has contributed to a 20 percent increase in sales, as well as a 30 percent to 40 percent reduction in technical support calls, says Hardy.
Factiva, a New York-based joint venture of Reuters Group PLC and Dow Jones & Co., uses FAST to analyze 5 million news items per month from 10,000 sources and feed that information to 1.8 million paying business subscribers. Factiva uses a four-step automated and manual process to ensure that everything is correctly categorized. Customers can receive the data either as an XML feed or a Web service for integration into their corporate intranets, or their CRM or competitive-intelligence systems.
"We work with the clients to integrate the content behind their firewall and build out their own intranet," says Clare Hart, president of the Dow Jones Enterprise Media Group. "FAST fit the profile for both the commercial product and the relationship we could have with out clients."
Spending on enterprise search is growing much faster than overall IT spending, according to Forrester's Brown. Nearly all major enterprise search vendors experienced a 65 percent to more than 100 percent growth rate last year, he says. The question is whether it will remain as a separate software category for long.
"Enterprise search is getting embedded in a lot of different solutions, like business intelligence, compliance, e-discovery, surveillance, security, fraud detection and quality management," Brown says. "There is a whole range of business-critical applications that go beyond just document retrieval, and search is playing an important role there."
Search vendors, therefore, may wind up becoming absorbed by larger companies. For example, Yahoo bought Inktomi in 2003, and Oracle Corp. bought TripleHop Technologies Inc. last year and plans to integrate that company's search functions into its database line.
"There are indicators that information access is moving toward absorption into applications," says Gartner's Knox. "This seems to be how Microsoft is heading with Vista and Office 12, and may be what the bigger players such as IBM and Oracle go for."
Sidebar
Back to the Web
While many organizations switch from a Web search to an enterprise search engine, the FirstGov portal has gone the other way. FirstGov, which acts as the official home page for the U.S federal government, is managed by the General Services Administration.
When the site launched in the fall of 2000, it used a custom search engine created by a foundation established by Eric Brewer, founder of search engine pioneer Inktomi. Then, in the spring of 2002, the site started using an enterprise search engine from Fast Search & Transfer that would index the data contained within all the government agency sites.
"We were crawling all the government sites ourselves," says Bob Keating, the GSA's search and program manager. "The model didn't scale well to what we needed to do in searching the entire government."
This past January, the site reverted to a Web search model. FirstGov now uses Vivisimo Inc.'s Clusty Web search technology with Microsoft Corp.'s MSN Web index. MSN already crawls all the government sites for its own index.
"We are using the MSN index filtered for government sites rather than doing any crawling ourselves," says Keating. "Rather than reinvent the wheel, we get the data from somebody who already does it really well."
As a result, the GSA has replaced the 24 servers it was using for the FirstGov site with a service hosted by Pittsburgh-based Vivisimo, cutting its costs in half. In the process, the number of official government documents available through the site has increased from 8 million to 40 million. It also provides searchers with easier access to relevant information by clustering the search results by category. For example, the search results page now has a tab listing federal forms that match the query.
"We still use [Autonomy's K2] enterprise search for the GSA.gov site," says M.J. Pizella, associate administrator of the GSA's Office of Citizen Services and Communications. "But for FirstGov, we wanted to get information quicker, without having to get into the agency sites, and Web search allows us to do that."
Robb is a Computerworld contributing writer in Los Angeles.