Enterprise Search: Trends for 2008

By Adriaan Bloem at 2007-11-29 18:05:00 |

For those looking to get a quick glimpse at what's changed since earlier editions of the Enterprise Search Report, we'll offer a brief look at trends and options in the marketplace we discuss in the new 2008 edition of the report.

Overall, the search marketplace continues to surprise, sometimes delight, and often obfuscate prospective customers. Nevertheless, some key trends are emerging.

Refining the Experience

After nearly a decade of commentators criticizing the usability of enterprise search tools, within the past few years, vendors have begun to take note. I can't say that all innovation here constitutes improvement, but on the whole, we see a trend towards greater adoption of consumer search experiences.

Navigating Search

Search vendor Endeca may have applied for patents on "guided navigation," but the reality is that nearly every vendor can provide some degree of results categorization, be it clustering, browsable categories, or any other approach. The hype-du-jour is clicking a query together from the navigation built -- browsing the search results. Of course, distilling the terms for navigation is relatively easy when the source is structured content or when metadata can provide guidance. But it gets very much more difficult where unstructured data is in the mix: here vendors frequently differentiate according to their auto-extraction facilities that can glean facets and terms from your corpus without you having to tag documents. Your mileage will vary.

Analyzing Search

What is management without measurement? Seems obvious, but most search tools -- until recently -- had no built-in capacity to analyze the rich information found in their logs. Even today, there is wide variances among the tools.

Using analytics to improve search is a no-brainer. You can find out what people are actually looking for, what they are finding, what they are not finding, and how often. It's no replacement for focus groups and real user testing, but it's a fine complement. Unfortunately, even those systems with nice query-and-reporting interfaces often don't link directly to the administrative panels (e.g., for hit boosting or best bets) to redress the problems you find, but arming yourself with good metrics is a very useful start.

Uncrowding Search

Enterprise search vendors are consciously making their default search results look like Google's, even down to the green titles and blue hyperlinks. Why? Evidence suggests that searchers have more confidence in Googlesque results pages. Nevertheless, enterprise search is often more complicated than public web search; sometimes enterprise employees need more bells and whistles, including folders and categories and other data on the results page. That's one of the reasons why we take a scenario-based analysis to evaluating search technologies, and you should too.

Marketplace Ferment

The search marketplace continues to evolve in a herky-jerk fashion, with acquisitions (e.g., Convera to FAST and VisualSciences to Omniture) and bankruptcies (e.g., Mondosoft, Speed-of-Mind). Out of this ferment, some clear trends are emerging.

Escape from MOSS

Several enterprise search vendors did a great business by selling alternatives to the quite deficient search services within the first two versions of SharePoint. But now that Redmond has addressed SharePoint search aggressively -- at least on paper -- those companies are rapidly re-tooling to serve other customers. At least one of them, Mondosoft, did not execute well on the transition, and was purchased by a smaller competitor out of bankruptcy.

The irony is that MOSS Search is not a slam dunk. There is room for alternatives, even in Microsoft-oriented enterprises.

Decline of SaaS Search

Software-as-a-Service delivery models are hot all across the software landscape....with the exception of search. Early hosted player Blossom has faded to near obscurity. Hosted CMS vendor CrownPeak has pulled back from its (originally ambitious) plans for an advanced SaaS search offering. The grand-daddy of hosted search, Atomz, was acquired by e-commerce metrics company WebSideStory in 2005, who merged with niche analytics vendor VisualSciences, took that name, tried to sell off its assets, and was ultimately acquired by mainstream web analytics vendor Omniture. As of Q4, 2007, the future for VisualSciences' search customers remains quite unsettled. To be sure, some enterprise search vendors offer their "normal" product in a hosted environment, often in conjunction with a consulting or datacenter partner. However, these are not true, built-from-the-ground-up, multi-tenant SaaS offerings. They are managed services offerings using traditional, installed software.

There are a couple reasons for the demise of SaaS search. First, from day one it was really limited primarily to the website search scenario -- an important use case to be sure -- but not as lucrative or as high-value as multi-repository enterprise search, which depends heavily on the kind of WAN connectivity and in situ software connectors that SaaS vendors simply cannot provide today. Second, and perhaps more importantly: the dramatic rise of search appliance vendors (especially Google) has surely cut into demand for simple, web-oriented solutions of the type offered by SaaS vendors (Google itself put its basic hosted solution on hold for a few years while it waited for the Appliance to gather momentum). This doesn't mean you should avoid going the SaaS route; just sign on with your eyes open and make sure you understand the provider's future plans and focus.

Talking Search/BI Convergence

In the past year, the marketplace has seen substantial discussion and speculation about the "convergence" of big-time structured search (usually under the auspices of Business Intelligence tools) and big-time unstructured search (usually by traditional enterprise search products).

Enthusiasm was fueled when BI vendor Business Objects acquired text mining supplier, Inxight (Business Objects itself was subsequently acquired by SAP). To us, the convergence remains mostly talk. Most enterprise search tools that could retrieve information from unstructured datasets could also access and index databases as well. However, vice-versa was not always true: BI vendors -- and their customers -- don't always understand unstructured content.

Clearly, there are use-cases for this sort of convergence. For example, companies want to mine customer comments as well as customer data. But today, the two software segments remain quite distinct, mostly because customers are still trying to solve basic search and BI problems, before moving on to more advanced challenges.

Google Continues Long March to the Enterprise

This report contains a section focused on the Google Appliance, the Trojan horse of the enterprise search world. Google continues to improve its appliance -- and intimidate its competition -- even though (as you will see in our chapter), the appliance remains somewhat deficient as an enterprise search tool. Google's long-term plans for the appliance surpass simple search. Google can use it as a platform for other enterprise applications, perhaps some offered on a hosted basis. But it's a long way from the public web to the enterprise, and customers report that once you reach the limits of the appliance, you're pretty much stuck. It's a service in a box. Google found a great, underserved niche for simple search and has executed well on it. But it remains unclear whether Google the company truly understands the workaday needs of the enterprise.

Enduring Technical Challenges

One of the great surprises -- and, for many customers, disappointments -- of enterprise search is how technically challenging it becomes after you get beyond the basics. And you can get beyond the basics pretty quickly. Search and the discipline of information retrieval are among the most complex computer endeavors we face today. The nature of language itself is inherently baffling. Humans cannot make sense of some documents. Software doesn't do much better. But if technical challenges endure, then at least two trends in particular stand out today.

Platforms and Products Diverge

Is the fundamental problem with enterprise search a lack of power or a surfeit of complexity? The marketplace doesn't seem to be able to decide. Some vendors, such as IBM, continue to expand on their multifaceted search toolkits. Meanwhile, competitor Oracle is working to simplify its offering and conceal or abstract much of the underlying power. Neither approach is universally ideal, but it is a measure of a maturing market that vendors are tending to go one direction or the other. The good news for you the buyer is that you have clear choices: fulfill an immediate business need or develop long-term capacity. That debate is as old as enterprises have been adopting software. We make no judgments, but point out in our report that you have solid alternatives either way.

Application Management Conundrum

If search is a software application, then it should be managed like one. This sometimes comes as a surprise to even the most seasoned enterprise IT team who may labor under the misimpression that even a lower-end search product is "install and forget." Even the lowest-end search platforms -- and even appliances like Google -- can be configured and extended.

But, of course, configurations beg management and testing themselves. In short, enterprise search has the enduring need for proper software development lifecycles and code / configuration management.

Unfortunately, most (if not all) search platforms do not make configuration management simple. Most assume you will not have a development, staging, or QA instance of the search application itself. Many allow for overlapping code, command-line, and browser-based configurations and customizations, making it hard for administrators and developers to separate concerns.

As search tools get ever more powerful, the more it becomes incumbent on you to manage them with the same care that you manage your other complex, mission-critical applications.

Wrapping Up

So, looking at the problem of enterprise search, you'll continue to see new buzzwords, but mostly you will continue to face the enduring challenge of collecting, securing, understanding, and displaying content. Search technology does get incrementally better, but your repositories are getting monumentally larger. Treat search like the serious project it is before you invest in any tool.