Real Story Group. Make Better Technology Decisions.

Kas Thomas

In search of a standard search syntax

27-Oct-2008

Tags: Enterprise Search, Industry Standards, Google Search Appliance

I've spent a lot of time recently (along with colleagues Theresa Regli and Adriaan Bloem) researching various information-access technologies and search products, and I had a bit of an "Aha moment" the other day. It occurred to me that while there are well-proven languages for querying structured data (e.g., SQL, XQuery, XPath), there is no universally agreed upon syntax for crafting ordinary keyword searches.

You might be wondering: Why do we need a standard syntax for keyword search?  After all, studies have shown that across a fairly wide range of topics, keyword queries tend to average just 2.3 words in length (in English, at least).  At first blush, that wouldn't seem to allow much room for linguistic interpretation. But in fact, even on a two-word query, there has to be some underlying assumption about whether the search terms should be treated conjunctively (connected by "and") or disjunctively (connected by "or"). A common behavior is to treat the terms as if they are connected by "and." There's no guarantee, however, that just because System A does it this way, System B will handle it the same way. And that's the crux of the problem.

It gets to be more of a problem as searches become more complex. If you're a Google user, you've probably found yourself, more than once, using Google syntax on a non-Google system only to run into strange results, either because the host system didn't honor a phrase in quotation marks ("New Year") as a single search term, as Google does, or because putting a minus-sign in front of a term didn't cause the search engine to suppress documents with occurrences of that word (again, a Google behavior), or whatever.

As much as you might want to believe that Google has created a de facto "standard keyword search syntax," it just isn't so.  Few commercial search offerings implement the Google syntax, and in fact even Google doesn't implement the same syntax on its various search pages. (If you've used Google Code Search, you know what I'm talking about.) And there are rudimentary capabilities Google doesn't support at all, such as arbitrary grouping and nesting of terms separated by Boolean operators.

A "search syntax for the rest of us" would have none of the complication of a formal query language; it would be totally unobtrusive, seldom (if ever) noticed by casual users, but very much appreciated by power users. It would make simple things easy and difficult things possible. If you needed to do a fuzzy search with negation and range-matching, you'd be able to do so (without learning SQL). But you could also continue to do simple "Google-style" keyword searches without even knowing that an advanced syntax exists. The essential point is, a system that honors a Standard Search Syntax could be counted on to behave the same way as any other SSS-compliant system. No unwelcome surprises.

Does the industry really need one more standard, at this point? Frankly, yes.  We need the data-query equivalent of International Sign Language. Right now we're just waving our arms.