Real Story Group. Make Better Technology Decisions.

Adriaan Bloem

Searching for Terrorists

8-Jan-2010

Tags: Enterprise Search, , IDOL Server, Lucene and Solr, Government

What if finding that one bit of information is of vital importance? Well, it may not have been the main story angle taken on last month's attempted terrorist attack over Detroit in the United States, but we've certainly heard a lot about how better information management and cross-border collaboration might have prevented the attempted airline bombing.

Or as U.S. President Obama was quoted as saying, "This was not a failure to collect intelligence; it was a failure to integrate and understand the intelligence that we already had. The information was there, agencies and analysts who needed it had access to it, and our professionals were trained to look for it and to bring it all together."

How is it possible the dots weren't connected? We've grown accustomed to seeing movies and series where searches instantly bring up all the relevant information. Then there's hush-hush operations like ECHELON supposedly filtering all of our communications (we can say with some certainty it exists, but nobody really knows what it does). So we automatically assume that all the content is there, and through some inconceivably complex software, it leads to actionable information. You and I, of course, have no access to such technology, but certainly the intelligence agencies have it, right?

Maybe. In our Search & Information Access research, there's plenty of familiar software that we've heard about from major intelligence agencies in the U.S., UK and France. For instance, Autonomy will gladly tell you their IDOL is being used by many agencies (among them, the U.S. Department of Homeland Security, and also my local Dutch counter-terrorism agencies). Sinequa is used by the Ministry of Defense in France, and also correlates related crimes logged by Parisian police forces. Or there's the CIA, which has shown an interest in Lucene Solr; so they invested in Lucid Imagination (which specializes in that technology). And therein lies the point: in all these cases, the software doesn't just magically achieve the end result of successful crime or terrorism prevention. It takes work to be sure the technology is pulling the right information together, and even more work to make sure someone acts on it. 

So is a "lowered threshold for information considered important enough to put suspicious individuals on a no-fly list or revoke their visas" the real answer here? In information retrieval terms, that's increasing the recall at the cost of precision. And yes, a large part of this is an information access problem. It's about common problems such as uniquely identifying a person (especially if there are several different spellings of a name, as there was in this case). It's about connecting various databases and (geographically disparate) repositories, and federating search across them. And then once all that is achieved, and the relevant data is flagged: someone has to do something about it.

Your organization's need to retrieve, correlate and act on information may not be in an equally serious domain; still, you likely have a similar challenge. You need to not only decide what information is important, but how it should be weighted, correlated and acted upon. This is an ongoing challenge for those trying to get search right - and it's a challenge that never ends. Your information needs constant tending, trends need to be analyzed, and subsequent courses of action put in place.

It's the complete process of retrieving, correlating, and acting on relevant information -- not just finding it -- that makes all the difference.