Real Story Group. Make Better Technology Decisions.

Adriaan Bloem

Scared of Spiders?

3-Sep-2008

Tags: Enterprise Search, Implementation, Industry Standards, Information Architecture, Google Search Appliance

My girlfriend really doesn't like spiders, so I regularly get to roll up my newspaper and squash one. I'm not so much afraid of them, but that's starting to change: they can be lethal to a website.

In Enterprise Search, we tend to use "spider" and "crawler" as synonyms: the piece of software that grabs a website's URLs and fetches the pages. The thing is, though, the spider should crawl your websites; it shouldn't bring your websites to a crawl. I was reminded of this when reading about Twiceler, the spider employed by new public web search engine Cuil. It seems that in an effort to be a Google-killer, Cuil has been hammering websites into submission.

Google itself has created its fair share of problems as well (you may remember the spider of doom which deleted a complete site). And often, it's reasonable to point the finger at your CMS or website implementation -- your website shouldn't run away frightened when confronted with one of these arachnids.

But more importantly, it's very easy to do the same thing yourself with your own site search engine. I recently demonstrated this when I ran a proof of concept for a consulting customer: the four-threaded spider I let loose on their webserver brought the site down, and most of the resulting index consisted of the dreaded "500 internal server error." I apologized -- even though it went otherwise unnoticed (the outage lasted for less than a minute) -- and counted myself lucky to be working with a product that actually let me tune the crawler to less aggressive settings.

If you're going to run your own search engine, it's important to know what its spider does and whether you can keep it under control. You don't want it to interfere with normal operations and you probably don't want it to crawl away where you can't find it anymore. And this is something you'll want to know even before testing the software: read up on the details first.

And in case all of this arachnophobia gets you into the sci-fi mood: remember that robots don't always deter spiders, even though technically, they should...