Real Story Group. Make Better Technology Decisions.

Tony Byrne

Follow Tony on Twitter @TonyByrne

No substitute for real-world testing

14-Dec-2010

Tags: Digital and Media Asset Management, Document Management (ECM), Enterprise Collaboration & Social Software, Enterprise Search, Portals and Content Integration, Web Content and Experience Management, Selecting Technology

Sometimes our research subscribers will ask us to join them for vendor demos during a technology selection process, to help them by serving as "truth detectors."

After a decade of doing this, I can assure you that there are two types of demos that vendors give.  The first kind are the ones they roll out during canned analyst / media "briefings."  And then there's the demos they perform for customers during the sales process. Those two are as different as night and day.

I'm not talking here about the demo gods conspiring to interrupt a vendor's pitch. I've come to expect the occasional glitch in vendor demos, and you should too.

There's a deeper problem to consider upon reading the typical analyst report: software is simply not as good in real-world conditions as in artificial environments. This goes for SaaS and open source offerings too. The minute you tread off the beaten path -- and we encourage all customers to script highly relevant, custom demos -- you expose a tool's real shortcomings and difficulties.

There's a reason for this. Software development is all about trade-offs. What's useful for one customer will become a liability for another. To paraphrase Tolstoy, every solution becomes unhappy in its own way.

Vendors (and open source project leaders) frequently take umbrage at the weaknesses we cite in our evaluation reports. "How can you say that our UI is difficult to customize?" they'll exclaim. Well, because we saw it in the real world, and not in your lab.