Our product evaluations are here for one purpose alone -- to provide buyers with truly independent, technical, and critical assessments of the products they are considering.  What this means in practice is that some products get added to our services, and others quietly get dropped.

So for example in the past year we have added product evaluations in our ECM evaluation service for Docuware and FabaSoft for example, and will add Perceptive Software in an update later this month. At the same time, some product evaluations will not get updated, and some will eventually leave the service altogether; it's tough but that's the way it works.

You may think that big vendors have an unfair advantage here -- that firms such as IBM and Microsoft always get coverage -- but that is not actually the case. EMC, IBM, and Oracle do indeed get extensive coverage in our ECM evaluation service. They do so because they are regularly requested by our customers and we regularly see them on shortlists. It is exactly the same reason a smaller vendor like Alfresco is covered.  On the other hand, SAP gets covered by other analyst firms in their ECM services, but not by us. The reason is simple: no customers ever ask us, and we never see them on ECM shortlists.

Once selected for evaluation we appreciate the input of software vendor into the process, through conducting product demonstration and fact checking (for example), but if a vendor does not do this for us, we will evaluate their product regardless.

Some vendors are unused to and deeply resent truly independent and critical evaluations of their products. They are used to paying firms to write nice things about them. They are used to pulling the strings and having editorial control. Many IT vendors see analyst firms as nothing more than an extension of their PR activities. A few vendors deliberately ignore our requests for information and demos, but then at the moment of publication, engage in toddler-like tantrums complete with hissy fits and threats. They tell us our evaluations are full of factual inaccuracies, which on close examination are not questions of fact at all. They just don't like us pointing out their products' weaknesses. They tell us their customers love them, though it is of course their customers who have provided us with highly critical feedback.

Most vendors are more pragmatic. They take our evaluations as a form of free consulting and use our insights to improve their products and services. Though that's not our purpose, we certainly don't object to technology improving as a result of our work.

Deciding who to evaluate and who to leave out is a tough process at RSG, and can provoke a lot of internal discussion and some arguments, but our decisions are based solely on whether our customers want to read them, not on how much a software vendor spends with us.

There are many evaluation services out in the market, some of which purport to be "independent."  Yet as far as we know, the way we work is unique. As far as we know, no other analyst/advisory firm:

• Funds their own research (no supplier-paid trips to events, fancy meals, etc.)
• Refuses to work for suppliers/vendors (no consulting, advice, webinars, or report writing)
• Covers community open source and other products/vendors who would never purchase the research themselves

If not unique then we are a very rare breed indeed. The way we see it, all products have both strengths and weaknesses. There are no "best" products, though there is typically a "best fit" for you the buyer. At the end of the day, the only thing important to us at The Real Story Group is that you the buyer can purchase the right technology with your eyes wide open, in the full knowledge of its shortcomings as well as its merits.