Real Story Group. Make Better Technology Decisions.

Tony Byrne

Follow Tony on Twitter @TonyByrne

Five signs that vendors influenced an analyst report

14-Feb-2011

Tags: Portals and Content Integration, Implementation, Selecting Technology, Vendor Viability & Financials

There's a lot of skepticism today about the influence that vendors exert on traditional analyst reports, including product evaluation reports. That skepticism is well founded. But the problem is not that vendors bribe analyst firms outright. The game is more subtle than that. Vendors use their leverage over analyst firms to skew reports in favor of suppliers over customers.

[Disclosure: we offer vendors the opportunity to fact-check our evaluations of their products, but not influence our interpretations. We do not consult with, work for, advise, or shill for vendors.]

How do you know when a report or white paper was written more for vendors' interest than for you the customer? Here's five tell-tale signs.

1) When the report employs euphemisms for product shortcomings

Here's a dead give-away: when a report labels real weaknesses as "challenges."

This implies that dealing with buggy code or missing functionality is not actually career-deadening drudgery, but somehow a muscle-boosting activity, like training for a marathon or learning a new language. When you dig deeper you'll often find that the analyst actually meant challenges for the vendor to overcome. It's revealing that traditional analyst grammar puts the vendor as subject and the customer as object. Reports are often for the benefit of other vendors and investors.

A similar and only slightly better variant employs the word "caution." When we point out -- as we did recently for an Enterprise Search vendor -- that "the company is perceived by customers as arrogant and unhelpful," that's not a caution. It's a potential project snuffer for you the implementer.

2) When the report takes vendor promises uncritically

When confronted with a real weakness, vendors are no different from other companies: they make promises. Most commonly, they'll say, "we're going to fix this in the next version." Sometimes they do actually fix the problem, often they don't, but either way it gives them something to smooth a written critique in the meantime.

I've learned through hard experience that roadmap certainty is an oxymoron, for commercial products and open source projects alike. Anyone who lived through years of Interwoven TeamSite 6.x - 6.y can attest to eternal promises of "it's forthcoming" in a mystical (but ultimately mythical) Version 7. OpenText, Drupal, and Documentum customers can describe similar experiences.

So when you read "a fix is on the way," there's a good chance another kind of fix was already in...

3) When the report employs fake criticism

Analysts often avoid criticizing vendors by listing "challenges" that don't really matter to customers. Things like lack of visibility in markets that the vendor doesn't target in the first place, or a perceived lack of marketing acumen.

Analysts also love to poke at a vendor's "positioning." Positioning is a bullshit term with no relevance for you the customer. In vendor-speak, positioning translates to "watch what we say, not what we do."

Sometimes there's another game here. Citing a positioning or marketing problem represents a not-so-subtle critique of the vendor's product management -- a critique that might generate a sales call from that same analyst firm offering to advise the vendor for a very lucrative fee.

4) When the report doesn't grade on a curve

Typically a vendor or technology needs to be writhing on its deathbed to receive a less-than-average rating in a traditional analyst report. I won't ascribe all technology problems to poor vendor selection, but in a world where more than half of IT projects fail to meet their objectives, surely your choices are not just between "good" and "better."

Journalist reviews sometimes fall victim to this too: all the vendors rate between an 8 and a 10 on some fanciful numeric scale, leaving you the customer to divine real meaning from fractional differences.

Remember, the key criterion here is not a generic rating, but a supplier's fit for your particular circumstances.

5) When the report avoids taboo topics

In vendor-influenced reports, certain topics become taboo, and therefore go missing. How often do you see criticisms of things like scalability, security, customer support, and ease-of-use?

The perceived stakes here are very high. If enterprise customers notice a taboo problem in a major analyst evaluation, they may consider them deal-killers, so the terms become toxic, and analyst reports tend to avoid them. It's possible that industry analysts -- some of whom are former vendor product managers themselves -- simply get cautious about appearing to have dropped the big hammer on their brethren.

That's bad. We find these weaknesses among those most commonly raised by customers. To be fair, problems like scalability frequently represent implementation failings rather than inherent software shortcomings. Then again, it's the role of an analyst to distinguish between those two things. (That's why we tend to hire former implementers who can effectively sniff out what's what.)

You the customer should not make hasty judgements here without testing the solution first. Because the real story is this: scalability, security, usability, and support problems plague all of the nearly 150 tools we cover. The degree and nature of their toxicity will vary. You need to understand these weaknesses in context. Above all, as an industry we need to talk about them openly. That's what we try to do in our product evaluations.

What analysts (sometimes) respond...

If you sit down with traditional analysts over a beer they'll explain that there are many different types of reports, from a vendor-funded white paper to multi-vendor evaluations and surveys. Each variant is going to incur different types of vendor pressure, and most analysts fight their corner hard.

After the second beer, they may note that they're stretched too thin to dig into great detail on how various tools work ("and besides, customers don't want to know all that," they'll add).

After the third beer they may confide all sorts of things they find wrong with Vendors X, Y, and Z, but would never put in writing.

In such an environment, one way traditional analysts maintain objectivity is to remain charitable to all the vendors reviewed; hence the escape into euphemism.

What you should do

Even in our Web 2.0 world, software vendor evangelists and open source project cheerleaders dominate much of the public discourse about technology. Sure, more customer conversations are happening in public forums -- and that's a good thing -- but there's still a place for dispassionate, expert analysis (or we wouldn't be in this business!).

I encourage you to see through the fake gravitas of euphemism that characterizes most analyst (and some journalist) output today. Don't accept that you have to read between the lines to know what a reviewer really thinks. You're the one spending money on the tools, so:

• Insist on candor, if only to save your valuable time
• Ask hard questions, because real experts will have intelligent answers
• Look for tough critiques, the true sign of objectivity in a world where everyone claims "independence"

We'll try our best to meet you there.