How to be a smart consumer of analyst research

  • 11-Aug-2014

Well, it's not exactly "a specter haunting the analyst world," but the pay-to-play issue in our industry has risen to the fore once more because of a vendor dragging Gartner Group to court over their business practices.

What’s the pay-to-play problem?

Broadly speaking, pay-to-play refers to a (usually unspoken) quid pro quo between industry analysts and the vendors they track. This can take the form of vendors purchasing consulting or other services from the analyst firm that evaluates them. The issue then becomes whether the research produced is (or ever could be) truly objective and neutral.

Before I come to that, note that analyst firms fall on a spectrum when it comes to the pay-to-play model.

  • On the one end are the “purely play” analyst firms; vendors engage these firms to produce reports for them and as long as the vendor/analyst firm are upfront and disclose who paid for the report, it’s all par for the course
     
  • On the other end of the spectrum, are the no-play firms that don't advise vendors; these are few and far between (RSG is one such firm -- please let me know of others you are aware of); here also, there is no doubt as to who the analyst firm is serving: the enterprise tech customer
     
  • Things become a bit trickier in the broad middle, where a major analyst firm seeks business both from the buy-side (technology buyers) and the sell-side (technology vendors). 

In the third scenario, the interests of these two masters clearly tug in opposite directions.

Another hazard also arises within the sell-side camp: when one vendor is a customer of the analyst firm, while the other is not, will the analyst firms offer “equal opportunity” coverage to all vendors, irrespective of whether they cut them a check or not? That's the core issue of the current dispute.

Of course, the credibility of an analyst firm is its biggest calling card and many leading analyst firms try to guard against the explicit conflicts of interest – but not always with success. 

 What’s your mitigation strategy? 

As a technology-buyer, you can guard against potential biases (explicit or implicit) that may have crept into analyst research.

  1. Use the standard 2-by-2 industry matrices (whichever name they may go by, in the end they are all 2 x 2 grids) as only a starting point for research, and don’t rely on them  for final selection or decision validation
     
  2. By extension, do not be unduly swayed by the positions of vendors in these 2 x 2 charts; the differences between the vendors could be marginal, and in any case, the rankings are generic and not in situational context
     
  3. You should seek a variety of sources of advice; this may bump up your workload a bit, but smart customers diversify and synthesize external input

When it comes to making decisions about technology suppliers, you always want to perform due diligence commensurate to the size of your investment. Take the same approach with your analyst firm as well.

Our customers say...

"I was really excited to preview RSG's Digital Marketing Technology evaluations. It's great to have a resource that explains what vendors really do, rather than what they say they do."

Gino Bona, Digital Marketing Consultant

Other Marketing Automation & Campaign Execution posts

What Is a Real Quadrant?

  • July 16, 2019

Customers have a love-hate relationship with marketplace "quadrant" diagrams. You suspect there's something not right in the arbitrary positioning...and you're correct!...

MD