The Ethics of Software Criticism

  • 6-Sep-2010

If there is anything I follow just as closely if not more closely than the world of content management, it is the world of food. If I'm not talking to customers about DAM vendors or watching demos of the latest software, you can safely bet I'm either shopping for food, cooking, hosting friends for a lavish multi-course meal, or keeping my lists of the most interesting places to eat in my favorite cities, from Chicago to Cape Town. (Admittedly, much like I track many DAM vendors not yet written up in our DAM Report, I also track restaurants in places I haven't been yet - because one never knows when I'll find myself in Rio.)

After the most omnipresent question in my life ("which DAM vendor should we choose?"), surely a close second is the SMS that crowds up my phone: "Theresa, I'm in (insert city/wine-growing region here). Where should I go eat?" 

I take the recommendations I give to my friends about where to go eat very seriously. I think first and foremost about the person I'm recommending the place to. I wouldn't send a colleague who doesn't eat seafood to a great sushi restaurant. I always check back with my friend to find out how the dining experience was, whether I pointed them to a cheap taco stand in California or a Michelin 3-star in Spain. My friends know I'm fiercely opinionated and uncompromising when it comes to food; I'm an omnivore who can offer an honest assessment on a gamut that runs from the vegan fare at a Cambodian restaurant to a meat-fest with speared animal organs at an Argentinean smokehouse. I'm always looking to expand my knowledge and be worthy of my friends' trust as an unbiased, informed source of opinion. The information they feed back to me about their eating experiences is added to the arsenal, helping others in the future.

The same is true of Real Story Group's customers and the information I impart about DAM vendors and the larger world of content management. I've talked about this before, using the selection of the right bottle of wine for dinner as an analogy. There is no "best" vendor or tool. The vendors we end up recommending to people vary greatly based on our customer's use cases, requirements, budget, geography, and many other factors.

But what I haven't talked about before is our own sources of information. When I recommend a restaurant, I've either eaten there myself, or people whose opinion I trust have. At RSG, our analyst team all come from implementation backgrounds. We've all worked for systems integrators or on teams that implemented content management systems, we've all gotten our hands dirty with these tools. But much like I do for my restaurant lists, I'm always going back to sources I trust. We're often talking to our customers, especially after we help them select a system, to find out how it's going, what it's been like to work with a vendor, what's worked well and not so well when it came to implementing a tool.

One of our largest set of research subscribers here at The Real Story Group are systems integrators themselves, who come to us as trusted sources for information around the technology they advise on and implement. We in turn get a better understanding of how these tools perform in ongoing conversations with them.

When we make a statement about poor customer support or a tool's weakness, the vendors we evaluate often get in a huff and ask, "where did you hear that from?" We protect our sources, but it's always nice to smile and say, "we got that straight from your customer," or "we tested your 30-day trial version ourselves." Our sources are valuable; first-hand evidence and experience is gold.

I also have my own trusted sources on food. Recently I've been reading through the books of the snarky and irresistibly opinionated Anthony Bourdain, and so on the occasion of the 100th episode of his food-centric show on the Travel Channel, I was inspired to write this blog entry and ponder the many parallels between my professional life and the chief activity of my leisure time:

1) Questioning the source of my information. Has my source used many content management systems over the course of many years? Then I'm going to talk to him or her for a long time. Is my source working for a company that's a partner with only two vendors, and gets an all-expense paid trip to the vendor's annual conference at a spa in Davos? Then you can bet I'll take what they say with an entire block of salt, let alone a grain, and use it for my next salt-encrusted barramundi. Similarly, I won't put much credence in my non-seafood-eating colleague's opinion on a restaurant that specializes in seafood, or a vegetarian's experience at a pig roast in Portugal. Pre-determined bias and preference must always be taken into consideration.

2) Skepticism. I'm inherently skeptical, especially when someone is trying to sell something expensive, like a car or a house or a half-million dollar software package. There's a reason house inspectors exist before you have to commit the money or sign the contract.  We're the software inspectors.

3) An astonishing number of people who don't question. Last week a Parisian friend of mine tried to convince me that studies of Nespresso coffee have proven that Nespresso is "the closest tasting to the coffee of the real restaurants of Paris." My answer? The studies by whom - the Nespresso marketing team? Did Nespresso sponsor the study? Is there a vendor logo on the top of that "independent research" white paper you're reading? I'm always surprised when people don't notice this, or question this - you should. Be a smart consumer. Develop your own opinion. I don't care what George Clooney says, I think Nespresso coffee tastes like paper. I don't need to spend hundreds of dollars on a Nespresso machine, I can spend $50 on a stovetop moka and a coffee bean grinder, and my coffee will taste better.

4) Buying business. Recently one of our customers in New York told me that Autonomy takes prospective customers out to Per Se for dinner. Per Se is one of the nation's most lavish and expensive restaurants, where arguably the greatest chef in America - Thomas Keller of the French Laundry in Napa Valley - heads up the kitchen. The tasting menu starts at about $300, before drinks. The distressing fact here isn't the cost of the meal, it's that vendors buy people's business. (They also try to buy analysts: in the last month, I've turned down two vendors who offered me free meals and tens of thousands of dollars in speaker fees.)

So let me repeat: prospective customers. These are people who haven't spent a dime with Autonomy, yet. This isn't a thank you. It's a sales tactic. Tempting though it may be, don't let that sway you, don't let yourself be bought. The fact is, if Autonomy isn't the right vendor for you, you'll save a lot of money, and can put that towards going out to Per Se without a sales guy talking your ear off about meaning-based computing.

The ethics of critics and evaluators, and the importance of truly impartial opinion, is a topic close to my heart - one I could debate for hours over a multi-course meal and a great bottle of wine. But I'll let Anthony Bourdain sum it up for me