I have never been accused of being a chef, cook, or baker. So, I am probably the least likely to write a food-related blog post. (I usually leave that to my colleague, Theresa.) However, more and more I have found myself defending the practice of the "bake-off."
In the context of a technology selection project, a "bake-off" is really a competitive proof of concept where the 2 vendor finalists are asked to mimic a real implementation project and actually implement some parts of their proposed solution. The idea is to use your ingredients (content), your bakers (digital asset managers), and your kitchen (your servers and your worker's real environments.)
The argument that other consultants, analysts, and many vendors make is that bake-offs can be too time and resource intensive - and ultimately they are a waste of time. Well they are right. They CAN be a waste of time for the vendors that don't get selected or the consultants and analysts that get paid only when a final choice has been made by their client. However, bake-offs are almost never a waste of time for the enterprise that is actually buying the technology.
Consider this experience we had while helping one of the largest hospitals in the United States select a new Digital Asset Management system for their marketing assets. After the vendor demo phase, the selection team identified two finalists, but Vendor A was the clear leader in the clubhouse.
It was tempting to just abandon the plan to do a bake-off and just start the implementation with Vendor A. However, in the interest of doing their full due diligence, they went ahead with a bake-off in which they asked Vendor A and Vendor B to do a 3 week project to implement 3 of their key use case scenarios. During the 3 weeks, the hospital's marketing and IT team worked side-by-side with the vendor's implementation team. At the end of the bake-off, the team summarized their findings.
Some of the key lessons learned during the bake-off were:
- While Vendor A's user interface had dazzled in the vendor demos, when it came to building out the hospital's most common use case, it was Vendor B's workflow and interface that resonated most with the hospital's marketing team
- When it came to using the hospital's real content (high-res photos from numerous photo shoots), Vendor B was able to demonstrate much faster ingestion times
- When it came to using each of the DAM systems via the marketing team's mobile devices (a common use case), certain key functionality was not available using Vendor A's solution (This never came up in the vendor demos)
- When the hospital's marketers reached out to each of the vendor's help-desk, Vendor A's help-desk never responded! But, Vendor B's help-desk responded immediately
- And, something that should never be underestimated: the hospital's team liked working with Vendor B's team more than Vendor A's team
The bake-off should be just one step of intense testing of a vendor's fit for your unique needs. The best selection projects test the solution at multiple points via custom scenarios in an RFP, custom (not canned) demos, and finally the competitive proof-of-concept or bake-off.
As you hear often hear RSG analysts say, there is no perfect vendor or perfect product. And I don't mean to cast Vendor B as a perfect fit in this case. But, they were a really good fit for this client and they would not have discovered how good a fit unless they had engaged in the bake-off. Vendor A was paid for their time and Vendor B used the bake-off work as a head-start on the actual implementation project. The hospital has since implemented their new DAM system and are extremely happy with Vendor B's solution and services.
In this case a 3-week head-to-head bake-off was a sufficient amount of time to get the information they needed to make a decision. In some cases, you'll want a longer time and in others, a shorter time will suffice. But, if you are in the market for new technology, please do not skip the phase where your people test the new system with your content in your real environments. As we've seen with dozens of successful selections, devoting the proper resources to conducting a proper bake-off before you commit to a solution will ultimately leave a much better taste in your mouth.