Real Story: Avoid Massive Checklists
Welcome to the latest in RSG's series of tech selection case studies. This story illuminates the problem when a feature-based vendor evaluation gets taken to its logical extreme.
Here's the background. We once advised a major U.S. federal government agency to select a new portal technology platform to underpin a hub for small business advice. We came late to the process after an initial round of vendor demos had failed to differentiate clearly among the bidders.
The Curse of Feature Checklists
The core problem was Excel. Or more specifically, the entire RFP as a 10-tab worksheet, with some sheets going hundreds of rows deep. Most of the tabs held feature requests, notably categorized by agency or department rather than customer personas, with a long series of columns annotating those features.
Nearly all the features were listed as “must have.” They were rigorously cross-tabbed to a long yet vague set of business objectives, but otherwise there was no prioritization.
I don't know who I felt more sorry for: the client or the bidders.
The bidding vendors didn’t know what to demo, although several gamely tried. Mostly, they just talked about their (voluminous) proposal responses, most of which consisted of declaring, for each row, “We can do that!”
Ultimately, we were able to recraft a more user-centered approach, with a narrower scope, that vendors could reasonably demo against.
Interestingly, the process uncovered some key internal limitations that ended up shaping the final selection criteria. When an agency executive declared, "I want the owner of a car wash in Rhode Island to only see content tailored to them," it fell to a plucky underling to explain, "Uh, we actually don't have any content for that person." Fortunately, some serious introspection ensued, and the requirements were adapted accordingly.
Three Lessons Learned
Lesson #1: Narrative scenarios help you clarify what you want for your customers, but also help you clarify the work you need to do internally to meet them there. In other words, spend less time in a spreadsheet and more time in a word processor. ðŸ˜‚
Lesson #2: Take the time to articulate a real business case. Getting that understanding involves information gathering from various stakeholder groups.
Lesson #3 While it’s critical to identify your requirements, it will prove even more important to prioritize them. Non-critical requirements can hijack the product selection process, by distracting you and vendors from what’s really important.
Some Specific Tips
- Stay away from long, feature-based checklists, and instead create scenarios with key personas
- Always start with the customer user experience and work your way back into enterprise systems management — they go hand-in-hand, but in a customer-centric world you want to work outside-in and not enterprise-out
- In an adaptive process, you don’t need to be perfect at this (or any other) phase, so focus your inquiry into stakeholders’ most burning problems or largest opportunities and then iterate as you test various solutions
- Pay attention to signals suggesting essential internal change management, and start adapting internally, before you license any new technology
RSG can help you make solid technology selection decisions. Review our services and ping me with any questions.
If you're selecting digital workplace or marketing/engagement technology, be sure to check out RSG's hard-hitting vendor evaluation research.