Data-Driven Product Roadmaps: What Should I Measure?

Business Insider recently ranked the biggest product flops in history. Its list includes one failure that was so spectacular, it actually amassed some popularity. Pepsi had every reason to believe its clear, cola-tasting soda would be a hit – with market research proving so positive in the early 1990s, Coca-Cola launched a clear Tab competitor, according to Time. Yet even Van Halen couldn’t convince consumers that clear cola was something they needed, or even wanted, right now.

It’s easy to Monday morning quarterback such a product fail. But the best commentary comes from a refreshing mea culpa from the leader of the project himself. In describing the incident recently, David Novak, COO at the time, blamed Crystal Pepsi’s short run on a failure to listen to others in the organization who thought the beverage should taste more like Pepsi. He should have sought evidence to either prove or disprove some of the opinions that competed with his own to make better business decisions and evangelize the final decision with the team, according to an interview with Business Insider.

In other words, data was needed to validate, and perhaps even overcome, opinion.

It’s a situation that is familiar to you, I’m sure. As you evolve your product roadmaps, you’re taking input from lots of different people with different agendas – sales, marketing, executives, and even customers themselves. There is often something valuable in all of those thoughts, but figuring out what that is and delivering what is right for the customer while navigating that political minefield is hard.

We all know that having the right data is crucial to overcoming doubts and opinions, and to guide the product in the right direction. But your problem today likely isn’t that you don’t have enough data – it’s that you don’t have the right data. It’s easy to become flooded with vanity metrics when what you need are actionable ones. Here are some actionable metrics to start measuring that will help better inform and validate your product roadmap decisions.

Activity

Looking at the number of times your product was downloaded won’t tell you much, but looking at changes in the install base certainly will. The most basic metric to start with – activity – gives your business a strong foundation to begin targeting resources so that they will have the best return on investment. With a customizable view of how users are leveraging your software – correlated with parameters like operating system, region, language, and more – you can easily dig into product growth trends and recalibrate strategy when it isn’t tracking to growth goals.

Churn

With a robust view of data surrounding activity, you can begin to dig into one of the most valuable activity metrics – churn. Some of the greatest insights can be gained from finding out why customers who were interested in your software didn’t end up buying it. But it’s very difficult to get information from a churned user. Throwing a survey at him is a surely Sisyphean task, the data from which can never be truly validated. With robust, automated feedback collection, once a user leaves, it’s easy to figure out why. Breaking down a lost customer by breaking down the events that led to his exit – as granular as the number of sessions he spent with you down to the number of minutes in each of those sessions – will lend insight on where users are losing engagement with your product. For instance, one company discovered that complicated settings required by a configuration wizard had users giving up within minutes, before ever really using their product – a situation that was causing them to lose more than half of the customers who downloaded the product for a 30-day trial.

Feature usage

As a product manager, some of the most difficult decisions you have to make surround when and how to sunset functionality, especially when legacy features are leveraged by users in key accounts. Gathering data on how a subset of users is leveraging a product allows you to make a value-based analysis on whether keeping this feature is actuality a viable business decision. Plus, armed with data, marketing can develop offers to encourage customers to move to the new version, thereby creating openings for sales to offer new functionality available with the updated platform. Robust usage data not only informs planned obsolescence – it helps ensure that nuanced information is read the right way. Perhaps users aren’t leveraging a key feature during a trial of your software – is it because they don’t need it, or is it because they don’t know about it? With data that allows you to drill into usage patterns, you can target marketing campaigns to user personas to introduce users to functionality they may have overlooked.

This foundation of information will allow you to achieve what is on every product manager’s mind these days – deeper user engagement. With the right information, you can target education and training to help users get more value from their installations, and boost your reputation as a partner to your customers.

The solution is clear – embrace usage analytics, right now, to bolster your role as the product CEO, and stop giving into the highest paid person’s opinion to design products that no one wants. Leveraging data will allow you to make good technical decisions and operational decisions that will lead to irresistible products.

For more information on creating data-driven product roadmaps, please watch our webinar with Under10 Playbook’s Steve Johnson, “Using the Right Data To Drive Product Decisions.”