Product Launch Metrics
As a product manager, you’ll inevitably encounter a situation where you’ll need to introduce a brand-new product to the market. In cases like these, you’ll want to know upfront how to measure the success of your product launch.
Many resources focus on how to measure the success of a mature product, but few discuss how to determine whether a newly launched product has been successful. That’s why we’ve written this essay to help you decide which metrics to use for a newly-launched product.
First, we’ll cover why new products are different from mature products. We’ll then use this knowledge to determine which kinds of product launch metrics to measure. Finally, we’ll discuss how to prioritize these metrics, and we’ll share other insights to track as you launch.
But before we get into it, let’s start from the basics. Qualitatively speaking, what exactly does “success” mean for a product launch? What are we trying to learn when we launch a product?
What’s the hypothesis of the product we’re launching?
The whole point of a product launch is to ensure that our product scalably addresses pain. To be clear, when we launch our product, the goal is not to sign up as many people as possible, or to generate hype and buzz.
Rather, we should be using product launches to move past an initial beta phase and to see whether the product is well adopted “in the wild” without the product team’s intervention.
For context, product teams regularly use beta tests or pilot groups to quickly iterate on their product alongside users, and the value of these groups is that the product team can get real life user feedback.
But, the problem here is that they’re working with a captive audience where the users are highly incentivized to keep using the product.
The point of the product launch is to identify whether the product will do well without the product team’s direct involvement. So, it’s helpful for us to remember that beta tests and pilot groups are not product launches.
Using this knowledge, we can more carefully define a product launch as “a large-scale release of a product to the market, where the product team does not have sustained direct interactions with users.”
Now we know that the point of the product launch is to identify whether the product can sustainably create value on its own. We can use this lens to come up with metrics that prove our progress towards the goal.
But, why can’t we just use the same metrics that mature products use? Well, let’s discuss how newly launched products are different from mature products.
Why are product launches different from mature products?
Launching a product is fundamentally different from building on top of an existing product, because a new product launch means operating with significant uncertainty.
Specifically, you’ll face uncertainty on these axes:
Lack of pre-existing user behavior data
Insufficient evidence for whether you have the right audience
Insufficient evidence for whether you’ve targeted the right pain
Insufficient evidence for whether your proposed value proposition and feature set resonates with what the audience seeks
Due to these differences, we want to implement specific sets of product metrics to help us determine the success of our product launch.
After all, the goal of using metrics is to prove progress against qualitative objectives, and our objective is to determine “does this product scalably solve pain for our targeted audience?”
What are the kinds of metrics to measure for product launches?
When launching new products, we should focus on three categories of metrics:
Conversion
Retention
Referral
Within each category, I’ll provide one example of a metric that you can actionably move.
Each of these metrics proves out whether the audience we’ve selected feels that our product resonates with them, and whether our product is attractive enough for them to adopt and eventually promote to others.
First off, when we think about conversion, we need to remember that newly launched products need some way to monetize, and therefore have some "desired action" we want users to take.
Every product must ultimately capture value for your organization, and the best way to capture that value is to clearly define which user action is the highest priority within your product flow. Therefore, understanding the % conversion rate for that desired action is really important.
To be clear, conversion rate here is not “how do we convert inbound traffic into using our product”, as I define that as user acquisition rather than conversion.
User acquisition is not that important for a product launch, because the key hypothesis of a product launch is that this product solves pain. We can always add more users later through marketing, but actually identifying that the product is solving pain is not as straightforward.
The reason why I don’t recommend directly measuring monetization is because many products don’t monetize off of the end user immediately.
As an example, consumers can use Google Search for free, but advertisers pay to be exposed in these searches. Therefore, the key conversion to measure for Google’s search product is “how frequently are users searching on Google”, as it leads to downstream monetization.
For some folks, they might classify conversion as engagement - that is, they’re saying that users are taking some action in the product. That’s completely fine to use, but the key is to make sure that we’re not treating all engagement as equal.
What I mean is that your product will necessarily have quite a few different pieces of functionality. Too frequently, I see engagement metrics that don’t have a particular point of view - for example, “did a user click on this sort/filter button”, which has nothing to do with whether you can monetize off of this user’s behavior.
That’s why I’m purposefully using the terminology of conversion to identify whether we’ve created value for the users who have encountered our product.
So, based on your product’s functionality, decide which action is the one that creates the most value, and then measure that feature’s usage in terms of absolute numbers (how many times it’s used per week) as well as in relative numbers (how frequently it’s used vs. the total user base).
Second, retention is valuable to determine whether people have actually adopted our product and changed their behavior. We want to see repeat usage and repeat behavior.
By looking at the churn rate of different cohorts of users, we can quickly determine whether our product is actually truly driving engagement.
Why aren’t we looking at total engagement metrics? Well, the problem with total-level engagement metrics (e.g. weekly active users or WAU) is that this particular number can move up and to the right if we’re pulling in new users faster than current users are churning. But, it doesn’t tell us whether current users have adopted the product over time.
So, we want to measure the % of people who are coming back within a given time period (e.g. every week or every month).
Generally, the longer the time period, the more people will likely come back to your product, since you have a longer period of time to reactivate them. You’ll want to compare apples to apples: only compare 7-day retention in month 1 vs. 7-day retention in month 2, and don’t compare 7-day retention in month 1 vs. 30-day retention in month 2.
One caveat to keep in mind when it comes to retention: if you work on a highly seasonal product e.g. filing taxes, or a low frequency product e.g. applying for a mortgage, you may not be able to see retention from the consumer side within a reasonable period of time.
In that case, retention metrics need to be deprioritized because they’ll give you less signal, as you have to wait longer to see whether users come back to your product.
Therefore, if you’re tackling a use case that has very long cycle times, you’ll want to focus instead on the upstream actions that predict the conversion event, e.g. login or progress through the workflow.
Third, referral metrics are powerful ways to determine whether your product is so valuable that it derives word-of-mouth and virality.
Generally speaking, we don’t want to measure net promoter score (NPS) as a leading indicator of referral metrics, because NPS takes a while to resolve, and we don’t necessarily want to bombard users with surveys as soon as we launch the product.
We’d prefer to have them successfully use the product multiple times, rather than use the product a single time and then feel turned off when they’re met with a survey.
But, something that we can do right out the gate is to give users the ability to invite other users into the product, whether they’re doing so through email invites, through social media, or through the product itself.
We’ll want to make sure that we have this kind of capability because that can demonstrate that users are finding so much value in our product that they’re willing to put their social capital on the line to advocate for our product.
Therefore, we want to measure invite rate - how frequently users are inviting other users into the product. Over the next few months, we’ll want to see the invite rate slowly increase as we drive conversion rate and retention rate upwards as well.
So, we’ve now covered conversion metrics, retention metrics, and referral metrics.
For all of these metrics, keep in mind that launching a product means moving “from zero to one.” If your product org has historically launched similar products in the past, be sure to consider historical launches as a benchmark for success.
However, you typically won't have a good baseline beforehand for "how much it should change,” since new product launches are typically oriented towards different kinds of users or different kinds of use cases, and therefore you won’t have relevant prior learnings to draw from.
Let’s now talk about how to prioritize these three metrics when it comes to identifying “what should we ship next in our newly-launched product?”
How should I prioritize these metrics?
The most critical metric to prioritize first is conversion. The whole point of the product is to create so much value for users that you can monetize off of their behavior. Therefore, if you don’t have any sort of monetization or conversion, then your launched product won’t be able to grow sustainably.
Your first order of business is to drive up conversion rates through the product. You can do so by eliminating friction in the “primary workflow”, e.g. by taking away steps such as account creation by using single-sign on, or by automating data inputs.
Another way to drive up conversion rates is to have your supporting workflows point back to the primary workflow. As an example, let’s say that your primary workflow is to have a consumer fill out enough information to get mortgage rate quotes from different lenders.
You might have a secondary workflow that helps consumers understand what interest rates are, or you might have a secondary workflow that helps consumers understand how much their current home is worth. For those secondary workflows, make sure that the end of those workflows point towards the primary workflow so that the end user takes the action that you want them to.
As an example, when you first use Twitter, you’re asked to first share your interests, which is a secondary workflow. After you add enough interests, you’re then incentivized to like the different tweets that appear in your feed, which is the primary “conversion action” that drives value for Twitter as a business.
After you’ve been able to convert usage into revenue, you’ll then want to ensure that you’re getting retention. That way, you know that your product isn’t just being used due to a novelty factor; rather, it’s something that users truly find valuable. The whole point of your product launch is to generate sustainable long-term value.
As an example of what happens when you focus solely on conversion without considering retention, consider Clubhouse. While it had huge initial conversion rates on their core action of “users creating rooms with audio discussions”, it failed to retain them for sustained periods of time, leading to significant negative business impact.
Lastly, you’ll want to look at increasing referral metrics upwards. Word-of-mouth is still one of the strongest engines of organic growth for any product, and referrals prove that the engagement in your product is valuable enough for users to recommend it to other users.
To summarize: We don’t want to try to increase engagement when we’re not getting conversion upfront, as that’s not the fastest way to move the needle.
And, we don’t want to try to increase referral rates if we’re not getting engagement, as referrals won’t happen unless your product is valuable enough for users to use it multiple times.
Now we have a good grasp of the metrics that we should prioritize. But, metrics alone won’t tell us the full story of whether our product launch successfully “solved the targeted pain in our targeted audience in a scalable way.”
What else should I be analyzing during a product launch?
Qualitative feedback is particularly crucial for newly-launched products. During product launch, you’ll typically have much fewer users than you’ll have in the future, and so now is the time when you’ll be able to engage deeply with qualitative insights such as support tickets, comments, and user interviews.
Analyze the qualitative feedback to determine whether you’re targeting the right audience and whether they understand the value proposition of your product. You want to identify whether you’ve positioned your product effectively towards solving their needs, and whether they’ve understood that your product is solving the intended pain.
Remember that your intention as a PM is not the same as user perception of your product. You may believe that your product solves pain A, but your user believes it solves pain B.
No amount of data analytics or metrics will get you this sort of insight; only qualitative feedback will surface this misalignment.
And of course, be sure to use this qualitative feedback to identify where other gaps in your product might be. You can then swiftly close these gaps through rapid iteration, which should then ultimately impact conversion rates, retention rates, and referral rates.
Additional thoughts for product launches
Now that we know which metrics we want to optimize for, we can (and should!) incorporate these metrics into our product development processes.
Before you write a single line of code, you’ll ideally work with design upfront to make sure that what you’re building will help with conversion, retention, and referral as key metrics to move.
And, before you ship your product into a live environment, remember to work with engineering to instrument these metrics. You won’t be able to measure them if you don’t have the right tracking and dashboarding for them, after all.
Of course, metrics are only one piece of the puzzle when it comes to launching a product. If you’re looking for additional resources on go-to-market planning for product launches, this in-depth guide covers product launches from end to end.
By working backwards from our business objectives of conversion, retention, and referral, we can ensure that the products that we’re building will create sustainable value for both our end users and for our business stakeholders.
Thank you to Pauli Bielewicz, Mary Paschentis, Siamak Khorrami, Goutham Budati, Markus Seebauer, and Kendra Ritterhern for making this guide possible.
