How to accurately elucidate MVP test indicators?

Image Source：Campaign Creators on Unsplash

1. Does the MVP Need to be Flawless?

The idea of MVP gained popularity with the publication of Eric Ries' "The Lean Startup." In brief, MVP involves the development team providing a minimum viable product to gather user feedback and then rapidly iterating on this minimum viable product until the product reaches a relatively stable stage. MVP is crucial for entrepreneurial teams as it allows them to quickly verify their goals and rapidly experiment.

However, MVP is also a controversial and criticized approach. The most significant issue with MVP is that the user experience may not be perfect, and some MVP experiences may be terrible. This may cause an excellent idea to receive negative feedback from users due to the poor MVP experience.

Some have suggested the concept of SLC (Simple, Loveable, and Complete) specifically to address the flaws in MVP. However, it is worth noting that Eric Ries recognized the "imperfect" approach of MVP in "The Lean Startup." The Dropbox example mentioned in the book did not even have a product prototype but only a video.

Therefore, how should we understand these contradictions?

2. The Two Aspects Goal of MVP

In my view, the goal of MVP testing comprises two aspects:

Firstly, determining user interest, i.e., does the user possess a certain demand, or can the MVP spark user interest? How many individuals have these demands, and are they interested in a solution?

Secondly, ascertaining user satisfaction, i.e., whether a particular solution can meet user needs and address user problems. Naturally, one can also add whether users are willing to pay. However, in my opinion, payment itself constitutes part of the solution, i.e., the cost of the solution (including money, time, energy, learning, etc.).

3. Data Metrics for MVP Testing

Thus, the answer to our question of "how to correctly interpret MVP test indicators" naturally arises.

Eric Ries' MVP concept emphasizes the first aspect, i.e., he focuses on "starting the learning and cognition process as early as possible" and "completing the development-test-cognition feedback loop with the least effort in the shortest time." He is primarily concerned with how many individuals possess a particular demand and are interested in a solution. Thus, the metric for measuring this is not user satisfaction with the MVP, just as we cannot judge a newly invented car with contemporary eyes. It is fundamentally proposing a solution, as long as it is perceived by the user, regardless of whether a product exists.

Under these circumstances, the measurement metrics should include actions such as click rate, open rate, etc., which represent user interest. Alternatively, it could employ personalized techniques like videos, landing pages only, crowdfunding, etc., or metrics such as on-demand rate, registration rate, and booking rate. The key is to represent the level of user interest.

The above metrics aim to test the product concept of the MVP, i.e., whether the problem it aims to solve or the demand it addresses exists, and how large the population it covers is.

For MVP testing of product experience, it is necessary to focus more on relevant metrics that gauge user satisfaction with their needs, such as completion rate, retention rate, payment rate, etc. In this case, what we need to test is the usability of the solution itself and then gradually improve and optimize it. However, we cannot abandon the product at this stage solely because these metrics are too low. If the product concept test has passed, then the product experience test is merely a process of optimization, not an evaluation of the value of its existence.

Image Source: woshipm

4. The Cost of Misinterpretation

Therefore, it is crucial that we interpret test indicators in the right way to avoid misinterpretation.

One common mistake is to use experience metrics to measure whether a product concept can generate interest among target users.

The imperfection of MVP can sometimes cause issues for users, particularly when the MVP design is incongruent with its objectives. For instance, if you created a poorly designed paid-for-knowledge product and found out, after testing it, that user retention is low, can you conclude that there is no market for this type of product?

The answer is no, because:

• The size of the market does not depend on user retention but on user click-through rate or trial rate, which indicates how many people are interested. This is because the retention rate reflects the user's satisfaction with the product, and perhaps the user needs such a product, but you have not executed it well enough.
• The essence of knowledge-paid products is quality. It is evident that this MVP is substandard when it comes to core value communication. The imperfection of the MVP is only superficial, and its core value should not be undervalued.

It is like asking someone to try a cake, but instead, only giving them flour or a steamed bun. Even if you draw a beautiful fake cake for product concept testing, it is still better than plain flour or an actual steamed bun. (The image below vividly illustrates the differences between these three ideas).

Image Source: woshipm

5. Determination of MVP Test Indicator Size

When it comes to determining MVP test indicators, there is no set numerical requirement that must be met. For testing a product concept, it's important to assess how many users are interested in the product, the value they bring, and whether it meets expectations. Even if the product serves only one user, but that user brings substantial returns, it can still be considered a desirable product concept.

In contrast, for testing product experience, higher values are generally better. While we can look to peer experience values for guidance, financial indicators like ROI are key. For instance, a low user retention rate may result in a small lifetime value of users and may not cover their acquisition costs. In this case, optimizing the experience and improving retention is necessary.