If I would get a dime for every time someone runs an experiment without having a clear understanding of what exactly he or she is trying to learn, I would be really really rich. Just too often I will hear things like “everything we will learn from this experiment will be useful” or “let’s see what happens, we’ll start with a pilot/experient” (without following up with anything concrete). That is just a waste. Everything that can be measured can be improved and if you think clearly about what you want to learn, you will probably learn about it, or at least you will set up the metrics/measurements to learn. It is very difficult to just do something and then in hindsight sift through data (if you have any) and look for something that stands out. It is way easier to define things you want to learn beforehand, set up ways to measure it, and learn from that. Imagine going to school with the teachers saying “yeah, whatever they learn, that is just fine”. No. They have a plan. They have a structure, they measure how students progress. Well … I guess you get my point by now. It is very useful to set clear hypotheses, metrics, and ways to verify if you were correct.
When I was speaking at a conference by Growth Tribe, I stumbled upon another speaker who used the Experiment outline as a structured way to plan for and report on experiments. I fell in love with the model and used it ever since.
Experiment outline elements
The Experiment outline consists of four elements: on the left side “what we believe” and “to verify that we will” and on the right side how you are going to prove the value of your experiment in “two metrics that matter” and “we were right if”.
The 4 Experiment outline elements
We believe that ...
This is your hypothesis, what you think is the case, and the main reason why you are running the experiment in the first place. It might be because you think another button color will make a difference because you think a specific group of people will behave in a certain way (for example, because you based these insights on a simple empathy map or empathy map).
- “This user group is very likely to pay a monthly fee for an ideal Philips Skincare app”
To verify that we ...
What will you do to verify your hypothesis? What experiment will you run? Be clear about what you are going to do so people will be able to understand and/or able to replicate your experiment (and/or can give some context to your findings, maybe because even with all best intentions, you have a biased experiment):
- “Mimic the experiment from the Hacking Growth Book and send them a questionnaire”
Two metrics that matter
These will be the two measurable metrics that will define the outcomes. What two KPIs can you measure that will tell you if your experiment succeeded or not?
- Willingness to Pay
- Conversion Rate
We were right if ...
Set a threshold on each of your metrics that matter. When were you right with your hypothesis?
- Willingness to Pay is positive
- Conversion Rate >10%
Keep me posted!
I send out an irregular newsletter with my favorite excercises, presentations and thoughts. Subscribe to stay posted.
Let me help you
Acts, not ads.
I have 15+ years in changing companies to be less about ads and more about acts. I'd love to hear your story and see how I can help.