Experiment outline.

The problem.

If I got a dime for every time someone ran an experiment without a clear idea of what they were trying to learn, I would be really rich. Too often I hear "everything we learn from this will be useful", or "let's start with a pilot and see what happens", without anything concrete following.

That is a waste. It is very hard to just do something and then sift through the data afterwards, if there is any, looking for something that stands out. It is much easier to define what you want to learn beforehand, set up ways to measure it, and learn from that. Imagine a school where the teachers say "whatever they learn, that's fine".

The model.

1. What we believe. Your hypothesis: what you think is the case, and the main reason you run the experiment at all.
2. To verify that, we will. What you will do. Clear enough that others understand it, can repeat it, and can put your findings in context.
3. Two metrics that matter. The two measurable things that will tell you whether the experiment succeeded.
4. We were right if. A threshold on each of those metrics, set before you start.

Example.

• We believe this user group is very likely to pay a monthly fee for an ideal skincare app.
• To verify that, we will mimic the experiment from the book Hacking Growth and send them a questionnaire.
• Two metrics that matter: willingness to pay, and conversion rate.
• We were right if willingness to pay is positive and conversion is above 10%.

Why it matters now.

Every organisation is running AI pilots. Few can say afterwards what they learned from them. Four boxes on one page, filled in before the pilot starts, make the difference.

Origin: I came across it when a speaker used it at a Growth Tribe conference, and have used it ever since. First published here in May 2020.