A statement predicting the outcome of a defined change, that can be confirmed or rejected with data. A hypothesis is the main unit of experimentation process in the same way as a story in the product development process. There’s usually a process set up to generate new hypotheses based on business objectives, user feedback, data insights, etc. This process fills a backlog which should be then prioritised and scheduled.
Hypothesis has a structure that looks like this: If… Then… Because…
Example: If… we show products directly on products group page Then… we will see a 5% increase in conversion Because… visitors will need less clicks to see products
If…
The if part describes a specific change to our product that we think will improve a metric. The change shouldn’t be too big. For example, we could have a hypothesis like this: “if we would redesign shop page then conversion will improve”. It’s bad for following reason: It’s impossible to learn from it. If we would redesign complete shop page and it will indeed increase conversion how will we know what part of new design affected conversion? The point of experimentation is to learn what works and what not and in this case we wouldn’t learn anything since everything was changed. It’s especially true when hypothesis fails, because in this situation only thing that would make it not a waste of resources is if we learned something new. Good hypothesis in this case would look like this: “if we would make product titles on shop page bigger then conversion will improve”. Irrespectively to whether this hypothesis fails or not we will be able to attribute a change in metric to a specific change in specific part of the product.
Then…
The then part describes a minimal change in the metric that we would be happy to observe. Sometimes it’s referred as minimal effect of interest (MEI). It could be based on anything from data insight to pure intuition. The only requirement is that it should be specific number e.g. “…15% increase in conversion…”, “…5% reduction in churn…” and not just “conversion will increase”. There’s mathematical reason for that, we won’t be able to calculate experiment duration without choosing specific MEI first. There should only one main metric per hypothesis that we want to optimize.
Because…
The because part describes why we think specific change will improve metric. It includes our assumption about users and it’s important part of learning process. If hypothesis fails then it might be because our assumption is wrong and we can learn from it.