There are two questions we could ask after experiment has ended regardless of wether variant lost or won:
- did all cohorts responded to variant in the same way? If so, what does it say about users who reacted differently? For example, we could find out that clients of wedding photographers bought more product bundles then clients of event photographers. We could use such insight to improve targeting of our offer
- are there any metrics besides the one we want to optimize that were affected? If there are, maybe there’s hidden relation between those metrics and our change we didn’t know exists.
What if variant lost
With a/b tests we either improve a metric or learn something new. And in the situation when our hypothesis was rejected there is a couple of ways we can learn:
- what the fact that variant lost says about our assumptions? For example, hypothesis that offering product bundles will improve sales is based on the assumption that customers want more products for lower price (how can it not be true?). If hypothesis is proved to be false does it mean our customers don’t want more products?
- what are other possible explanations for lost variant besides that our assumption is wrong? Possible explanations could be:
- an effect on metric that we’re trying to detect (MEI) is higher than actual effect was made on it by variant. If we’re trying to detect 10% uplift in sales than any change less then that will be missed.
- a change in product is not significant for users. Maybe our hypothesis about product bundles failed because we offered 5% discount and it’s not enough. What if we offered 10%? 15%?