AB Tests solution

Requirements:

Basic experiment setup

Solution needs to provide a way to create a A/B test. Set start/end date, name

Frontend & backend

Solution works on the backend and frontend

Targeting

We need to be able to run experiment only on a subset of users (new users, clients, only users on specific plans, desktop/mobile, etc.)

Minimal effect on latency

Since ab testing infrastructure will be invoked on each user visit it must have as low effect on latency as possible

Feature toggling

There should be a way to enable/disable feature for specific user group

Feature roll out/back

There should be a way to rollout feature for all users or disable it completely after experiment is done

Sample size calculation

We need to calculate duration of the experiment and for that we need to know sample size

Statistical significance calculation

After experiment is finished we need to calculate difference between control and experiment group and whether it’s significant or not

Post-experiment analysis

All data tracked during an experiment must be tied to one of the experiment groups so we can gain deeper insights

1. Optimizely + Mixpanel Experiments + online stats calculators

Prices Optimizely - need to contact sales Mixpanel - need to contact sales Online stats calculators - free Solution Basic experiment setup - optimizely has solution for ab tests Frontend & Backend - optimizely has react sdk and nodejs sdk Targeting - optimizely has audience feature Latency - optimizely sdk caches all needed data in-memory Feature toggling - optimizely has feature toggles Feature rollout - optimizely has feature rollouts Sample size calculator - online stats calculators: for conversions for other metrics Statistical significance - optimizely has stats engine that builds results page Deeper analysis - optimizely won’t have all of our data, so we should use mixpanel for that. Mixpanel has experiment report which helps breakdown any data by control/experiment groups. But it requires enterprise plan

2. Optimizely + Mixpanel free plan + online stats calculators

Price Optiizely - contact sales Mixpanel - free Online stats calculators - free Solution Optimizely - same as in the previous solution For deeper analysis we would use mixpanel. They have built in solution called Mixpanel Experiments Report It has a special event $e x p er im e n t_{s} t a r t e d w hi c h w ec an se n df ore a c h u ser w h e n e x p er im e n t s t a r t e d . M i x p an e lw i llt h e n reco g ni ze i t an dd i s pl a ye a c h t r a c k e d e x p er im e n t in t h e U I . T h e n w ec an l oo kin t o d a t a t r a c k e dd u r in g e x p er im e n tp er i o d an d a l soc h ec kb re ak d o w n f or d i ff ere n t e x p er im e n t g ro u p s . W eco u l d b u i l d an y m e t r i cs an d mi x p an e lw i ll c a l c u l a t e d i ff ere n ce f or g ro u p s an d s t a t i s t i c a l s i g ni f i c an ce f or i tB u tt hi s f u n c t i o na l i t yre q u i rese n t er p r i se pl an . I t hink w eco u l d t r a c k e x p er im e n t s an d g ro u p s w i t h c u s t o m e v e n t s an df i e l d s an d t h e n u see x i s t in g b re ak d o w n f e a t u re t o l oo k f or d i ff ere n ces . A s f ors i g ni f i c an ceor an yo t h ers t a t s I co u l d u se 3 r d p a r t yc a l c u l a t ors l ik eo n es f ors am pl es i zec a l c u l a t i o nW eco u l d s t i ll se n d$ experiment_started just in case we buy enterprise plan so old experiments will be available

3. Google optimize

Prices Google optimize - need to contact sales Online stats calculators - free Solution Basic experiment setup - google optimize provides ability to setup a/b tests Frontend & Backend - optimizely has react sdk and nodejs sdk Targeting - optimizely has audience feature Latency - optimizely sdk caches all needed data in-memory Feature toggling - optimizely has feature toggles Feature rollout - optimizely has feature rollouts Sample size calculator - online stats calculators: for conversions for other metrics Statistical significance - optimizely has stats engine that builds results page Deeper analysis - optimizely won’t have all of our data, so we should use mixpanel for that. Mixpanel has experiment report which helps breakdown any data by control/experiment groups. But it requires enterprise plan

4. In-house solution + Mixpanel free plan + online stats calculators

We could build everything on our own.

Vlad Severin

Explorer