A/B test
On this page
Definition
An A/B test is a controlled experiment: users are randomly split into control and treatment, the control sees the current product and the treatment sees the change. The difference in the metric between arms estimates the effect of that change.
How to compute
Compare the metric’s mean or proportion across arms and test significance: a t-test for means, a z-test for proportions, bootstrap for non-Gaussian distributions. Read the effect together with a confidence interval, not from a p-value alone. Fix the metric, the MDE and the sample size before launch.
Pitfalls
Peeking at the data and stopping on the first significant result inflates the Type I error rate. Several metrics without a multiple-comparison correction produce false positives. A changed split, events logged differently per arm, or bots break randomisation — that is what the SRM check catches.