Plan the test, then trust the result.
How many visitors an A/B test needs, whether a difference is real or chance, and why checking a test every day finds winners that aren’t there.
Guess first
Two identical versions of a page, 1,000 visitors each a day, checked every day for 14 days and stopped the first day the p-value drops below 0.05. Out of 1,000 such tests, how many “find a winner” that isn’t there?
217 of 1,000 tests found a “winner” (21.7%)
Checked once, on the last day, 48 of the 1,000 identical tests cross p < 0.05: the 5% the test allows. Checked every day and stopped at the first crossing, 217 do, because every look is another chance for noise to cross the line. Fix the sample size first and look once, or use a method built for repeated looks.
Plan a test39,475visitors per version
78,950 in all · about 8 days at 10,000 a day
To tell 4% from 4.4% (a 10% lift), with 95% confidence and an 80% chance of catching a real lift that size.
Look for half that lift (5%) and you’d need 1,54,304 a version: about 3.9 times as many.
A converts4.00%
B converts4.52%
Lift+13.0%
Not significant at 95% confidence
If the two versions were the same, a difference at least this big would turn up by chance 6.9% of the time.
- p-value
- 0.069
- 95% interval for B − A
- −0.04 to +1.08 points
How it’s worked out: visitors per version for two proportions, two-sided, by the normal approximation; the two-proportion z-test with the pooled rate, and the 95% interval with each version’s own; the 1,000 A/A tests simulated here in your browser (1,000 visitors a version a day, both converting at 5%, 14 days). Nothing you type leaves this page.
Three rules that keep a test honest.
Size it before you start
The visitors you need follow from the rate now and the smallest lift worth finding. Halve the lift and you need about four times the visitors. A rule of thumb for 95% confidence and 80% power: 16 × p(1 − p) ÷ the difference², per version.
Look once, or plan the looks
Every look at a running test is another chance for noise to cross the line. Stopped at the first p below 0.05 over 14 daily looks, about one identical test in five “wins”. Sequential methods are built for repeated looks.
Significant isn’t the same as big
A p-value says how surprising the difference would be if there were none; the interval says how big it might be. With enough visitors a tiny difference turns significant, so judge the interval against what matters to the business.
Sources
- Kohavi, Tang and Xu, “Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing”, Cambridge University Press, 2020.
- Kohavi, Longbotham, Sommerfield and Henne, “Controlled experiments on the web: survey and practical guide”, Data Mining and Knowledge Discovery, 2009: the rule of thumb for the sample size.
- Armitage, McPherson and Rowe, “Repeated Significance Tests on Accumulating Data”, Journal of the Royal Statistical Society, Series A, 1969: why repeated looks raise false positives.