Did the change work, or was it chance?

Lesson 5 of 6About 9 minutes5 steps

An A/B test answers whether a change made a difference. Sizing it, reading it and not peeking at it every day are what an analyst is paid to get right.

  • Size a test before it starts
  • Read a p-value and an interval
  • Explain why peeking misleads

You need: Lesson 1.

  1. 1Statistics you use every day
  2. 2SQL that answers questions
  3. 3Charts that answer the question
  4. 4Power BI’s filter context
  5. 5A/B tests
  6. 6Guesstimates
0 of 5 steps

  1. Randomise, then compare

    Predict first

    Why split visitors at random, instead of comparing this month with last month?

    Just show me

    So that everything else (festivals, ads, the weather) hits both groups alike, and only the change differs.

    Choose one primary metric before starting (the conversion rate, say), split traffic at random and run both versions at the same time. A before-and-after comparison mixes the change with everything else that changed.

    Quick check

    You launch a new checkout during Diwali and sales jump. Proof it works?

    Without a control group, the festival and the change are tangled.

  2. How many visitors?

    Predict first

    Conversion is 4%. To reliably detect a lift to 4.4%, how many visitors does each version need?

    Just show me

    39,475 per version, at 95% confidence and 80% power. Look for half that lift and it’s about 3.9 times as many.

    Small rates and small lifts need big samples. Work the size out first, from today’s rate, the smallest lift worth finding, the confidence and the power, and run the test until it’s reached.

    Size your own test in the A/B test calculator

    Quick check

    Traffic is 2,000 visitors a day, and the test needs 80,000 in all. How long?

    80,000 ÷ 2,000 = 40 days; if that’s too long, test a bigger change or a busier page.

  3. Reading the result

    Predict first

    400 of 10,000 converted on A and 452 of 10,000 on B: a 13% lift. Significant at 95%?

    Just show me

    No: p = 0.069, above 0.05. The 95% interval for B − A runs from -0.04 to +1.08 points, which includes zero.

    The p-value is how surprising this gap would be if the versions were the same; the interval is how big the real difference might be. A big lift on small numbers can still be noise, and a tiny lift on huge numbers can be “significant” and still not worth shipping.

    The z-test, symbol by symbol

    Quick check

    p = 0.03, and the interval runs from +0.01 to +0.05 points. Ship?

    Significant says “probably real”, not “worth it”.

  4. Don’t peek and stop

    Predict first

    Check a test of two identical versions every day for 14 days, and stop at the first p below 0.05. How often do you “find a winner”?

    Just show me

    About 22%: 217 of 1,000 simulated A/A tests, against 48 when each is checked once at the end.

    Every look is another chance for noise to cross the line. Fix the sample size and look once at the end, or use a method built for repeated looks (sequential testing).

    Watch 1,000 A/A tests in the calculator

    Quick check

    Your manager wants daily results. Best answer?

    Watching is fine; stopping early on a lucky day is what inflates false wins.

  5. Guardrails and many metrics

    Predict first

    You track 20 metrics and one is significant at 0.05. A real win?

    Just show me

    Maybe not: at 0.05, about 1 test in 20 crosses the line by chance anyway.

    Decide the primary metric in advance, add guardrail metrics that mustn’t get worse (refunds, load time), and treat surprises in the other metrics as ideas for the next test, not conclusions.

    Quick check

    What’s a guardrail metric?

    It guards against a change that wins the goal and hurts something else.

In the interview: A/B tests.

Ask Prism about this lesson

Sources

  1. Kohavi, Tang and Xu, “Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing”, Cambridge University Press, 2020.
  2. Armitage, McPherson and Rowe, “Repeated Significance Tests on Accumulating Data”, Journal of the Royal Statistical Society, Series A, 1969.
  3. Our A/B test calculator, its own arithmetic for every number here.

Back: lesson 4, Power BI’s filter context