How sure are you, and how would you know?
An estimate without its uncertainty is half an answer. A confidence interval gives the range; a hypothesis test asks whether a difference could be chance. Both are easy to say wrongly in an interview.
- Say what 95% confidence means, correctly
- Read a p-value, and its two kinds of error
- Choose a t-test, a chi-squared test or a test of proportions
You need: Lesson 1.
- 1Distributions and sampling
- 2Confidence intervals and tests
- 3Training, testing and overfitting
- 4Judging a classifier
- 5Trees and ensembles
- 6Clustering and PCA
- 7How a neural network learns
-
What 95% means
Predict first
Take 1,000 samples of 30 die rolls and a 95% confidence interval from each. How many catch the true mean, 3.5?
Just show me
958 of 1,000 did: about 95 in 100, as the method promises.
The 95% is a promise about the method, not about one interval: of many samples, about 95 in 100 intervals catch the true value, and any single one either did or didn’t. So “a 95% chance the true mean is in this interval” is the classic wrong answer; “a method that catches it 95% of the time” is the right one.
Draw 1,000 intervals in the central limit theorem simulator
Quick check
A 95% interval for the average order runs from ₹1,780 to ₹1,920. Which is right?
It’s about the average and the method, not about single orders.
-
Narrower takes more
Predict first
To halve an interval’s width, how much more data do you need?
Just show me
Four times: the width shrinks with the square root of the sample size.
An interval is the estimate plus and minus a margin: about two standard errors for 95%, with Student’s t in place of the 2 for small samples. More confidence (99%) widens it; more data narrows it, but only by the square root.
The 95% confidence interval, symbol by symbol
Quick check
Your manager wants a narrower interval without more data. What gives?
Lower confidence narrows the interval; there’s no free precision.
-
The p-value
Predict first
An A/B test: 400 of 10,000 converted on A, 452 of 10,000 on B, and p = 0.069. What does that mean?
Just show me
If A and B truly converted alike, a gap at least this big would turn up about 7% of the time by chance: not rare enough for the usual 5% line.
A test assumes there’s no difference (the null hypothesis), then asks how surprising the data would be. Below a line set in advance, usually 0.05, we reject that assumption. The p-value is not the chance the null is true, not the size of the effect and not the chance the result will repeat.
The two-proportion z-test, symbol by symbol
Quick check
p = 0.001 for a tiny lift. What do you know?
A small p says chance alone is unlikely; the size needs the estimate and its interval.
-
Two ways to be wrong
Predict first
Testing a new checkout, which mistake ships a change that doesn’t really help?
Just show me
A Type I error: you saw a win that was chance. Missing a real win is a Type II error.
The line α (often 0.05) caps Type I errors. Power (often 80%) is the chance of catching a real effect of the size you planned for, so Type II errors happen 1 − power of the time. Lowering one raises the other unless you add data, which is why tests are sized before they start.
Quick check
A fraud model flags a genuine payment. Which error?
Flagging something that isn’t there is a false positive, a Type I error.
-
Which test?
Predict first
Compare the average order value in two cities. Which test?
Just show me
A t-test: it compares the means of a number between two groups.
Means of a number: a t-test (Welch’s when the spreads differ). Rates such as conversion: a test of two proportions, the z-test. Counts in categories, such as plan by city tier: a chi-squared test of independence. Skewed data with outliers: a rank-based test such as Mann–Whitney is safer.
Quick check
Is churn related to city tier (tier 1, 2 or 3)?
Two categories in a table of counts: chi-squared.
Lesson 2 done.
Next: Training, testing and overfitting. Every model is judged on data it hasn’t seen.
In the interview: confidence intervals and tests.
MediumWhat does a 95% confidence interval actually mean?
EasyWhat are hypothesis testing and confidence intervals, and how are they related?
MediumWhat is a p-value not?
MediumWhat do p-values and Type I and II errors mean?
EasyType I or Type II error: which is worse when testing a new checkout flow?
Mediumt-test, chi-squared test or a test of proportions: which do you use?
MediumThe test is significant but the lift is tiny. Should you ship?
Ask Prism about this lesson
Sources
- Wasserstein and Lazar, “The ASA Statement on p-Values: Context, Process, and Purpose”, The American Statistician, 2016.
- NIST/SEMATECH e-Handbook of Statistical Methods, “Confidence Limits for the Mean”.
- Diez, Çetinkaya-Rundel and Barr, “OpenIntro Statistics”, 4th edition, 2019 (CC BY-SA): inference, tests and errors.