Statistics interview questions, answered the way you’d say them.

The tosses are independent, so multiply: (1/2)⁴ = 1/16, or 6.25%. The usual follow-ups reuse two ideas. At least one head is the complement, 1 − 1/16 = 15/16, and exactly two heads counts the arrangements, C(4, 2) = 6 of the 16 equally likely outcomes, which is 3/8. Name the rule you’re using, the multiplication rule for independent events or the complement, before giving the number.

For a biased coin with heads probability p, the same steps give p⁴ for all heads.

OpenStax, Introductory Statistics 2e: 3.2 Independent and mutually exclusive eventsOpenStax, Introductory Statistics 2e: 4.3 Binomial distribution

Practise it with Prism

Skew is named after the long tail. A right-skewed, or positively skewed, distribution has a long tail of large values, like order values or incomes, and its mean usually sits above the median. A left-skewed one has a long tail of small values, like marks in an easy exam, and its mean usually sits below the median. Treat the mean-and-median rule as a guide, not a law: it fails surprisingly often, especially for discrete data.

In pandas, Series.skew() returns the sample skewness: positive for a long right tail, negative for a long left one.

NIST/SEMATECH e-Handbook of Statistical Methods: 1.3.5.11 Measures of skewness and kurtosisvon Hippel (2005), Mean, median, and skew: correcting a textbook rule, Journal of Statistics Educationpandas docs: Series.skew

Practise it with Prism

A hypothesis test starts from a null hypothesis, such as no difference in spend, sets a significance level α, computes a test statistic from the data and rejects the null when the statistic is too extreme to fit it. A confidence interval gives a range of plausible values for the quantity, from a method that captures the true value in, say, 95% of repeated samples. They’re two views of one calculation: built from the same statistic, a 95% interval contains exactly the values a 5%-level test wouldn’t reject.

Report the interval even when the test is significant: it shows how big the effect could be, which a p-value doesn’t.

NIST/SEMATECH e-Handbook of Statistical Methods: 7.1.3 What are statistical tests?NIST/SEMATECH e-Handbook of Statistical Methods: 7.1.4 What are confidence intervals?NIST/SEMATECH e-Handbook of Statistical Methods: 7.2.2.1 Confidence interval approach

Try it in the central limit theorem simulatorPractise it with Prism

The mean is the sum divided by the count: it uses every value, so outliers pull it, and it’s the one to use when totals matter. The median is the middle value once sorted, or the average of the middle two, and it resists outliers, so it suits skewed data such as incomes or delivery times. The mode is the most frequent value, the only one that works for categories, such as the most used payment method, and there can be more than one.

When the mean and the median differ a lot, report both and say which way the data are skewed.

NIST/SEMATECH e-Handbook of Statistical Methods: 1.3.5.1 Measures of location

Practise it with Prism

It summarises a distribution in a few numbers. The box runs from the first quartile to the third, so it covers the middle half of the data, with a line at the median. In Tukey’s convention, the default in tools such as Matplotlib, whiskers reach the furthest points within 1.5 × IQR of the box, and points beyond are drawn separately as possible outliers. Box plots are best for comparing groups side by side, such as delivery times by city, but they hide shape, such as two peaks.

Check which whisker rule your tool uses before you call a point an outlier: some draw whiskers to the minimum and maximum instead.

NIST/SEMATECH e-Handbook of Statistical Methods: 1.3.3.7 Box plotMatplotlib docs: matplotlib.pyplot.boxplot

Practise it with Prism

Usually the median, with the mean beside it. If a few bulk orders form a long right tail, they pull the mean above the median, so the mean can describe almost no real order, while the median, the middle value, barely moves. Still report the mean when totals matter, because total revenue is the mean order value times the number of orders.

Plot a histogram first, and when the mean and median differ a lot, say why in one line.

NIST/SEMATECH e-Handbook of Statistical Methods: 1.3.5.1 Measures of location

Practise it with Prism

Standard deviation measures how spread out individual values are, in the data’s own units: how much one customer’s basket differs from another’s. Standard error measures how precisely a sample statistic estimates the population value; for a mean it’s the standard deviation divided by √n, so it shrinks as the sample grows. Use the SD to describe data, and the SE to build confidence intervals and tests.

The SD is pulled up by extreme values, so for skewed data also report the interquartile range.

NIST/SEMATECH e-Handbook of Statistical Methods: 1.3.5.6 Measures of scaleOpenStax, Introductory Statistics 2e: 7.1 The central limit theorem for sample means

Try it in the central limit theorem simulatorPractise it with Prism

In a simple random sample every group of n people is equally likely; a stratified sample takes a proportionate share from each group, such as each city; a cluster sample picks whole groups, such as some stores, at random; a systematic sample takes every kth record from a random start. Sampling bias occurs when some members are less likely to be chosen, as with convenience samples or a survey only active app users see.

Non-response is a bias too: the people who answer a feedback survey often hold the strongest opinions.

OpenStax, Introductory Statistics 2e: 1.2 Data, sampling, and variation in data and sampling

Practise it with Prism

A Type I error is a false positive: you ship a checkout that doesn’t really help, at a rate set by the significance level α. A Type II error is a false negative: you miss one that does help, at rate β, and power is 1 − β. Which is worse depends on the costs: a costly, hard-to-reverse change deserves a smaller α, while a cheap, reversible one can accept more false positives to catch more real wins.

For a fixed sample, lowering α raises β; only more data or a bigger real effect lowers both.

NIST/SEMATECH e-Handbook of Statistical Methods: 7.1.3 What are statistical tests?statsmodels: NormalIndPower.solve_power

Practise it with Prism

Not from that alone: correlation shows the two move together, not that one causes the other. A confounder, such as income or internet access, could drive both, and the cause could even run the other way. To show cause and effect, randomise, so the groups differ only in the change you made; otherwise call it an association and list the confounders you couldn’t rule out.

Name a specific confounder and how you’d measure it, rather than saying “there could be other factors”.

OpenStax, Introductory Statistics 2e: 1.4 Experimental design and ethics

Practise it with Prism

A p-value is how likely you’d be to see a result this extreme or more if the null hypothesis held; it doesn’t tell you how likely the null itself is. A Type I error means rejecting a null that was actually true, a false positive, and its rate is the significance level, often 5%. Missing a real effect is a Type II error, and power equals 1 minus the Type II rate.

The full answer, in the guideTry it in the A/B test calculatorPractise it with Prism

Two ways. Rejection: map two uniforms to x and y in [−1, 1], keep the point if x² + y² ≤ 1 and draw again if not; about π/4, or 78.5%, of draws are kept. Polar: take the angle as 2π times one uniform and the radius as the square root of another, because area grows with r², so a plain uniform radius crowds points into the centre. Check either by confirming that about a quarter of the points fall within radius 0.5.

import math, random
def in_circle_rejection():
    while True:
        x, y = 2 * random.random() - 1, 2 * random.random() - 1
        if x * x + y * y <= 1: return x, y
def in_circle_polar():
    theta, r = 2 * math.pi * random.random(), math.sqrt(random.random())
    return r * math.cos(theta), r * math.sin(theta)
pts = [in_circle_polar() for _ in range(100_000)]
print(sum(math.hypot(x, y) <= 0.5 for x, y in pts) / len(pts))    # about 0.25

Expect a follow-up on cost: rejection throws away about 21% of draws here, and the share wasted grows quickly in more dimensions.

Wolfram MathWorld: Disk Point PickingPython docs: random

Practise it with Prism

Use Bayes’ theorem. Before tossing, each coin is equally likely; three heads has probability 1 with the double-headed coin and (1/2)³ = 1/8 with the fair one. So P(double-headed | 3 heads) = (½ × 1) ÷ (½ × 1 + ½ × 1/8) = 8/9, about 89%. Each head doubles the odds in favour of the double-headed coin, and a single tail would rule it out completely.

Prior:        P(double) = 1/2           P(fair) = 1/2
Likelihood:   P(HHH | double) = 1       P(HHH | fair) = 1/8
Posterior:    P(double | HHH) = (1/2 × 1) ÷ (1/2 × 1 + 1/2 × 1/8) = 8/9 ≈ 0.89
Odds form:    prior odds 1 : 1, doubled by each head -> 8 : 1

Expect the general case: after n heads in a row, the chance is 2ⁿ ÷ (2ⁿ + 1).

OpenStax, Principles of Data Science: 3.4 Probability theory (Bayes’ theorem)

Practise it with Prism

Binomial counts successes in a fixed number of independent trials with the same probability, such as how many of 1,000 visitors pay. Poisson counts events in a fixed interval at a known average rate, such as failed UPI payments per minute, and its variance equals its mean. Normal is the symmetric bell curve for continuous measurements, and the central limit theorem makes it a good approximation for averages of large samples.

State the assumptions out loud: binomial needs the same probability on every trial, and Poisson needs events that occur independently.

NIST/SEMATECH e-Handbook of Statistical Methods: 1.3.6.6.18 Binomial distributionNIST/SEMATECH e-Handbook of Statistical Methods: 1.3.6.6.19 Poisson distributionNIST/SEMATECH e-Handbook of Statistical Methods: 1.3.6.6.1 Normal distributionOpenStax, Introductory Statistics 2e: 4.3 Binomial distributionOpenStax, Introductory Statistics 2e: 4.6 Poisson distribution

Practise it with Prism

The distribution of a sample mean approaches a normal distribution as the sample size grows, whatever the population’s shape, with a standard error of σ/√n. That’s why you can put a confidence interval on average order value, or compare revenue per user with a t-test, even though individual orders are skewed. It’s about means of large samples, not individual values.

Don’t quote n = 30 as a law: how large is large enough depends on how skewed the data are.

OpenStax, Introductory Statistics 2e: 7.1 The central limit theorem for sample meansNIST/SEMATECH e-Handbook of Statistical Methods: 1.3.6.6.1 Normal distribution

Try it in the central limit theorem simulatorPractise it with Prism

If you repeated the sampling many times and built an interval the same way each time, about 95% of those intervals would contain the true value. The 95% describes the method, not this one interval, which either contains the true value or doesn’t. An interval is the estimate plus or minus a critical value times its standard error, so it narrows as the sample grows.

95% CI for a mean, large sample:  mean ± 1.96 × SD / √n
Example (assumed): mean basket ₹820, SD ₹600, n = 3,600
SE = 600 / 60 = ₹10, so the interval is 820 ± 19.6, about ₹800 to ₹840

“There’s a 95% chance the true value is in this interval” is the classic slip; describe the procedure instead.

NIST/SEMATECH e-Handbook of Statistical Methods: 7.1.4 What are confidence intervals?

Try it in the central limit theorem simulatorPractise it with Prism

A p-value indicates how incompatible the data are with a specified model, usually the null hypothesis. It is not the probability that the hypothesis is true, nor the probability that the result arose by chance alone, and it doesn’t measure the size or importance of an effect. That’s from the American Statistical Association’s 2016 statement, which adds that decisions shouldn’t rest only on whether p crosses a threshold such as 0.05.

Report the effect size and its confidence interval next to the p-value, and say how many tests you ran.

American Statistical Association: Statement on statistical significance and p-values (2016)

Try it in the A/B test calculatorPractise it with Prism

Not on significance alone. The standard error shrinks with the square root of the sample, so with millions of users even a tiny difference becomes significant, and a p-value says nothing about how big or important an effect is. Compare the estimated lift and its confidence interval with the smallest effect worth acting on, then weigh it against the cost of building and maintaining the change, and the guardrail metrics.

Agree the minimum effect worth shipping before the test starts, so the result can’t move the goalposts.

American Statistical Association: Statement on statistical significance and p-values (2016)Kohavi, Henne and Sommerfield (2007), Practical guide to controlled experiments on the web, KDD

Practise it with Prism

It’s when a trend that holds in every group reverses once the groups are combined, because a confounder is spread unevenly between them. In a 1986 study of kidney-stone treatments, treatment A succeeded more often for small stones (93% against 87%) and for large stones (73% against 69%), yet B looked better overall (83% against 78%). A was used mostly on large stones, which succeed less often, so compare like with like.

In product data the same trap appears when the mix shifts, for example when more traffic from a low-converting city lowers the overall rate while every city improves.

Dunn, Scientific Research and Methodology: 14.1 Proportions and percentages (the kidney-stone data)Julious and Mullee (1994), Confounding and Simpson’s paradox, BMJ

Practise it with Prism

State the hypothesis and one primary metric, such as checkout conversion per user, plus guardrails such as payment failures and refunds. Randomise users rather than sessions so each person sees one version, size the test from the smallest lift worth detecting at 80% power or more, and run whole weeks to cover day-of-week patterns. Check that the split matches the plan, then analyse once, at the planned end.

Run an A/A test on the platform first: it checks the split, and how often “significant” results appear by chance.

Kohavi, Henne and Sommerfield (2007), Practical guide to controlled experiments on the web, KDDDmitriev et al. (2017), A dirty dozen: twelve common metric interpretation pitfalls in online controlled experiments, KDDFabijan et al. (2019), Diagnosing sample ratio mismatch in online controlled experiments, KDD

Try it in the A/B test calculatorPractise it with Prism

Pick one overall evaluation criterion that reflects the goal and that the change could move during the test, such as orders per user rather than clicks on the new button. Add guardrail metrics that shouldn’t get worse, such as cancellations and page load time, and fix all of them before launch. Clicks that jump while retention and satisfaction stay flat are a warning, not a win.

Expect “why not revenue?”: it varies more than conversion, so it needs a bigger sample to detect the same change.

Kohavi, Henne and Sommerfield (2007), Practical guide to controlled experiments on the web, KDDDmitriev et al. (2017), A dirty dozen: twelve common metric interpretation pitfalls in online controlled experiments, KDD

Practise it with Prism

Often much less likely than people expect. Bayes’ theorem gives P(fraud | alert) = P(alert | fraud) × P(fraud) / P(alert). Suppose 0.1% of payments are fraud, the model flags 99% of frauds and wrongly flags 2% of genuine payments: in 1 lakh payments, 99 frauds and 1,998 genuine ones are flagged, so only about 4.7% of alerts are fraud, because the huge genuine group produces most alerts.

Assumed: 1 lakh payments, 0.1% fraud, 99% of fraud flagged, 2% of genuine flagged
Fraud flagged:     100 × 0.99     =    99
Genuine flagged:   99,900 × 0.02  = 1,998
P(fraud | alert) = 99 / (99 + 1,998) ≈ 4.7%

Counting in natural frequencies, as here, is easier to say out loud than the formula and harder to get wrong.

OpenStax, Principles of Data Science: 3.4 Probability theory (Bayes’ theorem)

Practise it with Prism

It depends on the metric. Compare two averages, such as revenue per user, with a two-sample t-test, the Welch version when the variances may differ; compare two conversion rates with a two-proportion z-test; and test whether two categorical variables, such as city and payment mode, are related with a chi-squared test of independence, which needs an expected count of at least 5 in every cell.

With large samples the t-test doesn’t need normal data, but look at extreme revenue outliers before you test.

OpenStax, Introductory Statistics 2e: 10.1 Two population means with unknown standard deviationsOpenStax, Introductory Statistics 2e: 10.3 Comparing two independent population proportionsOpenStax, Introductory Statistics 2e: 11.3 Test of independence

Practise it with Prism

I’d write down the hypothesis and a single primary metric, such as orders per user, and randomise by user rather than by session. The sample size follows from the minimum effect worth finding and the power I’m aiming for, the test runs for whole weeks so weekday patterns are covered, and the analysis is fixed before I look at results. I’d also track guardrails like cancellations and latency, and never stop early just because the p-value dips.

The full answer, in the guideTry it in the A/B test calculatorPractise it with Prism

Take n independent observations, each with variance σ². The sample mean is their sum divided by n, and multiplying a variable by 1/n multiplies its variance by 1/n². Because the observations are independent, the variance of the sum is the sum of the variances, nσ², so Var(x̄) = nσ²/n² = σ²/n. Its square root, σ/√n, is the standard error: how much the mean varies from sample to sample.

Var(x̄) = Var((X₁ + … + Xₙ) / n)
       = (1/n²) · Var(X₁ + … + Xₙ)      a constant comes out squared
       = (1/n²) · (σ² + … + σ²)         independence: no covariance terms
       = σ²/n,  so  SD(x̄) = σ/√n

The proof needs independence: with positively correlated data, such as repeat visits by the same users, the true standard error is larger than σ/√n.

Grinstead and Snell, Introduction to Probability: 6.2 Variance of discrete random variables (LibreTexts)OpenStax, Introductory Statistics 2e: 7.1 The central limit theorem for sample means

Try it in the central limit theorem simulatorPractise it with Prism

3/4. All three fit in a semicircle exactly when one of them starts a semicircle, going clockwise, that holds the other two. For a given point that chance is (1/2)², since each other point lands in that half independently, and two points can’t both do it except with probability zero, so the chances add: 3 × 1/4 = 3/4. For n points the same argument gives n/2ⁿ⁻¹.

import random
def same_semicircle(n=3):
    a = sorted(random.random() for _ in range(n))        # positions as fractions of the circle
    gaps = [q - p for p, q in zip(a, a[1:])] + [1 - a[-1] + a[0]]
    return max(gaps) >= 0.5                              # an empty arc of half the circle
trials = 200_000
print(sum(same_semicircle() for _ in range(trials)) / trials)    # about 0.75

Back the answer with a quick simulation, as in the code: it shows you check results rather than trust a memorised number.

Wendel (1962), A problem in geometric probability, Mathematica Scandinavica 11

Practise it with Prism

It depends on the significance level, the power you want, how variable the metric is and the smallest effect worth detecting. The required size grows with the variance and with one over the square of that effect, so halving the effect you want to detect needs about four times the users. For a conversion rate moving from 10% to 11%, at 5% significance and 80% power, that’s about 14,700 users per group.

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

h = proportion_effectsize(0.11, 0.10)          # effect size for 10% -> 11%
n = NormalIndPower().solve_power(effect_size=h, alpha=0.05, power=0.8, ratio=1)
print(round(n))                                # 14744 users per group

Divide by your daily eligible users to get the duration, then round up to whole weeks.

NIST/SEMATECH e-Handbook of Statistical Methods: 7.2.2.2 Sample sizes requiredstatsmodels: NormalIndPower.solve_powerstatsmodels: proportion_effectsize

Try it in the A/B test calculatorPractise it with Prism

A novelty effect is an early lift that fades because users explore something new; a primacy effect is the opposite, an early dip while regular users adjust. Plot the treatment effect by day, compare new users, who have no old habit, with returning ones, and run at least a full week. Be sceptical too: Microsoft researchers found most suspected cases were statistical artefacts of noisy early data, not real trends.

If the test spans a festival sale, check whether the effect holds outside those days before you call it novelty.

Kohavi et al. (2012), Trustworthy online controlled experiments: five puzzling outcomes explained, KDDKohavi, Henne and Sommerfield (2007), Practical guide to controlled experiments on the web, KDDDmitriev et al. (2017), A dirty dozen: twelve common metric interpretation pitfalls in online controlled experiments, KDD

Practise it with Prism

Checking repeatedly and stopping as soon as the p-value dips below 0.05 raises the false-positive rate above the 5% you planned for, because every extra look is another chance to cross the line by luck. Fix the sample size and duration in advance and analyse once at the end, or use a sequential method designed for continuous monitoring, such as always-valid p-values. Extending a test that missed, to chase significance, breaks the same rule.

Watching a dashboard is fine; acting on it mid-test is the problem, unless a guardrail shows real harm.

Dmitriev et al. (2017), A dirty dozen: twelve common metric interpretation pitfalls in online controlled experiments, KDDJohari, Pekelis and Walsh, Always valid inference: bringing sequential analysis to A/B testing (arXiv)

Try it in the A/B test calculatorPractise it with Prism

Probably not. At a 5% level, 20 independent tests give about a 64% chance that at least one is significant by luck alone, since 1 − 0.95²⁰ ≈ 0.64. Name one primary metric before the test and correct for the rest: Bonferroni divides α by the number of tests, while Benjamini-Hochberg controls the false discovery rate and is less strict. Treat a lone significant secondary metric as a hypothesis for the next test.

from scipy.stats import false_discovery_control

adjusted = false_discovery_control(p_values)   # Benjamini-Hochberg by default
significant = adjusted < 0.05

Slicing by city, device and app version is multiple testing too: at 5%, about one segment in 20 looks significant by chance.

Dmitriev et al. (2017), A dirty dozen: twelve common metric interpretation pitfalls in online controlled experiments, KDDSciPy: scipy.stats.false_discovery_control

Practise it with Prism

Possibly: it’s a sample ratio mismatch. In a published Microsoft example, 821,588 against 815,482 users diverged enough from 50/50 that a chi-squared test put the chance of a gap that large at under 1 in 500,000. An SRM is a symptom of a data problem, such as a bug in assignment, redirects or logging, and in most cases it invalidates the results, so find the cause before reading any metric.

from scipy.stats import chisquare

print(chisquare([821588, 815482]).pvalue)   # equal split expected by default; about 2e-06

Check the split from the first day, not at the end: researchers at Microsoft found about 6% of their experiments had an SRM.

Fabijan et al. (2019), Diagnosing sample ratio mismatch in online controlled experiments, KDD

Practise it with Prism