Fit every point, miss the next one.
A model learns from the points it’s shown and is judged on the ones it hasn’t seen. Fit a curve through 16 noisy points, from a straight line to a polynomial of degree 15, and watch its error on 200 new points from the same curve.
Guess first
A degree-15 polynomial can pass through all 16 points exactly. A degree-3 one can’t. On 200 new points from the same curve, which predicts better?
The error on new points (lower is better). Degree 15 bends through every point it learned from, and swings wildly between them: it learned the noise, not the curve. That’s overfitting. Degree 3 misses the 16 points by a little (0.24) and new points by about the same (0.31), close to the noise itself.
Try every degreeFit it
16 points it learns fromNew pointsThe true curveThe fit
Degree 3 does best on new points here: close to the noise, which no model can beat.
Degree3
Error on the points it learned from0.24
Error on new points0.31
On the points it learned fromOn new pointsThe noise: no model beats it
Degree of the polynomial. Tap or drag across the chart to choose; an arrow marks errors off the top.
The data: made up for this page, so the truth is known: points from the curve y = sin(2πx) + x/2 with random noise (standard deviation 0.3), and 200 new points from the same curve. The fit: least squares, or ridge regression when it’s on, computed here in your browser. Error: root mean squared error.
Too simple, too clever, just right.
Underfit, then overfit
Degrees 0 to 2 can’t bend enough to follow the curve: high bias. Past the best degree, the curve chases the noise in the 16 points: high variance. The error on the points it learned from only falls; the error on new points falls, then rises.
More data, or a penalty
With 100 points, even degree 15 stays near the true curve here (0.37 on new points). Ridge regression adds a cost for large coefficients and calms the swings: degree 15 on 16 points, with λ 0.1, misses new points by 0.40.
Choose on data it hasn’t seen
Training error always prefers the most complex model. Pick the degree, or any model’s complexity, by its error on held-out data: a validation set, or cross-validation.
Sources
- scikit-learn, “Underfitting vs. Overfitting”: polynomial features of rising degree, scored on held-out data.
- Hastie, Tibshirani and Friedman, “The Elements of Statistical Learning”, 2nd edition, 2009, section 7.3: the bias-variance decomposition.
- Hoerl and Kennard, “Ridge Regression: Biased Estimation for Nonorthogonal Problems”, Technometrics, 1970.