A model that memorises fails on new data.

The score here is AUC: the chance the model ranks a customer who left above one who stayed, where 0.5 is a coin toss and 1 is perfect (lesson 4 explains it). The ML guide keeps 1,000 of its 5,000 customers aside until the end; their score estimates how the model will do on customers it has never met, the only score that matters.

Degree 15 bends through all 16 points it learned from (an error of 0 there) and swings wildly between them: it learned the noise. That’s overfitting. Degrees 0 to 2 are too stiff to follow the curve, missing old and new points alike: underfitting. Degree 3 misses the 16 points by a little (0.24) and new ones by about as much, close to the noise itself (0.3).

Each level lets the tree split customers into smaller groups, until with no limit it has 641 leaves, many holding a single customer. The training score climbs all the way; the test score peaks early and falls. Limiting the depth, or asking for a minimum number of customers per leaf, is how trees are regularised.

Cross-validation uses all the training data for learning and for checking: split it into k folds, train k times, score each time on the fold left out, and average. Stratified folds keep the churn rate the same in each fold, which matters when only 6.9% churn, as here. For data in time order, validate only on later periods.

The two cures for overfitting: more data, and regularisation, a penalty for complexity. Ridge (L2) shrinks every coefficient; lasso (L1) can set some exactly to zero, dropping features. For trees it’s depth and leaf size; for networks, dropout and weight decay. Underfitting needs the opposite: a more flexible model or better features.