Draw the line, choose your mistakes.
A churn model scores 1,000 customers from 0 to 1: how likely each one is to leave. Everyone at or above your line gets a call from the retention team. Move it, and watch the confusion matrix, precision, recall and the ROC curve change together.
Guess first
69 of these 1,000 customers left. The model calls everyone who scores 0.5 or more. Which is more accurate: the model, or simply saying nobody leaves?
Saying nobody leaves is right about all 931 who stayed and misses all 69 who left. The model catches 45 of the 69, and calls 348 who were staying. When few customers leave, accuracy rewards ignoring them, so churn and fraud models are judged by precision and recall.
Move the lineSet the line
69 who left
931 who stayed
Score: how likely to leave, from the model
Called: at or above the lineNot calledEach row is drawn to its own height.
Call the 393 customers who score 0.50 or more, and you reach 45 of the 69 who were going to leave.
- Recallof the 69 who left, how many you called65%
- Precisionof the 393 you called, how many were leaving11%
- False alarm rateof the 931 who stayed, how many you called37%
- Accuracyof all 1,000, how many it judged right63%
- F1precision and recall, balanced0.19
ROC AUC 0.68. Every line you could draw, from calling everyone to calling nobody, gives one point: recall against the false alarm rate. Pick one customer who left and one who stayed, and the model scores the leaver higher 68% of the time. Guessing scores 0.5, the dashed diagonal.
Average precision 0.19. The same lines, as precision against recall. Guessing scores 0.069, the share who left, the dashed line. With few leavers, this curve shows the cost of false alarms that the ROC curve flattens.
The data: the ML guide’s churn table, synthetic, 5,000 customers of an Indian telecom; its 1,000 test customers, held out by train_test_split (stratified, random_state=42). The model: its LogisticRegression(class_weight="balanced"), scored by scikit-learn 1.8.0 on 1 Nov 2026. Everything else is computed here, in your browser.
One model, many confusion matrices.
Four boxes, two mistakes
A model gives a score; a line turns it into a decision. Each customer lands in one box: caught, missed, false alarm or left alone. A miss loses a customer; a false alarm costs a call. Which costs more decides the line.
Precision and recall pull apart
Lower the line and you catch more leavers (recall rises) but call more people who’d stay (precision falls). F1 balances the two. In practice a team sets the line by cost, or by how many calls it can make in a week.
Accuracy hides rare cases
With 6.9% leaving, saying nobody leaves is 93.1% accurate. ROC AUC scores how well the model ranks, across every line. When positives are rare, the precision–recall curve shows the false alarms more plainly.
Sources
- scikit-learn, “Metrics and scoring: quantifying the quality of predictions”: the confusion matrix, precision, recall, F1, ROC AUC and average precision.
- Fawcett, “An introduction to ROC analysis”, Pattern Recognition Letters, 2006: the ROC curve, and AUC as the chance a positive is ranked above a negative.
- Saito and Rehmsmeier, “The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets”, PLoS ONE, 2015.