Accuracy can mislead. Precision and recall don’t.

Lesson 4 of 7About 10 minutes5 steps

When only 7 in 100 customers leave, a model that says “nobody leaves” is 93% accurate and useless. Here is how to judge a classifier properly, on a real churn model’s scores for 1,000 customers.

  • Read a confusion matrix
  • Choose between precision and recall for a business
  • Explain the ROC curve and AUC

You need: Lesson 3.

  1. 1Distributions and sampling
  2. 2Confidence intervals and tests
  3. 3Training, testing and overfitting
  4. 4Judging a classifier
  5. 5Trees and ensembles
  6. 6Clustering and PCA
  7. 7How a neural network learns
0 of 5 steps

  1. The accuracy trap

    Predict first

    69 of 1,000 test customers left. Which is more accurate: the model at its default line of 0.5, or simply saying nobody leaves?

    Just show me

    Saying nobody leaves: 93.1% accurate, against the model’s 62.8%. Yet it catches none of the 69 who left; the model catches 45.

    With rare classes (churn, fraud, disease), accuracy rewards ignoring them. You need to count the kinds of right and wrong separately, which is what a confusion matrix does: who left and was flagged, who left and was missed, and the same for those who stayed.

    Customers by what happened and what the model said, with the line at 0.3
    FlaggedNot flagged
    Left 6960caught9missed
    Stayed 931643false alarms288left alone

    Precision 8.5%Recall 87%Accuracy 34.8%

    Customers by what happened and what the model said, with the line at 0.5
    FlaggedNot flagged
    Left 6945caught24missed
    Stayed 931348false alarms583left alone

    Precision 11.5%Recall 65.2%Accuracy 62.8%

    Customers by what happened and what the model said, with the line at 0.7
    FlaggedNot flagged
    Left 6919caught50missed
    Stayed 93188false alarms843left alone

    Precision 17.8%Recall 27.5%Accuracy 86.2%

    The ML guide’s logistic regression on its 1,000 test customers: flagged when a customer’s score reaches the line

    Quick check

    A fraud model is 99.8% accurate, and fraud is 0.2% of payments. Good?

    Compare with the trivial rule first, then look at precision and recall.

  2. Precision and recall

    Predict first

    At 0.5 the model flags 393 customers, and 45 of them really leave. Its precision?

    Just show me

    11.5%: 45 of the 393 it flagged. Its recall is 65.2%: 45 of the 69 who left.

    Precision: of those flagged, the share that were right. Recall: of those who really left, the share that were flagged. F1 is their harmonic mean, high only when both are. A cheap action, such as a discount by SMS, can live with low precision; an expensive one, such as a call from a manager, needs it higher.

    Precision and recall, symbol by symbol

    Quick check

    A cancer screening test: which comes first?

    Missing a case is the costly error; follow-up tests weed out the false alarms.

  3. The line is a business choice

    Predict first

    Lower the line from 0.5 to 0.3. What happens?

    Just show me

    Recall rises to 87% (60 of 69) and precision falls to 8.5% (60 of the 703 flagged).

    The model gives each customer a score; the line turns scores into decisions. Where it goes depends on costs: if keeping a customer is worth ₹2,000 and an offer costs ₹100, you can afford many false alarms. 0.5 is only scikit-learn’s default, not a law.

    Move the line yourself in the confusion matrix

    Quick check

    Offers are cheap and losing a customer is costly. Where should the line go?

    When a miss costs more than a false alarm, lower the line.

  4. ROC and AUC

    Predict first

    The model’s AUC is 0.68. What does that mean?

    Just show me

    Pick one customer who left and one who stayed: 68% of the time, the model gives the leaver the higher score.

    The ROC curve plots recall against the false-alarm rate at every line; the area under it, AUC, sums up the ranking in one number that doesn’t depend on the line: 0.5 is a coin, 1 is perfect. For rare classes, the precision-recall curve shows the trade-off you’ll actually feel, because ROC can look rosy when there are many negatives.

    Quick check

    Two models: AUC 0.68 and 0.71. Is the second better for your campaign?

    AUC sums up every line; the campaign runs at one.

  5. Choose the measure before the model

    Predict first

    A model flags UPI payments for 50 reviewers who can check 50 a day. What matters most?

    Just show me

    Precision in the top 50 a day: that’s what the reviewers will actually see.

    Write down the decision the model supports and what each mistake costs, then pick the measure that matches: precision at the team’s capacity, recall at a bearable false-alarm rate, or rupees saved. Report it beside a simple rule, so everyone can see what the model adds.

    Quick check

    Which baseline should every classifier be compared with?

    If a model can’t beat a trivial rule on the measure that matters, it isn’t ready.

In the interview: judging a classifier.

Ask Prism about this lesson

Sources

  1. scikit-learn, “Metrics and scoring: quantifying the quality of predictions”.
  2. Fawcett, “An introduction to ROC analysis”, Pattern Recognition Letters, 2006.
  3. Saito and Rehmsmeier, “The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets”, PLoS ONE, 2015.
  4. Our confusion matrix, the ML guide’s logistic regression scoring its 1,000 test customers.

Back: lesson 3, Training, testing and overfitting