A model reports a spectacular accuracy — 96%, 97%, 98% — and the team celebrates, but in production it flags almost nothing. The fraud queue is empty, the churn campaign has no names on it, the manual reviewers see no change. When someone checks, the model predicts the majority class for nearly every row.
A classification problem with heavy class imbalance — 97 legitimate rows for every 3 fraud rows, 90 loyal customers for every 10 churners — and accuracy chosen as the metric by default, without asking the business what a mistake costs. Common wherever the interesting event is rare: fraud, default, churn, machine failure, disease screening. The celebration usually happens in a demo; the discovery happens weeks later in operations. Linked module: 1.1 (Framing an ML problem).
Accuracy rewards agreeing with the majority, and on a 97/3 split the majority is
almost everything: a model that predicts "not fraud" for every single row scores 97%
while catching zero fraud. The model quietly learnt that lesson. Confirm it in three
lines of pandas: print the class balance with value_counts(normalize=True), print
the prediction distribution the same way, and count true positives — rare-class rows
the model actually caught. If predictions are nearly all majority class and the
catch count is near zero, the "97% accurate" model is indistinguishable from a
one-line rule that says no to everything. The framing failure happened earlier: the
metric was picked by the data team alone, after modelling, instead of with the
business, before it.
Change the metric before touching the model. Sit with the business owner and price the two mistakes: what does a missed fraud cost, what does a false alarm cost? Then choose a metric that reflects those costs — recall on the rare class when misses are expensive, precision when false alarms are, or a cost-weighted combination when both matter. Re-score the majority baseline on the new metric (it will score near zero — that is the honest bar), re-evaluate the existing model against it, and only then decide whether retraining or rebalancing is needed. Report the new numbers to whoever saw "97%", explicitly retiring the old claim.
Choose the success metric with the business before any modelling, and write it on
the framing sheet next to the class balance. Always print value_counts on the
target first — imbalance spotted early forces the metric conversation. Always score
the majority baseline on the chosen metric; if the baseline scores 97%, the metric
is telling you it cannot see the problem. Treat any headline accuracy on an
imbalanced target as unproven until recall and precision on the rare class are
shown beside it.
Q: "Your fraud model is 97% accurate. Should we ship it?" Strong answer: refuse to answer from accuracy alone — ask for the class balance first, point out that on a 97/3 split the always-say-no baseline also scores 97%, and ask for recall and precision on the fraud class plus the cost of each error type. Walking through the majority-baseline check convinces the interviewer you have seen this failure; quoting accuracy back gets the candidate rejected on the spot.