A fraud (or rare-event) model reports a superb ROC-AUC — 0.93, 0.95, 0.97 — and the deck calls it world-class. In production the review queue is nearly empty: at the deployed threshold the model flags a handful of cases and catches almost none of the real positives. The metric says brilliant; the operations floor says the model does nothing.
Heavily imbalanced classification — one fraud in 200 payments, one defaulter in 300 accounts, one machine failure in 500 shift-days. ROC-AUC was chosen because it is the default "serious" metric in tutorials and AutoML summaries, and no one looked at what any single threshold actually delivers. Common in payments, insurance claims, and medical screening pilots. The gap surfaces weeks after launch, when someone asks why the fraud queue has three rows. Linked module: 1.5 (Evaluation & metrics deep dive).
ROC-AUC measures ranking: how often a random positive scores above a random negative.
Its false-alarm axis divides by the number of negatives — and on rare-positive data
that denominator is enormous, so even thousands of wasted flags barely move the curve.
A model can rank positives loosely near the top of a 10,000-row pile, earn 0.95, and
still offer no threshold where precision and recall are both usable. Confirm in three
prints: the class balance (value_counts(normalize=True)), the PR-AUC
(average_precision_score — it will be a fraction of the ROC number), and a threshold
sweep at 0.1/0.3/0.5 showing flagged versus caught counts. If ROC is high while PR-AUC
is low and every gate catches a sliver, you are holding this incident.
Change the headline, then the decision. Report PR-AUC beside ROC-AUC and make PR-AUC the number leadership sees for rare positives. Replace the single score with a threshold sweep: flagged, caught, precision, recall at each gate, plus the catch rate of the review queue the business can actually staff (top-K by score, not a probability cutoff). Re-anchor expectations with the do-nothing baseline. Only after the honest numbers are on the table decide whether the model needs better features, more positive examples, or a different operating point — often the top-K queue is already useful even when the 0.5 gate is not.
On any imbalanced target, print the class balance before the first model and write the metric decision down: PR-AUC and a named operating point, agreed with the business. Ban naked ROC-AUC headlines on targets rarer than roughly one in twenty. Every results message carries three things: baseline, metric at the chosen gate, and the flagged-versus-caught table. Treat a large ROC-minus-PR gap as a standing alarm that rarity, not skill, is doing the flattering.
Q: "Our fraud model has 0.95 ROC-AUC. What do you check before shipping?" Strong answer: state what ROC-AUC measures in one sentence, explain why its false-alarm axis is diluted when negatives flood the data, then ask for the fraud rate, the PR-AUC, and a threshold sweep showing caught counts at usable gates. Offering the top-K review queue as the operational view separates candidates who have run this in production from those who memorised the definition.