An anomaly detector goes live and floods the queue: hundreds or thousands of flags a day against a review team sized for dozens. For the first week the team triages heroically, then falls behind, then stops opening the queue. Weeks later a genuine fraud surfaces through a customer complaint — and the post-mortem finds the detector HAD flagged it, in position four hundred and something of an unread list.
An IsolationForest (or any scoring detector) whose flag rate was chosen for
comfort, not capacity — contamination=0.08 or 0.10 "so nothing slips
through" on tens of thousands of daily events, feeding a team that can inspect
thirty. Common at fintechs, payment ops and quality teams where the model side
and the ops side never sat together, and the flag threshold was set in a
notebook without asking who reads the flags. The failure is organisational as
much as statistical: an alert that nobody reads protects nobody.
Linked module: 1.9 (Unsupervised learning).
Contamination is not a detection dial, it is a queue-size dial: it tells the model what fraction of rows to call anomalous, and the arithmetic is public. 0.08 on 12,000 daily transactions is 960 flags a day into a 30-case team — a 3 percent read rate, decided the day the threshold was set. Confirm the incident in the queue metrics: time-to-review climbing week over week, review coverage falling toward zero, and precision-in-queue (how many opened flags were worth opening) so low that reviewers rationally stopped. The missed fraud is then not a model failure — its score was strong — but a ranking buried under weak flags nobody could reach.
Resize the flag rate to the review capacity: if the team reads 30 cases a day,
the pipeline surfaces the 30 lowest scores — set contamination near 30/12,000
(0.0025) or skip the flag label entirely and take score_samples bottom-N.
Add triage tiers so attention follows severity: tier 1 auto-blocks the extreme
tail, tier 2 goes to human review, tier 3 is logged for weekly batch scans.
Close the loop: every reviewed flag gets a verdict (data error, genuine,
explainable), and the verdict stream tunes both the threshold and the features.
Then re-run history: yesterday's buried true fraud is the business case for the
change.
House rule: no anomaly detector ships without a named queue owner and the capacity arithmetic written in the design doc — flags per day, reviewers, cases per reviewer. Alert volume is reviewed monthly like a cost, because it is one: attention. And every queue carries the triage taxonomy from module 1.9's exercise — data-error, genuinely-unusual, explainable — so "flagged" is the start of a verdict, never the end of one.
Q: "Your fraud team misses a real case that the model actually flagged. Walk me through what went wrong and your redesign." Strong answer: state the queue arithmetic first (flag rate versus review capacity), name contamination as a queue-size dial rather than a safety dial, and redesign with bottom-N scoring sized to the team, triage tiers, and a verdict feedback loop. The differentiator is saying plainly that an unread alert provides zero protection — detection only counts when attention can reach it.