A fraud (or churn, or demand) model that validated beautifully collapses in its first weeks live: catch rates at a fraction of the offline number, review queues full of false alarms, and a dashboard that looks nothing like the validation report. Retraining on fresher data helps briefly, then the gap reopens. Offline PR-AUC said 0.7; the live behaviour implies something under 0.1.
Months of time-stamped transactions were pooled into one table and split with a
plain random train_test_split. The behaviour being predicted drifts: fraudsters
change tricks, customers change habits, festivals and salary cycles come and go.
Common wherever the modelling team inherits "the last six months as one CSV" and
the split function everyone knows is the one that gets called. The date column
being included as a feature makes the flattery worse, not better — the model
learns period-specific rules it can never reuse. Linked module: 1.7 (Leakage,
imbalance, split discipline).
This is temporal leakage: a random split scatters every week across both sides, so the model trains on rows from the future and is tested on the past it already saw. The offline score then measures interpolation inside known weeks, while production demands extrapolation into unknown ones. Confirm with one paired experiment: keep the model fixed, evaluate once on the random split and once on a time-ordered split (train on the earliest months, test on the latest). A gap like 0.72 against 0.09 is this incident in one line. Check the feature list too — a raw date or running row-count feature alongside a random split is a near-certain tell.
Re-evaluate everything on a time-based split and treat that number as the only real one; expect it to be painfully lower, and say so plainly in the corrected report. Then build a rolling backtest: train on months one to four, test on five; slide forward and repeat, so the report shows how the model ages. Ship with a retraining cadence chosen from the backtest curve, not from hope, and re-score the launch-period decisions if money moved on the inflated number.
House rule: if the rows have a timestamp, the default split is temporal, and a random split needs a written justification. Keep one held-out final month that no experiment may touch until sign-off. Watch the offline-versus-live gap on a dashboard after launch — temporal drift is not a one-time bug but a permanent condition, and the backtest cadence is the instrument that tracks it.
Q: "Your model's offline PR-AUC is 0.72 but production behaves like 0.1. Data and code are bug-free. Walk me through it." Strong answer: name the random split on time-ordered data, explain train-on-future test-on-past in one sentence, propose the paired random-versus-temporal evaluation as the confirming artefact, and finish with the rolling backtest and retraining cadence. Mentioning that drift makes this a recurring cost, not a one-off fix, separates practitioners from textbook readers.