A model posts a near-perfect offline score — 0.99 accuracy, recall in the high nineties — sails through review, and then does one of two things in production: predictions crash with a missing-column error, or every prediction comes back negative because the star feature is always empty. Offline the model looks like the best the team ever built. Live, it catches nothing.
A complaint-prediction model at a consumer-services company. The training table was
exported from the operations database months after the events, so every row carried
its full history — including a refund_processed flag. Refunds are issued only
after a complaint is raised and closed, but in the export the flag sits innocently
beside genuine features like delivery delay and support calls. Any table exported
"as of today" about events that finished long ago is at risk: default tables with
recovery_agent_visits, churn tables with account_closure_reason, medical tables
with treatment_given. Linked module: 1.7 (Leakage, imbalance, split discipline).
This is target leakage: a feature recorded after the outcome it is supposed to predict, which makes it the answer wearing a costume. The tell is a score jump that outruns every baseline the team has ever seen on similar data. Confirm in three steps. First, print the model's coefficients or feature importances — a leak towers over every honest signal (the demo's refund flag carried +3.78 against +0.82 for the best real feature). Second, take the suspect column to the data dictionary and ask when it is written: before or after the outcome event? Third, refit without the column — if the score falls off a cliff back to a believable number, the cliff is the measure of the leak. Do not skip step three; a "harmless" version of the column sometimes survives in a correlated cousin.
Remove every feature that is written after the outcome event, retrain, and re-report with the honest score — including telling stakeholders that the earlier number was an artefact, not a regression. Then fix the source: rebuild the training table "as of prediction time", so each row contains only what the system knew at the moment the prediction would have been made. Where the warehouse cannot reconstruct that, drop the column rather than gamble on it.
Run a prediction-time audit before the first training run: for every feature, write one line naming when the value becomes known relative to the prediction moment. Anything answered "after the outcome" or "unsure" is out until proven safe. Adopt the reflex the linked module drills — a sudden jump in score is a bug until proven otherwise — and make the coefficient/importance audit a standard artefact in every model review. One glance at that table catches most leaks before they cost a launch.
Q: "Your teammate's fraud model scores 0.99 offline but flags nothing in production. The model file is fine. What happened?" Strong answer: name target leakage in one sentence, explain that a post-outcome column existed in the training export but is empty at prediction time, and give the three-step confirmation — importance audit, data-dictionary check on write time, refit without the column. Finishing with "the honest 0.75 was always the real model" signals you understand the score was never a loss.