A forecasting model posts a backtest MAE that beats the seasonal-naive baseline by a margin nobody on the team has seen before — and then performs worse than that same cheap baseline in its first live month. Offline, errors looked small and symmetric; live, forecasts lag every turn in the series and the error roughly doubles. Feature importance is dominated by one rolling-mean column, far above every lag and calendar feature.
The feature table was built in pandas with rolling windows, and one of them
included information from the future: a centred window
(rolling(7, center=True)), a rolling mean computed on the unshifted target
(so today's window contains today's value), or a smoothing pass applied to
the whole series before the split. Offline, pandas happily computes these
columns for every historical row. Live, the serving code must build the same
feature at 6 a.m. for a day that has not happened — so it quietly computes
something different, or fills with NaN. Common wherever notebook feature code
is copy-pasted into a daily job. Linked module: 1.10 (Time series basics).
This is leakage wearing a moving average as a mask: tomorrow's orders were inside today's feature, so the backtest scored a model that could see forward. A centred 7-day window reaches three days into the future; an unshifted rolling mean leaks today's own target into its feature. Three tells. First, a feature whose correlation with the target towers over every honest lag. Second, the as-of test: recompute the feature table using only rows dated strictly before each prediction day and compare — leaked columns change value. Third, the backtest-versus-live gap itself, with live errors trailing turning points, because the live feature no longer contains the future that the training feature did.
Shift before rolling, everywhere: today's rolling features must end at
yesterday, as in series.shift(1).rolling(7).mean(), and no center=True
in any forecasting feature. Rebuild the whole table under that rule, then
re-run walk-forward validation and expect the score to drop toward honesty —
report the drop plainly rather than hiding it. Fix the serving mismatch at the
root by using one shared feature-building function for training and serving,
so the offline table and the 6 a.m. table are computed by the same code path.
Adopt the as-of rule for every feature: it must be computable, unchanged, at the moment of prediction using only the past. Audit with walk-forward discipline — recompute features inside each fold from that fold's past and compare against the precomputed table; any cell that differs is a leak. Treat a backtest that beats seasonal-naive by a suspicious margin the way module 1.7 treats any sudden jump: as a bug until an audit proves otherwise.
Q: "Backtest MAE 4, live MAE 9, and the model now loses to copy-last-week. The code has no train-test contamination in the split itself. What do you check?" Strong answer: go straight to feature construction — centred or unshifted rolling windows leaking the future — name shift-before-rolling as the rule, the as-of recomputation as the audit, and one shared feature function for train and serve as the structural fix. Connecting it to the general leakage reflex ("too good is a bug") earns the senior nod.