A forecasting model keeps predicting sudden crashes that never happen: every few weeks it forecasts near-zero orders for a day or two, planners cut stock, and actual demand arrives normal. Weekly report totals occasionally show impossible dips — a week at half volume in a growing business, or a zero-order day for a platform that has never had one. The model looks worst right after calendar gaps, and retraining makes the phantom crashes more confident, not less.
The raw event data had missing days: an app outage, a failed ingestion job, a
vendor API down, or a store onboarded mid-history. Somewhere in the pipeline
those holes became zeros — a fillna(0) after reindexing, a resample over a
sparse table, or a join that silently dropped days and a later step that
padded them. Common in any team that aggregates event logs into daily series
without an explicit gap policy. The model then trained on days that said
"zero orders" when the truth was "no data". Linked module: 1.10 (Time series
basics).
The pipeline confused two different facts: a zero (the platform was open and
nobody ordered) and a gap (nobody recorded what happened). Lag features
spread the poison: a fake zero enters lag_1, lag_7 and every rolling mean
that touches it, so the model learns "after a quiet day comes a crash" from
days that never existed. Confirm in minutes: list every zero-valued day and
cross-check it against the ops calendar and outage log — real zeros have a
reason (planned closure, region not yet launched). Then compare row count
against the full calendar: asfreq("D") on a datetime index exposes the
holes as NaN, which is what they should have been all along.
Rebuild the series with an explicit gap policy. Reindex to the full calendar
so gaps are NaN, then fill by rule, not by reflex: a documented closure day is
a true zero; an outage or ingestion gap gets a seasonal fill (same weekday
last week) or time interpolation, whichever the series' weekly shape prefers.
Add a was_filled flag column so the model and every reviewer can see which
rows are reconstructed. Retrain, and re-run the weekly aggregates — totals
built on seasonal fills land near truth, while the zero-filled version
undercounts entire weeks.
Make the gap audit a standing first step for any series: assert that the row
count equals the calendar length, print all zero-valued days with their
reasons, and refuse fillna(0) unless a human has labelled the day as a true
closure. Keep the outage log as data beside the festival calendar — both are
inputs, not trivia. In review, one question catches this whole class of bug:
"how does this pipeline tell no-data from zero?"
Q: "Your forecaster keeps predicting crashes that never happen, and the bad days cluster after data gaps. Diagnose it." Strong answer: name the no-data-versus-zero confusion, trace how one fake zero propagates through lag and rolling features into learned "crash" patterns, give the confirming cross-check against the outage log, and close with the policy — NaN first, fill by rule, flag filled rows. Mentioning that weekly totals were also corrupted (reporting, not only modelling) shows systems thinking.