A daily demand forecast that ran quietly for months collapses in one specific week: predictions come in 40-60% under actual orders, dark stores run out of stock by early evening, and the ops channel fills with screenshots of empty shelves. The MAE dashboard, flat for a quarter, spikes to three or four times its usual level — and the spike lines up exactly with Dussehra-to-Diwali. The week after, the model is back to normal as if nothing happened.
The model was trained on the history the team had: three or four quiet months, typically June to September, with no major festival inside the window. Common in young quick-commerce, kirana-platform and logistics teams whose data collection started recently, and in any project where "use whatever history we have" was the whole data strategy. The feature table carried lags, rolling means and day-of-week — but either no festival column at all, or one that was zero on every training row. Linked module: 1.10 (Time series basics).
A model can only learn patterns that appear in its training window, and no
festival ever did. A festival spike is not noise; it is seasonality on a
calendar the model was never shown. Two confirmations. First, print the
training date range against a festival calendar — if no Diwali, Dussehra or
Eid falls inside it, the model is festival-blind by construction. Second, if
an is_festival column exists, check its training values: a column that is
constant in training is unlearnable, whatever it does at prediction time.
A walk-forward fold placed over last year's festival week (when history
allows) shows the failure before customers do; lag features react a day late
by design, so they cannot rescue a spike that arrives overnight.
Three layers, cheapest first. One: add a festival calendar as data — columns for festival day, festival eve and festival week, built from a maintained list of dates, so the spike becomes a learnable feature the moment history contains one example. Two: until the model has lived through a festival cycle, do not let it fly solo — apply a manual uplift agreed with ops (last year's uplift factor, or a category-manager estimate) on calendar-flagged days. Three: add a humility window — during flagged weeks, widen safety stock, lower the confidence shown to planners, and route forecasts through human review. Then retrain after the festival with the new rows and keep the calendar features in.
Before training any forecaster, run a coverage audit: does the training window contain at least one full cycle of every season you will forecast — weekly, salary-cycle, and festival? Keep the festival calendar as a versioned data file, not tribal knowledge, and review it each quarter (dates move every year). Keep one walk-forward fold pinned over the most recent festival season as a standing regression test; a model that fails that fold does not ship in October.
Q: "Your demand model's MAE triples during Diwali week, then recovers on its own. Walk me through the diagnosis and the fix." Strong answer: state that the training window contained no festival so the pattern was unlearnable, name the constant-column check and the calendar-overlap check as proof, then give the layered fix — calendar features, manual uplift until history exists, humility window during flagged weeks. Adding that lag features always chase a spike one day late shows real time-series understanding.