A sequence model on time-ordered data reports a validation score that nobody in the team believes: 0.97 where the business baseline is 0.70, or a forecast error a third of the naive last-value rule. The training curves look textbook. Then the model goes live and performs at or below the lazy baseline from module 1.10. A second signature often appears first: the score barely moves when you shrink the model, delete features, or shorten the input window — a sign the model is reading something far easier than the stated task. A third: retraining on a later month gives a wildly different score with no change to the code.
Any sequence model over data that has a clock — demand per day, sensor readings,
UPI transaction streams, clickstream sessions, electricity load. The usual birth is
one habit carried over from tabular work: train_test_split(..., shuffle=True) on
rows that were never exchangeable. The second and more common birth is subtler and
survives careful people. Windows are cut first, over the whole history, and the
resulting windows are split at random afterwards, so a window ending on 14 March
lands in training while a window ending on 13 March lands in validation, and the
two overlap on almost every reading. A third birth is a feature computed over the
full series before the split — a rolling mean, a scaler fitted on everything, a
target encoded from all months — which is 1.7's leakage in sequence clothing.
A fourth is a label built from a future window while the input window is allowed to
run past the label's start. Linked modules: 2.7, 1.10 (time discipline), 1.7
(leakage and split discipline), 4.8 (drift in production).
Two different faults hide under one symptom, and you must separate them. Fault one is order destruction: shuffling the rows themselves removes the sequence, so the model is being asked to read a bag of readings while you believe it reads a series. Fault two is leakage: the order survives, but training and validation windows share readings, so the model has seen the answer's neighbourhood during training. Confirm fault one by printing the timestamps of the first ten rows of one training batch and checking they increase. Confirm fault two by computing, for every validation window, the maximum timestamp inside it and comparing it with the maximum timestamp seen in training: if any training window ends after any validation window begins, you have overlap. Then run the honest reference: 1.10's naive rule, predict the last value, on the same split. A deep sequence model that cannot clearly beat that rule has not earned its place, and a model that beats it by an implausible margin is usually reading the future. Finally, shuffle the target column on purpose and retrain: a well-built pipeline collapses to chance, while a leaking one keeps a surprising amount of its score.
Split by time first, then cut windows inside each split, and never in the other
order. Choose one cut date, send everything before it to training and everything
after it to validation, and hold a third, later block as the sealed test set that
is opened once. Add a gap between the splits equal to at least the input window
length plus the forecast horizon, so no training window can touch a validation
reading; this gap is the sequence-model version of 1.7's rule. Fit every scaler,
encoder and rolling statistic on the training block only, then apply it forward.
Keep shuffle=True inside the DataLoader, because shuffling whole windows is fine
and useful — the order that must survive is the order inside each window, not the
order the windows arrive in. For model selection use rolling-origin validation from
1.10: train to month k, validate on month k plus one, roll forward, and average.
Then re-run the naive baseline on the same split and put both numbers in the same
table, every time.
Write the split before the model. Make the first cell of every time-series notebook print the minimum and maximum timestamp of each split and the size of the gap between them, and treat an overlap as a build failure rather than a warning. Store the cut dates in the config, not in the notebook body, so every rerun uses the same boundaries. Keep the naive baseline in the results table forever; it is the cheapest lie detector you own. Add one automated test that asserts the maximum training timestamp is earlier than the minimum validation timestamp. In review, ask two questions of any sequence pipeline: what date splits these sets, and were the windows cut before or after that split?
Q: "Your demand forecaster validates at 0.97 and fails in production. Where do you look?" Strong answer: say leakage before anything else, then separate the two mechanisms — shuffled rows destroying order, and windows cut before the split so training and validation share readings. Give the confirming checks concretely: print timestamp ranges per split, look for overlap, and compare against the naive last-value baseline. Give the fix as a rule with a reason: split by time, then window, with a gap of at least one window plus the horizon, and fit every transform on the training block only. Bonus: point out that shuffling windows in the DataLoader is correct and often confused with the bug, and name rolling-origin validation as the way to select a model without spending the sealed test set.