A sequence model trains cleanly, the loss falls, and the headline accuracy looks respectable. Then somebody checks it against real inputs and it is wrong in a pattern: short inputs are handled well, long ones are handled badly, and the prediction almost always agrees with the last sentence of the text. Paste the same review with an extra paragraph of neutral filler on the front and the answer does not change. Move the deciding sentence from the end to the beginning and the answer flips to whatever the ending suggests. Nothing errors, no warning appears, and rerunning with more epochs makes the training loss smaller while the live behaviour stays the same.
Any RNN, LSTM or GRU that reads a whole input and predicts from the final hidden state — module 2.7's sentiment classifier, a complaint router at a bank or an NBFC, a session model over a customer's click history, a sensor model over a long window. The usual birth is a benchmark run on short inputs. In 2.7's measured recall task, every model scores 1.000 at sequence length 5 and 1.000 at length 20, so a team that tested at those lengths concludes the architecture is fine. At length 50 the plain RNN reads 0.539 training accuracy and 0.507 test accuracy on a task where a single token decides the label, and the LSTM with its default forget bias reads 0.858 training and 0.506 test. A second birth is a dataset whose average input is short while the important production traffic is long: the average accuracy is carried by the short rows, and nobody splits the score by length. Linked modules: 2.7, 2.4 (reading two curves), 1.5 (scores by slice), 2.8.
A recurrent layer carries everything it knows in one hidden state, and every step multiplies that state by the same matrix and squashes it. Blame travelling back from the loss makes the same trip in reverse, one multiplication per step. Measure it rather than argue about it. In 2.7's 50-step run at recurrent gain 0.5, the gradient norm arriving at step 50 is 7.577e-02 and the norm arriving at step 1 is 5.032e-17, a shrink of about 0.490 per step. At the ordinary initialisation, gain 1.0, the factor is about 0.911 per step, which still loses roughly 99 percent of the signal across 50 steps. The first tokens therefore receive almost no training signal, and a weight that receives no signal cannot learn. Confirm the diagnosis with three cheap checks. Split your test accuracy into buckets by input length and look for a falling line. Score again with the deciding phrase moved to the front, the middle and the end of the same input. Print the gradient norm at each step for one batch, exactly as 2.7 Step 3 does, and read how far back the number stays above one thousandth of its value at the last step.
Take the cheapest fix first. Open the forget gates at initialisation: set the LSTM
forget-gate bias to 1 or higher and leave the rest alone. In 2.7's 50-step recall
task that one change moved test accuracy from 0.506 to 1.000, because the per-step
factor on the cell state moved from about 0.629 to about 1.016. Second, shorten
the distance the signal has to travel — read the sequence in both directions with
a bidirectional layer, or pool over all hidden states instead of taking only the
last one, so that early steps get a direct route to the loss. Third, if long
dependencies are the point of the task rather than an accident, stop patching and
move to an attention-based model: attention links any two positions in one hop,
against 49 hops for a 50-step RNN. Fourth, whatever you choose, report accuracy by
input-length bucket from now on, and keep a fixed set of long test inputs whose
deciding evidence sits at the front. Watch the mirror fault while you work: at
recurrent gain 1.5 the blame at step 1 grows to 5.271e+00, about 68 times the value
at step 50, so add torch.nn.utils.clip_grad_norm_ at a threshold near 1.0.
Clipping controls explosion and does nothing at all for forgetting.
Benchmark at the length you will actually receive, and at twice that length. Make "accuracy by input-length bucket" a standing column in your evaluation table, beside the overall number, the same way 1.5 made per-class recall standing. Build one synthetic memory test for every sequence project — a task where a single early token decides the answer — and run it at three lengths before you trust any architecture on real data. Set the forget-gate bias explicitly in code rather than accepting the framework default, and write a comment saying why. In review, ask one question of any recurrent model: how far is the evidence from the end of the input, and what did we measure at that distance?
Q: "Your LSTM classifies short reviews well and long reviews badly. Nothing errors. What is happening and what do you do?" Strong answer: name the vanishing gradient as the mechanism in one sentence, then give the measurement rather than the theory — gradient norm per step, and accuracy split by input length. Quote the shape of the decay: multiply by a factor below one at every step, so 50 steps back the signal is gone. Give the fix ladder in order of cost: forget-gate bias, then bidirectional or pooled readout, then an attention-based model. Finish with the distinction that separates a candidate who has done this from one who has read about it — gradient clipping fixes explosion, not forgetting, and the two faults come from the same shared matrix.