A sequence model that scored well on a fixed-length toy task falls apart on real text of mixed lengths. The training loss falls, nothing crashes, and the accuracy is merely poor. Two extra signatures usually appear beside it. First, the score changes when you change the batch size or reshuffle the data, because a batch that happens to hold one very long row makes every other row in that batch worse. Second, the model predicts nearly the same answer for every short input, and its confidence is oddly flat. Sorting the data by length before batching "fixes" the score, which feels like a win and is actually the clearest proof of the bug.
Any real sequence dataset, since real sentences, sessions and sensor windows are
never all the same length. Module 2.7's own reviews are exactly eight tokens long,
which is a deliberate simplification; the moment you bring your own text, this
entry applies. The usual birth is one line: a batch is built by padding every row
to the length of the longest row with a pad id of 0, and the model is fed the
padded tensor with no mask. If the readout takes the last hidden state, as 2.7's
out[:, -1, :] does, a 6-token sentence padded to 120 hands the classifier the
state after 114 pad tokens rather than the state after the sentence. Other births
are a pad id that collides with a real vocabulary entry, a loss computed over
padded positions, and an attention model with no padding mask, where the softmax
spends real weight on empty slots. Linked modules: 2.7, 2.3 (shapes and dtypes at
the tensor boundary), 3.4 (tokenizers and their pad ids), 2.8, 2.10.
Padding is not neutral. A pad id of 0 is a valid index into the embedding table, so each pad token has a vector, that vector enters the recurrent step, and the hidden state keeps changing after the sentence has ended. Over 114 pad steps the state drifts toward whatever fixed point the pad vector produces, and the real content is squashed away by exactly the decay WR-136 measures. Confirm it with prints, not guesses. Print the shape of one batch and the list of true lengths beside it, and look at the gap between the maximum and the median. Feed one short sentence twice, once alone in a batch and once in a batch beside a very long row, and compare the two outputs: a correct pipeline gives identical numbers, a polluted one does not. Print the hidden state at the true final token and at the padded final position and compare them. Finally, check the loss: if you are training on token-level targets, count how many positions in the batch are padding and how many are real, because a loss averaged over both is being diluted by whatever fraction is empty.
Carry the true lengths with the data and use them. In PyTorch the supported route
is torch.nn.utils.rnn.pad_sequence to build the batch, then
pack_padded_sequence(x, lengths, batch_first=True, enforce_sorted=False) before
the recurrent layer and pad_packed_sequence after it; the layer then stops each
row at its own length and the returned final state is the state after the last real
token. If you keep the padded tensor instead, do not use out[:, -1, :]; index the
last real step per row with out[torch.arange(n), lengths - 1], or pool over the
real positions only using a boolean mask. Set the embedding's padding_idx to the
pad id so pad vectors stay at zero and receive no gradient. Pass ignore_index to
the loss for token-level targets. For attention models, pass the padding mask into
the attention call so padded keys are scored at minus infinity and receive zero
weight after the softmax. Then re-run the two-batch experiment above and require
the two outputs to agree to float noise before you trust any new number.
Treat lengths as a first-class column of the dataset, stored and passed with every batch, never recomputed by guessing where the zeros start. Add a batch audit to the top of every sequence training run, in the shape 2.3 taught: print the tensor shape, the minimum, median and maximum true length, the pad id, and the percentage of positions that are padding — if that percentage is above about 50, fix the batching before you tune anything else. Use a length-grouped sampler so batches hold similar lengths, which cuts both waste and drift, and keep the mask anyway, because grouping reduces the problem and does not remove it. Add one regression test that runs a single short input alone and inside a mixed batch and asserts the outputs match. In review, ask: where is the mask, and what does the readout index?
Q: "Your text classifier's accuracy changes when you change the batch size. Why?"
Strong answer: name padding without masking as the first suspect and explain the
mechanism — the longest row in each batch sets the padded width, so the amount of
pollution each short row receives depends on its batch-mates, which makes the score
batch-composition dependent. Give the confirming test in one sentence: run one
sentence alone and inside a mixed batch and compare outputs. Give the fix by name,
pack_padded_sequence with true lengths, or a mask plus per-row indexing of the
last real step, plus padding_idx on the embedding and ignore_index on the loss.
Bonus: note that sorting by length appears to fix the score without fixing the bug,
and that the same mask is what an attention model needs so the softmax does not
spend weight on empty positions.