A decoder-only model trains and the loss looks wonderful. On a character task
where a blind guess costs 3.611 nats, training loss reads 0.035 and validation
loss reads 0.032 after four hundred steps on a laptop CPU. Training and
validation agree almost exactly, so nobody suspects overfitting. Somebody posts
the curve in the team channel. Then the first sample is generated and the output
is order 122222122p22222222212129292122222222w222w2222222212wa2 — the prompt
survives and everything after it is noise. Longer training makes the loss smaller
and the samples no better. On a classification head instead of a generation head
the same fault shows up as near-perfect validation accuracy that collapses the
moment the model meets one real input.
Any decoder-only Transformer you wrote yourself or copied and edited: module 2.8's
tiny GPT, a next-token model over delivery notes, a next-event model over UPI
transaction logs, a session model over clickstream. The fault is born in one of
four ways. The mask was never applied, because the attention was written with
torch.nn.functional.scaled_dot_product_attention and the is_causal=True flag
was left out. The triangle was built the wrong way round, with torch.tril
where torch.triu was needed, or diagonal=0 where diagonal=1 was needed, so
each position can read one token ahead. The mask was created but never used,
sitting in a variable while s.softmax(-1) ran on the unmasked scores. Or the
mask was correct for training and dropped in an inference path that batches
differently. A fifth birth is a refactor: somebody swapped a custom attention for
nn.MultiheadAttention and did not carry the attn_mask across.
Linked modules: 2.8, 1.7 (leakage), 2.4 (reading two curves), 3.1.
The target for position t is the token at position t+1. A causal mask is the
only thing stopping position t from reading that token directly, so a model
without one is being asked a question with the answer printed beside it. That is
1.7's leakage in architecture form, and it produces the same signature: a score
that is too good, and behaviour that falls apart outside the training loop. At
generation time the future does not exist yet, so the model loses the input it
learned to depend on and emits whatever its now-meaningless features suggest.
Confirm it in three cheap steps, in this order. First, compare the loss against
the cost of a blind guess: ln(vocab_size) for a uniform next-token model, which
is 3.611 for a 37-character vocabulary. A model reading a tenth of that after a
few hundred steps on a tiny architecture is a bug report. Second, run the tamper
assertion: take one batch, record the logits, change only the LAST input token,
and run again. A correct causal model moves earlier logits by exactly
0.000e+00; module 2.8's unmasked model moved them by 1.265e+01. Third,
print one attention weight matrix for a short input and read row 2: a masked row
reads something like 0.108 0.115 0.777 followed by zeros, while a leaking row
has non-zero weights to the right of the diagonal.
Fix the mask, then prove it. In hand-written attention, build the mask with
torch.triu(torch.ones(T, T, dtype=torch.bool), 1) and apply it with
scores.masked_fill(mask, float("-inf")) before the softmax — diagonal=1 keeps
the position itself readable, which is correct, and diagonal=0 would wrongly
blind every position to its own token. Use minus infinity rather than a large
negative number or a zero weight, because exp(-inf) is exactly 0 and the
surviving weights still add to 1.000. With PyTorch's own functions, pass
is_causal=True to scaled_dot_product_attention, or pass
attn_mask=nn.Transformer.generate_square_subsequent_mask(T) to
nn.MultiheadAttention, and never rely on a default. Then add the tamper
assertion to your test suite as a real test, not a notebook cell: assert that
changing the last input token leaves all earlier logits bit-identical. Add a
second test that asserts the loss of a freshly initialised model is within about
0.1 of ln(vocab_size), which catches a leak that exists from step zero.
Retrain from scratch after the fix — a model trained with a leak has learned to
lean on a signal that will not exist, and its weights are not worth resuming.
Expect the honest loss to be much higher: module 2.8's correct model reads 0.264
where the leaking one read 0.032, and its samples are readable sentences.
Write the mask test before you write the mask. Keep ln(vocab_size) printed at
the top of every training log so the floor is always on screen next to the
current loss, the way 2.4 kept two curves on screen. Adopt the reflex that a
result far better than anything comparable is a defect until proven otherwise,
and say it out loud in review — the same reflex 1.7 taught on tabular leakage.
Generate one sample at every checkpoint, including step zero, because generation
exposes leaks that loss curves hide. When reviewing anyone's attention code, ask
two questions: which line applies the mask, and which test proves it is applied.
Q: "Your teammate's GPT reports a validation loss five times lower than the
published baseline and its generations are gibberish. What do you check?"
Strong answer: name the causal mask in the first sentence and explain why a
missing mask lets position t read its own target. Then give evidence rather than
theory — compare the loss against ln(vocab_size), and describe the tamper test:
change the last input token, and earlier logits must not move at all. Mention
that generation is the honest metric here, because the leaked input disappears at
inference time. Finish with the two details that separate someone who has fixed
this from someone who has read about it: diagonal=1 rather than diagonal=0,
and retraining from scratch instead of resuming a poisoned checkpoint.