A PyTorch training loop starts well — the loss falls faster than expected for the
first few dozen epochs — then betrays: the loss climbs back up, drops again, and
settles into a wander far above where an identical run converged before. Accuracy
plateaus below its known ceiling. On wider or deeper networks the same run ends in
inf or nan instead of a wander. Nothing crashes, no warning prints, and rerunning
produces the same strange curve.
First hand-typed PyTorch loops in module 2.3, and any refactor afterwards. Common
births of the bug: a tutorial was copied that called zero_grad somewhere else; the
loop was restructured and the line fell out; or the builder assumed gradients reset
themselves after step(). The 2.3 measured demo: the buggy loop on the XOR cloud
reads 0.4094 at epoch 50 — ahead of the healthy 0.5945 — then 0.5406 at epoch 100
while the healthy run has passed 0.4176, and 0.3561 at epoch 500 while the healthy
run sits at 0.1371 on its way to accuracy 1.000. Linked module: 2.3.
loss.backward() does not overwrite .grad — it adds into it, by design, so
advanced users can accumulate gradients across several small batches. Without
opt.zero_grad() at the top of the loop, epoch 2's step uses epoch 1's blame plus
epoch 2's, epoch 3's uses all three, and so on. Early on this acts like a growing
learning rate, which is why the buggy loop looks faster at first. Later the sum is
dominated by stale directions measured at weights that no longer exist, and the
walk stops tracking the valley. The confirming evidence is the stored gradient
itself: print net.hidden.weight.grad.abs().max() each epoch. Healthy training
shows it shrinking toward zero as the loss settles; this bug shows it growing —
2.3's demo climbs 0.032 to 0.827 over 500 epochs while the loss wanders. Falling
loss early, wandering loss later, growing stored gradients: that triple is this
bug's fingerprint.
Put opt.zero_grad() back as the first line of the loop body and rerun from a
fresh init — the wandered weights are not worth saving. Then fix the habit, not
only the file: the canonical order is zero_grad, forward, loss, backward, step,
and it is memorised precisely so a refactor cannot silently drop a step. If the
code intentionally accumulates gradients over several batches (a real technique
for large models on small machines, taught in 2.9), that must be a visible,
commented choice: accumulate N batches, step once, then zero — never an accident.
Type the five-step loop from memory in fixed order every time; never assemble it
by pasting fragments. Keep a max-gradient print in development loops — it costs
one line and exposes this bug, WR-121's blow-ups, and WR-123's dead-network zeros.
When a loss curve looks better than expected early, treat it with the same
suspicion as one that looks worse. In review, check any PyTorch loop for the pair
backward and zero_grad in the same scope before reading anything else.
Q: "Your loss fell quickly for 50 epochs, then wandered upward and never
converged. No errors printed. What do you check?" Strong answer: name the
mechanism — .backward() accumulates into .grad, so a missing zero_grad
makes every step use the sum of all past gradients; explain why it looks fast
early (summed blame acts like a growing learning rate) and fails late (stale
directions); give the confirming print (stored gradient magnitude growing instead
of shrinking); recite the canonical five-step order. Bonus: mention deliberate
gradient accumulation as the legitimate cousin, and that it zeroes after each
optimizer step.