A training loop starts normally, then the printed loss climbs fast — 0.7, 1.1, 4.8,
then a huge number — and within about 20 epochs it prints inf, then nan, and
stays nan for every epoch after. NumPy may add RuntimeWarning: overflow encountered in exp or divide by zero encountered in log. Accuracy freezes at a
useless value and no amount of further training changes anything.
First hand-written training loops in module 2.2, and again with optimisers in 2.3
and 2.4. The builder set the learning rate high — 5, 10, or "let's try 100 to make
it faster" — or copied a rate that worked on a different, smaller problem. Losses
computed without clipping (log applied to a raw sigmoid output) fail loudest.
Deeper and wider networks fail faster because each layer amplifies the oversized
update. Cousin of WR-120: that entry is the forward pass exploding from bad init;
this one is the update step doing the same damage. Linked module: 2.2.
Each gradient-descent step moves a weight by learning rate times slope. When the
step is too large, the ball does not descend the valley — it overshoots to the far
wall at a higher point, where the slope is steeper, so the next step is larger
still. Weights grow geometrically, the pre-activations grow with them, exp
overflows, the sigmoid pins to exactly 0.0 or 1.0, and log(0) returns -inf.
One inf in the loss becomes nan in the next gradient, and nan poisons every
number it touches from then on — nan never heals. Confirm by printing the loss
every epoch from epoch 1: a healthy run drifts downwards; this failure shows the
climb (2.2's measured example: 0.70, 0.87, 1.11 by epoch 3 at rate 10). Printing
np.abs(W1).max() per epoch shows the same geometric growth in the weights.
Cut the learning rate by 10x and rerun; repeat until the loss falls instead of
climbing. Do it as the triptych habit from the lesson: train the same seed at
three rates spanning two orders of magnitude (0.01, 1.0, 10) for a few hundred
epochs each, read the three curves, and pick the stable one — the test costs a
minute. Keep the display loss safe with np.clip(p, 1e-12, 1 - 1e-12) inside the
log, so saturation prints a large finite number instead of nan. Know the
professional guard by name: gradient clipping — capping the gradient's size before
the update — which module 2.4 teaches properly. And once a run has printed nan,
restart it from a fresh init; the poisoned weights are not recoverable.
Never launch a long run at an untested learning rate — triptych first. Keep a
per-epoch loss print in every development loop; the geometric climb is visible
several epochs before the first inf. Treat the first inf or nan as a stop
sign, the same discipline WR-020 taught for overflow and WR-120 for the forward
pass. When a rate works, record it next to the seed and dataset it worked on —
rates do not transfer between problems.
Q: "Your training loss printed nan at epoch 18. Walk me through your diagnosis and
fix." Strong answer: name the mechanism — learning rate too high, each step
overshoots to a steeper point, weights grow geometrically until exp/log
overflow, and one inf becomes a permanent nan; diagnose from the per-epoch
loss print showing a climb before the blow-up; fix by cutting the rate 10x and
re-testing three rates side by side; name gradient clipping as the production
guard. Bonus: connect it to overflow from 0.7 (WR-020) and note that nan
requires a restart, not a patch.