The training loop runs cleanly — no errors, no warnings, no nan — but the loss
climbs every single epoch, smoothly and steadily: 0.693, then past 1.0, then up to
a flat ceiling (in 2.2's reproduction: 0.693 to 1.157 by epoch 10, pinned near
13.539 by epoch 25). Accuracy drifts to coin-flip or worse. The code looks
finished and confident, and it is confidently walking uphill.
Hand-written backward passes in module 2.2. The classic causes: the builder derived
the output blame on paper and wrote (y - p) where the loss's slope is (p - y);
or applied the minus sign twice — once inside the gradient and once in the update
line — so the two cancel; or copied a formula from a blog that defined the loss
with the opposite sign convention. The forward pass is completely correct, so
spot-checking predictions reveals nothing. Linked module: 2.2.
The shape of the curve is the tell. A learning rate set too high makes the loss
bounce — up, down, up — because each step overshoots in a different direction
(WR-121). A flipped sign makes the loss climb monotonically and smoothly, because
every update is exactly the correct step, mirrored: the network is performing
flawless gradient ascent. Confirm with the lesson's numerical check: nudge one
weight by 1e-5 both ways, compute rise over run, and compare with the coded
gradient. If the two numbers match to six decimals but with opposite signs, the
backward pass has a sign flip. If the magnitudes differ too, the bug is elsewhere
(wrong term, missing ReLU gate). The check takes two forward passes per weight and
settles the question with evidence.
Find the flipped term at its source and fix it there — in 2.2's case, restore
(p - y) in the output blame. Do not "fix" it by flipping the update line from
-= to += to compensate: two wrongs produce a network that trains today and an
unmaintainable file that breaks the next time anyone touches either line. Then
rerun the full gradient-check table (one weight per layer, plus a bias) and
confirm every row matches in value and in sign before training again.
Make the gradient check a ritual, not a ceremony: keep the numerical() helper
permanently in the file and rerun the table after every edit to backward, the
loss, or an activation. Check signs, not only magnitudes. Distrust any smooth
uphill run immediately — a loss that rises every epoch without bouncing is a sign
error until proven otherwise. The whole ritual costs under a minute on a
17-weight network and catches the one bug that no error message will ever report.
Q: "Training loss increases every epoch — no nan, no warnings. What is your top suspect and how do you confirm it?" Strong answer: a sign error in the gradient or the update, making the loop do gradient ascent; distinguish it from a too-high learning rate by curve shape — smooth climb versus bouncing; confirm with a numerical gradient check and look for matching magnitude with opposite sign; mention the double-minus trap where gradient and update each carry the flip and mask the bug. Bonus: state the fix-at-source rule and why compensating in the update line is a maintenance landmine.