A forward pass prints huge activations that grow layer by layer — hundreds, then
lakhs, then 1e+38 — until values print as inf, a sigmoid output pins to exactly
0.0 or 1.0, and any loss computed on top prints inf or nan. Sometimes
NumPy adds RuntimeWarning: overflow encountered in exp or
invalid value encountered in matmul. The code structure is correct; the numbers
are not.
First hand-built networks in module 2.1 and early training runs in 2.2–2.4. The
builder initialised weights with something like rng.normal(0, 5, shape) or
rng.uniform(0, 10, shape) — "random numbers are random numbers" — or scaled all
weights up while debugging. Deeper stacks fail faster: each layer multiplies the
previous layer's magnitude, so three layers can turn ones into lakhs. Linked
module: 2.1; the same disease returns as exploding gradients in 2.2.
Every layer roughly multiplies its input's magnitude by a factor set by the weight
scale and layer width. With weights drawn at scale 5 and a 100-wide layer, each
layer multiplies typical values by a large constant, so magnitudes grow
geometrically: about a hundred after layer one, thousands after layer two, lakhs
after layer three, inf soon after. This is 0.7's overflow lesson (WR-020) wearing
network clothes — np.exp and float64 both give up past a threshold. Confirm by
printing a magnitude per layer, e.g. print(i, np.abs(a).max()): an exploding pass
shows the maximum climbing by a near-constant multiple each layer, which is the
geometric signature. A healthy pass shows magnitudes staying within an order of
magnitude of the input.
Re-initialise with small random weights: rng.normal(0, 0.1, shape) is a sound
default at this module's scale, and dividing by the square root of the layer's
input width (the idea behind standard initialisers you will meet in 2.4) keeps
magnitudes steady across widths. Keep biases at zero to start. Rerun with the
per-layer magnitude print still in place and confirm the maximum stays tame
through every layer. Do not "fix" the symptom by clipping values to a ceiling —
clipping hides the disease and destroys the information the layers carry.
Initialise small by default and treat weight scale as a real setting, never a
throwaway. Keep a one-line magnitude probe in every hand-built forward pass while
developing; it costs nothing and turns explosions into a readable trend. Any inf,
nan, or a sigmoid saturated at exact 0 or 1 should stop you the way 1.3's
convergence warning does — a stop sign, not noise to scroll past.
Q: "Your forward pass prints inf by layer three. What happened and what do you do?" Strong answer: weights initialised too large, so each layer multiplies magnitudes geometrically until float overflow; diagnose by printing the max absolute value per layer and watching the multiple; fix with small-scale initialisation (and mention width-aware scaling as the principled version). Bonus: connect it to overflow from 0.7 and to exploding gradients arriving in 2.2 — the same growth, running backwards.