Binary cross-entropy loss prints 0.6931 at epoch 1 and refuses to move — thousands of epochs later it still reads 0.693-something (2.2's reproduction settles at 0.6929). Every prediction hovers at 0.5, accuracy exactly equals the class balance of the data (0.51 on the XOR cloud), and there are no errors or warnings of any kind. The run looks alive; the network is learning nothing.
First training runs in module 2.2. The builder initialised all weights to zeros ("start clean, start fair") or to one shared constant, sometimes after seeing WR-120's warning against large weights and overcorrecting to none at all. The same plateau appears later whenever gradients cannot reach the weights — frozen layers, saturated activations — so the number is worth memorising now. Linked modules: 2.2, 2.4.
First, read the number itself: 0.693 is ln 2, the exact cross-entropy score of a
model that answers 0.5 to every question. A binary model pinned there has learned
nothing whatsoever. (The pattern generalises: a k-class model with no learning pins
at ln k — 1.609 for five classes.) Then the mechanism, with zero init: every
hidden pre-activation is 0, ReLU of 0 outputs 0, so the hidden layer is silent for
every point. Backprop's ReLU gate passes blame only where the neuron was active,
so zero activation means zero blame flows back: dW1 and dW2 are exactly 0.0,
and only the output bias moves, chasing the class average. Symmetric constant init
is the same disease in disguise — every neuron in the layer computes the same
value, receives identical blame, and updates identically forever: four neurons
pretending to be one. Confirm with two prints: p[:5] shows a wall of 0.5s, and
np.abs(dW1).max() prints exactly 0.0 epoch after epoch.
Break the symmetry: initialise weights with small random values —
rng.normal(0, 0.5, shape) is sound at this module's 2-4-1 scale — small enough
to avoid WR-120's explosion, random enough to give every neuron its own starting
point and therefore its own blame. Biases may stay at zero; they receive distinct
gradients once the weights differ. Rerun and watch the loss leave 0.693 within the
first hundred epochs (2.2's seeded run reads 0.4176 by epoch 100). If it still
does not move, the blockage is elsewhere — check that the update lines actually
touch every parameter.
Never zero-initialise or constant-initialise a weight matrix; zeros are for biases. Memorise 0.693 as an alarm number the way 0.5 accuracy is one — when any loss pins at a suspicious constant, compute the "predict the average, always" loss for your problem and compare. Log loss at epoch 1 in every run: a healthy small random init starts near ln 2 and leaves; starting far above it points at WR-120, staying at it points here.
Q: "A binary classifier's loss sits at 0.693 for the whole run. What does the number itself tell you, and what do you check?" Strong answer: 0.693 is ln 2, the always-0.5 cross-entropy — the model has learned nothing; name the two classic causes, zero init and symmetric init, and the mechanism — silent ReLUs pass zero blame backwards, identical neurons update identically; confirm via printed predictions and a max-gradient print; fix with small random initialisation. Bonus: generalise to ln k for k classes and connect the plateau to any situation where gradients cannot reach the weights.