A training run looks like a triumph. The printed loss falls epoch after epoch and ends near zero — 0.0007 is a real measured example — and accuracy on the training data reads 1.00. Then the model meets data it has not seen and collapses: test accuracy sits barely above the do-nothing baseline, the demo flags the wrong customers, and a plain logistic model from Phase 1 beats it. Nothing crashed, no warning printed, and the log contains exactly one column of numbers.
Any training loop from module 2.3 onwards, and hardest on small tables where a
64-unit network can memorise every row in under a hundred epochs. The usual births:
the loop was copied from 2.3 (which had no validation split), the data was split
two ways instead of three, or the run was launched before dinner with range(1, 201)
and read only when it finished. The 2.4 measured demo, on 281 Ludhiana churn rows
with a 6-64-64-1 network and Adam at 0.01: at epoch 20 the two losses read
0.3735 and 0.4096; at epoch 100 they read 0.0596 and 0.9111; at epoch 400 they read
0.0007 and 2.3038. Test accuracy at epoch 400 is 0.712 against 0.824 at epoch 16.
Linked modules: 2.4, 1.7 (split discipline).
Training loss measures how well the network reproduces rows it has already been shown, so it can always be driven to zero given enough capacity and enough epochs. It is a progress bar, not a score. Overfitting is the state where the network stops learning the pattern and starts learning the rows, and it is invisible in the training column by construction — the training loss keeps improving the whole time the model gets worse. Only a second measurement on data excluded from the weight updates can see it. Confirm the diagnosis in one run: split the training rows again into train and validation, retrain, and print both losses every epoch. Overfitting shows as a V in the validation column while the training column falls straight; the epoch where the V bottoms is the model you actually wanted. If validation tracks training all the way down and both stay high, the illness is the opposite one (underfitting) and the prescription reverses.
Split three ways before anything else: training rows for the weights, a validation
set to steer decisions, and a test set opened once at the very end. Plot or print
both losses from epoch one — one extra loss_fn(net(Xva), yva) inside
model.eval() and torch.no_grad(). Then add early stopping with two parts, and
both parts matter: patience, the number of epochs of no improvement you tolerate
(20 is a fair default), and the rewind, copy.deepcopy(model.state_dict()) at every
new best plus load_state_dict before you save. The rewind is not decoration: in
the measured demo the weights at the stop epoch score 0.760 while the restored best
weights score 0.824. Do not reach for dropout, weight decay or a bigger network
until early stopping is in place — it is the cheapest fix and, on small tables, often
the largest.
Never let a training loop print one column. Make the two-column epoch table your template, the way the five-step loop from 2.3 is your template. Treat the test set as sealed: touching it to pick an epoch, a rate or an architecture converts it into a second validation set and its number stops meaning anything. Keep the reflex that a training loss near zero is a warning, not a win — on real data, a model that scores perfectly on rows it has seen has usually stopped generalising. In review, ask for the validation column before you ask anything else about a model.
Q: "Your training loss is 0.001 and your test accuracy is 71 percent. What happened, and what do you do?" Strong answer: name overfitting and say why the training column could never have shown it — it measures memory, not learning; then describe the three-way split and the two-column epoch table as the diagnostic; then prescribe early stopping with patience and best-weight restore first, giving the reason (cheapest, and it needs no change to the model); then name more data, dropout, weight decay and augmentation as the next levers, one at a time. Bonus: mention that the epoch where validation loss bottoms is the model you want, and that keeping the weights from the stop epoch instead of the best epoch throws away most of the benefit.