Validation scores are noisy in ways training scores are not: the same checkpoint evaluated twice prints different numbers, or the validation accuracy sits several points below what the same model scores in a separate script. Separately or together: memory usage climbs steadily during evaluation — a laptop that trains comfortably at 4 GB creeps past 7 GB while merely scoring the model, slows to a crawl, and may be killed by the OS. No exception is raised; everything looks like it worked.
Any PyTorch project that measures a model, starting in module 2.3 and biting
hardest from 2.4 onwards when dropout and batch norm join the network. The
builder evaluated inside the training loop and skipped one or both switches:
model.eval() was never called (or the model was left in train mode after the
last batch), or the scoring forward passes ran outside torch.no_grad(). On
2.3's plain ReLU net the scores happen to stay correct — which is exactly why the
habit decays before 2.4 punishes it. Linked modules: 2.3, 2.4.
Two independent switches are involved, and each failure has its own signature.
First: model.train() and model.eval() flip layer behaviour. Dropout randomly
silences neurons in train mode, so a model scored in train mode gives a different
answer on every call — that is the noisy, too-low validation score. Batch norm in
train mode updates its running statistics from whatever data passes through, so
scoring in train mode quietly contaminates the model with validation data.
Second: autograd records every operation on tensors that require gradients, and
holds intermediate activations in memory so a backward pass could run later.
Evaluation never calls backward, so without torch.no_grad() those recorded
graphs pile up batch after batch — that is the ballooning memory. Confirm the
first with print(model.training) before scoring (True means train mode), and
the second by watching memory during eval: flat means recording is off, climbing
means it is on.
Adopt the ritual as one unbreakable pair: model.eval(), then with torch.no_grad(): around every scoring forward pass, then model.train() before
the next training epoch. Wrap it once in an evaluate(model, X, y) helper so it
cannot be half-remembered at each call site; the helper's first line can even
assert the switch: enter eval, score, restore train. For pure inference scripts —
a saved model answering requests — set model.eval() once at load time and keep
every forward pass under no_grad. Rerun any validation numbers produced in
train mode; they were measurements of a different, randomised model.
Never score a model inline; always call the helper. Treat model.training as
part of the measurement's preconditions, the way 1.7 treated the train/test split.
When validation numbers look noisy, check the mode before questioning the data or
the seed. And keep the habit even while the network is plain ReLU — 2.4 adds
dropout and batch norm, and the loops written today are the loops those layers
will run in.
Q: "Your validation accuracy jumps around between epochs far more than your
training loss suggests, and evaluation eats memory. Two causes, two fixes." Strong
answer: separate the switches — model.eval() controls layer behaviour (dropout
off, batch norm frozen to running stats), torch.no_grad() controls autograd
recording (no graphs held, flat memory); map each symptom to its switch — noise
to train-mode dropout, memory to graph recording; give the ritual pair and the
helper-function discipline. Bonus: note that train-mode scoring also pollutes
batch-norm statistics with validation data, a quiet leakage cousin of 1.7.