Training runs, the progress bar moves, no error appears, and the loss does not change. In the 2.6 measurement it printed 1.3237 at epoch one and 1.3161 at epoch six, wandering in the third decimal because the batches were shuffled and going nowhere at all. Test accuracy sits at chance or below — 0.230 on three classes there — and stays there however long the run continues. A milder version of the same fault: training does work, but far more slowly than expected, because only part of what you meant to train is actually receiving updates. A third version appears at the end, when the saved checkpoint scores exactly what the untrained one scored.
Transfer learning, module 2.6 onwards, in the minutes after the freezing loop is
added. Three births cover almost every case. One: the freeze loop runs over
model.parameters() and the new head is created before it, so the head is frozen
too and nothing at all is trainable. Two: the head is replaced after the freeze
loop, which is correct, but the optimizer was built earlier over a list that no
longer contains the head's new weights, so the head that exists is not the head
being optimised. Three: the optimizer is handed model.parameters() while
everything is frozen, and backward() then raises
RuntimeError: element 0 of tensors does not require grad. Measured example from
2.6: with every parameter frozen the audit reports 0 trainable and 11,178,051
frozen weights and the run cannot start at all; with the optimizer built one line
too early it reports 1,539 trainable weights, 513,000 weights held by the
optimizer and 0 of them live, a loss stuck near 1.32, and test accuracy 0.230. The
correct configuration reports 1,539 trainable and 1,539 held, and reaches 0.953.
Linked modules: 2.6, 2.3 (optimizer construction), 2.4 (reading a flat curve).
requires_grad = False tells autograd to compute no gradient for that tensor, so
its .grad stays None and the optimizer skips it. That is exactly what you want
for a frozen body and exactly what you do not want for your head. Freezing every
single tensor at least fails loudly, because backward() refuses to run when no
input to the loss requires a gradient. The dangerous case is the partial one: an
optimizer holding only frozen or replaced tensors is a legal object, a training
loop over it is a legal loop, and nothing warns you. Count trainable
parameters with
sum(p.numel() for p in model.parameters() if p.requires_grad): zero means the
head was frozen along with the body. Then count what the optimizer actually holds,
sum(p.numel() for g in opt.param_groups for p in g["params"] if p.requires_grad):
a healthy setup has the same number on both counts, and a mismatch means the
optimizer is pointing at tensors that were replaced. Finally check that the loss
is not merely small but constant — a genuinely converged model wobbles in the last
decimals, and a dead one repeats the same digits. Rule out the neighbours: a loss
pinned at the chance value with a nonzero trainable count is 2.4's dead-network
family, WR-123, not this one.
Fix the order first, because order is the real bug. Load the backbone, freeze the
whole body, then replace the head, then build the optimizer over the head alone:
for p in model.parameters(): p.requires_grad = False, then
model.fc = nn.Linear(model.fc.in_features, num_classes) — a new layer arrives
with requires_grad = True, so it is live by construction — then
opt = torch.optim.Adam(model.fc.parameters(), lr=1e-3). Never build the
optimizer over model.parameters() on a partly frozen model; pass the trainable
parameters explicitly, or filter them with
filter(lambda p: p.requires_grad, model.parameters()). Then add the guard that
turns this into a five-second failure instead of a lost evening: assert the
trainable count is above zero and equals the count the optimizer holds, and print
both lines before the first epoch. When the fix lands the loss moves on epoch one.
Print a four-line freeze report before every transfer run: trainable parameters, frozen parameters, number of optimizer parameter groups, and the learning rate of each. Build the model in one fixed order — load, freeze, replace head, build optimizer — and keep that order in your recipe file from 2.4. Treat any flat loss as a setup bug rather than a hard dataset; a real model always moves in the first epoch, even when it moves the wrong way. In review, ask which object the optimizer was constructed from and whether the head existed at that moment.
Q: "You froze a backbone, trained for ten epochs, and the loss never moved. Walk
me through it." Strong answer: name requires_grad = False and say precisely what
it does — no gradient, so .grad stays None and the optimizer has nothing to
apply. Then give the two counts you would print, trainable parameters and
parameters the optimizer actually holds, and explain what each mismatch means.
Name the order that prevents it: freeze, then replace the head, then build the
optimizer, because a freshly created layer is trainable by default and an
optimizer captures whatever it was given at construction time. Bonus: point out
that a loss which never leaves its starting neighbourhood is a different symptom
from a loss that falls and then climbs, and that the two have different causes.