A frozen-backbone classifier is working. Somebody unfreezes the whole network "to squeeze out a bit more", keeps the head's learning rate, and reruns. The training loss looks healthy and often falls faster than before, so the change looks like a success from the training column alone. The test score tells the other story: in the 2.6 measurement it fell from 0.953 to 0.837, and the same six epochs spent unfreezing only the last block reached 0.937. Rerunning with more epochs does not recover it. The checkpoint is now a damaged copy of a good model, and if the frozen run was never saved, the good version is gone.
Transfer learning on a small dataset, module 2.6 onwards, and the standard next
step after a frozen-feature baseline works. The usual birth is one deleted loop:
the for p in backbone.parameters(): p.requires_grad = False block is removed, or
a later model.train() refactor rebuilds the optimizer over model.parameters()
with the head's rate still set. It also arrives when a tutorial written for
100,000 images is copied onto 600. Measured example from 2.6, resnet18 on 600
training photos across three classes, six epochs each: the last block unfrozen at
0.0001 reaches 0.937 on 300 test photos, the whole network unfrozen at 0.00001
reaches 0.927, and the whole network unfrozen at the head's own 0.001 reaches
0.837. The frozen-feature baseline on the same photos is 0.953. Linked modules:
2.6, 2.4 (learning rates and schedules), 2.2 (gradient size), 3.10 (fine-tuning
decisions on language models).
The head starts random and the body starts excellent, so they need opposite treatment. A random head produces large, badly aimed gradients for the first few hundred steps, and those gradients flow backwards into every unfrozen layer. At a rate suited to the head, one epoch of that is enough to overwrite features that took a million photos and many GPU-days to build. What remains is a network that is no longer pretrained and is far too large for 600 images, so it overfits from a damaged starting point. Two signatures tell it apart from ordinary overfitting. First, the training loss falls faster than the careful run while the test score falls with it, because the network is fitting the training photos with features it is busy destroying. Second, the same run with the body's rate divided by a hundred recovers almost everything, which no genuine capacity problem would do. Rule out the neighbouring bug before concluding: if the loss did not move at all, the fault is the opposite one, WR-135.
Do it in two stages and never in one. Stage one: freeze the body, replace the
head, and train the head alone until the validation curve flattens. Save that
checkpoint, because it is your fallback and often your final answer. Stage two, if
the data justifies it: unfreeze the last block only — layer4 on a ResNet — and
build the optimizer with two parameter groups,
[{"params": model.layer4.parameters(), "lr": 1e-4}, {"params": model.fc.parameters(), "lr": 1e-3}], so the borrowed weights move a tenth or a
hundredth as far as the new ones. Train few epochs, three to six on a small set,
and compare against the saved frozen checkpoint on the same test images before
accepting the change. Two guards make the mistake hard to repeat: print the count
of trainable parameters and the learning rate of every parameter group before the
first step, and refuse to ship a fine-tune that does not beat the frozen baseline.
Write the rule down and keep it: new weights get the normal rate, borrowed weights get a much smaller one, and frozen weights get none. Always keep the frozen-feature score as the number a fine-tune must beat, so an unfreeze is a measured experiment rather than an act of faith. Match the depth you unfreeze to the data you have — with a few hundred images per class the last block is usually the limit, and with a few dozen you should not unfreeze at all. This is 2.4's one-knob-at-a-time law applied to depth, and 2.4's early stopping with a rewind is what saves you when the experiment goes wrong.
Q: "Fine-tuning made your model worse than the frozen version. Explain what happened and what you would change." Strong answer: describe the gradient path — a randomly initialised head sends large gradients into pretrained layers, and at the head's learning rate they overwrite features built from millions of images. Name the clue that separates it from ordinary overfitting: the training loss falls faster than the careful run while the held-out score falls with it. Give the fix as a two-stage recipe with differential learning rates and a named parameter group per stage, and say what you would print before training to prove the setup is right. Bonus: state the amount of data at which each stage becomes worth it, and note that the frozen checkpoint should always be kept as the baseline to beat.