The same saved model, scored on the same test images, reports a different accuracy every time it is run — 0.8575, then 0.8665, then 0.8410. Nothing crashes and no warning prints. The reported number also sits stubbornly below what the validation curve suggested, so the team assumes the model is weaker than it looked and starts tuning it. A related report: two teammates run the identical evaluation script on the identical checkpoint and cannot agree on the score.
Any vision project from module 2.5 onwards, the moment data augmentation is added.
The usual birth is one transform= argument reused for both datasets, or a single
transforms.Compose([...]) variable passed to the training loader and the
evaluation loader because it was defined once at the top of the file. It also
arrives when the augmented dataset object is reused by slicing rather than
rebuilt. Measured example from 2.5: a CNN trained with RandomCrop(28, padding=2)
and RandomHorizontalFlip(p=0.5) scores 0.8730 on 2,000 raw test images; with the
same transform applied at evaluation time it reports 0.8575, 0.8665 and 0.8410 on
three consecutive calls. Linked modules: 2.5, 1.7 (split discipline), 2.4
(train/eval mode), 3.7 (evaluation harnesses).
Augmentation exists to widen what the model sees while it is learning. It is a
training-time device, and it belongs inside the training loop only. A test set has
one job: to be a fixed, unchanging ruler. Attach a random transform to it and the
ruler now changes length on every call, so the number it gives is no longer a
measurement of the model — it is a measurement of the model on a randomly damaged
copy of the test set. Two signatures confirm the diagnosis instantly. First,
repeated runs on identical inputs disagree, which no deterministic evaluation ever
does. Second, the score is biased downward, because crops and flips make the task
harder than the one the model will actually face. Rule out the neighbouring bug
before you conclude: 2.4's WR-125 (scoring in train() mode, so dropout and batch
norm are live) produces the same "different number every call" symptom, so check
model.eval() first, and if the scores still wander, the transform is your culprit.
Build two pipelines and give them different names, always:
train_tf = transforms.Compose([RandomCrop(28, padding=2), RandomHorizontalFlip(), ToTensor()]) and eval_tf = transforms.Compose([ToTensor()]). The evaluation
pipeline may resize, centre-crop and normalise — those are deterministic — and may
contain nothing whose name starts with Random. Pass eval_tf to the validation
and test datasets, and never to a training loader. Then make the rule enforceable
rather than remembered: add one assertion in the evaluation function that no
transform in the pipeline has a class name beginning with "Random", and score the
same checkpoint twice in your test suite, asserting the two numbers are equal.
When the fix lands, the score stops moving; in the 2.5 run it settles at 0.8730
on every call.
Keep the same law you learned for scaling in 1.7: anything that is fitted or randomised belongs to the training rows and travels nowhere else. Write the two pipelines side by side in the same file, so the difference is visible in one glance during review. Any evaluation number that changes between runs of the same checkpoint on the same data is a bug report, not noise — chase it before you tune anything. In review, ask which transform list the test loader received; if the answer takes more than a few seconds to find, the code is already unsafe.
Q: "Your reported test accuracy changes on every run of the same saved model.
What is happening?" Strong answer: name the two candidates in order — evaluation
running in train() mode with dropout and batch norm live, and random
augmentation applied to the evaluation set — and say how to tell them apart.
Then explain why an augmented test set is not merely noisy but wrong: it measures
the model against a task it will never meet, and the number is usually pessimistic.
Then give the fix as two named pipelines plus an assertion that the evaluation
pipeline is deterministic. Bonus: connect it to leakage discipline — augmentation
on the test set is the mirror image of fitting a scaler on all the data, and both
come from the same habit of treating the splits as interchangeable.