There was one great run. The screenshot shows validation loss 0.39 and test accuracy
0.82, and everyone agrees it happened. Rerunning the same file gives 0.79, then 0.77,
then 0.78 — never the number in the screenshot. Sometimes there is a saved .pt file
whose weights no longer match any version of the model class, so it will not even
load. Days go into hunting a difference in the code, and no difference is there. The
run is real; the recipe for it is gone.
Any project between the first working loop in 2.3 and the tracking discipline in 4.4.
Typical shape: numbers typed inline instead of read from a config, no seed anywhere,
a notebook whose cells were run out of order, and a results note that reads "big net,
Adam, dropout" with no values. The 2.4 measured demo makes the size of the effect
visible. Five unseeded runs of one identical script on 281 Ludhiana churn rows gave
test accuracies spread across two to three points, while three runs with
torch.manual_seed(7) set before the model was built gave an identical best
validation loss of 0.4022 and an identical test accuracy of 0.824 every time. A
second trap lives inside the fix: seeding after the layers are constructed changes
nothing, because nn.Linear draws its weights at construction time. Linked modules:
2.4, 4.1 (repo hygiene), 4.4 (tracking).
Three separate sources of randomness feed a training run, and a missing seed lets all
three drift. Weight initialisation is the largest on small data — a lucky init can be
worth two or three accuracy points on its own, which is exactly the size of the
"improvement" people chase. Data shuffling in the DataLoader is the second, and
dropout masks are the third. Confirm the problem before blaming the code: run the
unmodified script five times and record the scores. A spread across runs of the same
file proves the variation is randomness, not a change anyone made, and it also tells
you how large a real improvement must be before it means anything — a two-point gain
inside a three-point noise band is not a result. Then check the seeding line's
position, not only its presence: torch.manual_seed(7) placed below net = build()
looks correct in review and reproduces nothing.
Set the seed at the top of the run, before any layer is constructed and before the
DataLoader generator is made: torch.manual_seed(cfg["seed"]), plus
np.random.default_rng(cfg["seed"]) for any NumPy work and an explicit
generator=torch.Generator().manual_seed(cfg["seed"]) on the DataLoader. Move every
number out of the code and into a config.json the script reads and prints on
startup — seed, layer sizes, dropout, learning rate, weight decay, batch size,
epochs, patience. Save the config beside the checkpoint, never the checkpoint alone,
and save weights as a state_dict (2.3's rule) so a class rename cannot orphan them.
Finally, log the run: one line per experiment with date, config file, best validation
loss, test score and what changed since the previous line. If the good run is already
lost, accept it — rerun the config space from a seeded baseline rather than spending
days on a number you cannot recover.
Treat a run without a seed and a saved config as a run that did not happen. Make the config-print the first line of every script's output, so the log itself records what produced it. Before celebrating any gain, measure the noise band: three seeded runs at different seeds tell you how big a difference has to be to count. Keep the run log in the repository, not in chat. Module 4.4 replaces the text file with MLflow and a model registry, and everything you build here maps onto it one-to-one — parameters, metrics, artifacts.
Q: "A teammate reports a 0.82 result you cannot reproduce from the same repository. How do you handle it?" Strong answer: do not assume dishonesty or a code difference — name the three randomness sources (init, shuffling, dropout) and run the unmodified script several times to measure the spread first; explain that on small data the init alone can move the score by two or three points, so the reported number may be a lucky draw rather than a better model; then prescribe the fix — seed set before the model is constructed, a config file the script reads and prints, a state_dict plus config saved together, and a run log. Bonus: point out the subtle failure where a seed call sits after the layers are built, and say why a claimed improvement must exceed the measured noise band before it can be reported.