A training job is killed and restarted from a checkpoint, and everything looks correct: the file loads, no exception is raised, and training continues. The numbers, however, are wrong. The loss jumps upward at the first print after the resume, then settles into a curve that never quite catches the one it left. In module 2.9's measured example, an uninterrupted run read 0.685192, 0.383528 and 0.350039 at steps 160, 200 and 240, and the resumed run read 0.871505, 0.512969 and 0.388188 — a largest gap of 0.186313 with no error message anywhere. A second signature is worse because it is invisible: the resumed run scores slightly better than the reference, because it is training on rows it has already seen. Teams usually blame the model, the learning rate or the data before they suspect the load path.
Any run split across sessions: a free GPU notebook, a shared lab machine with a
time limit, or a laptop training over several evenings. It also appears in
fine-tuning work from module 3.10, where the adapter is saved every epoch and the
trainer state is not. The bug is written by good intentions. torch.save(model. state_dict(), path) is the line every tutorial shows, it is short, it loads
without complaint, and it is exactly half the job. Adaptive optimizers make the
gap worse than it looks: Adam keeps two running buffers per weight, and restoring
them as zeros means the first steps after a resume take a differently-scaled
update on every parameter. Linked modules: 2.9, 3.10, 2.4 (seeds and
reproducibility), 2.8 (the model used in the measured example).
Do not reason about it — measure it, because a short reference run settles the question in two minutes. Run N steps uninterrupted on a small subset and record the loss at fixed intervals. Then run the same N steps with a kill and a resume in the middle, and compare the two traces at every shared step. Identical to the printed digits means the state is complete. Any difference at all, even in the fourth decimal, means something was not restored, and the size of the difference tells you roughly how much. Now find which item is missing by reading the saved dictionary rather than the saving code: load the checkpoint and print its keys. Four keys go missing in practice. The optimizer state is the most common and produces the largest jump. The step counter is next, and its signature is a run that trains for longer than intended, or a learning-rate schedule that restarts from its warmup. The data-order state comes third: without it the shuffle restarts, so rows already seen are seen again, which quietly turns a validation score into an optimistic one. The random state is last, and it matters wherever dropout, augmentation or sampling is used.
Save a dictionary, not a tensor. The five items are the model's state_dict, the
optimizer's state_dict, the step or epoch number, the state of whatever
generator draws the batch order, and the framework's random state — plus the
scheduler and the gradient scaler if the run uses them. Restore all of them in
one function, and have that function return the step it restored so the loop can
start from the right place. Then write the assertion that makes it permanent: a
small test that trains 20 steps straight, trains 20 steps with a kill and resume,
and fails the build if the two final losses differ at all. Keep the checkpoint
format versioned with a "format": 2 key, so an old file loaded by new code
raises a clear error instead of silently restoring three of five items. If you
inherit a run whose checkpoints hold weights only, do not pretend it is
resumable: either restart it, or restart it with the fixed script and accept the
cost once. And log the resume line — resumed at step 4000 (full state restored)
— so the next reader can see what came back and what did not.
Write the resume path on the same day you write the save path, and test it before
either is needed. Make the test a habit rather than a document: every training
script in your repo gets a twenty-step continuity check, and it runs in seconds
on a laptop. Prefer one save_state and one load_state function used by every
project, so the list of five items is maintained in a single place. Never save a
bare state_dict to a file called checkpoint.pt, because the name promises
something the contents do not deliver — call it weights.pt if that is what it
is. Print the checkpoint's keys at load time during development. And treat any
unexplained jump in the loss immediately after a restart as this bug until you
have proved otherwise, because it almost always is.
Q: "Your training resumed from a checkpoint and the loss jumped. The weights loaded fine. What happened?" Strong answer: name the missing state before naming anything else. Say that a checkpoint is five things, not one, and that the optimizer's moment buffers coming back as zeros is the usual cause of a visible jump. Add the second, quieter failure: a restarted data order means repeated rows, which can make the resumed run look better rather than worse, and that is the dangerous case. Give the measurement rather than an opinion — an uninterrupted reference trace against a kill-and-resume trace, compared at every step, with any non-zero difference treated as a defect. Finish with the guard you would leave behind: a twenty-step continuity test in CI, and a versioned checkpoint format so an old file fails loudly instead of restoring silently.