A long training run on a free GPU disappears. The notebook tab shows a disconnected kernel, or a message that the session ended, or nothing at all because the browser was closed hours ago. The output cell is empty, the working directory is gone, and the only trace of five hours of work is a screenshot of a loss curve somebody posted in the group chat. Reopening the notebook starts a new machine with an empty disk. The weekly quota, however, has been spent: the hours were used, the model was not saved, and the deadline did not move. A milder version of the same failure looks like this — the run finished, the final line printed a good validation loss, and the weights file was written to a folder that does not survive the session either.
Any free-tier GPU session: Kaggle, Colab, a shared college machine with a job scheduler, or a laptop that a family member closes at midnight. Module 2.9 reproduces the exact failure on a laptop CPU in under ten seconds, so nobody has to burn quota to learn it. The setup that produces it is almost always the same: a script written on a laptop, where interruptions do not happen, then copied into a free session unchanged. Two beliefs keep it alive. The first is that a session that has run for four hours will probably run for six, which is not how best-effort compute works. The second is that saving at the end is the same thing as saving, which is true only for runs that end. Linked modules: 2.9, 3.10 (fine-tuning on a Kaggle quota), 2.8 (the tiny GPT used in the drill), 4.4 (tracking).
There is nothing to debug in the usual sense, so the diagnosis is an audit of the
script before the next run. Ask four questions in order. One: does the script
write anything to disk before its final line? If the only torch.save sits after
the loop, the run has no survivors. Two: where does it write? A path inside the
session's temporary working directory disappears with the session, so the file
must land in a location that persists — an attached dataset or output folder on
Kaggle, a mounted drive, or a copy pushed out before the session ends. Three: how
much work sits between two saves? Multiply the save interval by the seconds per
step, and that product is the worst-case loss in minutes; if you cannot state it,
you have not chosen a cadence, you have inherited one. Four: has the resume path
ever executed? A save block that has never been read back is untested code, and
WR-143 is what happens when it turns out to be wrong. The confirming experiment
is small: run 240 steps on a subset, kill the process at a random point, restart,
and see whether the step counter continues.
Choose a loss budget first, in minutes, and derive everything from it. Ten minutes
is a reasonable default for a free session: at 1.8 seconds per step that is a save
every 333 steps, and the arithmetic is the whole decision. Write the full state,
not only the weights — model, optimizer, step number, data-order state and random
state — because a partial save produces a different run (WR-143). Save to a
temporary filename and rename it, so a session that dies during the write leaves
the previous good checkpoint intact instead of a truncated file. Put the resume
block at the top of the script, before the loop, so the same file runs unchanged
in session one and session four; a script with a --resume flag that somebody has
to remember is a script that will be run without it. Confirm the checkpoint path
survives the platform's session teardown, and on Kaggle save the folder as a
dataset and attach it to the next notebook. Then run the drill: kill the run
twice at points you did not choose and require the resumed trace to match an
uninterrupted reference exactly. Only after that trace matches should a run be
allowed to spend quota.
Treat "this run cannot be interrupted" as a bug report about the script, never a
fact about the platform. Keep one resumable training template and start every new
project from it, so the save-and-resume block is never written under deadline
pressure. Print the resume line loudly at startup — resumed at step 4000 or no checkpoint found, starting fresh — because a silent restart is how a whole week
goes missing. State the cadence in the run log next to the score, so a teammate
reading it can see how much a death would have cost. Before any run longer than
about thirty minutes, do the kill-and-resume test on a subset; it takes two
minutes and it is the only proof that the recovery code works. Finally, keep the
laptop path alive: a job that can also run small and slow on your own machine can
always be debugged, whatever the free tier is doing today.
Q: "You had five hours of free GPU training and lost all of it when the session ended. What do you change?" Strong answer: do not start with the platform. Say that a free session ending is an expected event, so the script was wrong, not the provider. Name the cadence calculation out loud — loss budget in minutes divided by seconds per step gives the save interval — and name the five things a checkpoint must hold. Mention the atomic write, because it shows you have thought about a death during the save itself, and mention that the checkpoint must land somewhere that outlives the session. Close with the test: kill your own run before you trust it, and compare the resumed trace against an uninterrupted one. That last sentence is what separates a candidate who read about checkpointing from one who has actually recovered a run.