The weekly free GPU budget is gone and there is no score to show for it. The history reads the same way five times: a session starts, the dataset downloads, training begins, and forty minutes later the run dies with a traceback. The traceback is the same one every time — a shape mismatch in the model's first layer, a missing column, an out-of-memory error at the first large batch, or a label tensor of the wrong dtype. Each attempt cost two or three GPU-hours before it reached the failing line, mostly spent downloading and preparing data. The final session of the week is the worst: the bug is fixed, the code is correct, and there are twenty minutes of quota left. A related version has no traceback at all — the runs completed, and every one of them trained on the wrong split.
The first serious project on a free tier, usually a fine-tune or a vision model, where the dataset is large enough that the student stops running things locally. The pattern is created by a reasonable-sounding decision: the data is too big for the laptop, so the whole pipeline moves to the GPU session, including the parts that never needed a GPU. Now every syntax error, every path mistake and every dtype mismatch is discovered at GPU prices. Two more habits make it worse. The failing line usually sits late in the script, after the download and the first epoch, so each iteration is slow. And the run is launched and then left alone, because watching it feels like wasting time, so a crash at minute six is noticed at minute ninety. Linked modules: 2.9, 3.10, 2.4 (one change per run), 4.4 (the run log), 0.5 and 1.7 (data and split discipline).
Separate the two questions the run is answering at once: does the pipeline work, and does the model learn? Only the second needs a GPU. Reproduce the crash on your own laptop with the smallest possible slice — 2 percent of the rows, two steps, one epoch — and it will almost always reproduce, because shape, dtype, path and split bugs do not care how many rows there are. If it does not reproduce, the fault is genuinely about scale, and that is a much smaller list: memory, a file that only appears in the full dataset, or a class that only appears in the tail. Read the traceback for its class before its message. Shape and dtype errors are code bugs and belong on the laptop. Out-of-memory is a capacity bug and belongs in module 2.9's memory ladder, not in another full attempt. File-not-found is a staging bug about where the data lives between sessions. Then check the run log, and if there is no run log, that is the finding — five attempts with nothing written down is how the same bug gets paid for five times.
Put a smoke run in front of every run that spends quota, and make it a rule rather than a suggestion. The smoke run uses a tiny subset, runs the entire path end to end — load, stage, build model, two training steps, one validation pass, one checkpoint write, one resume — and finishes in a few minutes on the laptop. It must exercise the checkpoint and the resume, because those are the paths that otherwise get tested by a real session death. Add cheap assertions before the loop: batch shape, label dtype, class count, and the row counts of each split printed side by side. Then move the pipeline order around: stage the dataset once and cache it (module 2.9's staging step) so a restart does not re-download, and put the cheap failing checks first so a bad run dies in seconds instead of forty minutes. Fix one thing per attempt and write a line in the run log for each, because 2.4's one-change-at-a-time rule is worth far more when each change costs GPU-hours. If a run must be launched unattended, print a heartbeat line every N steps so a stall is visible in the log rather than at the end.
Budget quota in writing before the week starts: how many full runs, how many
repeats, and how much held back for the mistake you have not made yet. Keep a
--smoke flag on every training script that swaps in the tiny subset and two
steps, so proving the pipeline is one word on the command line. Never let a
script's first GPU run also be its first run — anything that has not executed
once on a laptop is untested code. Cache the dataset and attach the cache to the
next session, so a restart costs a load and not a download. Keep the run log even
for failures, with the traceback's first line in the note column, because five
identical notes are a pattern that a human notices immediately. And treat the
smoke run's few minutes as the cheapest insurance available: on module 2.9's
measured week, a disciplined schedule spent 12.8 GPU-hours and ended with a score
and 17.2 hours spare, against 15 hours spent and nothing to show.
Q: "You had 30 GPU-hours for the week and spent them on runs that all crashed. What is your process from now on?" Strong answer: name the split first — pipeline correctness and model quality are two different questions, and only one of them needs a GPU. Describe the smoke run concretely: tiny subset, two steps, one validation pass, one checkpoint write and one resume, running on a laptop in minutes, and say that it must exercise the checkpoint path. Add the cheap assertions before the loop and the reordering that makes a bad run fail in seconds. Mention staging the dataset so restarts do not re-download, and the run log so the same failure is never paid for twice. If you can quote the arithmetic — five crashed attempts at three hours each against a three-minute local check — you have shown the interviewer that you think about compute as a budget, which is exactly what a team with one shared GPU wants to hear.