Training on a poor man's machine
Tejaswini shares one laptop with her cousin in Jamshedpur. Her final-year project trains a model on delivery notes, and the free GPU session she waited two days for ended at hour five. Nothing was saved. Her mentor does not ask about the model at all. He asks one question: how much work can you afford to lose in one moment? She has no answer. Tonight she builds a run that expects to die, and loses ten minutes instead of a night.
- Free compute is the only training budget most freshers have. Knowing which channel to use, and how much of it you may plan on, is the difference between a finished project and four restarts.
- Checkpoint-and-resume is an interview answer and a survival skill. Tonight you kill your own run twice and prove the result is identical, which is a stronger story than any certificate.
- Every trick here works on an 8GB laptop with no GPU. Every number below was printed by the code in this module on that machine.
Module 2.4 gave you a training recipe, and module 2.8 gave you a model worth training. This module answers the question both of them dodged: where does the run actually happen, and what happens when the machine takes it away? Your limit is not talent and it is rarely the model. It is a machine with 8GB of memory, a free session that can end at any second, and a deadline. Those three limits have engineering answers, and each one is measurable.
Start with the map of free compute, because the wrong channel wastes a week. Your laptop CPU is always there, never queued, and needs no login. Kaggle publishes a free GPU budget of about 30 hours a week. That is the one number here you can divide across a month and plan against.
Colab's free tier is different in kind: Google describes it as best-effort, with limits that are not published and that vary with demand. So Kaggle is the plan and Colab is a bonus, and any course that quotes you a fixed Colab number is inventing it.
The published-versus-best-effort split decides how you schedule work. A published budget can be turned into a calendar: six evenings of five GPU-hours, or two long runs plus repeats. A best-effort channel cannot, because a plan that depends on a gift is not a plan. Your laptop sits under both of them as the floor that never fails. Almost all data work, all debugging and every first run belong there, and moving them to a GPU buys nothing.
Now the hard truth that shapes every script you write from tonight. A free session ends without warning, at any point, and it takes unsaved work with it. A checkpoint — a copy of the training state written to disk mid-run, so a later session can carry on from it — is therefore not an optimisation. It is the part of the code that decides whether a dead session costs you ten minutes or a night.
There is a wrong way and a right way to spend six hours of free GPU. The wrong way runs one long job, saves at the end, and loses everything when the session ends at hour five. The right way saves the full state every N steps and resumes in the next session.
Both rows below were killed at step 130 of a 240-step run on the same laptop. The first kept 0 steps. The second kept 120 and reloaded in 0.5 seconds. It then printed 0.685192, 0.383528 and 0.350039 — the uninterrupted run's own numbers.
What goes into the file matters as much as how often you write it. Save the model weights alone and the resumed job is a different job. The optimizer's moment buffers start again at zero, and the batch stream restarts from the top.
In the measured run below that partial save moved the loss at step 160 from 0.685192 to 0.871505, a gap of 0.186313 that no log will ever explain. A full checkpoint holds five things: model weights, optimizer state, the step number, the data-order state, and the random number state. Write it to a temporary name and rename it, so a death mid-write cannot leave you a broken file.
Fitting the run in memory is the second problem, and it has a ladder of answers. The cheapest is a smaller batch, but a small batch is a noisier step, as 2.4 measured. Gradient accumulation — running several small batches and adding up their gradients before one optimizer step — gets you the quality of a big batch inside a small machine. The effective batch — the number of rows that go into a single weight update — is then the micro-batch times the number of accumulation steps. Below, a batch of 256 needed 1778 MB and four micro-batches of 64 needed 680 MB, and their gradients agreed to 5.29e-07.
The rest of the ladder is worth naming even where you cannot measure it tonight. Cutting the context from 128 to 64 took the same job from 680 MB to 484 MB, because attention grows with the square of the length. Freezing the lower blocks, which is 2.6's transfer-learning trick, halved the trainable parameters and so halved the optimizer state.
Mixed precision — holding most tensors in 16-bit form while keeping a 32-bit copy of the weights — belongs in your Kaggle script. So does gradient checkpointing, which recomputes activations instead of storing them. Neither helps much on a CPU, and claiming otherwise is how blog posts waste your evening.
The third problem is fitting the run in time, and the rule is to measure before you optimise. Put two stopwatches in your loop, one around fetching a batch and one around the maths, and print the split. In the profile below, 42 percent of the time went to preparing text, so the model was idle almost half the run. Two DataLoader workers took that to 10 percent, and tokenising the rows once took it to nearly zero. Adding workers on top of that fixed pipeline made it slower again.
Dataset staging — preparing the data once, saving the result, and loading it in every later run — is the habit behind that fix. Name the cache after a fingerprint of its inputs, so changing a parameter produces a new file instead of silently reusing the old one. Write it with a temporary name and rename it, exactly like a checkpoint. Four restarts cost 12.5 seconds of rebuilding without a cache, and 3.2 seconds with one, and on a real dataset those seconds are minutes of your quota. Never download or rebuild inside a loop that may restart.
Two habits close the loop. A run log — one line per run holding the config, the score and the path to the checkpoint — answers the question that ends most student projects. That is module 4.4's tracking discipline in a form you can keep tonight in a CSV.
Before any run that spends quota, do a smoke run on a tiny subset. A bug that crashes at line 40 costs the same in three minutes or three GPU-hours. Module 3.10 applies all of this to fine-tuning, where the checkpoint is a 100 MB adapter. This module is the general case.
Work inside ai-lab with torch and numpy. Keep corpus.py and tiny_gpt.py from module 2.8 in the same folder, because every step here imports them. Nothing downloads and nothing needs a GPU. Timings come from a plain CPU laptop with four threads, so your seconds will differ; every loss and every gradient number is seeded, so those should land within about 1% of your screen — seeded torch runs vary slightly across builds — while counts and shapes should match exactly. Three of the five steps start child processes on purpose, because a session death has to be a real process death to teach anything.
If your numbers are close but not identical, triage in a fixed order before touching your code. Everything off by a little — every value drifting in the last decimal or two — points at a library version: run pip show torch, compare it with the pinned install line from 2.3, and match the pin, because the printed numbers were produced on that version. One number wildly off while the rest agree points at stale state: restart from a fresh kernel or a fresh python and run the file top to bottom, since seeded draws only reproduce when they happen in the same order from a clean start. Only when both checks pass and the numbers still disagree should you suspect your own logic — diff your script line by line against the printed code.
Step 1 — kill the run, resume it, and prove it is the same run. Save as step1_resume.py. Each child process is one free session:
import json, os, random, subprocess, sys, time, torch
import numpy as np
from corpus import build_corpus, make_dataset, get_batch
from tiny_gpt import TinyGPT
CKPT, LOG, STEPS, EVERY, KILL = "run_ckpt.pt", "traces.json", 240, 40, 130
def build():
"""Same model, same optimiser, same batch stream, every single time."""
torch.manual_seed(7)
chars, _, train, _ = make_dataset(build_corpus())
model = TinyGPT(len(chars), 64, 64, 2, 2)
opt = torch.optim.AdamW(model.parameters(), lr=3e-3)
gen = torch.Generator().manual_seed(7) # the batch stream has state too
return model, opt, gen, train
def save(path, model, opt, step, gen, full=True):
state = {"model": model.state_dict(), "step": step}
if full: # everything the next session needs
state["opt"] = opt.state_dict() # Adam's two moment buffers per weight
state["batch_gen"] = gen.get_state() # where the batch stream had reached
state["torch_rng"] = torch.get_rng_state()
state["numpy_rng"] = np.random.get_state()
state["python_rng"] = random.getstate()
torch.save(state, path + ".tmp") # write to a temp name, then rename:
os.replace(path + ".tmp", path) # a death mid-write can never corrupt it
def load(path, model, opt, gen):
ck = torch.load(path, weights_only=False)
model.load_state_dict(ck["model"])
if "opt" not in ck:
return ck["step"], "MODEL ONLY"
opt.load_state_dict(ck["opt"])
gen.set_state(ck["batch_gen"])
torch.set_rng_state(ck["torch_rng"])
np.random.set_state(ck["numpy_rng"])
random.setstate(ck["python_rng"])
return ck["step"], "full state"
def train(label, checkpoint=True, full=True, kill_at=None):
t0 = time.perf_counter()
model, opt, gen, data = build()
start, trace = 0, []
if checkpoint and os.path.exists(CKPT):
start, kind = load(CKPT, model, opt, gen)
print(f" [{label}] resumed at step {start} ({kind} restored) "
f"in {time.perf_counter() - t0:.1f} s")
else:
print(f" [{label}] no checkpoint on disk, starting at step 0")
for step in range(start, STEPS):
xb, yb = get_batch(data, 64, 32, gen)
_, loss = model(xb, yb)
opt.zero_grad(); loss.backward(); opt.step()
done = step + 1
if done % EVERY == 0:
trace.append((done, float(loss.detach())))
print(f" [{label}] step {done:4d} loss {float(loss.detach()):.6f}", flush=True)
if checkpoint:
save(CKPT, model, opt, done, gen, full)
if kill_at and done == kill_at:
print(f" [{label}] *** session ended at step {done}, no warning ***", flush=True)
record(label, trace); os._exit(9) # no cleanup, no atexit: a real death
print(f" [{label}] finished {STEPS} steps, {time.perf_counter() - t0:.1f} s in this session")
record(label, trace)
def record(label, trace):
log = json.load(open(LOG)) if os.path.exists(LOG) else {}
log.setdefault(label, []).extend(trace)
json.dump(log, open(LOG, "w"))
if __name__ == "__main__":
if len(sys.argv) > 1: # a child process = one free session
train(sys.argv[2], checkpoint=sys.argv[1] != "plain", full=sys.argv[1] != "partial",
kill_at=int(sys.argv[3]) if len(sys.argv) > 3 else None)
sys.exit(0)
def session(*a):
subprocess.run([sys.executable, __file__, *a], check=False)
for f in (CKPT, LOG):
if os.path.exists(f): os.remove(f)
print("A - one uninterrupted session, no checkpoints (the reference run)", flush=True)
session("plain", "A")
print("\nB - checkpoint every 40 steps; the session dies at step 130", flush=True)
session("full", "B", str(KILL))
print("B - a new session, same script, nothing edited", flush=True)
session("full", "B")
os.remove(CKPT)
print("\nC - same crash, but the checkpoint held the MODEL only", flush=True)
session("partial", "C", str(KILL))
print("C - a new session, same script, nothing edited", flush=True)
session("partial", "C")
log = json.load(open(LOG))
print(f"\ncheckpoint file on disk: {os.path.getsize(CKPT) / 1024:.0f} KB\n")
print(f"{'step':>6s} {'A uninterrupted':>16s} {'B full-state resume':>20s}"
f" {'C model-only resume':>20s}")
for i, (s, a) in enumerate(log["A"]):
b, c = dict(log["B"]).get(s), dict(log["C"]).get(s)
f = lambda v: f"{v:.6f}" if v is not None else "(lost)"
print(f"{s:6d} {a:16.6f} {f(b):>20s} {f(c):>20s}")
same = all(abs(dict(log["B"])[s] - a) == 0 for s, a in log["A"] if s in dict(log["B"]))
gap = max(abs(dict(log["C"])[s] - a) for s, a in log["A"] if s in dict(log["C"]))
print(f"\nB continues A exactly: {same} (largest difference "
f"{max(abs(dict(log['B'])[s] - a) for s, a in log['A'] if s in dict(log['B'])):.1e})")
print(f"C is a different run: largest difference {gap:.6f}")
Output:
A - one uninterrupted session, no checkpoints (the reference run)
[A] no checkpoint on disk, starting at step 0
[A] step 40 loss 2.039067
[A] step 80 loss 1.758386
[A] step 120 loss 1.250016
[A] step 160 loss 0.685192
[A] step 200 loss 0.383528
[A] step 240 loss 0.350039
[A] finished 240 steps, 1.9 s in this session
B - checkpoint every 40 steps; the session dies at step 130
[B] no checkpoint on disk, starting at step 0
[B] step 40 loss 2.039067
[B] step 80 loss 1.758386
[B] step 120 loss 1.250016
[B] *** session ended at step 130, no warning ***
B - a new session, same script, nothing edited
[B] resumed at step 120 (full state restored) in 0.5 s
[B] step 160 loss 0.685192
[B] step 200 loss 0.383528
[B] step 240 loss 0.350039
[B] finished 240 steps, 1.2 s in this session
C - same crash, but the checkpoint held the MODEL only
[C] no checkpoint on disk, starting at step 0
[C] step 40 loss 2.039067
[C] step 80 loss 1.758386
[C] step 120 loss 1.250016
[C] *** session ended at step 130, no warning ***
C - a new session, same script, nothing edited
[C] resumed at step 120 (MODEL ONLY restored) in 0.5 s
[C] step 160 loss 0.871505
[C] step 200 loss 0.512969
[C] step 240 loss 0.388188
[C] finished 240 steps, 1.2 s in this session
checkpoint file on disk: 435 KB
step A uninterrupted B full-state resume C model-only resume
40 2.039067 2.039067 2.039067
80 1.758386 1.758386 1.758386
120 1.250016 1.250016 1.250016
160 0.685192 0.685192 0.871505
200 0.383528 0.383528 0.512969
240 0.350039 0.350039 0.388188
B continues A exactly: True (largest difference 0.0e+00)
C is a different run: largest difference 0.186313
Read the three rows of the final table together, because that comparison is the whole module. Run A is the reference: one session, 240 steps, nothing saved. Run B was killed at step 130 by os._exit(9), which skips every cleanup handler and is therefore an honest imitation of a session that ends on someone else's schedule. Its next session found run_ckpt.pt, restored five things in 0.5 seconds, and then printed 0.685192, 0.383528 and 0.350039 — not close to run A, but bit-for-bit identical, with a largest difference of 0.0e+00. Ten steps of work were lost, because the last save was at step 120 and the death came at 130, and that gap is exactly the cadence you chose. Now run C, which is the same crash with a checkpoint that held only model.state_dict(). It restored the weights correctly and still diverged immediately: 0.871505 where run A had 0.685192, and a final gap of 0.186313. Two things were missing. Adam's moment buffers came back as zeros, so the first steps after the resume used a differently-scaled update, and the batch generator restarted from its seed, so the model saw rows it had already seen. Notice also the save function: it writes run_ckpt.pt.tmp and renames it, because a death during torch.save would otherwise leave you a truncated file and no way back. WR-143 is run C with a week of GPU quota behind it.
Step 2 — make a batch that does not fit, fit. Save as step2_accum.py. Each measurement runs in its own process so the memory reading is clean:
import resource, subprocess, sys, time, torch
from corpus import build_corpus, make_dataset, get_batch
from tiny_gpt import TinyGPT
BLOCK, D, LAYERS, TARGET, CAP_MB = 128, 128, 4, 256, 900
def peak_mb():
"""Highest memory this process ever held. macOS reports bytes, Linux kilobytes."""
r = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
return r / (1 << 20) if sys.platform == "darwin" else r / (1 << 10)
def setup(ctx=BLOCK, freeze=0):
torch.manual_seed(7)
chars, _, train, _ = make_dataset(build_corpus())
model = TinyGPT(len(chars), ctx, D, 4, LAYERS)
for i in range(freeze): # 2.6's trick, used here to save memory
for p in model.blocks[i].parameters():
p.requires_grad_(False)
if freeze:
model.tok.weight.requires_grad_(False)
model.pos.weight.requires_grad_(False)
live = [p for p in model.parameters() if p.requires_grad]
return model, torch.optim.AdamW(live, lr=3e-4), train, sum(p.numel() for p in live)
def one_update(model, opt, data, micro, accum, ctx, seed=0):
"""One optimiser step over micro*accum rows, done accum micro-batches at a time."""
gen = torch.Generator().manual_seed(seed)
opt.zero_grad()
t0, total = time.perf_counter(), 0.0
for _ in range(accum):
xb, yb = get_batch(data, ctx, micro, gen)
_, loss = model(xb, yb)
(loss / accum).backward() # divide, or the gradient is accum times too big
total += float(loss.detach()) / accum
dt = time.perf_counter() - t0
opt.step()
grad = torch.cat([p.grad.flatten() for p in model.parameters() if p.grad is not None])
return total, dt, grad
if __name__ == "__main__":
if len(sys.argv) > 1: # child: one clean measurement per process
micro, accum, ctx, freeze, out = (int(sys.argv[1]), int(sys.argv[2]),
int(sys.argv[3]), int(sys.argv[4]), sys.argv[5])
model, opt, data, live = setup(ctx, freeze)
loss, dt, grad = one_update(model, opt, data, micro, accum, ctx)
torch.save({"loss": loss, "dt": dt, "peak": peak_mb(), "live": live, "grad": grad}, out)
sys.exit(0)
def run(micro, accum=1, ctx=BLOCK, freeze=0):
subprocess.run([sys.executable, __file__, str(micro), str(accum), str(ctx),
str(freeze), "m.pt"], check=True)
return torch.load("m.pt", weights_only=False)
print(f"model: {LAYERS} blocks, width {D}, 4 heads, context {BLOCK}.")
print(f"The batch we want is {TARGET} rows. Pretend cap for this laptop: {CAP_MB} MB.\n")
print(f"{'batch':>7s} {'peak MB':>9s} {'over cap':>9s} {'s/update':>9s} {'loss':>9s}")
runs = {}
for micro in (16, 32, 64, 128, 256):
r = runs[micro] = run(micro)
print(f"{micro:7d} {r['peak']:9.0f} {'YES' if r['peak'] > CAP_MB else 'no':>9s}"
f" {r['dt']:9.2f} {r['loss']:9.4f}")
fits = max(m for m in runs if runs[m]["peak"] <= CAP_MB)
k = TARGET // fits
print(f"\nlargest batch that fits under {CAP_MB} MB: {fits}")
print(f"so reach {TARGET} rows per update by accumulating {k} micro-batches of {fits}\n")
big, acc = runs[TARGET], run(fits, k)
print(f"{'route to a 256-row update':<30s} {'peak MB':>9s} {'s/update':>9s} {'loss':>9s}")
print(f"{'one batch of 256':<30s} {big['peak']:9.0f} {big['dt']:9.2f} {big['loss']:9.4f}")
print(f"{f'{k} x {fits}, accumulated':<30s} {acc['peak']:9.0f} {acc['dt']:9.2f}"
f" {acc['loss']:9.4f}")
print(f"{'difference':<30s} {acc['peak'] - big['peak']:+9.0f} {acc['dt'] - big['dt']:+9.2f}"
f" {acc['loss'] - big['loss']:+9.4f}")
g1, g2 = big["grad"].double(), acc["grad"].double()
print(f"\nis it really the same update? comparing {g1.numel():,d} gradient numbers")
print(f" norm, one big batch {g1.norm():.6f}")
print(f" norm, accumulated {g2.norm():.6f}")
print(f" largest difference {(g1 - g2).abs().max():.3e}")
print(f" cosine similarity {torch.nn.functional.cosine_similarity(g1, g2, 0):.9f}")
print(f" forgetting /accum would give norm {g2.norm() * k:.6f}, {k}x too large\n")
print(f"{'other ways to fit, same 256 rows':<36s} {'trainable':>11s} {'peak MB':>9s} {'s/update':>9s}")
for label, kw in (("full model, context 128", {"micro": fits, "accum": k}),
("context cut to 64", {"micro": fits, "accum": k, "ctx": 64}),
("first 2 blocks + embeddings frozen",
{"micro": fits, "accum": k, "freeze": 2})):
r = run(**kw)
print(f"{label:<36s} {r['live']:11,d} {r['peak']:9.0f} {r['dt']:9.2f}")
print("\nmixed precision and gradient checkpointing are the two levers NOT in this table:")
print("both need a GPU session to be worth anything, so they belong in your Kaggle script.")
Output:
model: 4 blocks, width 128, 4 heads, context 128.
The batch we want is 256 rows. Pretend cap for this laptop: 900 MB.
batch peak MB over cap s/update loss
16 401 no 0.04 3.7633
32 491 no 0.07 3.7596
64 680 no 0.13 3.7618
128 1053 YES 0.25 3.7620
256 1778 YES 0.58 3.7570
largest batch that fits under 900 MB: 64
so reach 256 rows per update by accumulating 4 micro-batches of 64
route to a 256-row update peak MB s/update loss
one batch of 256 1778 0.58 3.7570
4 x 64, accumulated 680 0.44 3.7570
difference -1099 -0.13 +0.0000
is it really the same update? comparing 819,200 gradient numbers
norm, one big batch 1.136602
norm, accumulated 1.136602
largest difference 5.290e-07
cosine similarity 1.000000000
forgetting /accum would give norm 4.546408, 4x too large
other ways to fit, same 256 rows trainable peak MB s/update
full model, context 128 819,200 680 0.44
context cut to 64 811,008 484 0.20
first 2 blocks + embeddings frozen 401,536 516 0.31
mixed precision and gradient checkpointing are the two levers NOT in this table:
both need a GPU session to be worth anything, so they belong in your Kaggle script.
The first table is the measurement almost nobody takes before they start. Memory grows roughly with the batch: 401 MB, 491, 680, 1053, 1778, so under a 900 MB cap the largest batch that fits is 64 — and 256, the batch you wanted, needs nearly twice the cap. The middle table is the fix and its price. Four micro-batches of 64 reach the same 256-row effective batch in 680 MB instead of 1778 MB, a saving of 1099 MB, and the loss is 3.7570 in both rows because the same generator hands out the same 256 rows either way. On this laptop the accumulated route was also 0.13 s faster, which is honest and slightly surprising: the huge batch paid for its own memory pressure. Do not expect that on a GPU, where the big batch usually wins on speed and loses on memory. The gradient block is the proof that this is not an approximation — 819,200 numbers, a largest element-wise difference of 5.290e-07, and a cosine similarity that prints as 1.000000000. Read the last line of that block twice, because the /accum divide is the single most common bug in accumulation code: leave it out and every gradient is 4 times too large, which behaves exactly like a learning rate you never set. The bottom table prices two more levers: halving the context took 680 MB to 484 MB, since attention memory grows with the square of the length, and freezing the first two blocks plus the embeddings took the trainable count from 819,200 to 401,536, which halves the optimizer state as well.
Step 3 — find out where the time actually goes. Save as step3_profile.py. Two stopwatches, four setups, one honest table:
import re, time, torch, torch.nn as nn
from torch.utils.data import Dataset, DataLoader
from corpus import build_corpus
SEQ, BATCH, STEPS, DOCS, LINES = 256, 32, 60, 800, 400
def make_docs():
"""800 delivery-note documents, 400 lines each: one long file per depot per week."""
lines = build_corpus(DOCS * 4, seed=11).split("\n")
docs, labels = [], []
for i in range(DOCS):
chunk = lines[(i * LINES) % (len(lines) - LINES): (i * LINES) % (len(lines) - LINES) + LINES]
docs.append("\n".join(chunk))
labels.append(1 if "returned" in "\n".join(chunk) else 0)
vocab = {w: i + 1 for i, w in enumerate(sorted({w for d in docs for w in d.split()}))}
return docs, labels, vocab
class RawDocs(Dataset):
"""The beginner Dataset: the text is cleaned and tokenised on every single request."""
def __init__(self, docs, labels, vocab):
self.docs, self.labels, self.vocab = docs, labels, vocab
def __len__(self):
return len(self.docs)
def __getitem__(self, i):
words = re.sub(r"[^a-z0-9: ]", " ", self.docs[i].lower()).split()
ids = [self.vocab.get(w, 0) for w in words][:SEQ]
ids += [0] * (SEQ - len(ids))
return torch.tensor(ids), self.labels[i]
class StagedDocs(Dataset):
"""Same rows, tokenised once into one tensor. __getitem__ now only slices."""
def __init__(self, raw):
pairs = [raw[i] for i in range(len(raw))]
self.x = torch.stack([p[0] for p in pairs])
self.y = torch.tensor([p[1] for p in pairs])
def __len__(self):
return len(self.x)
def __getitem__(self, i):
return self.x[i], self.y[i]
class Classifier(nn.Module):
def __init__(self, vocab):
super().__init__()
self.emb = nn.Embedding(vocab, 128)
self.enc = nn.TransformerEncoderLayer(128, 4, 512, batch_first=True, dropout=0.0)
self.head = nn.Linear(128, 2)
def forward(self, x):
return self.head(self.enc(self.emb(x)).mean(1))
def profile(loader, model, opt, label, steps=STEPS):
"""Two stopwatches on one loop: waiting for a batch, and doing the maths."""
data_s = compute_s = 0.0
it, n = iter(loader), 0
t0 = time.perf_counter()
while n < steps:
t = time.perf_counter()
try:
xb, yb = next(it)
except StopIteration:
it = iter(loader); xb, yb = next(it)
data_s += time.perf_counter() - t
t = time.perf_counter()
loss = nn.functional.cross_entropy(model(xb), yb)
opt.zero_grad(); loss.backward(); opt.step()
compute_s += time.perf_counter() - t
n += 1
total = time.perf_counter() - t0
print(f"{label:<36s} {data_s:8.2f} {compute_s:9.2f} {total:8.2f}"
f" {100 * data_s / total:9.0f}% {total / steps * 1000:8.0f}")
return data_s, compute_s, total
if __name__ == "__main__":
torch.set_num_threads(4)
docs, labels, vocab = make_docs()
raw = RawDocs(docs, labels, vocab)
t0 = time.perf_counter()
staged = StagedDocs(raw)
stage_s = time.perf_counter() - t0
print(f"{DOCS:,d} documents, {len(docs[0]):,d} characters each, vocabulary {len(vocab):,d}.")
print(f"Tokenising all of them once took {stage_s:.1f} s.\n")
def fresh():
torch.manual_seed(7)
m = Classifier(len(vocab) + 1)
return m, torch.optim.AdamW(m.parameters(), lr=3e-4)
print(f"{'setup':<36s} {'data s':>8s} {'compute s':>9s} {'total s':>8s}"
f" {'data share':>10s} {'ms/step':>8s}")
m, o = fresh(); a = profile(DataLoader(raw, BATCH, shuffle=True), m, o,
"raw text, num_workers=0")
m, o = fresh(); b = profile(DataLoader(raw, BATCH, shuffle=True, num_workers=2,
persistent_workers=True), m, o,
"raw text, num_workers=2")
m, o = fresh(); c = profile(DataLoader(staged, BATCH, shuffle=True), m, o,
"staged tensors, num_workers=0")
m, o = fresh(); d = profile(DataLoader(staged, BATCH, shuffle=True, num_workers=2,
persistent_workers=True), m, o,
"staged tensors, num_workers=2")
print(f"\nthe same {STEPS} steps, priced three ways against the first row:")
for name, r in (("workers on raw text", b), ("stage once, no workers", c),
("stage once AND workers", d)):
print(f" {name:<26s} {a[2]:6.2f} s -> {r[2]:5.2f} s {a[2] / r[2]:5.2f}x")
print(f" compute seconds barely move: {a[1]:.2f} {b[1]:.2f} {c[1]:.2f} {d[1]:.2f}")
print(f"\nin 5 free GPU hours you would get {5 * 3600 / (a[2] / STEPS):,.0f} steps the slow way"
f" and {5 * 3600 / (c[2] / STEPS):,.0f} the staged way.")
print("pin_memory is not in this table on purpose: it only speeds up copies from RAM to a")
print("GPU, so it does nothing on a CPU laptop. Switch it on in a Kaggle GPU session.")
Output:
800 documents, 45,339 characters each, vocabulary 153.
Tokenising all of them once took 0.4 s.
setup data s compute s total s data share ms/step
raw text, num_workers=0 1.13 1.56 2.69 42% 45
raw text, num_workers=2 0.19 1.73 1.92 10% 32
staged tensors, num_workers=0 0.01 1.59 1.60 0% 27
staged tensors, num_workers=2 0.46 1.57 2.03 23% 34
the same 60 steps, priced three ways against the first row:
workers on raw text 2.69 s -> 1.92 s 1.40x
stage once, no workers 2.69 s -> 1.60 s 1.69x
stage once AND workers 2.69 s -> 2.03 s 1.33x
compute seconds barely move: 1.56 1.73 1.59 1.57
in 5 free GPU hours you would get 400,816 steps the slow way and 675,528 the staged way.
pin_memory is not in this table on purpose: it only speeds up copies from RAM to a
GPU, so it does nothing on a CPU laptop. Switch it on in a Kaggle GPU session.
Read the compute column first, because it never moves: 1.56, 1.73, 1.59, 1.57 seconds in all four setups. The model is doing identical work every time, so every difference in the total is the data pipeline, and that is what profiling is for. In the first row, 42 percent of the run went to cleaning and tokenising text while the model sat idle — a run that is nearly half a text-processing job wearing a training costume. Two worker processes fetch the next batch while the model works on this one, and that alone took the data share from 42 percent to 10 percent and the run from 2.69 s to 1.92 s. Staging the tokenised rows once, at a one-time cost of 0.4 seconds, beat the workers outright: 1.60 s, a 1.69x speed-up, with the data share rounding to zero. Now the row that matters most for your habits. Adding two workers on top of the already-staged pipeline made the run slower — 2.03 s against 1.60 s — because each worker costs a process to start and a batch to hand back, and there was no work left to hide behind. That is why "add num_workers=4" is advice and not engineering: profile first, then pick the fix your own table asks for. The GPU-hour line converts the same result into the only currency that matters here, roughly 400,000 steps against 675,000 in the same five hours.
Step 4 — stage the data once, so a restart costs nothing. Save as step4_staging.py. Eight child processes stand in for eight sessions:
import hashlib, json, os, subprocess, sys, time, torch
from corpus import build_corpus, make_dataset
CACHE = ".cache"
def fingerprint(name, params):
"""A cache is only safe if its name changes when its inputs change."""
blob = json.dumps({"name": name, "params": params}, sort_keys=True).encode()
return f"{name}-{hashlib.sha256(blob).hexdigest()[:12]}.pt"
def stage(name, params, build):
"""Build once, save atomically, reuse for ever. Safe to call at the top of every run."""
os.makedirs(CACHE, exist_ok=True)
path = os.path.join(CACHE, fingerprint(name, params))
t0 = time.perf_counter()
if os.path.exists(path):
data = torch.load(path, weights_only=False)
print(f" cache HIT {path} loaded in {time.perf_counter() - t0:5.2f} s")
return data, "hit", time.perf_counter() - t0
print(f" cache MISS {path} building...")
data = build(**params)
torch.save(data, path + ".part") # never write the real name directly
os.replace(path + ".part", path) # rename is atomic: no half-written cache
dt = time.perf_counter() - t0
print(f" built and saved in {dt:5.2f} s")
return data, "miss", dt
def build_tokens(n_lines, seed):
"""The expensive step: generate the corpus and turn every character into a number."""
chars, stoi, train, val = make_dataset(build_corpus(n_lines, seed))
return {"chars": chars, "stoi": stoi, "train": train, "val": val}
def session(tag, params, rebuild_always=False):
if rebuild_always: # the wrong way: rebuild inside every run
t0 = time.perf_counter()
data = build_tokens(**params)
dt = time.perf_counter() - t0
print(f" no cache at all, rebuilt in {dt:5.2f} s")
kind = "rebuild"
else:
data, kind, dt = stage("tokens", params, build_tokens)
print(f" [{tag}] {len(data['train']):,d} training characters, "
f"vocabulary {len(data['chars'])}")
print(json.dumps({"tag": tag, "kind": kind, "seconds": round(dt, 2)}),
file=open("staging_log.jsonl", "a"))
if __name__ == "__main__":
P = {"n_lines": 120000, "seed": 5}
if len(sys.argv) > 1:
session(sys.argv[1], P, rebuild_always=sys.argv[2] == "rebuild")
sys.exit(0)
for f in ("staging_log.jsonl",):
if os.path.exists(f): os.remove(f)
if os.path.isdir(CACHE):
for f in os.listdir(CACHE): os.remove(os.path.join(CACHE, f))
def run(tag, mode):
subprocess.run([sys.executable, __file__, tag, mode], check=True)
print("THE WRONG WAY - no cache, four sessions, the same work four times", flush=True)
for i in (1, 2, 3, 4):
print(f"session {i}:", flush=True); run(f"no-cache-{i}", "rebuild")
print("\nTHE RIGHT WAY - stage once, then every later session loads it", flush=True)
for i in (1, 2, 3, 4):
print(f"session {i}:", flush=True); run(f"cached-{i}", "cache")
print("\nthe cache must go stale LOUDLY - change one parameter and watch the name change")
print(f" n_lines=120000 -> {fingerprint('tokens', {'n_lines': 120000, 'seed': 5})}")
print(f" n_lines=200000 -> {fingerprint('tokens', {'n_lines': 200000, 'seed': 5})}")
print(f" seed 5 -> 6 {fingerprint('tokens', {'n_lines': 120000, 'seed': 6})}")
rows = [json.loads(l) for l in open("staging_log.jsonl")]
bad = sum(r["seconds"] for r in rows if r["kind"] == "rebuild")
good = sum(r["seconds"] for r in rows if r["kind"] != "rebuild")
print(f"\n{'session':<14s} {'no cache':>10s} {'staged':>10s}")
for i in range(4):
print(f"{'session ' + str(i + 1):<14s} {rows[i]['seconds']:9.2f}s"
f" {rows[i + 4]['seconds']:9.2f}s ({rows[i + 4]['kind']})")
print(f"{'total':<14s} {bad:9.2f}s {good:9.2f}s")
print(f"\nfour restarts cost {bad:.1f} s of rebuilding, or {good:.1f} s with a cache"
f" - {bad / good:.1f}x")
print(f"cache file size: {sum(os.path.getsize(os.path.join(CACHE, f)) for f in os.listdir(CACHE)) / 1e6:.1f} MB")
Output:
THE WRONG WAY - no cache, four sessions, the same work four times
session 1:
no cache at all, rebuilt in 3.15 s
[no-cache-1] 12,248,663 training characters, vocabulary 37
session 2:
no cache at all, rebuilt in 3.08 s
[no-cache-2] 12,248,663 training characters, vocabulary 37
session 3:
no cache at all, rebuilt in 3.12 s
[no-cache-3] 12,248,663 training characters, vocabulary 37
session 4:
no cache at all, rebuilt in 3.13 s
[no-cache-4] 12,248,663 training characters, vocabulary 37
THE RIGHT WAY - stage once, then every later session loads it
session 1:
cache MISS .cache/tokens-6e10bbba60b0.pt building...
built and saved in 3.22 s
[cached-1] 12,248,663 training characters, vocabulary 37
session 2:
cache HIT .cache/tokens-6e10bbba60b0.pt loaded in 0.01 s
[cached-2] 12,248,663 training characters, vocabulary 37
session 3:
cache HIT .cache/tokens-6e10bbba60b0.pt loaded in 0.01 s
[cached-3] 12,248,663 training characters, vocabulary 37
session 4:
cache HIT .cache/tokens-6e10bbba60b0.pt loaded in 0.01 s
[cached-4] 12,248,663 training characters, vocabulary 37
the cache must go stale LOUDLY - change one parameter and watch the name change
n_lines=120000 -> tokens-6e10bbba60b0.pt
n_lines=200000 -> tokens-4c4f8feb1459.pt
seed 5 -> 6 tokens-144fe9d275f3.pt
session no cache staged
session 1 3.15s 3.22s (miss)
session 2 3.08s 0.01s (hit)
session 3 3.12s 0.01s (hit)
session 4 3.13s 0.01s (hit)
total 12.48s 3.25s
four restarts cost 12.5 s of rebuilding, or 3.2 s with a cache - 3.8x
cache file size: 108.9 MB
The two halves of this output are the same four sessions with one habit changed. Without a cache every session pays 3.1 seconds again, and four restarts cost 12.5 seconds; with a cache only the first session pays, and the other three load in 0.01 second each. Scale those seconds honestly: this corpus is written by code in three seconds, but a real dataset is downloaded, unzipped, cleaned and tokenised, and on a free session that is fifteen to forty minutes of your quota each time the run restarts. The fingerprint lines are the part students skip and then regret. The cache file is named from a hash of its own inputs, so n_lines=120000 gives tokens-6e10bbba60b0.pt and n_lines=200000 gives tokens-4c4f8feb1459.pt — changing a parameter can never silently reuse the wrong data, and changing the seed cannot either. Notice stage() writes .part and renames, the same trick as the checkpoint, so a session that dies during the save leaves no half-built cache behind to poison tomorrow. One warning the size line makes concrete: 108.9 MB of cache is cheap on a laptop and worth checking against a free platform's storage rules before you rely on it.
Step 5 — the decision table, the quota, and the run log. Save as step5_decide.py. No torch, no GPU, and the rules are the ones you will actually apply:
import csv, math, os
KAGGLE_WEEKLY_GPU_HOURS = 30 # published, plannable. Check the current page before a big run.
LOSS_BUDGET_MIN = 10 # most work I am willing to lose to one session death
def decide(task, quota_left):
"""Returns (channel, reason, plan). Nothing here is a guess about Colab's limits."""
t = dict(task)
if not t["needs_gpu"]:
return ("laptop CPU", "no GPU work in this job, and a laptop is never queued",
plan(t, "laptop CPU"))
if t["gpu_hours"] <= quota_left and t["deadline_h"] >= t["gpu_hours"] + 2:
return ("Kaggle GPU", f"{t['gpu_hours']} GPU-h fits the {quota_left} h left this week",
plan(t, "Kaggle GPU"))
if t["gpu_hours"] > quota_left:
return ("Kaggle GPU next week + laptop subset now",
f"{t['gpu_hours']} GPU-h is more than the {quota_left} h left this week",
plan(t, "Kaggle GPU"))
return ("laptop subset now, Colab only as a bonus",
"the deadline is shorter than the run, so nothing queued can be trusted",
plan(t, "Colab GPU"))
def plan(t, channel):
steps_lost = math.ceil(LOSS_BUDGET_MIN * 60 / t["sec_per_step"])
return {
"smoke run first": f"{t['subset_pct']}% subset, {t['smoke_min']} min, on the laptop",
"checkpoint every": f"{steps_lost} steps (~{LOSS_BUDGET_MIN} min of work at "
f"{t['sec_per_step']} s/step)",
"resume proof": "kill the run once on the subset and check the loss continues",
"channel note": CHANNEL_NOTE[channel],
}
CHANNEL_NOTE = {
"laptop CPU": "always available, never queued, no quota; slower per step",
"Kaggle GPU": f"about {KAGGLE_WEEKLY_GPU_HOURS} GPU-hours a week are published, so this "
"channel can be scheduled; sessions still end without warning",
"Colab GPU": "free tier is best-effort: Google does not publish the limits and says they "
"vary with demand, so it can never be the plan",
}
TASKS = [
{"name": "compare 4 classical models on a 45 MB UPI-fraud table",
"needs_gpu": False, "gpu_hours": 0, "deadline_h": 14, "sec_per_step": 0.4,
"subset_pct": 10, "smoke_min": 3},
{"name": "fine-tune a small image model on 18,000 kirana shelf photos",
"needs_gpu": True, "gpu_hours": 6, "deadline_h": 120, "sec_per_step": 0.9,
"subset_pct": 2, "smoke_min": 6},
{"name": "train a character GPT for a college demo starting in 3 hours",
"needs_gpu": True, "gpu_hours": 2, "deadline_h": 3, "sec_per_step": 0.25,
"subset_pct": 5, "smoke_min": 4},
]
def quota_table(spent_rows):
print(f"{'run':<34s} {'GPU-h':>7s} {'left of 30':>11s} {'outcome':>26s}")
left = KAGGLE_WEEKLY_GPU_HOURS
for name, hours, outcome in spent_rows:
left -= hours
print(f"{name:<34s} {hours:7.1f} {left:11.1f} {outcome:>26s}")
return left
RUNLOG = "runs.csv"
FIELDS = ["run_id", "date", "channel", "config", "steps", "val_loss", "checkpoint", "note"]
def log_run(**row):
new = not os.path.exists(RUNLOG)
with open(RUNLOG, "a", newline="") as f:
w = csv.DictWriter(f, FIELDS)
if new: w.writeheader()
w.writerow(row)
if __name__ == "__main__":
print("WHERE SHOULD THIS RUN? three real student jobs, one rule set\n")
quota_left = 4.0 # hours left in this week's Kaggle budget
for t in TASKS:
ch, why, p = decide(t, quota_left)
print(f"TASK {t['name']}")
print(f" deadline {t['deadline_h']} h | GPU needed: {t['needs_gpu']} | "
f"estimate {t['gpu_hours']} GPU-h | {quota_left} h of quota left")
print(f" -> {ch}")
print(f" because {why}")
for k, v in p.items():
print(f" {k:<18s} {v}")
print()
print("WHAT A WEEK OF QUOTA BUYS, AND HOW IT GETS BURNT\n")
left = quota_table([
("smoke run, 2% subset, laptop", 0.0, "caught a shape bug"),
("smoke run, 2% subset, Kaggle", 0.2, "ran clean, 40 steps"),
("full run, session 1 of 2", 4.0, "died at 3h50m, resumed"),
("full run, session 2 of 2", 2.3, "finished, val 0.244"),
("one repeat with a new seed", 6.3, "val 0.251, same story"),
])
print(f"\n{left:.1f} GPU-hours still unspent this week.")
print("The same week without a smoke run: five crashed starts at 3 GPU-h each = 15 h gone,")
print("and not one score to show for it. That is WR-144.\n")
print("THE RUN LOG - 4.4's tracking, in a form a student can keep tonight\n")
for r in [
dict(run_id="r01", date="2026-03-02", channel="laptop", config="d64 L2 lr3e-3 b32",
steps=240, val_loss=0.351, checkpoint="ckpt/r01-step240.pt", note="subset smoke"),
dict(run_id="r02", date="2026-03-02", channel="kaggle", config="d128 L4 lr3e-4 b64x4",
steps=4000, val_loss=0.244, checkpoint="ckpt/r02-step4000.pt", note="died 3h50m, resumed"),
dict(run_id="r03", date="2026-03-03", channel="kaggle", config="d128 L4 lr1e-3 b64x4",
steps=4000, val_loss=0.318, checkpoint="ckpt/r03-step4000.pt", note="higher lr, worse"),
]:
log_run(**r)
rows = list(csv.DictReader(open(RUNLOG)))
print(f"{'run':<5s} {'channel':<8s} {'config':<22s} {'steps':>6s} {'val':>7s} {'checkpoint':<26s}")
for r in rows:
print(f"{r['run_id']:<5s} {r['channel']:<8s} {r['config']:<22s} {r['steps']:>6s}"
f" {r['val_loss']:>7s} {r['checkpoint']:<26s}")
best = min(rows, key=lambda r: float(r["val_loss"]))
print(f"\nbest so far: {best['run_id']} at val {best['val_loss']}, "
f"weights at {best['checkpoint']}")
print("three lines of CSV answer the question that ends most projects: "
"which file was the good one?")
Output:
WHERE SHOULD THIS RUN? three real student jobs, one rule set
TASK compare 4 classical models on a 45 MB UPI-fraud table
deadline 14 h | GPU needed: False | estimate 0 GPU-h | 4.0 h of quota left
-> laptop CPU
because no GPU work in this job, and a laptop is never queued
smoke run first 10% subset, 3 min, on the laptop
checkpoint every 1500 steps (~10 min of work at 0.4 s/step)
resume proof kill the run once on the subset and check the loss continues
channel note always available, never queued, no quota; slower per step
TASK fine-tune a small image model on 18,000 kirana shelf photos
deadline 120 h | GPU needed: True | estimate 6 GPU-h | 4.0 h of quota left
-> Kaggle GPU next week + laptop subset now
because 6 GPU-h is more than the 4.0 h left this week
smoke run first 2% subset, 6 min, on the laptop
checkpoint every 667 steps (~10 min of work at 0.9 s/step)
resume proof kill the run once on the subset and check the loss continues
channel note about 30 GPU-hours a week are published, so this channel can be scheduled; sessions still end without warning
TASK train a character GPT for a college demo starting in 3 hours
deadline 3 h | GPU needed: True | estimate 2 GPU-h | 4.0 h of quota left
-> laptop subset now, Colab only as a bonus
because the deadline is shorter than the run, so nothing queued can be trusted
smoke run first 5% subset, 4 min, on the laptop
checkpoint every 2400 steps (~10 min of work at 0.25 s/step)
resume proof kill the run once on the subset and check the loss continues
channel note free tier is best-effort: Google does not publish the limits and says they vary with demand, so it can never be the plan
WHAT A WEEK OF QUOTA BUYS, AND HOW IT GETS BURNT
run GPU-h left of 30 outcome
smoke run, 2% subset, laptop 0.0 30.0 caught a shape bug
smoke run, 2% subset, Kaggle 0.2 29.8 ran clean, 40 steps
full run, session 1 of 2 4.0 25.8 died at 3h50m, resumed
full run, session 2 of 2 2.3 23.5 finished, val 0.244
one repeat with a new seed 6.3 17.2 val 0.251, same story
17.2 GPU-hours still unspent this week.
The same week without a smoke run: five crashed starts at 3 GPU-h each = 15 h gone,
and not one score to show for it. That is WR-144.
THE RUN LOG - 4.4's tracking, in a form a student can keep tonight
run channel config steps val checkpoint
r01 laptop d64 L2 lr3e-3 b32 240 0.351 ckpt/r01-step240.pt
r02 kaggle d128 L4 lr3e-4 b64x4 4000 0.244 ckpt/r02-step4000.pt
r03 kaggle d128 L4 lr1e-3 b64x4 4000 0.318 ckpt/r03-step4000.pt
best so far: r02 at val 0.244, weights at ckpt/r02-step4000.pt
three lines of CSV answer the question that ends most projects: which file was the good one?
Three tasks, three different answers, and none of them is "use the GPU because a GPU is better". The tabular job needs no GPU at all, so it runs on the laptop where there is no queue and no quota to spend. The image job is six GPU-hours against four hours of quota left, so the rule splits it: subset on the laptop now, the real run next week when the budget resets. The third task is the interesting one, because the deadline is three hours and the run is two, and the only honest advice is to shrink the job and treat Colab as a bonus if it appears. Note what every plan carries regardless of channel: a smoke run on a subset, a checkpoint cadence derived from a ten-minute loss budget, and a resume test before the long run. The quota table then prices the doctrine — a disciplined week spends 12.8 GPU-hours and ends with a score and a spare 17.2, while the same week of crashing at line 40 spends 15 hours and ends with nothing, which is WR-144. The run log is three lines of CSV, and it is the only thing in this module that answers "which file was the good one?" a month later, when r02 is the checkpoint worth defending in a viva.
Work inside ai-lab with torch and numpy, and keep corpus.py and tiny_gpt.py from module 2.8 in the folder, because every exercise imports them. Nothing downloads and nothing needs a GPU. Every seed is fixed, so where a loss or a gradient number is given your output should land within about 1% of it — seeded torch runs vary slightly across builds — while counts and shapes should match exactly; memory in megabytes and wall-clock seconds will differ with your machine, and that is expected. Where your machine's numbers differ from this lesson's, report your own and say so — a measured number you can defend beats a copied one every time.
Kill your own run twice, and prove nothing was lost. Create resume_drill.py, starting from Step 1's harness but typed rather than copied, because the save and load pair is the part you must own. Part one: run 400 steps of the 2.8 tiny GPT uninterrupted, printing the loss every 50 steps, and save that trace as your reference. Part two: add full-state checkpointing every 50 steps, then run it again and kill it twice at random points you did not choose in advance — press Ctrl-C, or have a friend close the terminal, and note the step number each time. Restart after each kill with no edits to the script. Part three: put both traces side by side and compute the largest absolute difference at every shared step. Part four: break it on purpose — save only model.state_dict(), repeat one kill and one resume, and record the new trace beside the other two. Part five: write four comment lines naming what each of the five saved items does, and one line stating how many steps of work your chosen cadence can lose in the worst case. Check: your interrupted-and-resumed trace matches the uninterrupted trace with a largest difference of exactly 0.0e+00, at every printed step, across both kills; your resume lines show the step counter continuing from the last save rather than restarting at 0; the model-only version diverges visibly at the first print after the resume, as Step 1's 0.871505 against 0.685192 did, and your comment names the optimizer state and the batch stream as the two causes; your worst-case line reads 49 steps for a cadence of 50; and your checkpoint code writes to a temporary name and renames, with a comment saying what that protects against.
Find the biggest batch that fits, then beat it with accumulation. Create memory_lab.py. Part one: measure peak memory for the 4-block, 128-wide model at batches 8, 16, 32, 64, 128 and 256, one measurement per child process, and record peak megabytes, seconds per update and the loss. Pick a cap that is honest for your machine — 900 MB if you have 8GB of RAM and a browser open — and mark which rows break it. Part two: take the largest batch that fits and reach an effective batch of 256 with accumulation, then prove equivalence: gradient norms, largest element-wise difference and cosine similarity against the single big batch. Part three: run the same comparison once with the /accum divide deleted and record what happens to the gradient norm and to the loss after ten updates. Part four: extend the table with three more routes to the same 256 rows — context halved, first two blocks frozen, and both together — reporting trainable parameters, peak memory, seconds and loss for each. Part five: write a five-line MEMORY.md naming, in order, the levers you would pull on an 8GB laptop and the one you would only pull inside a Kaggle GPU session. Check: your memory column rises with batch size and your cap splits the table into rows that fit and rows that do not; the accumulated route reaches the same effective batch inside the cap, saving roughly the difference this lesson measured as 1778 MB against 680 MB; the two gradients agree to better than 1e-05 with a cosine similarity that prints as 1.000000000; deleting the divide multiplies the gradient norm by exactly your accumulation count, and your comment states that this behaves like a learning rate nobody set; freezing the first two blocks plus embeddings roughly halves your trainable count, as 819,200 fell to 401,536 here; and MEMORY.md names mixed precision or gradient checkpointing as the GPU-only lever, with one sentence saying why neither pays on a CPU.
Profile it, stage it, log it, and decide where it runs. Create plan_lab.py and runs.csv. Part one: put two stopwatches in a training loop over Step 3's document dataset and print data seconds, compute seconds, total and the data share, first with num_workers=0 on raw text, then with two workers, then on staged tensors, then on staged tensors with two workers. Part two: write the stage(name, params, build) helper with a fingerprint in the filename and an atomic rename, run your script four times, and paste the four cache lines into your notes. Then change one parameter and show the cache name changing rather than the data going stale. Part three: adapt Step 5's decide function and run it on three jobs of your own — one that needs no GPU, one that is larger than the quota you have left, and one with a deadline shorter than the run — printing the channel, the reason, and a checkpoint cadence computed from your own seconds per step. Part four: keep a real runs.csv for every run you did in exercises 1 and 2, one line each with config, steps, score and checkpoint path, then print the best row by score. Check: your compute-seconds column is nearly constant across all four profiling rows while the totals differ, and your comment states that this is what makes the data pipeline the suspect; at least one of your four rows is slower than the plain one, and you name the reason rather than deleting the row; your second run of the staging script prints a cache hit and loads in a small fraction of the build time; changing a parameter produces a different cache filename, and your comment says what would have happened with a fixed filename; each of your three decisions names a channel and a reason, and no decision quotes a numeric limit for Colab; and runs.csv has at least four rows and your best-row lookup prints a real checkpoint path that exists on disk.
Three free-compute failures cost students more evenings than every modelling mistake in this phase combined. Each has a War Room entry with the full diagnosis path:
- WR-142 — the session that died at 90 percent. A five-hour training run on a free GPU ended with no checkpoint ever written, so a night of work and a large slice of the week's quota vanished together; the fix is a checkpoint cadence chosen from a stated loss budget, plus a kill-and-resume test on a tiny subset before any long run is allowed to start.
- WR-143 — the resumed run that was not the same run. The checkpoint held the model weights but not the optimizer state, the step counter, the data order or the random state, so training silently continued as a different run: the same steps that read 0.685192 read 0.871505 instead, and the final gap of 0.186313 was blamed on the model for two days; the fix is saving all five items and asserting continuity against a short reference run.
- WR-144 — the quota burnt on one bug. A week of free GPU hours went into five full-size runs that each crashed at the same line, because nobody ran the pipeline once on a 2 percent subset first; the fix is a mandatory smoke run on the laptop, a shape-and-dtype assertion before the loop, and a rule that no run spends quota until a subset run has finished end to end.
Free compute has three channels, and only Kaggle publishes a budget you can plan against, at about 30 GPU-hours a week. Free sessions end without warning, so a checkpoint holding model, optimizer, step, data order and random state decides whether a death costs ten minutes or a night. A run killed at step 130 and resumed from step 120 printed 0.685192 and 0.350039, matching the uninterrupted run, while a model-only save drifted by 0.186313. Gradient accumulation reached a 256-row effective batch in 680 MB instead of 1778 MB, and freezing lower blocks bought more room. Profile before you optimise, stage the data once, smoke-run on a subset, and write one run log line at the end.
- Every Kaggle competition team on a free tier runs this way. Long trainings are split across sessions, and the checkpoint folder is saved as a dataset and attached to the next notebook. The solutions that win are the ones whose runs could survive being interrupted.
- Indian startups and college labs train on one shared GPU box. A job that cannot be stopped and resumed blocks everyone else on the roster. Checkpoint-and-resume is a condition of getting time, not a personal preference.
- Module 3.10's fine-tuning runs are this module with a bigger file. The adapter checkpoint of about 100 MB is written every epoch to survive a dead session. The same smoke-run-then-spend rule protects the same weekly quota.
Answer these before opening the gate assessment. Each question names its type.
type=interpretation— Two logs cover the same 240-step run, killed at step 130 and resumed. The first prints 0.685192 at step 160 and the second prints 0.871505. Both loaded the same weights file. State what the second run failed to restore, name the two effects that produce a gap this size, and give the one-line check that would have caught it before the long run.type=debug— A student's Kaggle notebook printsno checkpoint found, starting freshat the top of every session, even though the previous session printedsaved step 4000before it died. Name the two most likely causes, say which one you would test first and how, and give the between-sessions habit that removes the problem for good.type=output-prediction— A run saves full state every 200 steps and each step takes 1.8 seconds. The session ends without warning at step 1,530. Predict the step the next session resumes from, the minutes of work lost, and what the loss printed at step 1,600 should equal if the checkpoint was complete.type=trade-off— Micro-batch 32 with 8 accumulation steps against a single batch of 256, for the same model, first on an 8GB laptop CPU and then in a free GPU session. Give one concrete advantage and one concrete cost of each route in each setting, name the two numbers from this module you would quote, and say what you would measure before committing.type=scenario-judgement— A teammate has 3 GPU-hours of weekly quota left and a submission due in four days. They propose starting the full 5-hour training run tonight and "seeing how far it gets". Judge the proposal, name the two things missing from it, and state the plan you would put in its place with a checkpoint cadence and a first step.
Explain to a friend who has trained one model on their own laptop and has never used a free GPU: why does a training script need to be written as if the machine will be taken away mid-run, and what exactly has to be saved for that to work? Use at most 4 sentences, name one number from this module, and use no word harder than "save" and "continue". If they then ask "why not save only the model, it is the model I want", answer in one sentence about the optimizer and batch order, and quote the 0.186313 gap.