Fine-tuning + decision framework
Lakshmi supports the loan-desk chatbot at an agri-finance office in Vijayawada. The bot writes long, messy paragraphs when the ops team wants a fixed three-line ticket. A senior developer announces the fix: train our own model, like the big companies do. Lakshmi has an 8GB laptop, a weekly Kaggle GPU quota, and one free weekend. Before she spends any of it, she wants one honest answer: is training even the right tool here, or the most expensive way to look busy?
- Interviewers for GenAI roles now open with "prompt, RAG, or fine-tune — how do you choose?" — a decision table you can defend beats any buzzword answer.
- Free GPU quota is the only training budget most freshers have, and the checkpoint-and-resume habit is what separates a finished run from five wasted hours.
- Fine-tunes done badly ship models worse than the base they started from — the eval discipline you built in module 3.7 is what catches that before users do.
Every stubborn LLM problem lands on one of three tools: better prompts, RAG, or fine-tuning. Fine-tuning means training a model further on your own examples, so its weights change. Prompts change no weights at all; they steer the model at run time. RAG changes what the model can see, by fetching your documents into the context window. The skill this module builds is choosing between the three for a written reason you can defend.
Here is the working rule, and it settles most cases in a minute. If the failure is missing or fresh facts — this week's rates, your internal policy — pick RAG, because facts baked into weights go stale. If the failure is format, tone, or a behaviour that survives good prompting, fine-tuning earns a place on the table. Most problems need neither: a clear prompt with a few examples fixes more failures than any training run. The tree below asks the same questions in order.
Fine-tuning is also the costliest and least reversible of the three, so it goes last. It is the wrong answer for fresh knowledge, because updating weights means retraining, while updating a RAG folder means saving a file. It is the wrong answer for a one-off task, where a prompt finishes before your dataset is even ready. It is the wrong answer when no eval proves the prompt has hit a ceiling. Teams have spent three weeks tuning their way up to the level a better prompt reached in one afternoon.
When fine-tuning does win, you will not retrain all the weights. LoRA — low-rank adaptation — freezes the base model and trains a small add-on instead. That add-on is called an adapter — a pair of skinny matrices, roughly 100 MB on disk, holding everything your training changed. QLoRA goes one step further: it loads the frozen base in 4-bit form, so a 3B or even 7B model fits on a free-tier GPU. Your 8GB laptop still cannot run the training, but it can build, validate, and later serve everything around it.
The training run is the easy part; the dataset decides the outcome. Fine-tuning data is a list of instruction pairs, and an instruction pair is one example of an input plus the exact output you want back. The standard file is JSONL chat format: one JSON object per line, holding system, user, and assistant messages. For a style or format task, a few hundred clean pairs beat a hundred thousand messy ones. Every wrong or duplicated pair is a bug you compile into the weights.
Now the part most courses skip: where the GPU comes from. This programme plans on Kaggle's free tier, which publishes around 30 GPU hours per week — a budget you can schedule. Colab's free GPU is a bonus when it appears, never the plan, because nothing about it is promised. Both share one hard truth: free sessions can die at any moment, taking unsaved work with them. So every training run must save a checkpoint — a mid-run copy of the adapter weights — after every epoch.
That rule splits two kinds of students. The wrong way trains for six straight hours with no saves, loses the session at hour five, and restarts from zero with a smaller quota. The right way checkpoints every epoch, loses the same session, and resumes from the last saved adapter next time. Splitting one run across sessions is not a workaround here; it is a first-class skill, workable up to 7B models with QLoRA. Plan the run to die, and it can never really fail.
One habit closes the loop: never trust a tuned model until it beats its own base in your eval. Run the fixed cases from your 3.7 harness on both models, plus a small general slice you never trained on. A tuned model can score below its base — bad pairs, a wrong chat template, or too many epochs all cause it. The eval table is the only place that failure shows up before your users find it.
Everything below runs on your 8GB laptop, except the training run itself — that script is written for a Kaggle GPU session. The flow: decide with the table, build and validate the dataset locally, read the training script line by line, then serve and score the result with Ollama.
Step 1 — the decision table, applied. Three real requests land on Lakshmi's desk in one week. Run each through the two questions from the tree — is it a facts failure, is it a behaviour failure — and write the verdict down:
request failing on choice why
A "quote this week's gold-loan interest fresh facts RAG rates change weekly;
rates in answers" weights go stale
B "reply as a 3-line ticket: CATEGORY, format/behaviour fine-tune format still breaks after
SEVERITY, NEXT - every single time" (candidate) prompts plateau on a
30-case eval
C "summarise one loan file for a manager, neither - one-off prompt only no repeat use, no dataset,
needed today" done in an hour
Only B is even a candidate, and only because an eval showed the best prompt stuck at 9/30 valid formats. Without that eval number, B stays a prompting problem. This table — three rows, defended in writing — is the artefact an interviewer wants to see, and the one WR-049 was missing.
Step 2 — build and validate the dataset locally. For case B, Lakshmi hand-writes 30 instruction pairs from real tickets and validates them before any GPU is touched. One script does both:
import json, hashlib, re
SYSTEM = ("You are a support assistant. Reply as exactly one line: "
"CATEGORY: <word> | SEVERITY: low/medium/high | NEXT: <one action>.")
raw = [
("UPI payment failed twice but money left my account",
"CATEGORY: payments | SEVERITY: high | NEXT: raise a reversal ticket with both UTR numbers."),
("How do I download last month's loan statement?",
"CATEGORY: statements | SEVERITY: low | NEXT: send the statements page link with steps."),
# ... 28 more pairs from real tickets
]
seen, pairs = set(), []
for user, assistant in raw:
key = hashlib.md5(user.lower().strip().encode()).hexdigest()
if key in seen: # exact dedupe on the input
continue
seen.add(key)
pairs.append({"messages": [
{"role": "system", "content": SYSTEM},
{"role": "user", "content": user},
{"role": "assistant", "content": assistant}]})
FORMAT = re.compile(r"^CATEGORY: \w+ \| SEVERITY: (low|medium|high) \| NEXT: .+")
problems = 0
for i, p in enumerate(pairs):
roles = [m["role"] for m in p["messages"]]
if roles != ["system", "user", "assistant"]:
problems += 1; print(f"pair {i}: bad roles {roles}")
if not FORMAT.match(p["messages"][2]["content"]):
problems += 1; print(f"pair {i}: output breaks the ticket format")
if len(p["messages"][1]["content"].split()) < 4:
problems += 1; print(f"pair {i}: input too short to teach anything")
with open("pairs.jsonl", "w") as f:
for p in pairs:
f.write(json.dumps(p) + "\n")
print(f"{len(pairs)} pairs written, {problems} problems")
A first run on real hand-written data almost never comes back clean:
pair 7: output breaks the ticket format
pair 19: output breaks the ticket format
pair 22: input too short to teach anything
29 pairs written, 3 problems
Read what happened: one duplicate was silently dropped (30 in, 29 out), and three pairs failed checks. Fix all three by hand, rerun until 0 problems, then split 24 train / 6 eval — the 6 never touch training, exactly like 3.7's holdout. Twenty minutes of this loop is worth more than any extra GPU hour, because every bad pair that survives becomes model behaviour.
Step 3 — the training script, built to die. This file runs on Kaggle, not your laptop. Read the two marked lines before anything else — they are the compute doctrine in code:
# train_qlora.py -- runs in a Kaggle GPU session
from pathlib import Path
from trl import SFTConfig, SFTTrainer
# model: a small instruct model loaded in 4-bit, LoRA adapters (r=16) on all linear layers
OUT = Path("/kaggle/working/adapters") # persisted as a Kaggle dataset at session end
args = SFTConfig(
output_dir=str(OUT),
num_train_epochs=3,
per_device_train_batch_size=2,
gradient_accumulation_steps=4, # effective batch 8 on a small GPU
learning_rate=2e-4,
save_strategy="epoch", # <-- checkpoint the adapter EVERY epoch (~100 MB)
save_total_limit=2,
logging_steps=10,
)
ckpts = sorted(OUT.glob("checkpoint-*"),
key=lambda p: int(p.name.split("-")[1]))
last = str(ckpts[-1]) if ckpts else None
trainer = SFTTrainer(model=model, train_dataset=train_ds,
eval_dataset=eval_ds, args=args)
trainer.train(resume_from_checkpoint=last) # <-- resumes if a checkpoint exists
trainer.save_model(str(OUT / "final"))
This is a read-along, not a run-now — the model-loading line is deliberately a comment, and you will type the real thing inside a Kaggle session when your dataset earns it. SFTConfig is trl's own training-arguments class; current trl versions expect it where older tutorials pass TrainingArguments. Two lines carry the whole doctrine. save_strategy="epoch" writes the adapter to disk every epoch — about 100 MB, small enough to save as a Kaggle dataset before the session ends. The resume_from_checkpoint line looks for the newest checkpoint at startup and continues from it, so the script behaves the same whether this is session one or session three. Everything else is the standard QLoRA recipe from the source module: small instruct model, rank 16 adapters, two to three epochs, learning rate 2e-4.
Step 4 — what a session-split run looks like. Lakshmi's three epochs need about three GPU hours, and her Tuesday session dies early. The log across two sessions:
-- session 1 (Tuesday, ~2 GPU hours) --
no checkpoint found, starting fresh
epoch 1/3 train loss 1.84 eval loss 1.65 -> saved checkpoint-42
epoch 2/3 train loss 1.21 eval loss 1.32 -> saved checkpoint-84
[session terminated by provider]
-- session 2 (Thursday, ~1 GPU hour) --
found checkpoint-84, resuming from epoch 2
epoch 3/3 train loss 0.97 eval loss 1.28 -> saved checkpoint-126
final adapter saved: adapters/final (98 MB)
The Tuesday death cost nothing: two epochs of work lived on in checkpoint-84. Note the ritual between sessions — save the checkpoint folder as a Kaggle dataset, attach it to the next session, and the resume line does the rest. Also read the losses like a 3.7 graduate: eval loss is still falling at epoch 3, so the run is healthy; if eval loss had turned upward while train loss fell, the right move is to keep the earlier checkpoint, not the last one.
Step 5 — serve it and make it prove itself. Back on the laptop, the merged and quantised model (a GGUF file, exported in the Kaggle session) runs under Ollama next to its base. Same eval, both models:
$ ollama pull qwen2.5:1.5b # the untouched base, for comparison
$ ollama create ticket-bot -f Modelfile # FROM ./ticket-tuned-q4.gguf
$ python evalrun.py --model qwen2.5:1.5b # your 3.7 harness, model name as a flag
$ python evalrun.py --model ticket-bot
format-valid fields-correct general slice
base 9/30 14/30 10/10
ticket-bot 28/30 25/30 9/10
This is the shipping table. The tuned model beats its base where it was trained to — format validity 9 to 28 — and the general slice checks it did not forget how to hold a normal conversation. That one dropped general case is not noise to ignore: read the transcript, decide if it matters, and write the decision down. If the tuned column had come back below the base column, that is WR-051's story — and the dataset, not the GPU, is where you would look first.
Work in ai-lab. No exercise here needs a GPU — that is the point: everything around a training run is laptop work, and the run itself you rehearse with a toy. Each exercise ends with a check.
The decision-defence drill. Five requests, one verdict each — prompts, RAG, or fine-tune. (a) A mandi-price bot must quote today's crop rates. (b) A bank bot must output a fixed JSON incident format, and the best prompt scores 11/30 on your eval. (c) A one-time cleanup of 200 customer addresses, needed this week. (d) A kirana loyalty app wants a consistent Hinglish tone that good prompting cannot hold across long chats. (e) An HR bot answers policy questions from documents that change every quarter. In notes-3.10.md, write each verdict plus a two-sentence defence naming the failure type (facts or behaviour) and what the losing option would cost. Check: all five verdicts are written; each defence names the failure type; for every fine-tune verdict you can point at the eval evidence that earned it, and for every RAG verdict you can say what goes stale.
A 30-pair dataset, built and validated. Pick your own format or style task — a complaint-to-ticket format, a fixed bill summary, a polite two-line reply pattern. Hand-write 30 instruction pairs in JSONL chat format, then write validate_pairs.py with at least four checks: role order, output-format regex, exact-duplicate inputs, and a minimum input length. Fix what it finds, then split 24 train / 6 eval into separate files. Check: the validator prints 0 problems on your final file; when you deliberately re-add one duplicate and one format-breaking pair, it catches both by line number; the 6-pair eval file exists and you can say in one sentence why it must never enter training.
The checkpoint-resume dry run. Write toy_train.py: a loop of 10 fake epochs, each doing two seconds of fake work and updating a running total. After every epoch it writes state.json (epoch number plus the total); at startup it loads state.json if present and continues from there, else starts at epoch 1. Run it once untouched and record the final total. Delete state.json, run again, and kill the process around epoch 4 with Ctrl-C; rerun and watch it resume from epoch 5. Check: the resumed run's final total exactly matches the untouched run's total; your terminal log shows the kill and the resuming from epoch 5 line; in notes-3.10.md, map each toy part to the real thing in two sentences — state.json to the adapter checkpoint, the rerun to a fresh Kaggle session.
Three failures cover most first fine-tuning attempts. Each has a War Room entry with the full diagnosis path:
- WR-049 — three weeks of fine-tuning that RAG replaces in a day. A team tuned a model on rate documents so it would "know" them; the rates changed, the model stayed confidently stale, and the decision table they skipped would have said RAG on day one.
- WR-050 — the free-tier session that died at 90% with nothing saved. Five GPU hours of training vanished when the session ended, because no checkpoint was ever written; adapter checkpoints every epoch plus resume logic turn the same death into a non-event.
- WR-051 — the tuned model that lost to its own base. Duplicated, inconsistent, and mislabelled pairs trained a model worse than the one it started from — and only the base-vs-tuned eval table made the damage visible before launch.
Most LLM problems fall to better prompts, fresh-knowledge problems to RAG, and stubborn format or style problems to fine-tuning — in that order of cost. Fine-tuning changes the model's weights, and LoRA or QLoRA make it affordable by training a small adapter instead of the whole model. The dataset decides the outcome: clean, deduped instruction pairs in JSONL chat format, validated on your laptop, with an eval slice held back. GPU time comes from a planned Kaggle quota, so every run saves an adapter checkpoint each epoch and resumes after a dead session. A tuned model ships only when it beats its own base on your eval set without losing the general slice.
- Banking and insurance GCCs in India run the split this module teaches. A small model is fine-tuned for fixed ticket and report formats, while RAG feeds it policy knowledge — behaviour in the weights, facts in the retriever.
- UPI and fintech support platforms serve one shared base model with a separate adapter per team, because a 100 MB adapter per use case costs far less than a full model copy each.
- A Kaggle-trained adapter with a base-vs-tuned eval table is the portfolio piece that proves this full loop. This module rehearses every laptop-side part without producing one; when your dataset and eval are ready, the Kaggle checkpoint-and-resume path above is how you build yours.
Answer these before opening the gate assessment. Each question names its kind.
type=scenario-judgement— A teammate proposes fine-tuning your bot on this quarter's electricity tariff circulars so it "knows the new rates". Give your judgement, name the failure type this request actually contains, the tool that fits it, and the one question about next quarter that settles the argument.type=interpretation— Your shipping table reads: base 22/30 format-valid, tuned 12/30 format-valid, general slice base 10/10, tuned 6/10. State what happened in one sentence, name two likely causes from this module, and the first artefact you would open to tell them apart.type=output-prediction— You run the Step-2 validator on apairs.jsonlwhere pair 4's input is an exact lowercase copy of pair 1's, and pair 9's assistant reply readsCATEGORY: loans | NEXT: call back. Predict what the script prints about each of these two pairs and the final counts line.type=debug— Session 2 on Kaggle printsno checkpoint found, starting fresh, yet you watched session 1 printsaved checkpoint-84before it died. Name the most likely cause given whereOUTpoints, and the between-sessions step that fixes it for every future run.type=trade-off— For the fixed ticket format, compare living with a 3,000-token mega-prompt on an API model versus fine-tuning a small local model: one concrete gain and one concrete cost of each, which you try first, and the eval evidence that would move you to the other.
Explain to a cousin who runs a kirana shop and has never coded: why you refused to train a custom model to remember his price list, but did train one when his billing bot kept breaking the bill format — and why your training kept saving its progress every few minutes like a game. Use at most 4 sentences, plain words only. If he can repeat back which problem needed "new facts" and which needed "new habits", you own this module.