This module installs the judgment fine-tuning roles actually screen for: read the eval to tell a form gap from a knowledge gap, walk the adaptation ladder, decontaminate training data against the eval set, and let a frozen harness decide whether the adapter ships.
The incident-summary feature — which condenses a resolved support thread into two lines for the exec channel — keeps producing summaries in the wrong register, too chatty where terse wins. He has a Kaggle GPU and a weekend.
Not rank or learning rate: "what does the eval say the failure actually is?"
The M5 harness says the summaries are stylistically off, not factually wrong — a form failure, which is the one thing fine-tuning is genuinely for.
A clearer prompt spec closes half the gap, and the residue is exactly the kind of consistent-form problem weights fix better than context.
Karthik curates eight hundred pairs from accepted summaries.
The question that saves them from a lie: "Did you check the training set against the eval set?" He hadn't.
An embedding-similarity pass finds forty near-duplicates of eval items sitting in the training data — the exact contamination that would have reported a spectacular, meaningless lift and shipped a model no better than the base.
He removes them, trains, and lets the frozen harness decide whether the adapter earns production.
That sequence — measure the gap, walk the ladder, decontaminate, prove the lift honestly — is the whole skill. Fine-tuning is the most over-prescribed intervention in applied LLM work.
It appears in job postings — LoRA and QLoRA are named verbatim in DS/ML listings from Bengaluru to Boston — precisely because employers have been burned by both failure modes: teams that fine-tune when a better prompt would have done, and teams that prompt-engineer for months around a problem a weekend of adaptation would have solved. The skill being screened is not "can you run a training loop." It's "can you tell when a training loop is justified, run it under real constraints, and prove the lift honestly."
You arrive at this module with an asset most fine-tuning tutorials assume away: the eval harness you built in M5–M6. Every claim in this module gets measured with it. That inverts the usual tutorial flow — instead of "train, then eyeball some generations," the discipline here is "freeze the eval set first, train second, and let the harness decide whether the adapter ships." Everything below is engineered to fit the free-tier GPU reality most learners have: a single 16GB T4 or P100 on Kaggle, roughly 30 GPU-hours a week.
Fine-tuning is for form, not facts. It excels at teaching a model how to respond — style, format, domain vocabulary, task-specific behavior — and is the wrong tool for injecting knowledge that changes, which is retrieval's job. Most "we need to fine-tune" conversations end at a cheaper rung of the ladder, and the eval harness is how you find out which rung.
Every adaptation decision walks the same four rungs, and each rung should be exhausted — measured, not vibed — before you climb to the next.
| Rung | What it changes | Typical cost to try | Iteration speed | When it wins |
|---|---|---|---|---|
| Prompt & context engineering | Instructions, examples, structure | Hours; API spend only | Minutes per iteration | Behavior gaps a clearer spec or few-shot examples can close — most gaps |
| RAG | What the model knows at inference | Days; retrieval infra | Hours | Knowledge that is large, changing, or needs provenance/citations |
| Fine-tuning | What the model is — weights | Days–weeks; GPU time, dataset curation, eval cycles | Days | Consistent form: output style, domain dialect, structured tasks where few-shot plateaus, latency/cost pressure from long prompts |
| Distillation | Which model you run | Weeks; teacher labeling + student training | Days | A frontier model is provably good at the task and you need it at 1/20th the serving cost |
The economics deserve honesty because the training run is the cheap part. A QLoRA run on Kaggle costs a few GPU-hours — effectively free. What isn't free: the 20–40 hours of dataset curation done properly, the eval cycles before and after, the second training run when the first one regresses something, and the permanent operational tax of owning a model artifact — you now re-validate on every base-model upgrade, maintain a training pipeline, and lose the frictionless model-swapping a prompt-based system enjoys. Price the whole lifecycle, not the trainer.train() call.
Three signals that fine-tuning is genuinely the right rung:
And the anti-signal: "the model doesn't know about our products" is a RAG problem. Fine-tuning on Tuesday's product catalog produces a model that is confidently wrong by Friday, with no citation trail to catch it.
Scenario questions on adaptation choice hinge on one distinction: knowledge gaps (changing facts, provenance needed) point to RAG; behavior gaps (style, format, domain dialect, few-shot plateau at volume) point to fine-tuning. An answer that reaches for fine-tuning to fix factual staleness is the planted wrong option.
Supervised fine-tuning is only as good as its dataset, and the practitioner consensus is blunt: a small, ruthlessly curated instruction set beats a large noisy one. For a narrow task on a 7–8B model, several hundred to a few thousand excellent pairs is the working range. Your job is curation, and your background in data quality is the edge here — this is dataset construction with a labeling budget, a task you've done before under other names.
The pipeline:
Source from reality. The best training pairs come from the same place your golden dataset came from in M5: real traces. Mine your logs for inputs paired with outputs that were accepted, corrected, or highly rated. Pair domain documents with the artifacts experts actually produced from them. Synthetic augmentation — using a frontier model to draft additional pairs — is legitimate only with human review; unreviewed synthetic data teaches the teacher's tics along with the task.
Deduplicate semantically. Exact-dedup first, then embed everything and cluster (you built this muscle in M2): near-duplicate clusters get collapsed to their best exemplar. Fifty paraphrases of the same billing question teach the model one thing fifty times and starve everything else.
Decontaminate against your eval set. This is the step that separates honest fine-tunes from accidental fraud. Embed both your training candidates and your M5 eval set; flag any training example above a high similarity threshold to any eval item, and remove it. A model trained on near-copies of its test set will show a spectacular, meaningless lift.
Format for TRL. The trainer accepts conversational data directly and applies the model's chat template for you:
# One line of train.jsonl — conversational prompt-completion format
{"prompt": [{"role": "user", "content": "Summarize this incident report for the exec channel:\n<report>...</report>"}],
"completion": [{"role": "assistant", "content": "**Sev-2, resolved.** Payment webhooks delayed 41 min..."}]}
With prompt-completion data, TRL computes loss on the completion only by default — you're teaching the response, not the prompt. Hold out 10–15% as a validation split for the trainer (this is separate from your M5 eval set, which stays untouched in its vault until the final before/after comparison).
Eval-set decontamination is not optional housekeeping. Skipping it is the single most common way teams fool themselves — the harness reports a huge win, the model ships, and production quality is unchanged because the "win" was memorization of the test set.
Full fine-tuning of an 8B model is out of reach on free GPUs — the weights, gradients, and optimizer states of 8B parameters in mixed precision want well over 100GB. LoRA (Low-Rank Adaptation) sidesteps this: freeze the base weights, and learn a pair of small low-rank matrices A (d×r) and B (r×d) whose product is added to selected weight matrices. With rank r=16 on an 8B model you're training tens of millions of parameters instead of eight billion — under 1% — and the artifact you ship is an adapter of a few hundred megabytes.
The knobs that matter:
r (rank): capacity of the adaptation. 8–16 is the sweet spot for style/format tasks; 32–64 for harder behavioral shifts. Doubling r doubles adapter size and rarely doubles quality — treat it as a hyperparameter your validation loss arbitrates.lora_alpha: scaling factor; the effective update is scaled by alpha/r. The common convention is alpha = 2r, which keeps effective scale constant as you sweep r.target_modules: which weight matrices get adapters. Attention-only (q_proj, v_proj) is the minimal classic; modern practice targets all attention and MLP projections, which measurably helps at small extra cost.QLoRA adds the trick that makes 7–8B training fit in 16GB: load the frozen base model quantized to 4-bit NF4 (a data type shaped for normally-distributed weights), with double quantization shaving the quantization constants themselves, while the LoRA adapters and their gradients stay in 16-bit. The arithmetic on a T4: an 8B base at 4-bit is roughly 5GB; adapters, gradients, and optimizer state add well under 1GB; the rest of the 16GB absorbs activations — which gradient checkpointing (on by default in TRL) keeps manageable at 1K-token sequences with a small batch.
The worked configuration, using TRL's modern API:
import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer
bnb = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.float16, # T4 has no bfloat16 support
)
lora = LoraConfig(
r=16,
lora_alpha=32, # alpha = 2r convention
lora_dropout=0.05,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
task_type="CAUSAL_LM",
)
args = SFTConfig(
output_dir="qlora-incident-summarizer",
max_length=1024,
packing=True, # pack short examples into full sequences
per_device_train_batch_size=2,
gradient_accumulation_steps=8, # effective batch size 16
learning_rate=2e-4, # adapters train hot; ~10x full-FT rates
num_train_epochs=2,
fp16=True, # not bf16 — Turing GPU
eval_strategy="steps",
eval_steps=50,
logging_steps=10,
)
trainer = SFTTrainer(
model="Qwen/Qwen2.5-7B-Instruct",
args=args,
train_dataset=load_dataset("json", data_files="train.jsonl", split="train"),
eval_dataset=load_dataset("json", data_files="val.jsonl", split="train"),
quantization_config=bnb,
peft_config=lora,
)
trainer.train()
trainer.save_model("adapter-v1") # saves the adapter, not 7B of weights
Practical notes from the trenches: watch validation loss, not training loss — with a small dataset an 8B model can memorize your training set inside two epochs, and the validation curve turning upward is your stop signal. On Kaggle, save adapter checkpoints to the working directory regularly; sessions cap at nine hours and this run should finish in a fraction of that. If you hit CUDA OOM, the levers in order: shorter max_length, batch size 1 with more accumulation, then smaller r. Unsloth offers the same workflow with substantially faster kernels and lower memory if the stock stack is tight.
The alpha = 2r convention exists so that when you sweep rank, you're testing capacity — not silently changing the update scale at the same time. Change one variable per run; GPU-hours on a free tier are a budget like any other.
The adapter is the deliverable, and it composes with serving: vLLM can load LoRA adapters on top of a shared base model, so one deployed base can serve several task-specific adapters. That story completes in M9.
SFT teaches the model what good looks like; it never shows what bad looks like. When your error analysis says outputs are "correct but worse" — verbose where terse wins, hedging where directness wins, plausible-but-shallow where rigor wins — you have a preference gap, and pairs of (chosen, rejected) responses encode it better than more gold examples can.
Direct Preference Optimization (DPO) trains on triples of {"prompt": ..., "chosen": ..., "rejected": ...} with a loss that pushes the policy's relative log-probability of chosen-over-rejected above a frozen reference model's, temperature-controlled by beta (lower = stronger preference pressure, higher = stay closer to the reference). Unlike RLHF it needs no reward model and no sampling loop — it's a supervised-shaped training run you can do with TRL on the same Kaggle GPU, typically as a short second stage on top of your SFT adapter:
from trl import DPOConfig, DPOTrainer
args = DPOConfig(
output_dir="dpo-stage",
beta=0.1,
learning_rate=5e-6, # far gentler than SFT
num_train_epochs=1,
)
trainer = DPOTrainer(
model=sft_model, # your SFT-tuned PeftModel
args=args,
train_dataset=pref_ds, # prompt / chosen / rejected
processing_class=tokenizer,
)
trainer.train()
With a PEFT model, TRL derives the frozen reference by disabling the adapter — no second model copy in memory, which is what keeps this on-budget. Preference pairs come from the annotation ops you built in M6: reviewer corrections (original = rejected, corrected = chosen) are the cheapest honest source. A few hundred clean pairs is a real experiment. GRPO and other RL-based methods — now appearing by name in Indian job postings — sit beyond this module; DPO is the entry point that gets you conversant in the family.
This section is the module's reason for existing. The evaluation protocol, in order:
lm-evaluation-harness (a few hundred items keeps it inside free-GPU time), plus 30–50 held-out general prompts scored by your judge. Define the regression budget before training — e.g., ship only if task metric improves by your target and every general probe stays within a small tolerance of base.The output of all this is the one-page honest eval report — base vs. tuned on task metrics with confidence intervals, slice table, regression suite results, and a ship/no-ship recommendation. That artifact, more than the adapter, is what distinguishes you in interviews: "benchmarking model performance" appears in fine-tuning job specs because most candidates can train and few can prove.
The same protocol covers distillation, where the question becomes parity rather than lift: label a large input set with a frontier teacher — batch APIs cut labeling cost roughly in half — train a small student (an SLM via this module's SFT recipe, or even a classical GBM on extracted features for classification tasks), and require the student to hold a defined fraction of teacher quality on your harness at its fraction of the cost.
For teacher labeling at scale, schema-enforced outputs keep the training data clean:
import anthropic
from pydantic import BaseModel
from typing import Literal
class TriageLabel(BaseModel):
category: Literal["billing", "bug", "how_to", "churn_risk", "other"]
severity: Literal["low", "medium", "high"]
client = anthropic.Anthropic()
resp = client.messages.parse(
model="claude-opus-4-8",
max_tokens=256,
messages=[{"role": "user", "content": f"<ticket>{ticket_text}</ticket>\nLabel this ticket."}],
output_format=TriageLabel,
)
label = resp.parsed_output # validated TriageLabel instance
The other model worth adapting is not the generator — it's the retriever. Off-the-shelf embedding models were trained on general text; your domain's vocabulary ("rollover" in pensions vs. mobile data vs. dog training) is exactly where they blur distinctions your users rely on. If your M3/M4 retrieval benchmark shows recall stuck despite hybrid search and reranking, embedding fine-tuning is the next lever, and it's dramatically cheaper than generator fine-tuning — base models are a few hundred million parameters and train on free GPUs in minutes to an hour.
The recipe is contrastive: mine (query, positive) pairs from your own telemetry — queries paired with the chunk users actually clicked or the passage a correct answer cited — and train with a loss that pulls queries toward their positives and pushes them from in-batch negatives:
from datasets import load_dataset
from sentence_transformers import (SentenceTransformer, SentenceTransformerTrainer,
SentenceTransformerTrainingArguments)
from sentence_transformers.losses import MultipleNegativesRankingLoss
model = SentenceTransformer("BAAI/bge-base-en-v1.5")
train_ds = load_dataset("json", data_files="pairs.jsonl", split="train") # columns: anchor, positive
loss = MultipleNegativesRankingLoss(model) # in-batch negatives: bigger batch = harder task
trainer = SentenceTransformerTrainer(
model=model,
args=SentenceTransformerTrainingArguments(
output_dir="domain-embedder", num_train_epochs=1,
per_device_train_batch_size=64, learning_rate=2e-5, fp16=True,
),
train_dataset=train_ds,
loss=loss,
)
trainer.train()
Add mined hard negatives — retrieved-but-wrong chunks from your logs — as a third column when in-batch negatives get too easy. Evaluation reuses the labeled retrieval benchmark you built for embedding selection in M3: recall@k and MRR, off-the-shelf vs. tuned, on held-out queries. The same honesty rules apply — decontaminate training pairs against benchmark queries, and note the operational cost you're accepting: a custom embedder means re-embedding the corpus on every model update, and a frozen dependency you must now version and re-validate.
A few points of recall@k from a fine-tuned embedder often moves end-to-end answer quality more than any generator intervention — the generator can't cite what retrieval never surfaced. Check the retriever before assuming the generator is the model that needs adapting.
Your first 15 minutes Monday: before anyone says "let's fine-tune," write the one-line ladder verdict — is the gap a knowledge gap (changing facts, needs provenance → RAG) or a form gap (style, format, domain dialect, few-shot plateau at volume → fine-tune)? If it's knowledge, stop; fine-tuning on Tuesday's catalog is confidently wrong by Friday. That single distinction settles most "we need to fine-tune" conversations before a GPU is booked.
Check yourself: A teammate says "the model keeps getting our product names and specs wrong — let's fine-tune it on our catalog." Why is that the planted wrong answer, and what does the eval evidence need to show before fine-tuning is justified?
torch.cuda.max_memory_allocated()), and save the adapter. Then rerun with r=8: compare validation loss, adapter size, and wall-clock, and state which you'd ship.lm-evaluation-harness subset or 30 held-out general prompts). Deliverable: the one-page ship/no-ship report.bge-base, and measure recall@5 and MRR against your M3 benchmark, off-the-shelf vs. tuned, on decontaminated held-out queries. State whether the lift justifies owning a custom embedder.This module anchors Lab L11 — QLoRA fine-tune on a free GPU, the track's proof that you can run the full adaptation lifecycle under real constraints: curate ~800 instruction pairs with documented dedup and decontamination, QLoRA-tune a 3–8B model inside Kaggle's 16GB budget, and evaluate base-vs-tuned on a held-out harness including a general-capability regression check. The artifact — adapter weights plus the honest eval report — is exactly the evidence "LoRA/QLoRA + benchmarking" job postings are probing for. The playbooks provide the adaptation-decision worksheet (ladder + lifecycle cost model), the SFT curation checklist with the decontamination procedure, the QLoRA memory-budget calculator for 16GB GPUs, and the before/after evaluation protocol template. Do this at work this week → pb-27 Choose the Adaptation Path: Prompt, RAG, Fine-Tune, or Distill (Neha's ladder verdict with honest cost accounting), pb-28 Engineer an SFT Dataset: 1,000 Curated Pairs Beat 100,000 Noisy Ones (curation with the decontamination pass Priya demanded), and pb-29 LoRA and QLoRA Fine-Tuning on a Free 16GB GPU (the worked Kaggle-budget training run). Every path is free-tier: Kaggle/Colab GPUs, open-weight 7–8B bases, judge calls on cheap-tier models within the track's standard per-lab API budget.
SFTConfig, PEFT and quantization integration