Lab 11 · QLoRA Fine-Tune on a Free GPU

Fine-tuning is for form, not facts — this task (messy analyst notes → strict JSON) is exactly the kind of format-shaping where a small adapter earns its keep.

Time & tier

Advanced · ~2.5–3 hours (about 30–45 minutes of that is GPU training) · $0 — everything runs on Kaggle's free GPU quota (~30 h/week).

Setup

Create a Kaggle notebook: kaggle.com → Create → Notebook → Settings: Accelerator = GPU T4 x2 (we use one), Internet = On. In the first cell:

!pip install -q -U trl peft bitsandbytes datasets accelerate
import torch; print(torch.cuda.get_device_name(0))

This lab is fully self-contained — the dataset is synthesized in-notebook with a fixed seed. Paste and run the generator (this is your "curated ~800 pairs"; in a real engagement these come from logs plus human curation, deduped and decontaminated against your eval set — here the synthesis makes decontamination trivial and the split explicit):

import json, random
random.seed(13)
METRICS = ["conversion rate", "churn", "ARPU", "activation rate", "NPS", "CAC", "retention", "DAU"]
SEGMENTS = ["enterprise accounts", "self-serve users", "EU region", "mobile users", "trial cohort", "Q2 signups"]
CONF = ["high", "medium", "low"]
DRIVERS = ["pricing change", "onboarding revamp", "seasonality", "outage on the 12th", "new ad campaign", "competitor launch"]
STYLE = [
    "quick note - {m} for {s} moved {d} {v}% wow. confidence {c}. driver seems to be {drv}",
    "FYI {m} ({s}): {d} {v}% vs last week; {c} confidence; likely {drv}",
    "obs: {s} {m} {d}{v}pct. conf={c}. hypothesis: {drv}",
    "{m} shifted {d} {v}% in {s} this week — {c} confidence, attributing to {drv}",
]
SYSTEM = ("Convert the analyst note into JSON with keys: metric, segment, direction (up|down), "
          "change_pct (number), confidence (high|medium|low), driver.")
pairs = []
for _ in range(800):
    m, s, c, drv = random.choice(METRICS), random.choice(SEGMENTS), random.choice(CONF), random.choice(DRIVERS)
    d = random.choice(["up", "down"]); v = round(random.uniform(0.3, 24.0), 1)
    note = random.choice(STYLE).format(m=m, s=s, d=d, v=v, c=c, drv=drv)
    target = {"metric": m, "segment": s, "direction": d, "change_pct": v, "confidence": c, "driver": drv}
    pairs.append({"prompt": [{"role": "system", "content": SYSTEM}, {"role": "user", "content": note}],
                  "completion": [{"role": "assistant", "content": json.dumps(target)}]})
random.shuffle(pairs)
train, holdout = pairs[:720], pairs[720:]
for name, rows in [("train.jsonl", train), ("holdout.jsonl", holdout)]:
    with open(name, "w") as f:
        for p in rows: f.write(json.dumps(p) + "\n")
print("720 train / 80 holdout")

This is TRL's conversational prompt-completion format, so the trainer applies the chat template and computes loss on completions only — no formatting function needed.

Guided steps

1. 1.

   Baseline the base model first. No before, no honest after. Load Qwen/Qwen2.5-7B-Instruct (Apache-2.0) in 4-bit and score the 80-item holdout:

   Copy

   import json, torch
   from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
   
   BASE = "Qwen/Qwen2.5-7B-Instruct"
   bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                            bnb_4bit_compute_dtype=torch.float16,   # T4 has no bfloat16
                            bnb_4bit_use_double_quant=True)
   tok = AutoTokenizer.from_pretrained(BASE)
   model = AutoModelForCausalLM.from_pretrained(BASE, quantization_config=bnb, device_map="auto")
   
   KEYS = ["metric", "segment", "direction", "change_pct", "confidence", "driver"]
   def score(model, holdout_path="holdout.jsonl"):
       items = [json.loads(l) for l in open(holdout_path)]
       valid = fields = 0
       for it in items:
           msgs = it["prompt"]
           ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
           out = model.generate(ids, max_new_tokens=120, do_sample=False, pad_token_id=tok.eos_token_id)
           text = tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True).strip()
           gold = json.loads(it["completion"][0]["content"])
           try:
               pred = json.loads(text[text.index("{"): text.rindex("}") + 1])
               valid += 1
               fields += sum(pred.get(k) == gold[k] for k in KEYS) / len(KEYS)
           except (ValueError, json.JSONDecodeError):
               pass
       n = len(items)
       return {"json_valid": valid / n, "field_acc": fields / n}
   
   base_task = score(model); print("BASE:", base_task)
   

   Expect the base model to be decent but imperfect — extra prose around the JSON, key-name drift, change_pct as a string.

2. 2.

   General-capability baseline. Before touching a single weight, measure something your training data does not cover — ARC-Easy, 200 questions, generation-based:

   Copy

   from datasets import load_dataset
   arc = load_dataset("allenai/ai2_arc", "ARC-Easy", split="test").select(range(200))
   
   def arc_score(model):
       correct = 0
       for ex in arc:
           opts = "\n".join(f"{l}. {t}" for l, t in zip(ex["choices"]["label"], ex["choices"]["text"]))
           msgs = [{"role": "user", "content":
                    f"{ex['question']}\n{opts}\nAnswer with the letter only."}]
           ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
           out = model.generate(ids, max_new_tokens=4, do_sample=False, pad_token_id=tok.eos_token_id)
           pred = tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True).strip()[:1]
           correct += pred == ex["answerKey"]
       return correct / len(arc)
   
   base_arc = arc_score(model); print("BASE ARC-Easy:", base_arc)
   

3. 3.

   Train with QLoRA — the config that fits 16 GB. 7B weights in NF4 are ~4.7 GB; LoRA adapters, fp16 activations at max_length=512, batch 2 with gradient accumulation, and gradient checkpointing (TRL's default) keep peak memory around 10–12 GB. Free the baseline model first (del model; torch.cuda.empty_cache()), then:

   Copy

   from datasets import load_dataset
   from peft import LoraConfig
   from trl import SFTConfig, SFTTrainer
   from transformers import BitsAndBytesConfig
   
   ds = load_dataset("json", data_files="train.jsonl", split="train")
   trainer = SFTTrainer(
       model=BASE,
       train_dataset=ds,
       quantization_config=BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                                              bnb_4bit_compute_dtype=torch.float16,
                                              bnb_4bit_use_double_quant=True),
       peft_config=LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, task_type="CAUSAL_LM",
                              target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                                              "gate_proj", "up_proj", "down_proj"]),
       args=SFTConfig(output_dir="qlora-notes", max_length=512,
                      per_device_train_batch_size=2, gradient_accumulation_steps=8,
                      num_train_epochs=2, learning_rate=1e-4,
                      fp16=True, bf16=False,           # T4: fp16, never bf16
                      logging_steps=10, save_strategy="epoch", report_to="none",
                      model_init_kwargs={"device_map": "auto"}),
   )
   trainer.train()
   print(f"peak VRAM: {torch.cuda.max_memory_allocated()/1e9:.1f} GB")
   trainer.save_model("qlora-notes/final")
   

   ~90 optimizer steps; expect 25–45 minutes on a T4 (longer on P100). Watch mean_token_accuracy climb.

4. 4.

   Score the tuned model — task and regression. Reuse the trainer's model (it is the 4-bit base with adapters attached): tuned = trainer.model; tuned.eval(), then score(tuned) and arc_score(tuned).

5. 5.

   Write the honest eval report. A markdown table with four cells that matter:

   metric	base	tuned
   JSON-valid rate (holdout)	e.g. 0.86	≥ 0.98
   field accuracy (holdout)	e.g. 0.71	≥ 0.95
   ARC-Easy accuracy (regression)	e.g. 0.88	within ±3 pts

   Then three written judgments: (a) contamination statement — how you know holdout items were never trained on (here: disjoint split of a seeded synthesis, stated explicitly); (b) whether the ARC delta is inside your tolerance (±3 points on 200 questions is ~sampling noise; a 10-point drop means your LR/epochs cooked general ability); (c) whether prompt engineering alone could have bought this lift for less — run the base model once with a few-shot prompt and say so either way.

6. 6.

   Ship the adapter, not the model. qlora-notes/final holds only LoRA weights (tens of MB). Save it via Kaggle's output, and record base model + revision, LoRA config, data hash, and eval table alongside — an adapter without its provenance is not reproducible.

Verification

• •
  Training completes on one T4 with reported peak VRAM under 15 GB.
• •
  Tuned model: JSON-valid ≥ 0.98 and field accuracy beats base by a clear margin on the untouched holdout.
• •
  ARC-Easy delta between base and tuned is within ±3 points — the regression check ran on both models with identical prompts.
• •
  The report states the contamination check and the prompt-vs-tune comparison.

Completion artifact

Adapter weights (qlora-notes/final) + the notebook + EVAL.md (the table and three judgments). Self-grade (pass at 4+):

• Memory fit: config demonstrably fits 16 GB.
• Baseline first: baseline measured before training.
• Regression check: general-capability regression measured, not asserted.
• Contamination statement: present, stating how you know holdout items were never trained on.
• Provenance: adapter shipped with full provenance.

Stuck?

• CUDA OOM: drop max_length to 384 and per_device_train_batch_size to 1 (raise accumulation to 16); confirm nothing else holds the GPU (del model; torch.cuda.empty_cache() after the baseline).
• bf16 error or silent NaN loss: you are on a T4/P100 — fp16=True, bf16=False everywhere, including bnb_4bit_compute_dtype=torch.float16.
• bitsandbytes import error: restart the Kaggle session after the pip cell; the CUDA libs load at first import.
• Base scores of exactly 0: your JSON extraction slice is failing on chatty output — check the {/} bracketing logic against a printed sample first.
• Session about to hit the 9-hour cap: checkpoints save per epoch; resume with trainer.train(resume_from_checkpoint=True).

Client-safe variant

Never fine-tune on raw client data on infrastructure the client hasn't approved — training data is the most sensitive artifact in this whole track because the weights memorize it. The compliant pattern: curate and redact pairs inside the client environment, train on client-controlled GPUs (their cloud account; the same notebook runs on a single A10/L4 spot instance in ap-south-1 for well under $1/run), and deliver the adapter plus eval report, never the intermediate dataset. This lab's synthesize-from-schema trick is itself a client-safe technique: generate training pairs from the client's schemas and policies rather than their customers' actual text, and validate on a small human-reviewed real set that stays in-tenancy. Check the base model's license (Qwen2.5-7B-Instruct is Apache-2.0) against the client's OSS policy before training, and record it in the provenance file.