LoRA and QLoRA are named verbatim in Indian DS/ML job postings for a reason: they are what makes fine-tuning feasible without a GPU cluster.
Tier: Advanced · Workflow area: Fine-tuning & Model Adaptation
This playbook runs a complete parameter-efficient fine-tune on a free Kaggle T4/P100 (16GB) — LoRA in fp16 for 1–3B models, QLoRA 4-bit for 7–8B — with configs that actually fit, a before/after eval, and the fidelity trade-off measured rather than assumed.
Skip if a system prompt plus few-shot already hits your eval target — an adapter you did not need is pure maintenance debt.
transformers, peft, trl, bitsandbytes, accelerate (preinstalled or pip-installable in the notebook).Pick the model to fit the memory math. fp16 weights cost ~2 bytes/param: a 3B base ≈ 6GB, leaving room for LoRA fp16 training on 16GB. A 7–8B base ≈ 14–16GB in fp16 — weights alone exhaust the card, so 7–8B requires QLoRA (4-bit base ≈ 4–5GB). Start with 3B LoRA for the mechanics, then repeat at 7–8B QLoRA.
Baseline before training. Run the untuned base model on your full eval set and store per-example scores. No baseline, no claim — "after" numbers without "before" numbers are decoration.
Configure QLoRA quantization (7–8B path).
from transformers import BitsAndBytesConfig
import torch
bnb = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.float16, # T4/P100 have no bf16 — fp16, not bfloat16
)
Configure the LoRA adapter.
from peft import LoraConfig
lora = LoraConfig(
r=16, lora_alpha=32, lora_dropout=0.05,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
task_type="CAUSAL_LM",
)
r=16/alpha=32 is the sane default; move r to 8 if memory is tight or 32 if the task is far from the base distribution. Targeting all linear projections (not just q/v) is the 2026 default and measurably helps.
Train with 16GB-safe settings. SFTTrainer with: per_device_train_batch_size=2, gradient_accumulation_steps=8 (effective batch 16), max_seq_length=1024 (2048 only on 3B), learning_rate=2e-4, cosine schedule with warmup_ratio=0.03, num_train_epochs=2, fp16=True, gradient_checkpointing=True, optim="paged_adamw_8bit". On Kaggle, save adapter checkpoints every epoch to /kaggle/working — sessions die at 9 hours.
Watch the loss like a statistician. Training loss should fall smoothly and eval loss should track it; eval loss climbing while train loss falls = overfitting your 1k pairs — stop at the earlier checkpoint. Loss spiking to NaN on fp16 usually means learning rate too high; halve it.
Run the before/after eval. Same eval set, same grader, base vs tuned. Report per-slice deltas, not just the average. For the QLoRA path, also compare the 4-bit base (untuned) against the fp16 base on 20 cases to isolate the quantization cost from the tuning gain — this is the fidelity trade-off number most people never measure.
Package the artifact. Save only the adapter (model.save_pretrained — tens of MB, not GB), with a model card: base model + revision, dataset version, full config, before/after table, and known regressions from pb-33.
Unsloth wraps this exact stack with ~2x speed and lower VRAM on the same free GPUs — same LoRA concepts, one-line model loading; use it when Kaggle hours are the binding constraint.
torch.cuda.max_memory_allocated()) stayed under 15GB — headroom matters for reproducibility on busier sessions.bf16=True on a T4/P100. These cards do not support bfloat16 — training silently falls back or crashes; use fp16=True with the paged optimizer to absorb instability.For client data or offline mandates:
HF_HUB_OFFLINE=1) so training needs no egress.Uploading a client-derived dataset to a free notebook platform is a data transfer to a third party — get that decision made by the data owner, not by the person holding the GPU quota.