Preference Tuning with DPO: When SFT Alone Is Not Enough

Tier: Advanced · Workflow area: Fine-tuning & Model Adaptation

DPO (Direct Preference Optimization) does this without training a separate reward model, and GRPO-style methods now appear by name in Indian job postings. This playbook runs DPO on a small preference set, on the same free-GPU budget as pb-29.

When to use

• •
  Your SFT adapter (pb-29) passes correctness evals but loses side-by-side preference comparisons against a reference.
• •
  Failure modes are relative ("A is better than B") rather than absolute ("B is wrong") — tone, concision, safety-of-phrasing, choosing among valid answers.
• •
  You have or can generate honest preference pairs: same prompt, chosen response, rejected response.

Skip if SFT-stage problems remain (wrong facts, broken format) — DPO polishes preferences, it does not repair correctness; fix the SFT dataset first.

Prerequisites

• •
  The SFT-tuned model from pb-29 as your starting point (DPO on a raw base model wastes most of its budget re-learning format).
• •
  500–2,000 preference triples (prompt, chosen, rejected) — quality bar as in pb-28, decontaminated against evals.
• •
  TRL with DPOTrainer, plus the pb-29 GPU setup (Kaggle 16GB, 4-bit base + LoRA for 7–8B).
• •
  A held-out preference set (~100 triples) never used in training.

The workflow

1. 1.

   Mine preference pairs from reality first. Best sources: production A/B outputs with user choices, reviewer edit history (pre-edit = rejected, post-edit = chosen), and side-by-side ratings from pb-36's annotation workflow. Synthetic fallback: sample two responses at temperature 0.9 from your SFT model, have a strong judge pick chosen/rejected with your rubric, and human-review 20% of verdicts before trusting the rest.

2. 2.

   Audit the pairs for degenerate patterns. Check length statistics: if "chosen" is systematically longer than "rejected", DPO will learn verbosity, not quality — length-match or subsample until the chosen/rejected length ratio averages ~1.0. Same check for format artifacts (markdown headers, bullet density).

3. 3.

   Configure DPO conservatively.

   Copy

   from trl import DPOConfig, DPOTrainer
   
   cfg = DPOConfig(
       beta=0.1,                      # KL strength: raise to 0.3 if drifting, lower to 0.05 if not moving
       learning_rate=5e-6,            # 10-40x smaller than SFT — this is not a typo
       per_device_train_batch_size=1,
       gradient_accumulation_steps=16,
       num_train_epochs=1,
       max_length=1024, max_prompt_length=512,
   )
   

   Train a fresh LoRA adapter on top of the merged SFT model (or stack adapters); with PEFT, TRL uses the frozen base as the implicit reference model, saving the second model's VRAM — that is what keeps 7–8B DPO inside 16GB.

4. 4.

   Monitor the DPO-specific metrics. rewards/accuracies (how often the model prefers chosen over rejected) should climb toward 0.7–0.9; rewards/margins should grow steadily. Accuracy pinned at ~0.5 means your pairs are indistinguishable or mislabeled; accuracy at 1.0 within an epoch means pairs are trivially separable (often the length artifact from step 2).

5. 5.

   Evaluate three ways. (a) Preference accuracy on the held-out triples. (b) Win-rate: calibrated judge (pb-36) compares DPO-model vs SFT-model outputs on fresh prompts, order-randomized. (c) Regression: re-run the pb-33 correctness suite — DPO must not have traded correctness for likability.

6. 6.

   Decide SFT-only vs SFT+DPO with numbers. Ship DPO only if the win-rate improves meaningfully (e.g., ≥ 60% with confidence interval clearing 50%) and correctness regressions are within your tolerance. Record the verdict and configs in the model card.

Tool variants

• HF TRL DPOTrainer: the reference implementation; pairs directly with the pb-29 PEFT stack.
• Unsloth DPO: same API surface, faster and lighter on free GPUs.
• GRPO/online methods (TRL): when you have a programmatic reward (verifiable answers, passing tests) instead of static pairs — heavier compute; treat as the next step after DPO, not the first.

Verification

• •
  Chosen/rejected length ratio ≈ 1.0 (or an explicit note that length correlates with genuine quality in your task).
• •
  Held-out preference accuracy meaningfully above 0.5 with your ~100 untouched triples.
• •
  Judge-based win-rate over the SFT model computed with position randomization (A/B and B/A), not single-order.
• •
  pb-33 regression suite delta documented; no silent correctness loss.

Pitfalls

• SFT-scale learning rates. 2e-4 destroys a model in DPO; preference gradients are strong signals — 5e-7 to 5e-6 is the working range.
• Judge-generated pairs with no human audit. You are distilling the judge's biases (verbosity, sycophancy) straight into weights; the 20% human check is mandatory.
• Reward hacking the metric. Rising reward margins with falling human ratings means the model found a loophole in the pairs; re-audit before training longer.
• Skipping the correctness regression. The most common DPO failure is a more agreeable model that is slightly more wrong.
• Over-training. One epoch is the norm on small sets; DPO overfits into repetitive, sycophantic phrasing quickly.

Restricted-environment variant

• •
  Preference pairs mined from client interactions carry the same sensitivity as the interactions themselves — anonymize with pb-28's placeholder discipline before any external platform sees them.
• •
  If judges cannot run on external APIs, use a self-hosted 8B judge for pair labeling, double the human-review sample to 40%, and note the weaker-judge caveat in the model card.
• •
  Run training inside the client boundary per pb-29's restricted variant; adapters trained on preferences leak stylistic fingerprints of the source data.

Success criteria

• •
  A DPO adapter whose held-out preference accuracy and judge win-rate both beat the SFT-only model.
• •
  Zero unexplained correctness regressions against the pb-33 suite.
• •
  A pair-audit note proving the chosen/rejected separator is quality, not length or format.
• •
  A model card documenting when your team should (and should not) reach for DPO again.