Engineer an SFT Dataset: 1,000 Curated Pairs Beat 100,000 Noisy Ones

Tier: Advanced · Workflow area: Fine-tuning & Model Adaptation

This playbook takes you from raw logs and domain docs to a deduplicated, decontaminated, chat-formatted SFT dataset ready for TRL — with an audit trail for every filtering decision.

When to use

• •
  You passed the adaptation decision gate (pb-27) and fine-tuning is justified for a format/style/procedure gap.
• •
  You have raw material — support tickets, analyst reports, SQL review threads, internal Q&A — that encodes how your org wants the task done.
• •
  An existing fine-tune underperforms and you suspect the dataset, not the hyperparameters (this is the correct suspicion ~80% of the time).

Skip this playbook if you cannot describe what a "gold" output looks like in one paragraph — write that spec first; curation without a quality bar is just data laundering.

Prerequisites

• •
  A one-paragraph output spec: format, tone, length, what must always/never appear.
• •
  Source material: production logs, domain documents, or expert time to author examples.
• •
  Python with datasets, pandas, and a strong API model for assisted drafting (budget ≤ $3).
• •
  Your eval set from pb-34/pb-27 — you need it now, before training, for decontamination.

The workflow

1. 1.

   Write the data card first (20 minutes). One page: task definition, gold-output spec, sources used, licensing/PII status of each source, target size (start at 800–1,200 pairs), and the split policy. Every later step logs its counts into this card.

2. 2.

   Mine candidates from real usage. Pull successful interactions from logs (high user rating, accepted answer, no follow-up correction) and expert-authored documents. Convert each to an (instruction, input, output) triple. Aim for 3–5x your target count as raw candidates.

3. 3.

   LLM-assisted drafting with mandatory human review. For gaps in coverage, have a frontier model draft pairs from domain docs — then a human reviews every synthetic pair against the spec, editing or rejecting. Track the reject rate; if it is under 20%, your reviewer is rubber-stamping, not reviewing.

4. 4.

   Deduplicate. Exact dedup on normalized text, then near-dedup: embed instructions with a sentence-transformer, drop pairs with cosine similarity > 0.92 to a kept pair. Log how many you removed — heavy duplication is why 100k-scale scraped sets underperform.

5. 5.

   Decontaminate against your eval set. Embed both training instructions and eval-set inputs; flag any training pair with similarity > 0.85 to any eval case and remove it. Also run substring checks on distinctive entities. Record the removed count in the data card — this number is what makes your later eval claims believable (pb-33).

6. 6.

   Balance and stress the distribution. Bucket pairs by sub-task and difficulty; cap any bucket at ~25% of the set. Deliberately include 10–15% "hard negatives": ambiguous inputs where the gold output says what is unknowable or refuses gracefully — without these, the tuned model never learns to hedge.

7. 7.

   Format for TRL chat templates. Emit JSONL in messages format — {"messages": [{"role": "system", ...}, {"role": "user", ...}, {"role": "assistant", ...}]} — and let TRL's apply_chat_template handle model-specific tokens. Never hand-write <|im_start|>-style markers; template mismatch is the most common silent fine-tune killer.

8. 8.

   Final quality gate. Sample 50 random pairs; score each pass/fail against the spec. Ship only if ≥ 90% pass. Version the dataset (git-LFS, DVC, or HF datasets with a revision tag) alongside the data card.

Tool variants

• HF datasets + pandas: the default; dataset.map for formatting, dataset.train_test_split(seed=42) for splits.
• Argilla or Label Studio: when multiple reviewers grade synthetic pairs; gives you per-reviewer accept rates and disagreement queues.
• LLM-assisted pipeline (any frontier API): drafting and first-pass quality scoring; keep the human veto on every pair that enters the final set.

Verification

• •
  Data card shows the full funnel with counts: candidates → post-review → post-dedup → post-decontamination → final.
• •
  Synthetic-pair reject rate ≥ 20%, proving review was real.
• •
  Zero training pairs above the similarity threshold to any eval case; decontamination code is committed next to the data.
• •
  50-pair random audit passes at ≥ 90%; the failing pairs are quoted in the card with reasons.
• •
  The JSONL loads through TRL's chat template for your target base model without warnings.

Pitfalls

• Curating for volume. Adding 5,000 mediocre pairs to 1,000 good ones drags the model toward mediocre — quality is the median, not the max.
• Skipping decontamination. Eval leakage produces a fine-tune that looks brilliant and ships broken; this is the most common way teams fool themselves (pb-33 checks for it independently).
• All-easy examples. A set with no ambiguity teaches the model that every question has a clean answer — hallucination by training design.
• Hand-formatting chat tokens. One wrong special token trains the model to emit garbage delimiters; always go through the template API.
• Unversioned data. If you cannot diff dataset v3 against v2, you cannot explain why the new adapter behaves differently.

Restricted-environment variant

For client data or PII-bearing logs:

• •
  Run PII detection (Presidio or regex passes for emails, phones, IDs) before any pair leaves the source system; replace with typed placeholders (CUSTOMER_NAME, ACCOUNT_ID).
• •
  Do LLM-assisted drafting with a self-hosted model (Ollama/vLLM) if client terms bar external APIs; quality drops slightly, so raise the human-review bar.
• •
  Keep the raw-to-curated mapping inside the client boundary; the exported artifact is the anonymized JSONL plus the data card.

Success criteria

• •
  A versioned JSONL of ~800–1,200 pairs with a complete data card and reproducible filtering scripts.
• •
  Documented decontamination against the exact eval set you will use in pb-33.
• •
  A teammate can audit any pair back to its source and the decision that admitted it.
• •
  The fine-tune trained on this set (pb-29) beats the base model on your eval without regressing general behavior.