Build a Fine-Tuning Dataset Curation Pipeline

Tier: Advanced · Workflow area: Data for fine-tuning & evaluation

The pipeline that gets you from raw logs and documents to training-ready JSONL is data engineering, start to finish — dedup (exact and near-dup via embeddings), quality filtering (heuristics first, LLM scoring second), PII scrubbing, format validation, and leakage-checked train/eval splits. Netflix-scale post-training runs on exactly these steps; so does a 5,000-example LoRA job on a single GPU.

When to reach for this

• •
  The ML team wants to fine-tune on your org's data — support conversations, internal Q&A, domain documents — and needs a defensible dataset, not a dump.
• •
  An earlier fine-tune underperformed and the suspicion is data quality (it usually is).
• •
  Compliance asks what, exactly, went into the model — and you need a datasheet, not a shrug.

The workflow

1. 1.

   Normalize everything into one candidate schema. Whatever the sources, land them as uniform records first: {id, source, messages: [{role, content}], meta}. Every later stage reads and writes this shape, and source survives to the datasheet.

2. 2.

   Dedup in two passes. Exact first — hash normalized text (casefold, collapse whitespace) and drop collisions. Then near-dup: embed each record, index in pgvector or FAISS, and drop pairs above ~0.95 cosine similarity, keeping the higher-quality member:

   Copy

   dup_pairs = index.range_search(embs, threshold=0.95)   # near-dup candidates
   

   Expected effect on real corpora: 10–40% removed. Duplicates don't just waste tokens — repeated examples get memorized, and near-dups that straddle a train/eval split fake your eval scores.

3. 3.

   Filter with cheap heuristics before any LLM. Length bounds, language ID, boilerplate detection, truncated-conversation checks, ratio-of-code-to-text where relevant. Heuristics kill the obvious 20% for free; record per-rule drop counts — that's the first column of your datasheet funnel.

4. 4.

   Score the survivors with an LLM judge, in a loop. A structured rubric score per record, batched over the API:

   Copy

   import anthropic
   from pydantic import BaseModel
   
   class QualityScore(BaseModel):
       correctness: int      # 1-5: is the answer actually right?
       completeness: int     # 1-5: does it fully address the question?
       tone_fit: int         # 1-5: matches target assistant style?
       reject_reason: str | None
   
   client = anthropic.Anthropic()
   for rec in candidates:                        # checkpointed loop
       response = client.messages.parse(
           model="claude-opus-4-8",
           max_tokens=1024,
           thinking={"type": "adaptive"},
           messages=[{"role": "user", "content":
               "Score this training example against the rubric. Be harsh: "
               "a wrong answer is correctness=1 regardless of style.\n\n"
               f"{rec.to_json()}"}],
           output_format=QualityScore,
       )
       save_score(rec.id, response.parsed_output)
   

   Calibrate before trusting: hand-score 50 records yourself, check agreement with the judge, and set the keep-threshold where your bar sits. Route borderline scores to a human-review queue rather than auto-dropping — the boundary cases define your dataset's character.

5. 5.

   Scrub PII — detect, transform, verify. Regex/NER for the structured kinds (emails, phones, PAN/Aadhaar-shaped IDs, card numbers), plus an LLM pass for free-text leakage (names in prose). Replace with consistent placeholders (<CUSTOMER_NAME>) rather than deleting, so conversations stay coherent. Then verify: run detection again on the scrubbed output and sample-audit 100 records by eye. Scrubbing you didn't verify is scrubbing you don't have.

6. 6.

   Validate format against the training target. Whatever the fine-tuning framework expects (chat-format JSONL, prompt/completion pairs), validate every line with a schema before shipping: parses as JSON, roles alternate correctly, no empty contents, token length within the trainer's limit. One malformed line at row 48,213 is a crashed training run at hour two.

7. 7.

   Split with leakage checks. Random split is not enough: near-dup pairs must land on the same side (use the step-2 similarity graph's connected components as split units), and group-level leakage matters — all conversations from one customer/ticket/document stay on one side. Verify by embedding-searching each eval record against the train set and inspecting anything above ~0.9 similarity.

8. 8.

   Emit the datasheet. Counts at every funnel stage (raw → exact-dedup → near-dedup → heuristics → judge → PII → final), per-source composition, score distributions, split sizes, and the audit results. This document is what makes the dataset defensible to the ML team and to compliance.

Verify before you trust it

• Funnel accountingstage counts sum exactly; no records silently vanish between stages.
• Judge calibration≥85% agreement with your 50 hand-scored records at the chosen threshold.
• PII re-scan of the final setzero detections, plus a clean 100-record human sample.
• Leakage probeno eval record has a >0.9-similarity neighbor in train; group IDs never span the split.
• Dry runthe trainer's own loader ingests the JSONL end-to-end without error before you hand it over.

Pitfalls

• Splitting before deduping. Near-dups across the split inflate eval scores and everyone celebrates a model that memorized its test.
• LLM-scoring the raw corpus. Judge calls on records heuristics would have killed is pure spend; order the funnel cheap-to-expensive.
• Deleting PII instead of placeholding. Holes in conversations teach the model to produce incoherent turns; consistent placeholders preserve structure.
• Uncalibrated judges. An unchecked rubric quietly optimizes for confident-sounding examples over correct ones — exactly the wrong training signal.
• No provenance. When the fine-tuned model says something weird in prod, source + id on every record is how you find and fix the offending slice.

Tools

• Python + embeddings (pgvector/FAISS)near-dup detection and leakage probes.
• Argillahuman review queues for borderline judge scores and PII audit sampling.
• Anthropic/OpenAI batch APIs + structured outputsthe judge-scoring loop with schema-enforced rubrics.

Client-safe variant

Training data is the highest-stakes data product on an engagement: it's a permanent copy of client content that will be baked into model weights. Get written scope for which sources may be used, run the entire pipeline inside the client's environment, and make the PII stage non-negotiable even if the model will be self-hosted — weights leak training data under extraction attacks. If external LLM judging isn't approved, substitute a self-hosted judge via vLLM and widen the human-review band; the funnel and datasheet stay identical. The datasheet itself is the compliance artifact — deliver it signed alongside the dataset, and keep raw-to-final lineage queryable in the client's warehouse so any example in the released set can be traced to its source on demand.