Distillation: Turn a Frontier Model into a Small, Cheap One

Tier: Advanced · Workflow area: Fine-tuning & Model Adaptation

Distillation uses the expensive model as a teacher — labeling data at scale — and trains a small student to match it: an 8B open model, or even a gradient-boosted classifier when the task is secretly classification. Production teams (the Netflix/Uber-style platform pattern) use this constantly to cut serving cost 10–100x while keeping most of the quality. This playbook builds the teacher→student pipeline and, critically, the eval parity report that says exactly what you gave up.

When to use

• •
  A frontier-model solution works but F4 (cost/latency) fails: high-volume classification, extraction, routing, scoring, or templated generation.
• •
  The task is narrow and stable — distillation freezes teacher behavior into the student; volatile tasks re-distill constantly.
• •
  You need on-prem/self-hosted inference for compliance and the frontier model cannot be deployed there.

Skip for low-volume tasks (the API bill is smaller than the engineering time) and for open-ended reasoning tasks where small students fall off a quality cliff — check the small-model baseline from pb-27 first.

Prerequisites

• •
  A passing teacher pipeline with its eval set and grader (never distill a pipeline that has not cleared evals — you would be mass-producing its mistakes).
• •
  5k–50k unlabeled real inputs from production logs for the teacher to label (volume depends on task complexity; classification needs less, generation more).
• •
  Teacher API budget (batch APIs typically halve cost) and the pb-29 training stack, or sklearn/XGBoost for the classical-student path.

The workflow

1. 1.

   Decide the student class by task shape. Fixed label set or score → LLM→GBM/logistic path: teacher emits labels, student is XGBoost/logistic over embeddings or engineered features; serving cost approaches zero. Structured or free text out → LLM→SLM path: teacher emits target outputs, student is a 1–8B model SFT-tuned on them (pb-28/29 mechanics apply unchanged).

2. 2.

   Design the labeling prompt for training data, not for chat. The teacher prompt should produce the exact output format the student will be trained to emit — schema-validated JSON, canonical label strings — plus a confidence field. Pin the teacher model version and temperature 0; log every call (prompt hash, model, timestamp) so the dataset is reproducible.

3. 3.

   Label with quality controls. Run the teacher over your 5k–50k inputs via the batch API. Then: (a) drop or re-queue low-confidence labels; (b) label 3–5% of inputs twice (paraphrased prompt or second teacher) and measure agreement — under ~90% agreement means the task definition is fuzzy, fix the prompt before scaling; (c) human-audit 100 random labels against the rubric. Decontaminate against the eval set exactly as in pb-28.

4. 4.

   Train the student. GBM path: embed inputs with a small open embedder (or the pb-31 tuned one), fit XGBoost with standard CV, calibrate probabilities (Platt/isotonic) — calibph matters because you will threshold on it. SLM path: SFT per pb-29 (LoRA on a 3–8B base, teacher outputs as gold), 1–2 epochs.

5. 5.

   Write the eval parity report. On the original human-grounded eval set (not teacher labels — the teacher is now an interested party), report teacher vs student vs untrained-small-model baseline: task metric, per-slice deltas, worst-100-case review, plus cost/1k requests and p95 latency for each. The deliverable is one table: quality gap vs cost gap.

6. 6.

   Route around the student's weaknesses. If parity fails only on identifiable slices, ship a router: student handles the easy majority, low-confidence or hard-slice inputs escalate to the teacher. Even a crude confidence threshold typically preserves ~95% of quality at a fraction of cost — report the blended cost/quality point too.

7. 7.

   Set the re-distillation trigger. Log student confidence and sampled teacher-agreement in production; when agreement drifts below a set floor, or the upstream task shifts, re-run the pipeline. Distillation is a build you re-run, not a one-off.

Tool variants

• Frontier API (batch mode) + HF small models: the LLM→SLM path; batch APIs cut labeling cost ~50%.
• Frontier API + sklearn/XGBoost: the LLM→GBM path for classification/scoring; deploys anywhere, including Lambda-class CPU.
• vLLM (self-host) for the student: serve the distilled 3–8B on one T4/A10 spot instance; pairs with the quantization trade-off study if you push further.

Verification

• •
  Teacher self-agreement ≥ ~90% on the double-labeled sample; human audit of 100 labels documented.
• •
  Parity evaluated on human-grounded data, with the untrained small model as a third column proving the distillation added value.
• •
  Cost and latency measured, not estimated: real per-1k-request numbers for teacher, student, and router configurations.
• •
  Worst-100 student cases reviewed and categorized; escalation rules cover the discovered weak slices.

Pitfalls

• Distilling a broken teacher. The most expensive failure: 50k labels of a prompt that never cleared evals. Gate on pb-27/pb-33 evidence first.
• Evaluating the student against teacher labels. That measures imitation, not quality; final numbers come from human-grounded evals.
• Ignoring the small-model baseline. If the untuned 8B already matches the teacher, you needed a model swap, not a distillation project.
• Distribution-shifted labeling inputs. Label real production inputs, not synthetic ones — the student learns the input distribution you feed the teacher.
• No drift trigger. A student silently ages as the task shifts; sampled teacher-agreement in production is your smoke alarm.

Restricted-environment variant

• •
  If client data cannot reach a frontier API, invert the topology: run the largest approved self-hosted model (e.g., 70B on client infra, or via a contracted private endpoint) as the teacher and distill into the 3–8B student — same pipeline, smaller quality ceiling, so widen the human audit to 300 labels.
• •
  Anonymize per pb-28 if partial API use is permitted for label-only calls; confirm the provider's no-training-on-inputs terms in writing.
• •
  The labeled dataset and the student model both derive from client data — classify and store both accordingly.

Success criteria

• •
  A student model serving the task at ≥ 10x lower cost per request with a quantified, accepted quality gap (or a router hitting the agreed blended point).
• •
  Parity report on human-grounded evals with per-slice honesty, reviewed by the task owner.
• •
  Reproducible labeling pipeline: pinned teacher version, logged prompts, quality-control counts.
• •
  A production drift trigger with an owner and a threshold, written down.