Most fine-tunes "work" in the demo and die in production, and the cause is almost always evaluation malpractice: eval examples that leaked into training, comparisons against a strawman baseline, and nobody checking what the model forgot.
Tier: Advanced · Workflow area: Fine-tuning & Model Adaptation
Indian fine-tuning-specialist postings now name "benchmarking model performance" explicitly — this playbook is that skill: a three-part honest evaluation (contamination audit, matched base-vs-tuned harness, general-capability regression suite) that produces a ship/no-ship verdict you can sign.
This playbook is not optional after fine-tuning — treat "we fine-tuned it" without this evidence as an unreviewed PR.
lm-eval-harness (or a custom equivalent) for general-capability checks; Kaggle/Colab GPU is enough for 3–8B inference.Audit contamination independently — do not trust the data card. Re-run the check yourself: embed all training instructions and all eval inputs, flag cosine similarity > 0.85; plus n-gram overlap (13-gram collision is a standard tripwire) between training pair texts and eval texts. Any hit gets the eval item replaced or the training pair documented-and-removed for a retrain. Report the contamination rate even if zero — "0 of 1,014 pairs flagged at 0.85" is evidence; silence is not.
Build the matched harness. One script, two models (base and tuned), and everything else identical: same chat template, same system prompt, same decoding (temperature 0 for graded tasks), same max tokens, same grader version. The number-one source of fake fine-tune gains is an unmatched comparison — a tuned model with a tuned-friendly prompt against a base model with a lazy prompt.
Give the base model its best shot. Run the base model with few-shot examples and an optimized prompt as an additional arm. The honest question is not "tuned vs naive base" but "tuned vs the best you can get without training" — pb-27's rung 1. If prompt-engineered base ≈ tuned, the adapter is not paying for its maintenance.
Score with slices and uncertainty. Report per-slice metrics (sub-task, difficulty, input length) and bootstrap confidence intervals over eval items. A +6-point average hiding a −15-point slice on the highest-volume sub-task is a no-ship; averages are where regressions hide.
Run the general-capability regression suite. Fine-tuning on 1k narrow pairs can degrade instruction-following, refusal behavior, and out-of-domain competence (catastrophic forgetting still bites at LoRA scale, just less). Run base vs tuned on: (a) 2–3 lm-eval-harness tasks near your domain (e.g., an MMLU subset, GSM8K if numeric); (b) 20 hand-picked out-of-domain prompts scored by your judge; (c) 10 safety/refusal probes relevant to your deployment. Flag any drop > 2–3 points for investigation.
Check behavioral regressions specific to your stack. Schema-validity rate under the production parser, latency and tokens/response (tuned models often get verbose or terse), and — if the tune combines with RAG — the full Ragas run (pb-35) on the combined pipeline, since a tuned generator can increase hallucination by answering from tuned "knowledge" instead of retrieved context.
Write the verdict memo. One page: contamination audit result; the three-arm table (base, base+best-prompt, tuned) with CIs and slices; regression suite deltas; ship / ship-with-conditions / retrain decision; and the monitoring hooks that carry these metrics into production (pb-37's CI gates pick them up from here).
The comparison that matters is tuned-model vs prompt-engineered base on decontaminated data with matched everything — any easier comparison is marketing.
An honest "no-ship, retrain with fixed data" verdict is a successful evaluation — the failure mode is shipping on flattering numbers, not failing to ship.