After a fine-tuning run that looked healthy — training loss fell nicely, no errors — the tuned model performs WORSE than the untouched base model on the task it was trained for. Outputs repeat one phrasing regardless of input, contradict themselves across similar questions, or apply labels inconsistently. Sometimes nobody notices until users complain, because the tuned model was shipped on the strength of its loss curve.
A first fine-tune where the dataset was assembled quickly and never read: pairs copied from tickets or generated by another model, no deduplication, no format validation, no one checking that two similar inputs carry consistent outputs. Small style datasets (30–500 pairs) are the most fragile — each bad pair is a big fraction of the lesson. Linked modules: 3.10 (dataset validation, base-vs-tuned eval) and 3.7 (eval discipline).
Every training pair is an instruction: "when you see this, do exactly this." Duplicated pairs over-weight one pattern until the model parrots it; inconsistent pairs teach a coin-flip; mislabelled pairs teach the mistake itself. Training loss cannot reveal any of this — it happily falls while the model memorises garbage, which is why the run "looked healthy". The instrument that catches it is the base-vs-tuned eval from 3.10: run the same fixed cases against both models, plus a general slice. A tuned column below the base column is the defect made visible; per-case diffs then point at which pairs taught the wrong lesson. Rule out the sibling cause too: a chat template mismatch between training and serving produces the same "tuned got worse" signature.
Withdraw the tuned model; the base plus your best prompt is the better product today. Then repair the dataset, not the hyperparameters: run a validator that checks role order, output format, exact and near-duplicate inputs, and minimum input length; read every pair (at 30–500 pairs this is an hour, not a project); fix inconsistent outputs so similar inputs agree; re-split train/eval so no eval pair or near-copy sits in training. Retrain only after the validator reports zero problems, and ship only when the tuned column beats the base column without losing the general slice.
Validate the dataset before every run — the 3.10 rule "the dataset decides the outcome" is a gate, not advice. Record the base model's score on your eval set before training; that number is the floor any tune must beat. Never judge a run by training loss: judge it by the base-vs-tuned table on held-out cases. Keep the eval slice locked away from training, and re-run the general slice after every retrain to catch forgetting early.
Q: "Your fine-tuned model scores worse than the base model on your own eval. Walk me through your investigation." Strong answer: start at the dataset, not the GPU — name duplicates, inconsistency, and mislabelling as compiled bugs; mention the chat-template mismatch as the other classic cause; explain why training loss hid the problem; and close with the discipline that catches it, a base-vs-tuned eval on held-out cases run before shipping. Weak answer: "train for more epochs" — more epochs teach the garbage better.