The eval score climbed all month — 10/15, then 13/15, then a proud 15/15 — while production complaints about wrong and unhelpful answers stayed flat or rose. New questions from real users fail routinely, yet every run of the official eval set is perfect. The team celebrates the score; the users do not.
A small eval set — often the first ten or fifteen cases ever written — reused unchanged for weeks, with every prompt edit tuned directly against it. No case has been added since day one, and no case is held back from tuning. Common in the month after a team first adopts evals, when the score becomes the goal instead of the instrument. Linked module: 3.7 (Evaluating LLM systems).
The set rotted into a mirror. Iterating a prompt against the same fixed cases overfits to them: each edit bends the prompt toward those fifteen wordings, not toward the task. The score measures "how well the prompt fits these fifteen questions", which drifts further from "how well the bot serves users" with every tuning round. A perfect score is the tell, not the trophy — a healthy set sits below 100%, because headroom is information. Confirm the rot cheaply: collect ten fresh questions from real usage, run them, and compare the fresh pass rate to the official one. A wide gap is the disease in one number.
Split and refresh. Move a third of the cases into a holdout — a slice you run only when deciding whether to adopt a change, never while tuning. Then harvest production: every real failure users hit becomes a new case, added with its expected answer and, for RAG, its expected chunk. Version the set (cases-v2) and never compare scores across versions as if they were the same exam. Re-establish the baseline on the refreshed set and expect it to drop — that drop is the honest number returning.
Treat the eval set as a living asset: append real failures on a schedule, version every change, and keep the holdout untouched by tuning. Watch for the two smells — a score at or near 100%, and a set whose newest case is weeks old. When either appears, refresh before trusting another run. The demo trap and the eval trap are the same trap at different sizes: optimising against what you can see, then mistaking that for quality.
Q: "Your eval passes 15/15 but users keep reporting wrong answers. What is going on?" Strong answer: name overfitting to the eval set — tuning against fixed cases until the score stops measuring reality — then give the two-part exit: a held-out slice excluded from tuning, plus a standing pipeline that turns production failures into new versioned cases. Bonus for "a perfect score means the set has stopped teaching me anything."