The weekly eval report says 94 percent. It has said something near 94 percent for two months, and every prompt change is signed off against it. Meanwhile the HR helpdesk is filling with tickets: the assistant invents policies that do not exist, quotes circulars nobody follows any more, and answers questions about other departments with total confidence. The two stories cannot both be true, and yet nothing in the harness is broken — every case it runs, it runs correctly. The problem is what it never runs.
Any team whose gold set was written by the person who built the system, in one sitting, by looking at the documents and asking what they answer. That is the natural way to write one and it produces the same set every time: easy, in-scope, single-document questions phrased in the document's own words. Fires hardest after a corpus grows past one department, because the new failure modes — supersession, cross-department leakage, refusal — arrive with the growth and the gold set does not. Linked modules: P3 (Enterprise RAG assistant + eval harness), 3.7 (Evaluating LLM systems), 4.8 (Monitoring).
The score is honest and the set is not representative. Ninety-four percent of a set that contains only questions the corpus answers means the system is good at questions the corpus answers. It says nothing at all about the three behaviours users are complaining about, because no case in the set can produce those behaviours.
Break the missing coverage into named classes and the gap becomes obvious. Out-of-scope cases — questions the drive does not answer, where the only correct output is a refusal — were absent, so hallucination was untestable. Superseded cases — questions whose answer changed between versions — were absent, so the assistant's preference for a withdrawn circular was invisible. Cross-department cases — questions asked by a role that must not see the answer — were absent, so a confidentiality breach could not fail the run. Contradiction cases, where an old memo and the live policy both retrieve, were absent too. On the P3 reference corpus these four classes behave nothing like the easy ones: the contradiction class scored 0.31 on retrieval where the in-scope class scored 0.40, and the cross-department class scored 0.00 while every other number looked healthy.
There is a second, quieter failure underneath. Even where the set had cases, one blended score was reported for all of them, so a drop in retrieval and a drop in answer quality were indistinguishable. Confirm both in an afternoon: tag every existing case with a class, count the classes, and re-report last week's run per class with retrieval hit-rate and answer groundedness in separate columns. Empty rows are your missing coverage, and a class whose number is far from the headline is the reason your users disagree with your dashboard.
Immediate: stop quoting the aggregate. Re-report the last run per case class, with retrieval and groundedness split, and publish the classes that have zero cases as zero cases rather than as silence.
Structural: define the case classes before writing any more cases, and give each a minimum count that a release must satisfy. On the P3 build the required classes are in-scope, superseded, contradiction, out-of-scope, cross-department and injection, and the harness fails if any class falls below its floor — a coverage gate, checked before the score is even computed. Write the traps first, because they are the cases nobody writes voluntarily: questions with no answer in the corpus, questions whose answer changed, questions asked by the wrong role. Then split the meters permanently: hit-rate answers "did retrieval find it", groundedness answers "did the answer stay inside what we found", and a system can fail either one alone. Finally, make the run a gate rather than a report — no prompt edit, model swap or chunking change ships until the per-class table is compared to the previous run, case by case, because two runs can share a total and disagree on twenty cases.
Write the gold set from failure modes, not from documents. The question is never "what do these files say"; it is "what could go wrong here, and what question would catch it". Add a class the day a new failure is discovered in production, and add the specific user question that exposed it, because a bug that has happened once must never be able to happen silently again.
Two habits keep the set honest as it ages. Keep a holdout slice you never tune against, since a set you optimise against for two months slowly becomes a mirror of your own system. And keep the class counts on the front page of the report, above the score, so a reader sees the shape of the test before the number — a set with no refusal cases and no version cases should embarrass its owner in public, which is exactly the effect you want.
Q: "Your eval says 94 percent and your users say the system is bad. Who is right?" Strong answer: both, and the set is the reason. Say that a score is only as representative as the cases behind it, then name the classes that are almost always missing — out-of-scope where refusal is the only correct output, superseded where the answer changed between versions, cross-department where the asker must not see the answer. Explain what you would compute first: the same run re-reported per class, with retrieval hit-rate and answer groundedness in separate columns, and the empty classes shown as empty. Give the fix as a coverage gate with minimum counts per class plus a per-case regression comparison before any change ships. Close on the honest note that makes the answer memorable: the classes you leave out of a gold set are exactly the failures you are choosing not to see.