Engineer Retrieval Quality with a Golden Set

Tier: Advanced · Workflow area: Assistants & RAG · ~45 minutes

This playbook builds the evaluation discipline for any KB-grounded assistant — the pb-38 build, a vendor virtual agent, Copilot over SharePoint: construct a golden Q&A set from real tickets and searches, run it through the system, diagnose failures by category, and iterate chunking, article rewrites and synonym coverage with before/after numbers. (UC-41)

When to use

• Before exposing any RAG assistant to usersthe golden-set score is the go/no-go evidence, and the baseline every later change is judged against.
• After every meaningful changenew embedding model, re-chunk, big KB update, vendor "improvement" release: re-run the set, compare, then believe.
• When users say the bot is dumbthe golden set converts anecdotes into a failure taxonomy you can actually fix.

When NOT to use

• With invented questionsquestions you write from the articles score flatteringly high, because they reuse the articles' own vocabulary. Real users ask in their words ("wifi broken again", not "troubleshoot 802.1X authentication"); mine tickets and search logs or the set measures nothing.
• As a one-time gatean evaluation run once is a snapshot; the value is the trend across changes.
• Chasing the last few points to 100%a golden set that saturates has stopped discriminating; some questions genuinely need humans. Set a target (say 85% good), route the rest to escalation, and add harder questions as the system improves.

The recipe

1. 1.

   Mine 50 real questions. From ticket subjects/descriptions, portal search logs, and (if live) bot transcripts, pull questions in users' actual phrasing. Cover the volume distribution — mostly top intents, a tail of rare-but-answerable, plus 5–8 questions the KB deliberately cannot answer (the refusal set). Deduplicate near-identical phrasings with AI's help, keeping the most colloquial variant of each.

2. 2.

   Attach verified ground truth to each row. For every question: the correct answer (verified by a human against current reality — not just against the article, which may be wrong), and the source article(s) that should be retrieved. Golden-set columns: id, question, expected_answer, expected_sources, category, notes. The refusal set's expected answer is the refusal. This verification pass is the expensive step and the whole point — a golden set with unverified answers just measures agreement with your KB's errors.

3. 3.

   Run the set and capture everything. Script it against your system (trivial for the pb-38 build; for vendor bots, run manually or via API): for each question record the retrieved chunks/sources, the generated answer, and the citations. Fifty questions is a lunch break.

4. 4.

   Grade with an LLM judge, spot-checked by you.

   Copy

   For each row: QUESTION, EXPECTED_ANSWER, EXPECTED_SOURCES,
   RETRIEVED_SOURCES, GENERATED_ANSWER.
   Grade:
   - retrieval_hit: did RETRIEVED_SOURCES include EXPECTED_SOURCES? (Y/N)
   - answer_quality: CORRECT / PARTIAL / WRONG / REFUSED, judged
     against EXPECTED_ANSWER only — not your own knowledge
   - faithful: is every claim in the answer supported by the retrieved
     chunks? (Y/N)
   Output one row per question plus a summary table of counts.
   

   Manually re-grade 10 of the 50 each run; if you disagree with the judge more than once or twice, tighten the grading prompt before trusting the totals. (Ragas-class frameworks automate these metrics — faithfulness, answer relevancy, context precision/recall — worth adopting once the manual loop has taught you what the numbers mean.)

5. 5.

   Classify every failure with the two-question test. For each non-CORRECT row ask: Did retrieval bring back the right chunks? and Is the correct answer actually in the KB?

   • •
     Retrieval failure — right answer exists, wrong chunks returned. Fix levers: chunking size/overlap, synonym coverage, embedding model, top-k, hybrid (keyword+vector) retrieval.
   • •
     Generation failure — right chunks returned, wrong/unfaithful answer produced. Levers: grounding prompt, model choice, temperature, chunk formatting in context.
   • •
     Content failure — the KB doesn't contain the answer, or contains it wrongly. Levers: write or rewrite the article. No retrieval tuning fixes a missing article.

   Typical first-run distribution is heavily retrieval and content — which is why prompt-fiddling alone rarely moves the score.

6. 6.

   Fix the biggest cluster with its matching lever, one change at a time. The classic retrieval-failure fix: vocabulary mismatch. Users say "wifi keeps dropping", the article says "intermittent 802.1X disconnection". Two remedies, both cheap: add a user-phrasing summary block to the top of the article ("Users describe this as: wifi dropping, wifi broken, laptop keeps disconnecting"), or enable hybrid retrieval so literal keywords get a vote. For chunking-shaped failures (answers split across chunk boundaries), adjust size/overlap or add heading-aware splitting. Change one lever, note it in a changelog.

7. 7.

   Re-run, compare, keep or revert. Same set, new scores, side by side. Retrieval hit rate up, faithfulness flat, refusals intact? Keep. Anything regressed — including the refusal set, which quietly breaks when you widen retrieval — revert and try the next lever. Each accepted change gets one changelog line: date, change, score before/after. That log is your evidence when someone asks whether the bot is actually getting better, and your map of which levers move your system.

Make it yours

• pb-38 local buildfull loop as written; you own every lever. This is the ideal place to learn the loop before applying it to systems you control less.
• Vendor virtual agent (JSM, Freshservice, Copilot Studio)you cannot re-chunk, but the loop still works: your levers are article rewrites, user-phrasing blocks, synonym coverage, grounding-scope changes and intent phrasings. Run the golden set through the real chat interface monthly; vendors change models under you, and your set is how you notice.
• Ragas-class automationonce the manual loop is habitual, script it: golden set in a repo, evaluation in CI, run on every KB re-index. Quality regressions become failed builds instead of user complaints.

Quality check

• Provenance is realevery golden question traces to a ticket, search log or transcript; every expected answer was human-verified against reality within the last two quarters.
• The refusal set runs every time. Widened retrieval and loosened prompts break refusal silently; it regresses more often than anything else.
• One lever per iteration. Two simultaneous changes with a better score teach you nothing about which to keep.
• Scores are published with the changelog. A quality number nobody sees is a number nobody defends in the renewal meeting or the rollout decision.