Lab 12 · Retrieval Quality Engineering

Evaluate the lab-11 bot against a 40-question golden set, classify every failure as retrieval vs generation vs content, fix each class with its own tool — chunking, synonyms, KB rewrites — and re-measure to prove which fix moved which failures.

Farhan's bot from lab-11 answers ten questions respectably. His MSP wants to put it in front of client users, and "respectably on ten questions" is not an evidence base. This lab is the discipline that separates a demo from a system: a golden set of questions real users would ask — including the vocabulary they actually use, which is not the vocabulary the KB is written in — a failure taxonomy that tells you which layer broke, and targeted fixes measured by re-running the same set. This is also the skill that transfers furthest: when a vendor claims "our bot is 92% accurate," the golden-set-and-taxonomy method is how you check whose questions, measured how.

The golden set below is seeded: it contains vocabulary-mismatch questions engineered to defeat naive retrieval, multi-article questions that need stitched answers, out-of-scope questions that must be refused, and one question whose KB answer is itself wrong — because the KB you inherited in lab-11 still contains the stale split-tunnel article, and a perfect bot over wrong content gives perfect wrong answers.

Time & tier

~75 minutes hands-on · Advanced tier (maps to module M6). $0 — the lab-11 stack as-is (Ollama, Chroma, your ingest.py/ask.py), plus a spreadsheet or markdown table for scoring. Complete lab-11 first; this lab tunes that bot.

Setup

Save the golden set as golden-set.txt. Format: id | question | expected. Expected values: A:kb0X (answer grounded in that article), A:kb0X+kb0Y (multi-article), R (must refuse). Questions marked with the same theme deliberately vary vocabulary.

G01 | How do I reset my password? | A:kb01
G02 | Forgot my login, locked out | A:kb01
G03 | The sign-in box keeps rejecting me | A:kb01
G04 | New phone, need my authenticator back | A:kb02
G05 | Where do I re-enroll MFA? | A:kb02
G06 | The approve notification never arrives | A:kb02
G07 | What is VPN error 809? | A:kb03
G08 | VPN won't connect from my hotel | A:kb03
G09 | Remote access broken on public wifi | A:kb03
G10 | Should I turn off my firewall for the VPN? | A:kb03
G11 | How do I get to the finance share? | A:kb04
G12 | Need permissions on the team folder | A:kb04
G13 | How long till drive access is granted? | A:kb04
G14 | Why is my internet slow on VPN? | A:kb05
G15 | Are we allowed to split tunnel? | A:kb05
G16 | My Teams calls stutter on VPN, what do I do? | A:kb05
G17 | Locked out AND lost my phone | A:kb01+kb02
G18 | New starter needs drive access and VPN | A:kb03+kb04
G19 | Hotel wifi: VPN fails, then calls choppy | A:kb03+kb05
G20 | Password reset but MFA still on old phone | A:kb01+kb02
G21 | How do I request a laptop? | R
G22 | Office wifi password? | R
G23 | Can I get local admin? | R
G24 | How do I book a meeting room? | R
G25 | What's the guest printing process? | R
G26 | My email keeps asking me to sign in | A:kb01
G27 | Authenticator gone after factory reset | A:kb02
G28 | Error 809 persists after trying the KB steps | A:kb03
G29 | Who approves shared drive requests? | A:kb04
G30 | Is all my traffic monitored on VPN? | A:kb05
G31 | Can the desk unlock me before 30 minutes? | A:kb01
G32 | Temporary access pass - how long valid? | A:kb02
G33 | Which ports does the VPN need? | A:kb03
G34 | Can I ask for NTFS permissions directly? | A:kb04
G35 | Video quality bad on VPN - enable split tunnel? | A:kb05
G36 | I typed my password right three times, still locked | A:kb01
G37 | Does drive access need my manager? | A:kb04
G38 | Set up MFA without any old device | A:kb02
G39 | What changed in March on the network? | A:kb05
G40 | Expense report policy? | R

Scoring: each question earns pass (correct grounded answer with citation, or correct refusal) or a failure class: RET (wrong/no chunks retrieved), GEN (right chunks retrieved, wrong answer generated), CON (bot faithfully reported KB content that is itself wrong/stale). To assign RET vs GEN you must look at the retrieved chunks, not just the answer — add a debug print of retrieved sources to ask.py if you have not already.

Guided steps

Run the baseline. All 40 questions through the unmodified lab-11 bot. For each, record in eval-sheet.md: answer summary, sources retrieved, pass/RET/GEN/CON. Budget half your lab time here; the discipline is not changing anything while measuring. Expected output: a baseline pass rate typically in the 55–75% band — if you scored 95%+, check whether you graded citations and refusals strictly (G35 in particular; see step 4).
Classify before fixing. Tally failures by class. The taxonomy decides the tool: RET failures are fixed by chunking, synonyms and retrieval breadth; GEN failures by prompt and model settings; CON failures only by fixing the KB. Write the tally and your fix plan — which fix, aimed at which question ids — before touching anything. This pre-commitment is what makes your re-measure honest.
Diagnose the vocabulary-mismatch cluster. G03, G06, G09, G12, G26, G36 phrase problems in user language distant from KB vocabulary. Check their retrievals: which pulled the wrong article or nothing relevant? The fix is not embedding magic — it is lab-05's lesson mechanized: add a synonym/user-phrasing block to each KB article ("users say: sign-in box, locked out, keeps rejecting..."), re-ingest, re-run just this cluster. Expected output: most of the cluster flips to pass, and you can name which article's synonym block did it.
Face the content failure. G35 asks exactly what stale kb06 recommends ("enable split tunnel?"). If your bot answered "yes, enable it [kb06]" — retrieval worked, generation worked, and the answer is still wrong: CON, the class no bot-side fix touches. The fix is retiring/correcting kb06 (mark it superseded by kb05, or delete it) and re-ingesting. Also re-check G14/G16/G19/G30 afterward — the conflict was polluting their retrievals too. Log the before/after for the whole affected group.
Tune the multi-article questions. G17–G20 need two articles each. Check retrieval breadth: with top-3 and small chunks, one topic often crowds out the other. Experiment with exactly one variable at a time — top-k to 4–5, or chunk size — and re-run only G17–G20 after each change, watching the single-article questions for regressions. Expected output: at least three of four multi-article questions passing without breaking previously passing questions; record the trade-off you chose and what it cost.
Verify the refusal wall held. Re-run G21–G25 and G40 after all changes. Tuning retrieval breadth can weaken refusals — with a larger top-k, weakly-related chunks reach the prompt and the model gets adventurous. Expected output: six exact refusals. Any regression here outweighs a multi-article gain; if it happened, rebalance and say so.
Run the full re-measure. All 40 again, same strict grading, new column in eval-sheet.md. Expected output: measurable improvement concentrated exactly where your fix plan predicted — and possibly a question or two that got worse, which you report rather than quietly re-tune, because the eval sheet is evidence, not marketing.
Write the failure-taxonomy report.taxonomy-report.md, one page: baseline vs final pass rate; failures by class before/after; a table of which fix moved which question ids; the one fix with the best effort-to-impact ratio; and the two-sentence answer to "how would you evaluate a vendor's accuracy claim?" that this lab has earned you.

Warning

The unforgivable move in evaluation is fixing and measuring simultaneously — tweaking the prompt, re-running the three questions you remember failing, and declaring improvement. Baseline first, classify, pre-commit the fix plan, change one variable, re-run the full set. Anything less is tuning to the test while grading yourself.

Verify

eval-sheet.md contains all 40 baseline results with retrieved sources logged, then a full re-measure column.
Every failure carries a class (RET/GEN/CON) assigned from retrieval evidence, not answer vibes.
The fix plan was written before fixes, and the report's fix-to-question mapping matches it.
G35's content failure was caught as CON and fixed in the KB, with the polluted-question group re-checked.
The vocabulary cluster improved via synonym blocks, with the responsible article named per question.
All six refusals hold after tuning; any regression is documented with the rebalance.

Self-grading rubric (pass = 7/10): honest baseline with strict grading (0–2) · taxonomy assigned from retrieval evidence (0–2) · CON failure caught and fixed in the KB, not the bot (0–2) · vocabulary cluster fixed and attributed (0–2) · refusal wall intact post-tuning (0–1) · report maps fixes to question ids (0–1).

▸Answer key — seeded failure classes and expected findingsoptional deep dive — click to expand

Seeded RET failures (vocabulary mismatch): G03, G06, G09, G12, G26, G36 — user phrasings ("sign-in box keeps rejecting me," "approve notification never arrives," "remote access broken") sit far from KB vocabulary in embedding space. Typical baseline: 3–5 of these fail with wrong or empty retrievals. G26 is the sharpest: "email keeps asking me to sign in" tends to retrieve nothing better than kb01, and kb01 is the least-wrong target here — grade it pass if grounded in kb01 without invention. Fix: synonym/user-phrasing blocks per article (kb01 and kb02 blocks do most of the work), which is KB authoring, not model work.

Seeded multi-article questions: G17–G20. Baseline top-3 usually passes one or two. G19 is the hardest (kb03 + kb05 span two problems in one story). Expect top-k 4–5 to lift these; watch for the side effect on refusals (step 6).

The seeded CON failure: G35 — stale kb06 answers it "enable split tunneling," and a well-grounded bot cites it faithfully. This is the lab's centerpiece: retrieval correct, generation correct, answer wrong, and the only fix is editorial. G14/G16/G30/G39 are the pollution-check group; before the KB fix, kb06 chunks intrude on their retrievals. After retiring kb06, all should ground cleanly in kb05. If in lab-11 you already retired kb06 (some learners do after the contradiction step), G35 grades as pass at baseline — note it and credit yourself; the check is whether you knew why.

Seeded refusals: G21–G25, G40. G25 ("guest printing") and G40 ("expense policy") are the leak-prone ones — plausible office topics the model can improvise. Six exact refusals is the bar, before and after tuning.

GEN failures are the least seeded (this KB is small and the questions grounded), but they appear when: the model summarizes kb03's port detail wrong on G33 (UDP 500/4500 — check the number survived generation), or blends unlock timing on G31 (30 minutes, desk can act sooner). Numeric fidelity is where small local models slip; grade strictly.

Scoring: baseline complete with sources = 2 · G35 classed CON and fixed editorially = 2 · ≥4 of the vocabulary cluster flipped with attribution = 2 · ≥3 multi-article passing without regressions = 1 · refusals 6/6 post-tuning = 1 · re-measure full-set and honest about regressions = 1 · vendor-claim answer references whose-questions-measured-how = 1. 8+/10 means you can evaluate anyone's bot — including the one a vendor is selling you.

Key Takeaway:

Pass rate is a number; the taxonomy is the insight — retrieval failures are fixed in chunking and synonyms, generation failures in prompts, content failures only in the KB itself — and the evaluator who knows which layer broke is the person every vendor accuracy claim has to get past.

© Anvi Innovate · BXUP — Be Exponentially Upskilled · BXUP is an independent learning platform, not affiliated with or endorsed by AWS, Microsoft, Google, NVIDIA, IBM, Salesforce, Databricks, Oracle or Anthropic. All trademarks belong to their respective owners. · bxup.in