You inherit a broken RAG app and 60 production traces. Component attribution from traces — deciding whether a failure belongs to retrieval, to grounding, or to generation — is the debugging skill that separates RAG engineers from RAG tutorial-followers.
Your job: attribute every failing trace to retrieval-miss (evidence never retrieved), grounding-miss (evidence retrieved, answer ignores or contradicts it), or generation-miss (evidence retrieved and used, answer mangled — wrong unit, truncated, or an unnecessary refusal), then fix the top two root causes and prove metric recovery.
~2.5 hours hands-on · Advanced tier · $0 API spend (deterministic retriever; optional Ragas confirmation ≤ $0.20).
Run in the Lab 05 directory (corpus/ and qa.jsonl required). Save the broken app exactly as written — every line is potentially evidence:
"""rag_app.py — the RAG service under investigation. DO NOT fix anything yet."""
import pathlib
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
CHUNK_SIZE = 250 # characters
TOP_K = 1
PROMPT = (
"Answer the user's question using the context below. If the context does "
"not contain the answer, answer from your general knowledge instead.\n"
"Context: {context}\nQuestion: {question}"
)
def build_index(corpus_dir="corpus"):
files = sorted(pathlib.Path(corpus_dir).glob("*.md"))[:9] # cap for speed
chunks, src = [], []
for f in files:
text = f.read_text()
for i in range(0, len(text), CHUNK_SIZE):
chunks.append(text[i:i + CHUNK_SIZE])
src.append(f.name)
vec = TfidfVectorizer(stop_words="english").fit(chunks)
return vec, vec.transform(chunks), chunks, src
def retrieve(query, index, k=TOP_K):
vec, mat, chunks, src = index
query = query[:50] # trim noisy long queries
sims = (vec.transform([query]) @ mat.T).toarray()[0]
order = np.argsort(-sims)[:k]
return [{"source": src[i], "text": chunks[i]} for i in order]
Then generate the traces. The generator runs the buggy retrieval for real (retrieval failures genuinely emerge from the bugs) and synthesizes answer behavior deterministically, writing a hidden label key:
"""make_traces.py — run the buggy app over 60 questions; emit traces + hidden key.
Deterministic (seed 21). DO NOT read answer_key.json."""
import json
import random
import rag_app
rng = random.Random(21)
qa = [json.loads(l) for l in open("qa.jsonl")]
PREFIX = ["", "Under company policy, ", "Quick question - "]
questions = []
for i, row in enumerate(qa):
n = 3 if i % 2 == 0 else 2 # 24 facts -> 60 questions
for p in range(n):
q = row["question"] if p == 0 else PREFIX[p] + row["question"][0].lower() + row["question"][1:]
questions.append({**row, "question": q})
index = rag_app.build_index()
traces, key = [], {}
for tid, row in enumerate(questions):
retrieved = rag_app.retrieve(row["question"], index)
evidence = any(row["ground_truth"] in c["text"] and c["source"] == row["gold_doc"]
for c in retrieved)
truth_num = row["ground_truth"]
if not evidence:
label = "retrieval-miss"
fake = "".join(str((int(ch) + 3) % 10) if ch.isdigit() else ch for ch in truth_num)
answer = f"According to standard policy, the answer is {fake}."
else:
r = rng.random()
if r < 0.60:
label = "correct"
answer = f"Per {row['gold_doc']}, the answer is {truth_num}."
elif r < 0.85:
label = "grounding-miss"
fake = "".join(str((int(ch) + 5) % 10) if ch.isdigit() else ch for ch in truth_num)
answer = f"The policy states {fake}."
else:
label = "generation-miss"
answer = ("I could not find this in the provided context."
if rng.random() < 0.5 else
f"The relevant figure is {truth_num.split()[-1]} (units unclear); "
"see the full policy for details.")
traces.append({"trace_id": tid, "question": row["question"],
"retrieved": retrieved, "answer": answer,
"gold_doc": row["gold_doc"], "ground_truth": truth_num})
key[tid] = {"label": label}
with open("traces.jsonl", "w") as f:
f.writelines(json.dumps(t) + "\n" for t in traces)
with open("answer_key.json", "w") as f:
json.dump(key, f, indent=1)
print("wrote 60 traces")
"""eval_retrieval.py — evidence recall of rag_app over the traced questions."""
import json
import rag_app
index = rag_app.build_index()
rows = [json.loads(l) for l in open("traces.jsonl")]
hits = sum(any(r["ground_truth"] in c["text"] and c["source"] == r["gold_doc"]
for c in rag_app.retrieve(r["question"], index)) for r in rows)
print(f"evidence recall: {hits}/{len(rows)} = {hits/len(rows):.2f}")
Expected before any fix: 35/60 = 0.58.
2. Triage all 60 traces into labels.csv (trace_id,label) using this protocol, applied in order per trace: (a) is the ground truth present in any retrieved chunk from the gold doc? No → retrieval-miss. (b) Yes: does the answer's value contradict the retrieved evidence? → grounding-miss. (c) Evidence used but the answer is a refusal, unit-less fragment, or otherwise unusable? → generation-miss. (d) Otherwise correct. Do the first 15 by hand to internalize the protocol; you may then script the mechanical part — but hand-verify 10 scripted labels.
3. Hunt root causes in the code. For the retrieval-miss pile, group by gold_doc and by question prefix. Two patterns should jump out: certain documents never appear in any retrieved set (why?), and paraphrased questions with long prefixes fail where the plain form succeeds (why?). Read rag_app.py line by line against those hypotheses. List every defect you find, ranked by traces affected.
4. Fix the top two causes only (per the count from step 3), rerun eval_retrieval.py, and record the delta. Then fix the remaining retrieval defects (chunking without overlap, TOP_K=1) and rerun again — target ≥0.95. Keep each fix a separate commit with the metric in the commit message.
5. Fix the generation side on paper. The permissive prompt line ("answer from your general knowledge instead") is the app's grounding bug — rewrite PROMPT to require context-only answers with an explicit "say you don't know" escape and a "quote the number with its unit" instruction. If you have API budget, verify with Ragas faithfulness on 10 traces; otherwise state the expected effect in the report.
6. Score your triage against answer_key.json — only now. Report agreement and analyze every disagreement (some refusal-with-evidence cases are legitimately arguable between grounding-miss and generation-miss; your protocol note is what makes the call defensible).
Fix retrieval before judging generation — a perfect prompt cannot answer from evidence that was never retrieved.
failure-clinic-report.md: the taxonomy table with counts, labels.csv, ranked root-cause list mapped to code lines, the fix diff (or PR link), and the metric-recovery trail. Rubric (pass = 8/10):
[:9] but recall barely moved. You fixed cause #1 but not #2 — diff the failing questions' lengths against the survivors.This is exactly the drill for inheriting a client's underperforming RAG system: export 50–100 traces (redact PII from questions first), run the three-bucket triage, size root causes by count, and fix in evidence order. The deliverable format here — taxonomy, ranked causes, per-fix metric recovery — is presentable to a client steering committee without modification, and the deterministic TF-IDF probe pattern means the triage itself needs no LLM access to client data.
Seeded defects in rag_app.py, ranked by impact: (1) [:9] in build_index silently drops the last three documents alphabetically — training.md, travel.md, wfh.md — causing 15 retrieval-misses; (2) query[:50] truncation destroys the paraphrased questions with prefixes, causing 10 more; (3) TOP_K=1 leaves zero headroom; (4) CHUNK_SIZE=250 with no overlap can split fact sentences; (5) the PROMPT explicitly licenses answering from general knowledge — the grounding defect behind the 6 grounding-miss traces — and lacks format instructions, behind the 5 generation-miss traces (2–3 of which are refusals-despite-evidence, legitimately arguable as grounding). Trace labels: 25 retrieval-miss, 6 grounding-miss, 5 generation-miss, 24 correct. Evidence recall: 35/60 buggy; 60/60 with full index, no truncation, k=4, 700–1000-char chunks.