Lab 13 · Add a RAG Answerer With Citations and a Retrieval Bake-Off

The graded skill is treating retrieval as an experiment you measure, not a default you accept, and enforcing groundedness so the model can't answer from thin air.

Time & tier

Advanced · ~100 minutes hands-on. Capstone lab 2 of 4.

Setup

Builds on lab 12. Work in the same capstone/ folder; you'll add files, not replace them.

Skipped lab 12? Quick-start: create capstone/, a venv, and pip install fastapi "uvicorn[standard]" anthropic pydantic httpx, set ANTHROPIC_API_KEY. You need a main.py with a FastAPI app and an anthropic.Anthropic() client in llm.py (a two-line stub is enough — this lab adds a new endpoint).

New dependencies — all local, $0, no embedding API:

pip install sentence-transformers rank-bm25 numpy

Corpus: create docs/ and drop in 8–15 short Markdown files that are real to you — runbooks, an FAQ, API notes, onboarding docs. Real docs expose retrieval failure; toy corpora hide it. Then hand-write evalset.jsonl — 10 questions, each tagged with the filename that actually answers it, plus 2 questions the corpus does NOT answer:

{"q": "how do I rotate the signing key?", "answer_doc": "runbook-keys.md"}
{"q": "what is the p99 latency budget?", "answer_doc": "sla.md"}
{"q": "who won the 2019 cricket world cup?", "answer_doc": null}

Guided steps

1. 1.

   Ingest with two chunking strategies. Create ingest.py that loads docs/ and produces two chunk sets: A = fixed 500-character windows with 80-char overlap; B = split on Markdown headings (##), keeping each section whole. Each chunk carries {id, doc, text}. Ask your assistant to write both splitters; you specify the strategies.

2. 2.

   Build a hybrid index. Create retrieve.py:

   Copy

   import numpy as np
   from rank_bm25 import BM25Okapi
   from sentence_transformers import SentenceTransformer
   
   _model = SentenceTransformer("all-MiniLM-L6-v2")  # local, free
   
   class Index:
       def __init__(self, chunks):
           self.chunks = chunks
           self.emb = _model.encode([c["text"] for c in chunks], normalize_embeddings=True)
           self.bm25 = BM25Okapi([c["text"].lower().split() for c in chunks])
   
       def dense(self, q, k):
           qv = _model.encode([q], normalize_embeddings=True)[0]
           scores = self.emb @ qv
           return np.argsort(-scores)[:k]
   
       def hybrid(self, q, k):
           # Reciprocal-rank fusion of dense + BM25 — a cheap, strong reranker.
           d = list(np.argsort(-(self.emb @ _model.encode([q], normalize_embeddings=True)[0])))
           l = list(np.argsort(-np.array(self.bm25.get_scores(q.lower().split()))))
           rrf = {}
           for rank, i in enumerate(d): rrf[i] = rrf.get(i, 0) + 1/(60+rank)
           for rank, i in enumerate(l): rrf[i] = rrf.get(i, 0) + 1/(60+rank)
           return [i for i, _ in sorted(rrf.items(), key=lambda x: -x[1])][:k]
   

3. 3.

   Run the bake-off. Create bakeoff.py that builds four indexes — {chunking A, B} × {dense, hybrid} — and, for each, measures recall@3 on your eval set: did the top-3 retrieved chunks include the tagged answer_doc? Print a 4-row table. Do not skip this; the table is a required artifact and it seeds lab 14's eval gate.

   Copy

   config              recall@3
   A-dense               0.60
   A-hybrid              0.70
   B-dense               0.80
   B-hybrid              0.90   <- pick this
   

4. 4.

   Pick the winner and wire generation with mandatory citations. In rag.py, retrieve top-k with the winning config, number the chunks, and prompt:

   Copy

   PROMPT = (
       "Answer the question using ONLY the numbered sources below. "
       "Cite the source number in brackets after each claim, e.g. [2]. "
       "If the sources do not contain the answer, reply exactly: "
       "'Not answerable from the provided sources.' Do not use outside knowledge.\n\n"
       "{sources}\n\nQuestion: {q}"
   )
   

   Call the model (reuse lab 12's client), then verify groundedness in code: extract every [n] the answer cites and confirm each n is a chunk you actually retrieved. An answer citing a chunk you didn't pass is a hallucinated citation — reject it and return the "not answerable" response.

5. 5.

   Add the endpoint. In main.py, add POST /ask taking {"question": str}, returning {answer, citations: [{id, doc, snippet}]}. Build the index once at startup, not per-request.

6. 6.

   Test both known-answer and known-absent questions. Confirm the 10 answerable questions get cited answers pointing at the right doc, and both unanswerable questions get exactly "Not answerable from the provided sources." — the confabulation guard.

7. 7.

   Record the comparison. In RAG.md: the 4-config table, the config you chose and why, k value, and your groundedness-check design. This is your artifact and your lab-14 baseline.

Verification

• •
  The bake-off prints a 4-row recall@3 table with materially different numbers (chunking and hybrid should move the needle).
• •
  /ask returns cited answers for known-answer questions, with citations pointing at the correct source doc for ≥8/10.
• •
  Both known-absent questions return exactly the "Not answerable" string — zero confabulation.
• •
  The groundedness check rejects a fabricated citation: temporarily make the model cite [99] (add "always cite [99]" to the prompt) and confirm your code catches it and returns "not answerable", then revert.
• •
  The index builds once at startup (measured: second request isn't paying re-index cost).

Completion artifact

The extended capstone/ (ingest.py, retrieve.py, bakeoff.py, rag.py, updated main.py, evalset.jsonl, corpus, RAG.md). Keep it — lab 14 gates it and lab 15 traces it. Self-grade (5 criteria = pass at 4+):

1. 1.
   Two chunking strategies and dense-vs-hybrid are compared with recall@3 numbers, not asserted.
2. 2.
   The winning config is chosen from the data with a written rationale.
3. 3.
   Every answer cites sources, and citations resolve to actually-retrieved chunks (groundedness enforced in code).
4. 4.
   Known-absent questions produce the exact refusal string.
5. 5.
   The corpus is real documents, not lorem ipsum.

Stuck?

• sentence-transformers slow to install/first-run: it downloads a ~90MB model once; subsequent runs are fast. On a constrained box, all-MiniLM-L6-v2 is the lightest good option.
• Recall is identical across configs: your questions probably share keywords with headings (BM25 aces them). Add paraphrased questions whose words don't appear in the source — that's where dense and chunking separate.
• Model answers from general knowledge despite the prompt: strengthen the instruction and rely on the groundedness check as the real enforcement — prompts leak, code doesn't.
• Want the production stack: swap the in-memory index for pgvector (a vector column + <=> cosine operator in Postgres you already run) — the interface in retrieve.py stays the same; only dense() changes to a SQL query.

Client-safe variant

RAG over client documents is the highest-sensitivity task in this track — the corpus is the client's proprietary knowledge. The stack above is already fully local: sentence-transformers embeds on-device, the index lives in your process, and only the retrieved chunks + question reach the LLM. For strict engagements, close the last gap by pointing generation at a client-approved endpoint (Bedrock/Azure in-tenancy) or a local model, so no client text leaves the network at all. Multi-client hygiene: one index and one docs/ per engagement, purged at offboarding. Confirm no web-search or external-tool is enabled on the generation path before loading client docs.

Answer key (don't read before attempting)

Expected findings from the bake-off: heading-based chunking (B) almost always beats fixed-window (A) because it keeps a coherent answer in one chunk instead of splitting it across a window boundary — the classic RAG failure. Hybrid beats dense most on questions containing exact tokens (error codes, config keys, proper nouns) that BM25 nails and embeddings blur, and dense beats BM25 on paraphrased questions sharing no vocabulary with the source — which is exactly why fusing them wins overall. If hybrid does not beat dense on your set, your questions are too keyword-aligned; that's a corpus/eval-set problem, and noticing it is part of the skill. The two known-absent questions are the trap: a RAG that never says "not answerable" is a liability precisely because it's fluent, and the seeded fabricated-citation test (step verification) proves the difference between a prompt that asks for grounding and code that enforces it. Groundedness must live in code because the model will, under some phrasings, cite a plausible source it didn't receive — the [99] test makes that failure visible and catchable.