This module installs the habit that separates a RAG system from a RAG demo: a labeled retrieval test set and its metrics land in the same commit as the pipeline, so chunking and embedding are chosen by experiment on your own corpus rather than argued over a whiteboard.
It goes well — too well. Neha's support assistant, retrieving over the team's runbooks and wiki, answers the five questions she picked in the standup and the room nods.
Priya, who has seen demos before, types "what's the retry policy for failed ingest jobs?" The assistant returns a confident paragraph about job scheduling that never mentions retries.
"So how would you even know that's wrong at scale?" Priya asks. "You picked the five questions it could answer." Neha doesn't have a number to give her, and that absence is the whole problem.
She and Karthik do the unglamorous thing the tutorials skip: they mine fifty real questions from support threads and Slack searches — including messy keyword queries like "vacation policy india" that carry almost no semantic structure.
They hand-label the gold chunks for each, and wire recall@k and MRR into the same commit as the pipeline.
Priya asks "how do you know it works," and the answer is a scorecard — every chunking and embedding decision from that point scored against those fifty queries instead of argued over a whiteboard.
That reordering — measure before you optimize — is the one decision that separates this module from the tutorial ecosystem. You know what RAG is — the Common Core covered the anatomy. This module is about how experienced engineers build it differently, and the difference is a single decision made on day one: instrument before you optimize.
RAG appears in roughly 65% of applied-LLM job listings in this program's research, and Ragas is named verbatim in postings — but what employers are actually screening for is the discipline underneath the tool names: can you say, with numbers, whether your retriever finds the right evidence, and can you attribute a bad answer to the component that caused it? Most RAG systems are built in the opposite order — pipeline first, vibes-based tweaking second, metrics never — which is why most RAG systems plateau at "usually fine" and stay there.
You will build the baseline and its measurement harness in the same commit, choose your chunking strategy and embedding model by experiment on your own corpus rather than by blog post, and learn the decision framework for when retrieval is even the right architecture versus stuffing a long context window. Everything runs on a laptop with open models at zero API cost, with cheap-tier judge calls as the only optional spend.
A RAG system without a labeled retrieval test set is not a system — it is a demo. The test set is the first artifact you build, it costs an afternoon, and every decision in this module and the next is scored against it.
The pipeline itself is deliberately boring: parse → chunk → embed → index → retrieve → generate. Boring is the point — every clever addition (hybrid search, reranking, query rewriting) belongs in Module 4, after you can measure what it buys. What makes this baseline different from the tutorial version is that the eval harness is in the same first commit.
from sentence_transformers import SentenceTransformer
from qdrant_client import QdrantClient, models
from langchain_text_splitters import RecursiveCharacterTextSplitter
# 1. Chunk
splitter = RecursiveCharacterTextSplitter(
chunk_size=1800, chunk_overlap=250, # ~450 tokens, ~15% overlap
add_start_index=True,
)
chunks = []
for doc in corpus: # [{id, text, source}, ...]
for i, piece in enumerate(splitter.split_text(doc["text"])):
chunks.append({"chunk_id": f'{doc["id"]}#c{i}',
"doc_id": doc["id"], "text": piece})
# 2. Embed + index (open model, runs on CPU/laptop)
embedder = SentenceTransformer("BAAI/bge-small-en-v1.5")
vectors = embedder.encode([c["text"] for c in chunks], normalize_embeddings=True)
client = QdrantClient(path="./index") # local, no server needed
client.create_collection(
"kb", vectors_config=models.VectorParams(
size=vectors.shape[1], distance=models.Distance.COSINE))
client.upsert("kb", points=[
models.PointStruct(id=i, vector=v.tolist(), payload=c)
for i, (v, c) in enumerate(zip(vectors, chunks))])
# 3. Retrieve
def retrieve(query: str, k: int = 5):
qv = embedder.encode(query, normalize_embeddings=True)
hits = client.query_points("kb", query=qv.tolist(), limit=k).points
return [h.payload for h in hits]
Generation is a single LLM call over the retrieved chunks with a grounding instruction ("answer only from the provided context; say so if the context is insufficient"). Resist the urge to engineer the prompt yet — an unmeasured prompt tweak is a random walk.
This is the artifact Module 4's opening line will call "something rare," and it is the highest-ROI afternoon in the entire track. The format is trivial; the value is total:
{"query": "what is the retry policy for failed ingest jobs?",
"gold_chunk_ids": ["runbook-ingest#c3", "runbook-ingest#c4"],
"query_type": "procedural", "source": "real-user"}
Build ~50 queries to start, and build them honestly:
Twenty lines, no framework required — and these twenty lines are the referee for every experiment that follows:
def evaluate_retrieval(test_set, retrieve_fn, k=5):
recalls, rranks = [], []
for case in test_set:
hits = [h["chunk_id"] for h in retrieve_fn(case["query"], k=k)]
gold = set(case["gold_chunk_ids"])
recalls.append(len(gold & set(hits)) / len(gold))
rr = 0.0
for rank, cid in enumerate(hits, start=1):
if cid in gold:
rr = 1.0 / rank
break
rranks.append(rr)
return {"recall@k": sum(recalls)/len(recalls),
"mrr": sum(rranks)/len(rranks)}
Recall@k answers "is the evidence on the table at all?" — the ceiling on everything downstream, because no generator can ground an answer in a chunk that was never retrieved. MRR answers "how high does the evidence rank?" — which matters because chunks buried at position 7 behind six distractors get ignored by generators in practice. Track both, always at a fixed k, on every change.
Retrieval metrics see the retriever; Ragas sees the system. Wire it in now — not because the baseline needs a verdict, but because component-attributed metrics are how you will localize every future regression:
from ragas import EvaluationDataset, evaluate
from ragas.metrics import Faithfulness, LLMContextRecall
rows = []
for case in test_set:
ctx = retrieve(case["query"], k=5)
answer = generate(case["query"], ctx) # your generation call
rows.append({
"user_input": case["query"],
"retrieved_contexts": [c["text"] for c in ctx],
"response": answer,
"reference": case["reference_answer"], # add these to ~30 cases
})
result = evaluate(
dataset=EvaluationDataset.from_list(rows),
metrics=[Faithfulness(), LLMContextRecall()],
llm=judge_llm, # cheap-tier model as judge
)
Read each score against the component it implicates: context recall low → the retriever failed (or the chunking destroyed the evidence — next section); faithfulness low with context recall high → the evidence arrived and the generator ignored or contradicted it — a generation-side problem no retriever tuning will fix. That attribution logic is the seed of Module 4's full failure taxonomy and Module 5's evaluation engineering; here it needs to become a habit. Judge calls are the only API spend in this module — a cheap-tier judge over a 50-case set costs pennies per run, comfortably inside the track's lab budget.
Metrics come before tuning, in the literal git history. If your first commit has a pipeline and no eval harness, you have already built the thing this module exists to prevent.
Version the test set like code — it will grow (every interesting production failure becomes a case), and a metric is only comparable across experiments when you know which test-set version scored it.
Chunking is where retrieval quality is won or lost before a single vector is computed, because it decides what a "retrievable unit" even is. It is also the decision the tutorial ecosystem hands you as folklore. The 2026 practitioner consensus gives you a starting grid — recursive splitting around 400–512 tokens with 10–20% overlap is the default to beat, and very large chunks (multiple thousands of tokens) reliably degrade retrieval precision — but consensus numbers were measured on someone else's corpus. Yours has its own answer, and you now own the harness that finds it in an hour.
Hold everything else fixed — embedder, k, test set — and vary one axis at a time:
configs = [
("recursive", 300, 0.10), ("recursive", 450, 0.15),
("recursive", 800, 0.15), ("sentence", 450, 0.10),
("semantic", None, None), ("layout", None, None),
]
for name, size, overlap in configs:
chunks = chunk_corpus(corpus, strategy=name, size=size, overlap=overlap)
index = build_index(chunks, embedder)
m = evaluate_retrieval(test_set, make_retriever(index), k=5)
log_result(name, size, overlap, len(chunks), m)
A real-shaped result table from an internal-docs corpus (~1,200 documents, mixed runbooks and policy pages), with the reasoning a senior engineer extracts from it:
| Strategy | Size (tokens) | Overlap | Chunks | recall@5 | MRR | Reading |
|---|---|---|---|---|---|---|
| Recursive | 300 | 10% | 9,410 | 0.74 | 0.61 | Precise matches, but procedures split mid-sequence — steps 4–7 retrieved without steps 1–3 |
| Recursive | 450 | 15% | 6,220 | 0.81 | 0.68 | The consensus default earns its reputation here |
| Recursive | 800 | 15% | 3,480 | 0.77 | 0.60 | Recall holds; rank slips — big chunks blur into each other in embedding space |
| Sentence-aware | 450 | 10% | 6,540 | 0.82 | 0.69 | Statistically indistinguishable from recursive-450; not worth added complexity |
| Semantic (embedding-split) | ~var | — | 5,910 | 0.79 | 0.66 | Slower to build, no measured edge on this corpus |
| Layout-aware (headings) | ~var | — | 5,120 | 0.85 | 0.72 | Winner — this corpus has strong heading structure; sections are the natural unit |
Three lessons generalize even though the numbers will not:
Two engineering notes the table cannot show:
chunk_size × chunk_count as an index-cost column when strategies tiefewer, better chunks are cheaper to embed, store, and search.Keep the losing configs' full results. When Module 4 adds hybrid retrieval and reranking, re-running this exact grid tells you whether the reranker changed which chunking wins — it sometimes does, because rerankers absorb precision problems that small chunks were compensating for.
Never compare chunking configs by re-labeling gold chunks per config from scratch — subtle labeling drift will swamp the effect you are measuring. Label gold evidence spans in the source documents once, then map spans to whichever chunks contain them, mechanically, per config.
The MTEB leaderboard is the right place to start and the wrong place to stop: it ranks models on public benchmarks, and your corpus — with its acronyms, product codenames, and internal jargon — is not a public benchmark. The workflow is shortlist-then-verify.
| Model | Dims | recall@5 | MRR | Index size (6.2k chunks) | Embed cost | Notes |
|---|---|---|---|---|---|---|
| bge-small-en-v1.5 | 384 | 0.81 | 0.68 | 9.5 MB | $0 (local) | The baseline; embarrassingly hard to beat |
| bge-large-en-v1.5 | 1024 | 0.84 | 0.71 | 25 MB | $0 (local, slower) | +3 recall points for 2.7× storage and ~4× encode time |
| API embedding (large) | 3072 | 0.83 | 0.70 | 76 MB | ~$0.13/M tokens | No local infra; per-token cost forever, incl. every reindex |
| Multilingual-e5-large | 1024 | 0.80 | 0.66 | 25 MB | $0 (local) | Wins only if the corpus actually needs it |
How to read a table like this: the leaderboard ordering (API-large above bge-large above bge-small) partially survived contact with the corpus and partially did not — which is the norm, not the exception. The deltas are small enough that the decision is really about the other columns: dimension count is a permanent tax on storage and search latency; API embeddings put a meter on every future reindex (and Module 4's ingestion pipelines reindex often); local models keep the whole loop free and private, which matters for client-data compliance in services work. And a three-point recall gap is exactly the size of improvement that Module 4's hybrid retrieval buys for free on top of any embedder — so do not overpay for it here.
One more verification before committing: run the candidate on your corpus's hard slice — the acronym-heavy and jargon-heavy queries you tagged when building the test set. General-purpose embedders degrade most exactly there, and that degradation is invisible in the aggregate. (When domain vocabulary is the dominant failure mode, fine-tuning the embedder on mined triples is the eventual answer — that is Module 7 territory; hybrid lexical search in Module 4 is the cheaper first response.)
"Which embedding model should you use?" is never answerable from a leaderboard in this track's assessments. The creditable answer names the shortlist logic, the domain benchmark, and the non-quality columns — dimensions, cost, reindex economics — that break ties.
Every retrieval-quality intervention you will ever apply is one of four levers. Naming them now gives you the map for the rest of the track:
The levers interact — which is precisely why the harness exists. A reranker changes the optimal first-stage k; hybrid search changes how much embedding quality matters; better chunking can make a query-rewriting problem vanish. You will never tune them jointly by feel; you will re-run the grid and read the table. Module 4 picks up levers 3 and 4 on the first line, using this module's test set as its acceptance harness — and adds the component-level failure diagnosis that tells you, per failing query, which lever to reach for.
Frontier models now take a million tokens of context, which makes "just stuff the documents in" a legitimate architecture — sometimes. Retrieval-versus-stuffing is one of the four decisions practitioners keep re-litigating, so make it with a framework instead of a reflex:
| Dimension | Favors long-context stuffing | Favors RAG |
|---|---|---|
| Corpus size | Fits comfortably in the window (say, ≤200–500K tokens) | Orders of magnitude larger than any window |
| Query volume | Few queries, ad-hoc analysis | Many queries against the same corpus — per-query token cost dominates |
| Freshness | Static snapshot is fine | Corpus changes continuously; index updates beat re-stuffing |
| Access control | Everyone may see everything | Per-user document permissions (retrieval can filter; a stuffed prompt cannot) |
| Attribution | "Read it and answer" suffices | Answers must cite specific evidence chunks |
| Latency & cost | Prompt caching amortizes repeated stuffing of an identical prefix | Retrieval keeps per-query input tokens small and stable |
Do the cost arithmetic once and it stops being abstract. Stuffing a 300K-token corpus into a $3/M-input model costs about $0.90 per query uncached; at 1,000 queries a day that is ~$900/day, and even a 90%-discounted cache-read rate leaves ~$90/day if the prefix never changes — one document edit invalidates the cached prefix and reprices the day. The RAG pipeline you just built spends ~3K input tokens per query — about $0.009, two orders of magnitude less — plus an indexing cost that amortizes across all queries. Stuffing wins at the other extreme: a 60K-token contract analyzed once, with follow-up questions, is cheaper and better stuffed than chunked, because no retriever can lose evidence that is all present.
The honest answer for many products is both: long-context for small, high-stakes, read-everything tasks; RAG for the large living corpus; and — because you have a harness — the boundary between them located by measurement on your own workload, not by vendor blog post. Run the comparison in Exercise 5 and you will have your own number.
RAG is not the default and neither is long context. Corpus size, query volume, freshness, permissions, and attribution decide — and with a labeled test set, the decision is an experiment, not an argument.
Your first 15 minutes Monday: before you touch chunking or embeddings on any RAG system you own, open a file and paste in ten real questions from your support channel with the gold chunk id hand-labeled next to each. That ten-line file is the start of the referee — if a change can't be scored against it, you have no business shipping the change.
Check yourself: A teammate says their new RAG assistant "feels much better" after they switched to a bigger embedding model from the top of the MTEB leaderboard. What is the one question that decides whether to believe them, and why can't the leaderboard answer it?
This module's playbooks package the reusable assets: the measured-baseline repo template (pipeline + harness in one commit), the test-set construction guide with query-type coverage checklist and the synthetic-query curation rules, the chunking-experiment grid runner, the embedding-selection decision card with the MTEB-shortlist logic, and the RAG-vs-long-context decision matrix with the cost worksheet. Do this at work this week → pb-21 Measured RAG Baseline — and Whether You Need RAG at All (Neha's answer to "how do you know it works"), pb-22 Chunking-Strategy Benchmark: Choose by Eval, Not Vibes (the grid runner over your own corpus), and pb-23 Embedding-Model Selection: MTEB Shortlist to Domain Benchmark (shortlist-then-verify with the non-quality tie-breakers). The anchoring graded lab is L5: stand up a measured RAG baseline on a provided corpus — graded on test-set quality (provenance, coverage, gold-span labeling), harness correctness, and the written experimental reasoning in your chunking and embedding tables, not on leaderboard-style scores. L5's artifacts are load-bearing: Module 4's labs upgrade this exact pipeline to hybrid retrieval and reranking and use your labeled test set as the before/after referee, and Module 5 seeds its golden dataset from the failures you log here.
EvaluationDataset, and the component-attribution semantics used throughout this module.