Settle the chunking question the way employers now expect it settled — with a measured evaluation, not folklore.
You will implement four chunking strategies (fixed-window, recursive, structure-aware, and hierarchical parent-child) over a generated platform handbook salted with deliberately confusable distractor sections, then score all four against the same golden retrieval set with recall@1, recall@3, and MRR using local embeddings. The strategies genuinely diverge on this corpus — fixed-window loses at recall@1 while recursive and parent-child tie at the top — and your deliverable is a decision memo that picks a winner on evidence and states the trade-offs (chunk count, context size per query) that the headline metric hides. Chunking choice alone swings retrieval quality by several points on the same corpus; "we measured on our corpus" is the sentence that separates a RAG engineer from a tutorial-follower.
~120 minutes hands-on · Advanced tier (maps to module M5 · RAG Pipelines I — Ingestion, Chunking, Embeddings).
Fully local, $0. Embeddings run on your CPU via sentence-transformers; the first run downloads the small all-MiniLM-L6-v2 model (~90 MB) once, then everything is offline.
mkdir chunk-lab && cd chunk-lab
python3 -m venv .venv && source .venv/bin/activate
pip install sentence-transformers
The corpus extends Lab 06's world: a data-platform handbook in markdown with 24 headed sections — 12 carrying gold facts your questions target, and 12 distractors that share vocabulary with the gold sections (multiple retention policies, multiple refresh cadences) so retrieval has to discriminate, not just keyword-match. Save make_handbook.py:
"""Generate a structured platform handbook (markdown) + a golden retrieval set."""
import json, os, random
random.seed(31)
OUT = os.path.dirname(os.path.abspath(__file__))
SECTIONS = [
("Ingestion SLAs", "Batch loads must land by 06:00 IST. The nightly ingest job retries three times with exponential backoff before paging the on-call engineer.", "What happens when the nightly ingest job fails?", "retries three times"),
("Warehouse cost policy", "Any query scanning more than 500 GB per run requires a review ticket. Dashboards must read from aggregate tables, never raw event data.", "When does a query need a cost review ticket?", "500 GB"),
("PII handling", "Email addresses and phone numbers are hashed with SHA-256 before landing in the analytics schema. Raw values live only in the vault schema with row-level security.", "How are emails protected in analytics tables?", "SHA-256"),
("Vector index refresh", "Embedding indexes are rebuilt incrementally every 15 minutes using content hashes; a full rebuild happens on the first Sunday of each month.", "How often are embedding indexes rebuilt incrementally?", "15 minutes"),
("Incident severity", "A SEV-1 is declared when data loss is suspected or a regulatory report would be late. SEV-1 requires an incident commander within 10 minutes.", "How fast must an incident commander join a SEV-1?", "10 minutes"),
("Schema changes", "Column drops require a 30-day deprecation window announced in the #data-changes channel, with usage checked against query logs.", "How long is the deprecation window for dropping a column?", "30-day"),
("Backfill safety", "Backfills above 10 million rows run in a sandboxed warehouse with a hard credit cap of 20 credits, then swap via table rename.", "Where do large backfills run?", "sandboxed warehouse"),
("dbt conventions", "Staging models are views named stg_<source>__<entity>; marts are tables. Every model needs a unique key test before it can be promoted.", "What must every dbt model have before promotion?", "unique key test"),
("Access requests", "Analyst access to the finance mart is granted through the broker service and expires after 90 days, requiring re-attestation by the data owner.", "How long does finance mart access last?", "90 days"),
("Streaming lag", "Kafka consumer lag above 100k messages for 5 minutes triggers auto-scaling of the consumer group, capped at 12 instances.", "What triggers consumer auto-scaling?", "100k messages"),
("Data contracts", "Producers publish schemas to the registry; breaking changes are blocked by CI unless a new major version is declared with a migration note.", "What blocks a breaking schema change?", "CI"),
("Disaster recovery", "The warehouse is replicated cross-region with a 4-hour RPO. Failover drills run every quarter and are graded against a 2-hour RTO.", "What is the recovery point objective for the warehouse?", "4-hour"),
]
DISTRACTORS = [
("Log retention", "Application logs are retained for 30 days in hot storage and 180 days in cold storage before deletion."),
("Audit log retention", "Audit logs for privileged actions are kept for 400 days to satisfy the security certification."),
("Checkpoint retention", "Pipeline checkpoints are retained for 14 days and pruned by the nightly janitor job."),
("Metric retention", "Aggregated metrics are kept for 5 years; downsampled to hourly after 90 days."),
("Dashboard refresh", "Executive dashboards refresh every 30 minutes from the aggregate layer."),
("Cache refresh", "The feature-store online cache refreshes every 5 minutes from the offline store."),
("Model retraining", "Forecasting models retrain weekly on Sunday nights, gated by an accuracy check."),
("Report deadlines", "Regulatory reports are due by the 5th business day of each month."),
("Access reviews", "Warehouse role memberships are reviewed twice a year by team leads."),
("On-call rotation", "The data on-call rotation is weekly, with handoff at 10:00 IST on Mondays."),
("Freshness alerts", "Freshness monitors alert when a gold table is more than 2 hours behind its schedule."),
("Sandbox cleanup", "Personal sandbox schemas are dropped automatically after 21 days of inactivity."),
]
FILLER = ("This section also documents ownership, escalation paths, and links to the runbook. "
"Owners review this page every quarter and update examples as tooling evolves. "
"Historical context and rejected alternatives are archived in the decision log. ")
lines, gold = ["# Data Platform Handbook\n"], []
for i, (title, body, question, anchor) in enumerate(SECTIONS, 1):
lines.append(f"## {i}. {title}\n")
lines.append(FILLER * random.randint(2, 4))
lines.append("\n" + body + "\n")
lines.append(FILLER * random.randint(2, 4) + "\n")
gold.append({"question": question, "anchor": anchor, "section": title})
for i, (title, body) in enumerate(DISTRACTORS, len(SECTIONS) + 1):
lines.append(f"\n## {i}. {title}\n\n" + (FILLER * 2) + "\n" + body + "\n")
open(f"{OUT}/handbook.md", "w").write("\n".join(lines))
json.dump(gold, open(f"{OUT}/golden_set.json", "w"), indent=2)
print(f"handbook: {len(open(f'{OUT}/handbook.md').read())} chars, {len(SECTIONS)+len(DISTRACTORS)} sections, {len(gold)} golden questions")
Run it — expected: handbook: ~25165 chars, 24 sections, 12 golden questions. Skim golden_set.json: each entry is a question, an anchor string that appears in exactly one section, and the section name. The anchor is how scoring checks whether retrieved context could actually answer the question.
Implement the four chunkers. Save bakeoff.py. Each chunker is a few honest lines — the lab implements them raw so you understand exactly what LangChain's or LlamaIndex's splitters (or chonkie) do under their configuration surface:
"""Four chunking strategies, one retrieval eval. Metrics: recall@1, recall@3, MRR over the golden set."""
import json, os, re
OUT = os.path.dirname(os.path.abspath(__file__))
TEXT = open(f"{OUT}/handbook.md").read()
GOLD = json.load(open(f"{OUT}/golden_set.json"))
def chunk_fixed(text, size=400, overlap=50):
return [{"text": text[i:i+size]} for i in range(0, len(text), size - overlap)]
def chunk_recursive(text, size=400):
"""Split on paragraphs first, then sentences, packing up to `size` chars."""
chunks, buf = [], ""
for para in re.split(r"\n\n+", text):
units = [para] if len(para) <= size else re.split(r"(?<=[.!?])\s+", para)
for u in units:
if len(buf) + len(u) > size and buf:
chunks.append({"text": buf.strip()}); buf = ""
buf += u + " "
if buf.strip(): chunks.append({"text": buf.strip()})
return chunks
def chunk_structure(text):
"""One chunk per markdown section, heading kept with its body."""
parts = re.split(r"(?m)^(## .+)$", text)
chunks, i = [], 1
while i < len(parts) - 1:
chunks.append({"text": (parts[i] + "\n" + parts[i+1]).strip()}); i += 2
return chunks
def chunk_parent_child(text, child_size=200):
"""Children = small windows inside each section; retrieval scores children, answer context = parent section."""
chunks = []
for parent in chunk_structure(text):
p = parent["text"]
for j in range(0, len(p), child_size):
chunks.append({"text": p[j:j+child_size], "parent": p})
return chunks
def evaluate(strategies, k=3):
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
print(f"{'strategy':16s} {'chunks':>6s} {'recall@1':>9s} {'recall@3':>9s} {'MRR':>6s}")
for name, chunks in strategies.items():
texts = [c["text"] for c in chunks]
emb = model.encode(texts, normalize_embeddings=True)
q_emb = model.encode([g["question"] for g in GOLD], normalize_embeddings=True)
r1 = r3 = 0; rr = 0.0
for qi, g in enumerate(GOLD):
scores = (emb @ q_emb[qi]).tolist()
top = sorted(range(len(texts)), key=lambda i: -scores[i])[:k]
contexts = [chunks[i].get("parent", chunks[i]["text"]) for i in top]
ranks = [r for r, ctx in enumerate(contexts, 1) if g["anchor"].lower() in ctx.lower()]
if ranks:
if ranks[0] == 1: r1 += 1
r3 += 1; rr += 1.0 / ranks[0]
n = len(GOLD)
print(f"{name:16s} {len(chunks):>6d} {r1/n:>9.2f} {r3/n:>9.2f} {rr/n:>6.2f}")
if __name__ == "__main__":
evaluate({
"fixed-400": chunk_fixed(TEXT),
"recursive-400": chunk_recursive(TEXT),
"structure-aware": chunk_structure(TEXT),
"parent-child": chunk_parent_child(TEXT),
})
Before running, predict the winner and write your prediction down — comparing your intuition to the measurement is half the value.
Run the bake-off. python bakeoff.py. Reference results from this corpus (yours should match, since generation is seeded):
strategy chunks recall@1 recall@3 MRR
fixed-400 72 0.83 1.00 0.92
recursive-400 69 1.00 1.00 1.00
structure-aware 24 0.92 1.00 0.96
parent-child 135 1.00 1.00 1.00
Read the table like an engineer, not a leaderboard. Four observations to work through, in your own words, in the memo. First: everyone hits 1.00 at recall@3 — on a 24-section corpus, five candidates deep, retrieval is easy; the discriminating metric is recall@1, which is what matters when you send only 1–3 chunks to a token-budgeted LLM. Second: fixed-400 loses (0.83) for a mechanical reason — a hard character window slices mid-sentence and mid-fact, so an anchor like "retries three times" can straddle two chunks, diluting both embeddings; overlap softens but does not fix this. Third: structure-aware at 0.92 with only 24 chunks — section-sized chunks average ~1,000 chars, so each embedding mixes the gold fact with filler prose, occasionally letting a lexically-similar distractor section outrank the right one at position 1. Fourth: parent-child ties recursive at 1.00 by decoupling match precision from context completeness — the 200-char child is a sharp probe, and the returned parent guarantees the LLM sees the whole section.
Surface what the headline metric hides. Extend the eval loop (or compute by hand) the average context characters delivered per query: recursive returns ~3 × 400 ≈ 1,200 chars; parent-child returns up to 3 parents ≈ 3,000 chars. Same recall, ~2.5× the prompt tokens per query, forever, on every query your platform serves. Also note index size: 135 child vectors vs 69 — storage and index-build cost scale with chunk count. Add both columns to your memo's table.
Stress your winner once. Whichever strategy you lean toward, probe its failure mode: for recursive, re-run with size=150 (fragments) and size=1200 (section-sized blobs) and watch recall@1 move — the "right" answer is parameter-sensitive; for parent-child, ask what happens when a "section" is 40 pages (parents blow the context window — production systems cap parent size or add an intermediate level). One paragraph of findings.
Write the decision memo. chunking-decision.md, one page, four parts: the results table (with your added context-size and chunk-count columns); the decision — for this corpus, recursive-400 as primary (top recall at the smallest context footprint) with parent-child as the documented alternative when answer synthesis needs whole-section context (policies, runbooks); the reasoning per rejected strategy, each tied to a number in the table; and the re-evaluation triggers — re-run this bake-off when the corpus changes character (tables, code, transcripts), when the embedding model changes, or when downstream moves from extractive Q&A to summarization. A memo with numbers, decisions, and triggers is the artifact interviewers ask about; India AI-data-engineer postings explicitly reward documented retrieval evaluations.
Optional — upgrade the harness to RAGAS. Your anchor-substring check is a transparent, dependency-free proxy for RAGAS's context recall/precision. If you have a Claude API key, pip install ragas and re-score one strategy with RAGAS's context metrics using an LLM judge to grade whether retrieved context supports the answer — the judged version handles paraphrase (where substring matching under-credits) and is the version employers name. Same harness shape, sturdier grader; the CI pattern is to gate merges on context-recall regression against the golden set.
Chunking is an empirical decision with a three-line eval harness — recall@k and MRR against a golden set on your corpus. The bake-off's real lesson is the trade-off recall hides: parent-child buys precision-with-context at 2.5x the prompt tokens, and only a measured table lets you buy it knowingly.
make_handbook.py reports 24 sections and 12 golden questions; the handbook contains at least four sections mentioning retention (the confusable distractors).bakeoff.py runs offline after the first model download and prints all four strategies with chunk counts 72 / 69 / 24 / 135 (±2 acceptable if you altered sizes).chunking-decision.md contains the extended table (context chars per query, chunk count), a named winner, per-loser reasoning tied to numbers, and at least three re-evaluation triggers.The chunk-lab/ folder: generator, bakeoff.py, results, and chunking-decision.md. Self-grading rubric (pass = 8/10):
HF_HUB_OFFLINE=0 and retry once on a network connection; after the first download everything is cached locally.chunks[i]["text"] for parent-child too — the .get("parent", ...) fallback matters; also confirm distractors made it into the handbook (24 sections, not 12).normalize_embeddings=True on both encode calls — unnormalized dot products break the ranking.model.encode(texts, batch_size=16)).This lab is client-safe by default: sentence-transformers runs entirely on your machine, so client documents never leave the boundary even during evaluation — which is precisely why local embedding models are the standard first choice for regulated-client RAG evaluation work. If the optional RAGAS step is used on client material, run the LLM judge through the client's approved tenancy (Amazon Bedrock or Azure OpenAI) or a local model via Ollama, and build the golden set from redacted or synthetic stand-in documents that mirror the real corpus's structure. The decision memo pattern is unchanged — and doubly valuable on engagements, because "we chose chunking strategy X on measured recall against your corpus" is a sentence a client architect can audit.