Choose a Chunking Strategy with a Measured Eval

Tier: Core · Workflow area: RAG data pipelines

This playbook sets up a small eval harness and lets recall@10 on a golden set — not folklore — pick between fixed, structure-aware, semantic, and parent-child chunking.

When to reach for this

• •
  You are indexing a new corpus and need to pick chunk size and strategy deliberately.
• •
  Retrieval misses answers you know are in the corpus, and parsed text quality is already verified (see pb-27).
• •
  Your corpus is heterogeneous — Markdown docs, code, tables — and one splitter clearly cannot fit all.
• •
  A stakeholder asks "why 512 tokens?" and the honest answer is currently "a tutorial said so."

The workflow

1. 1.

   Build a retrieval golden set first. 30–50 real questions, each mapped to the source document (and ideally passage) containing the answer. Pull them from support tickets, search logs, or SME interviews. This artifact outlives every other choice in the pipeline — it also drives pb-29, pb-32, pb-33, and pb-34.

   Copy

   {"q": "What is the penalty for late delivery under the MSA?",
    "doc_id": "contracts/msa-2025.pdf", "answer_span": "1.5% per week, capped at 10%"}
   

2. 2.

   Define 3–4 candidate strategies. A sane bracket: recursive fixed-size (400 tokens, 15% overlap), structure-aware (split on Markdown headings, then recursively), and parent-child (embed 128–256-token children for precision, return 512–1,024-token parents for context). Add semantic chunking only if the first three plateau — it costs an embedding pass at index time.

   Copy

   from langchain_text_splitters import RecursiveCharacterTextSplitter, MarkdownHeaderTextSplitter
   
   fixed = RecursiveCharacterTextSplitter(chunk_size=1600, chunk_overlap=240)  # ~400 tokens
   structural = MarkdownHeaderTextSplitter(headers_to_split_on=[("#", "h1"), ("##", "h2")])
   

3. 3.

   Index each strategy into its own local collection. Chroma keeps this a laptop exercise; the same loop targets pgvector, Qdrant, or Pinecone in production:

   Copy

   import chromadb
   from sentence_transformers import SentenceTransformer
   
   model = SentenceTransformer("BAAI/bge-small-en-v1.5")  # local; hosted equivalent: any provider embedding API
   client = chromadb.PersistentClient("eval_db")
   
   for name, chunks in strategies.items():
       col = client.get_or_create_collection(name)
       col.add(ids=[c["id"] for c in chunks],
               embeddings=model.encode([c["text"] for c in chunks], normalize_embeddings=True).tolist(),
               metadatas=[{"doc_id": c["doc_id"]} for c in chunks])
   

4. 4.

   Score recall@k per strategy. The metric that proves this step worked: recall@10 against the golden set — the share of questions whose gold document appears in the top 10 retrieved chunks. Track recall@5 alongside it, since that is closer to what the LLM actually sees.

   Copy

   def recall_at_k(col, golden, k=10):
       hits = 0
       for item in golden:
           res = col.query(query_embeddings=model.encode([item["q"]], normalize_embeddings=True).tolist(), n_results=k)
           hits += any(m["doc_id"] == item["doc_id"] for m in res["metadatas"][0])
       return hits / len(golden)
   

   Expected output: a table like fixed-400: 0.72 · structural: 0.80 · parent-child: 0.84. A 5–9 point spread between strategies is typical and is exactly the signal you are here to capture.

5. 5.

   Stress the winner on its weak spots. Slice recall by document type (tables, code, long narrative). Parent-child often wins overall but structural may win on heading-rich manuals; a per-corpus-slice routing decision ("Markdown → structural, everything else → parent-child") is legitimate and common.

6. 6.

   Freeze and document the decision. Record strategy, parameters, recall table, and eval date next to the pipeline code. Re-run the eval whenever the corpus composition shifts materially — chunking decisions rot as corpora evolve.

Verify before you trust it

• •
  Recall@10 computed for every candidate on the same golden set, same embedding model, same k — one variable at a time.
• •
  The winning margin exceeds noise: with a 40-question set, a 2-point gap is one question; treat gaps under ~5 points as a tie and prefer the simpler strategy.
• •
  Manually inspect 5 retrieved-but-wrong results — misses caused by parsing artifacts are a pb-27 problem, not a chunking problem.
• •
  Chunk-count and token-histogram sanity check: no strategy silently produced 5-token fragments or 4,000-token monsters.

Pitfalls

• Optimizing chunk size by generation quality alone. End-to-end answer quality confounds retrieval, prompting, and the model. Measure retrieval recall in isolation first.
• Overlap as a crutch. 50% overlap doubles index size and mostly masks a bad split boundary. If overlap above ~20% is what wins, the base strategy is wrong.
• Splitting tables and code mid-structure. A half-table chunk is worse than useless — it retrieves confidently and answers wrongly. Route structured content to structure-aware splitters.
• Golden set that only contains easy questions. If every question is keyword-matchable, every strategy scores 0.9 and the eval discriminates nothing. Include paraphrased and cross-section questions.
• Changing embedding model and chunking simultaneously. You will not know which change moved the number. One variable per experiment.

Tools

• LangChain / LlamaIndex splittersrecursive, Markdown/code-aware, and hierarchical node parsers; OSS and the fastest way to bracket strategies.
• chonkielightweight OSS chunking library with semantic and late-chunking implementations when the standard bracket plateaus.
• Chroma / pgvectorlocal eval store now, production store later; the eval loop transfers unchanged.

Client-safe variant

The eval harness contains client questions and passages, so treat the golden set as client-confidential data and keep the whole loop local: sentence-transformers embeddings and a Chroma or Postgres store run entirely inside the client environment with no external calls. Report only aggregate metrics (recall tables, chunk statistics) in deliverables — never raw golden-set questions, which frequently quote sensitive documents verbatim. If the client's policy allows hosted embeddings, run the same harness with the approved provider's API and present both scorecards so the client can weigh the recall delta against the data-exposure decision explicitly.