Chunking-Strategy Benchmark: Choose by Eval, Not Vibes

Tier: Core · Workflow area: RAG & Retrieval Engineering

When to use

• •
  Immediately after the pb-21 baseline, as the first upgrade candidate — chunking changes are cheap to test and often worth more than model upgrades.
• •
  Your corpus has structure the default splitter ignores: headings, tables, code blocks, legal numbering, transcripts.
• •
  Diagnosed low context_recall/context_precision in pb-21 — both frequently trace to chunk boundaries slicing answers in half.

Prerequisites

• •
  The pb-21 baseline: grounding set (questions + gold passages), metrics harness, frozen scorecard.
• •
  langchain-text-splitters (or LlamaIndex node parsers), chonkie or a semantic splitter, Docling for layout-aware splitting of PDFs.
• •
  Enough embedding throughput to re-embed the corpus ~10–15 times (local bge on a T4 makes this trivial; hosted embeddings make it a real cost — budget it).

The workflow

1. 1.

   Define the matrix. Strategies × parameters, pruned to what your corpus can use:

   Strategy	Sizes (tokens)	Overlap	Notes
   Recursive (char/sep-aware)	256 / 450 / 800	10%, 20%	The default to beat; ~400–512 + 10–20% overlap is the strong 2026 prior
   Sentence-window	3–5 sentences, window ±2	—	Embed small, retrieve with neighbors attached
   Semantic (embedding-drift boundary)	breakpoint percentile 90/95	—	Splits where topic shifts; costs an embedding pass to chunk
   Layout-aware (headings/tables)	section-bounded, max 800	—	Requires Docling/unstructured structure from pb-18

   Keep the total under ~12 configurations; beyond that you are fishing.

2. 2.

   Hold everything else constant. Same embedding model, same top-k, same generator, same prompts as the baseline. One variable class per experiment — this is the same discipline as pb-20, applied to retrieval.

3. 3.

   Re-map gold passages per configuration. The grounding set references document spans; a chunk is "gold" if it overlaps a gold span by ≥50% of the span. Automate this mapping — hand-relabeling per config is how these benchmarks die.

   Copy

   def is_gold(chunk, span):  # both carry (doc_id, start_char, end_char)
       if chunk.doc_id != span.doc_id: return False
       inter = min(chunk.end, span.end) - max(chunk.start, span.start)
       return inter / (span.end - span.start) >= 0.5
   

4. 4.

   Score the matrix. For each config: recall@5, MRR, context_precision/context_recall + faithfulness via RAGAS, plus two often-decisive operational columns — index size (chunk count) and mean retrieved-context tokens per query (your per-query cost driver).

5. 5.

   Check the context cliff. Add one deliberately oversized config (~2,500-token chunks). Published 2026 benchmarks and internal results consistently show retrieval quality falling off a cliff around this size — reproduce it on your corpus so the team stops proposing "just use bigger chunks."

6. 6.

   Read results by document type, then decide. Slice metrics by corpus segment (prose vs tables vs code). It is common — and fine — to ship two chunkers routed by document type. Promote the winner only if it beats the baseline scorecard outside the bootstrap CI; ties go to the simpler strategy.

Tool variants

• LangChain:RecursiveCharacterTextSplitter.from_tiktoken_encoder(...), SemanticChunker; LlamaIndex:SentenceWindowNodeParser, SemanticSplitterNodeParser, HierarchicalNodeParser (pairs with pb-25's parent-document retrieval).
• Chonkie: fast, minimal library covering token/sentence/semantic/late chunking behind one interface — convenient for matrix runs.
• Layout-aware: Docling's document model → section-bounded chunks with table integrity preserved (never split a table row from its header).
• Harness: the pb-21 metrics code unchanged; RAGAS for the answer-side metrics, judge pinned.

Verification

• •
  All configs evaluated on the identical question set with automated gold-mapping; no config's gold set was hand-adjusted.
• •
  The winning config beats the baseline on recall@5 and does not regress faithfulness or cost/query beyond agreed bounds.
• •
  The context-cliff config confirms degradation (or, if it doesn't on your corpus, that finding is documented — it's rare and worth knowing).
• •
  Decision recorded with the full matrix table; the losing configs' scores are kept so nobody re-proposes them next quarter.

Pitfalls

• Scoring only end-to-end answers. Chunking effects show up first and largest in retrieval metrics; answer-level scores add noise from the generator and hide 2–3-point recall differences.
• Overlap as a free lunch. Overlap inflates index size and retrieval of near-duplicate chunks (hurting context_precision); pair any overlap increase with a dedup-at-retrieval check.
• Splitting tables and code mid-structure. A half-table chunk is worse than no chunk; route structured content to the layout-aware splitter.
• Comparing configs with different embedding models. That's pb-23's experiment; mixing the two variables makes both unanswerable.
• Chasing the last point of recall. Once recall@5 is in the low 90s, the marginal chunking win is smaller than what pb-24 (hybrid + rerank) delivers — know when to stop.

Restricted-environment variant

• •
  The entire benchmark is local by construction if you use local embeddings and a local judge — chunking experiments are the safest RAG work to run on client corpora.
• •
  Hosted-embedding shops: run the matrix on a 10–20% stratified corpus sample first, then re-embed only the top-2 configs on the full corpus.
• •
  Keep chunk artifacts (per-config chunk tables with spans) in the engagement environment; they contain the client corpus verbatim.

Success criteria

• •
  A completed matrix with ≥8 configurations scored on retrieval + answer + cost metrics, reproducible from one script.
• •
  A shipped chunking decision that beat the baseline outside confidence intervals — or a documented finding that the default was already optimal.
• •
  The context-cliff demonstration saved for the next "bigger chunks" debate.
• •
  Gold-mapping automation that makes the next chunking experiment a one-hour job.