This module installs the ordering that makes a retrieval pipeline defensible: build the golden set before the experiments, then let one identical eval harness settle the parser, the chunking strategy, and the embedding model with a measured number instead of folklore.
Retrieval quality is decided long before anyone queries a vector database — it is decided in the parsing, chunking, and embedding pipeline that a data engineer builds and owns. This module turns "make our documents searchable" from a demo into an engineered system where every design choice is backed by a measured number on your corpus.
Sprint one, the naive build. Meena is an eight-year data engineer at a services company's Bengaluru GCC.
Her client hands her the mandate every DE is getting this year: build the retrieval layer for a support copilot over the client's knowledge base — 12,400 documents: product-manual PDFs full of spec tables, HTML runbooks, contract scans, and a decade of policy memos.
She does what every first-timer does: pypdf to extract raw text, split every 500 tokens, embed with a default hosted model, load a vector store.
The demo looks magical for four minutes — then the client's SME asks for the torque limit on the CX-340 mount, and the copilot answers with a value from a different product's table.
She digs in and finds the crime scene in her own pipeline: the PDF's spec table was extracted as an interleaved word soup, the 500-token splitter had sliced it mid-row and glued the tail to the next section's prose, and the fragment embedded near everything and meant nothing.
She has no way to say how bad it is overall — there is no eval, so every fix would be folklore.
Sprint two, the engineered build. Meena restarts, this time like a data engineer.
She builds a 60-question golden set with the client's SMEs — real questions, each mapped to the document and section that answers it — because nothing else in the pipeline can be judged without it.
Parser bake-off on a stratified 50-document sample. Docling preserves the CX-340 table as an actual table (94% cell fidelity on her hand-checked sample) where raw text extraction had produced soup, and an OCR fallback catches the 7% of the corpus that turns out to be scanned images that pypdf had silently returned as empty strings.
Chunking bake-off. Fixed 500-token chunks score recall@10 of 0.61 on the golden set; structure-aware chunking that respects headings and keeps tables whole hits 0.74; parent-child (256-token children for precision, 1,024-token parents returned for context) hits 0.79.
Embedding bake-off on her corpus, not a leaderboard. The hosted frontier model and open-source BGE-M3 land within two recall points of each other, but a third of the client's tickets arrive in Hindi-English mix, and BGE-M3's multilingual recall wins there by nine points, which settles it.
The batch pipeline — rate-limit-aware batching, retries with a dead-letter queue, idempotent re-runs, and a cost log showing the full corpus embedded for $6.40 on the batch tier.
The decision memo she sends the client has a number attached to every choice. That memo — not the demo — is what gets her team the phase-two contract.
The concrete failure first: run a two-column PDF with an embedded table through naive text extraction and you get columns interleaved line-by-line and table cells flattened into prose — Mount | CX-340 | 12 Nm becomes Mount CX-210 8 CX-340 12 CX-510 22 fused with the paragraph below it. No downstream stage can repair this. Chunking cannot un-shuffle words, and embeddings faithfully encode the garbage; if the parser destroys structure, retrieval quality is capped before you've made a single "AI" decision.
Layout-aware parsers are the fix, and you should treat them as swappable components behind an interface, not a framework commitment. Docling (fully OSS — the free-path default, which matters for India-based learners and client-billable work alike) produces structured output with headings, tables, and reading order; Unstructured.io partitions mixed formats into typed elements; LlamaParse is the managed API alternative with strong table handling. Behind whichever you choose sits the same pipeline contract:
Then measure parsers like you'd measure anything: a stratified sample (digital PDFs, scanned pages, HTML, the gnarliest spec tables), a hand-checked scorecard per parser — text accuracy, table-cell fidelity, heading/structure preservation, throughput, cost per 1K pages — and a decision recorded with numbers. That scorecard is a portfolio artifact and it is LAB-06's deliverable.
Architecturally, treat parsed output as a staged zone, exactly like raw-vs-staging in a warehouse: persist the parser's structured output (with the parser name and version stamped on it) separately from the chunks derived from it. Parsing is the slowest, most expensive stage to repeat — when you later change chunking strategy in LAB-07, you re-chunk from the parsed zone in minutes instead of re-parsing 12,400 PDFs, and when a parser upgrade lands, the version stamp tells you which documents to reprocess.
Parsing failures are silent — a scanned PDF extracts as an empty string, a mangled table extracts as fluent nonsense, and nothing throws. Gate ingestion with per-document extraction metrics (chars per page, table count, OCR-trigger rate) or you will discover parser failures from wrong answers in production.
Meena built her golden set before running a single experiment, and the ordering is the method: without it, every parser, chunking, and embedding decision is a matter of opinion. A golden set is 50–100 real questions, each mapped to the specific document — ideally the specific section or chunk — that answers it. The sources are wherever real questions already live: SME interviews (an hour with two domain experts yields thirty), support-ticket logs, internal search queries, and the questions stakeholders actually asked in Slack.
Stratify it deliberately, because an unstratified set silently optimizes for the easy majority. Cover every document type (the spec-table PDFs, not just the clean memos), every query shape (semantic paraphrase, exact-string lookup like part numbers, multi-hop questions needing two sections), and — for Indian enterprise corpora — every language mix the users actually type, including code-mixed Hindi-English. Ten of Meena's sixty questions were Hinglish ticket phrasings; those ten are what later settled her embedding decision.
The harness is smaller than people expect:
def evaluate(retriever, golden_set, k=10):
hits, precisions = 0, []
for q in golden_set:
results = retriever.search(q.question, top_k=k)
got = {r.chunk_id for r in results}
if got & q.answer_chunk_ids:
hits += 1
precisions.append(len(got & q.answer_chunk_ids) / k)
return {"recall@k": hits / len(golden_set),
"context_precision": sum(precisions) / len(precisions)}
RAGAS gives you the standard-vocabulary version of these metrics plus synthetic test-set generation to bootstrap coverage; Arize Phoenix and Langfuse track results across runs. Two disciplines keep the set honest over time:
The golden set is the single highest-leverage artifact in this module — sixty questions built in two days will adjudicate every pipeline decision you make for the next year, and India postings that ask for "documented RAGAS scores" are asking whether you have one.
Chunking is the data modeling of the AI era: you are choosing the unit of retrieval, and the choice alone swings recall by up to ~9% on the same corpus with everything else held constant. The strategies, as a decision matrix:
| Strategy | How it splits | Best for | Cost & complexity | Characteristic failure |
|---|---|---|---|---|
| Fixed-size (+ overlap) | Every N tokens, sliding overlap | Uniform prose; quick baselines | Trivial; cheapest | Slices mid-sentence, mid-table; splits answers across chunks |
| Recursive | Paragraph → sentence → token separators, first that fits | General text without reliable structure | Low | Boundaries drift; an early edit reshuffles later chunks |
| Structure-aware | On document structure: headings, sections, list/table boundaries; tables kept atomic | Manuals, contracts, runbooks, Markdown/code — most enterprise corpora | Medium; needs a structure-preserving parser upstream | Oversized sections still need a secondary splitter |
| Semantic | Splits where embedding similarity between consecutive sentences drops | Long unstructured prose with topic drift | High — embedding calls at index time | Cost; unstable boundaries on re-index; modest measured gains |
| Parent-child (hierarchical) | Small children (128–256 tok) indexed for precision; parent (512–1,024 tok) returned for context | Corpora needing both pinpoint matching and enough context to answer | Medium; store parent-child links + dual bookkeeping | Parent dedup when many children hit; more moving parts |
| Late chunking | Embed long context first, derive chunk vectors from token embeddings after | Cross-reference-heavy docs where chunks depend on far-away context | High; needs long-context embedding models | Newer tooling; fewer battle-tested implementations |
Two defaults fall out of practice. Structure-aware is the starting point for enterprise documents — it is why the parser's structure preservation mattered — and parent-child is the upgrade when your eval shows retrieval finding the right neighborhood but the LLM lacking context to answer: children match precisely, parents carry the surrounding table or section. The mechanics are less exotic than the names suggest:
def parent_child_chunks(section):
"""Index small children for precision; return the parent for context."""
parent = {"id": f"{section.doc_id}:{section.path}",
"text": section.text[:4000], # 512-1,024 tokens of context
"metadata": section.metadata}
children = [{"id": f"{parent['id']}#c{i}",
"text": child_text, # 128-256 tokens, embedded + indexed
"parent_id": parent["id"]}
for i, child_text in enumerate(split_tokens(section.text, size=256, overlap=32))]
return parent, children # retrieval hits children; the LLM receives deduped parents
In LangChain/LlamaIndex this is a stock splitter plus a parent-document retriever; chonkie is the lightweight standalone alternative; a custom splitter over Docling's structured output is often under a hundred lines and exactly fits your corpus. The one bookkeeping obligation is deduplication at query time — when four children of the same parent all hit, the LLM should see one parent once.
The framework only chooses candidates — the golden set chooses the winner. Run every candidate strategy through the identical harness from the previous section, compare recall@k and context precision per strategy, and pick with evidence. This is LAB-07, and "chunking strategy" is language employers now put in job postings verbatim; the interview-ready answer is never a strategy name, it is a measured comparison on a corpus.
There is no best chunking strategy, only a best strategy for a corpus — and the only way to know is a golden-set eval run identically across candidates. The decision memo with recall numbers per strategy is the artifact that separates engineers from tutorial-followers.
The trap first: MTEB rank is a leaderboard over other people's corpora, and models within a few points of each other there routinely reorder on a specific domain corpus — legal boilerplate, part-number-dense manuals, Hinglish support tickets. The method is the same golden set from the chunking bake-off, reused: embed the corpus with each candidate, run the retrieval eval, compare recall@k and context precision per model. Same harness, different variable — by module's end you will have used one eval to settle three different pipeline arguments, which is precisely the habit that makes the pipeline defensible.
The decision axes:
| Axis | Hosted API (text-embedding-3-large class, Voyage, Cohere) | Self-host OSS (BGE-M3 class, sentence-transformers) |
|---|---|---|
| Quality | Frontier-competitive out of the box | Competitive on many corpora — verify on yours |
| Cost model | Per-token, scales with corpus + query volume | GPU/CPU serving cost; near-zero marginal per token |
| Data boundary | Text leaves your tenancy — a real constraint under client MSAs and DPDP-sensitive data | Stays fully in-house; the compliance-friendly default for Indian services work |
| Multilingual | Varies by provider | BGE-M3 is strong across 100+ languages incl. Indic — often the deciding axis |
| Ops | None | Model serving, versioning, GPU capacity are now your job |
Run the cost arithmetic alongside the quality numbers, because the axes trade against each other. A 12,400-document corpus at ~40M tokens costs a few dollars to embed on a hosted API's batch tier — but query-side embedding runs forever at interactive prices, and a re-embed (model upgrade, chunking change) pays the corpus cost again each time. Self-hosting inverts the shape: a small GPU instance embeds the same corpus for roughly the price of its runtime hours, and marginal cost per query rounds to zero, but you now operate model serving. The crossover is volume- and boundary-driven — sustained query traffic or a client MSA that forbids text egress both push toward self-host, and for Meena the MSA alone would have decided it even without the multilingual result.
Dimensions are a storage-and-latency decision you should do arithmetic on, not default. At 3,072 float32 dimensions a vector is ~12 KB — 10M chunks is ~120 GB of vectors before index overhead; at 1,024 dimensions it is a third of that, with retrieval impact you can measure on the golden set rather than guess. Matryoshka-trained models make dimensionality a tunable: truncate to 256 dimensions for a cheap first-pass ANN search, rescore the shortlist at full dimension — most of the quality at a fraction of the memory and latency. And one operational fact worth internalizing now, before M7 makes you pay for it: embeddings from different models — or the same model after a version bump — live in incompatible vector spaces, so the model choice is pinned in pipeline metadata and changing it later means re-embedding the corpus. Choose like it costs something, because it does.
Reuse one golden set to settle every pipeline argument — parser, chunking, embedding model, dimensions. One harness, one metric vocabulary, many decisions; the moment two experiments use different evals, their numbers stop being comparable and the folklore comes back.
A for loop over an embeddings API dies at corpus scale in four predictable ways: rate limits, transient failures that torch hours of progress, zero cost visibility, and re-runs that re-embed everything. The production shape is the same one you'd build for any expensive external-API sink:
import hashlib, json, time
def embed_corpus(chunks, embed_client, state_store, run_id, batch_size=128):
"""Idempotent, resumable, cost-tracked batch embedding."""
todo = [c for c in chunks
if state_store.get(chunk_key(c)) != c.content_hash] # skip already-embedded
for batch in batched(todo, batch_size):
try:
vectors = embed_with_backoff(embed_client, [c.text for c in batch])
except PoisonInput as e: # oversized/garbage chunk
dead_letter(batch, e, run_id); continue
state_store.write_vectors(batch, vectors) # keyed by content hash
log_cost(run_id, tokens=sum(c.token_count for c in batch))
def chunk_key(c): # content-addressed: re-runs and edits re-embed only what changed
return f"{c.doc_id}:{hashlib.sha1(c.text.encode()).hexdigest()[:16]}"
The load-bearing decisions:
For non-urgent corpus work — which is all indexing work — the provider's batch tier at ~50% of synchronous pricing is the default: submit the corpus as a batch job, poll, collect. Meena's 12,400 documents cost $6.40 this way; lab corpora here are sized to stay under ~$5.
Where it runs is a genuine three-way choice. External API from Spark or a worker pool gives model choice and portability at the cost of data egress — the Spark version wraps the client in a pandas UDF and inherits your cluster's parallelism, but every partition is now hammering the same rate limit, so the backoff must be shared, not per-executor. Warehouse-native turns embedding into a SQL statement with orchestration, retries, and governance inherited from the platform:
-- BigQuery: embed a chunk table in place; Snowflake Cortex EMBED_TEXT is the analogous one-liner
CREATE OR REPLACE TABLE kb.chunk_embeddings AS
SELECT chunk_id, content AS chunk_text, ml_generate_embedding_result AS embedding
FROM ML.GENERATE_EMBEDDING(
MODEL kb.embed_model,
(SELECT chunk_id, chunk_text AS content FROM kb.chunks WHERE embedded_at IS NULL));
The data never leaves the tenancy — the compliance-friendly pattern for DPDP-sensitive and client-restricted corpora, at per-token platform pricing — and incremental behavior falls out of the WHERE clause. Self-host (sentence-transformers on your own GPUs, vLLM for serving) wins on sustained volume and hard data-boundary requirements, and Colab/Kaggle free GPUs cover the lab-scale version. The free path for this whole module: local corpus + Docling + a small OSS embedding model, or the BigQuery sandbox for the warehouse-native variant — no card required.
Whichever lane you choose, the pipeline's outputs are the same three things: vectors keyed by content hash, a per-run cost record, and metadata (embedding model + version, chunking config) stamped on every vector. That last stamp looks bureaucratic until the first model upgrade, when it becomes the only way to know which vectors belong to which space — M7 formalizes it as pipeline_version in the sync ledger, but the habit starts here.
Scale-out embedding questions are pipeline-engineering questions in disguise — the correct options feature idempotent content-hash keys, backoff, dead-lettering, batch-tier pricing, and cost logs; options that "just parallelize harder" or re-embed the world on every run are the distractors.
Your first 15 minutes Monday — if your RAG pipeline has no golden set, book an hour with two SMEs and capture 30 real questions, each mapped to the document and section that answers it, before you touch parser or chunking settings; if you already have one, add every retrieval failure real users hit this week as a new golden question.
Check yourself: Before running a single parser, chunking, or embedding experiment, Meena built one artifact — and the ordering is the method. What is it, and why must it come first?
This module is the first half of the track's differentiator: the ingestion-to-embeddings pipeline, with a measured justification for every choice — and it anchors three labs that build one continuous artifact. LAB-06 (document ingestion pipeline): build the parsing stage over a mixed corpus — PDFs with tables, HTML, a scanned page — compare two parsers on extraction fidelity, add dedup and metadata capture; artifact: the ingestion repo plus the parser-comparison scorecard. LAB-07 (chunking bake-off): implement fixed, recursive, structure-aware, and parent-child chunking on the LAB-06 corpus and evaluate all four against a RAGAS golden set; artifact: a chunking decision memo backed by context-precision/recall numbers per strategy. LAB-08 (embedding selection & scale-out): benchmark a hosted model against an open-source one on your corpus golden set, test Matryoshka truncation, then build the batch pipeline with retries, idempotency, and a per-run cost log; artifact: the benchmark notebook plus the production batch pipeline. India postings now ask for documented RAG pipeline projects with evaluation scores by name — these three artifacts, chained, are exactly that evidence. Do this at work this week → pb-27 Build a Document Parsing Stage for RAG Ingestion, pb-28 Choose a Chunking Strategy with a Measured Eval, and pb-29 Benchmark Embedding Models on Your Own Corpus. M6 picks up the corpus you just embedded and makes retrieval over it fast, hybrid, and reranked.