Enrich Chunks with LLM-Generated Context Before Embedding

Tier: Advanced · Workflow area: RAG data pipelines

In Anthropic's published measurements, contextual embeddings alone cut top-20 retrieval failures by roughly 35% (and combined with BM25 and reranking, the stack reached ~67%). The catch is cost — one LLM call per chunk — which batch processing and prompt caching turn from prohibitive to routine. This playbook builds the enrichment stage and proves the lift on your golden set.

When to reach for this

• •
  Retrieval failures cluster on chunks full of pronouns, relative references, or bare numbers that lose meaning out of context.
• •
  Long documents (contracts, filings, manuals) where section context determines what a passage is even about.
• •
  Hybrid search and reranking (pb-33, pb-34) are in place and you need the next recall increment from the index side.
• •
  You can afford one batched LLM pass per corpus re-index — this is an index-time cost, zero query-time latency.

The workflow

1. 1.

   Baseline first. Record current golden-set fail@20 from your pb-33/pb-34 pipeline. Contextual enrichment is an increment; without the baseline you cannot prove it earned its token bill.

2. 2.

   Write the enrichment prompt so the whole document is cacheable. The pattern: full document first (cached across all its chunks), then the chunk, then the instruction. With prompt caching, the document tokens are paid once per document, not once per chunk — this is what makes the technique affordable.

3. 3.

   Generate contexts with structured output. Local-first note: enrichment quality tracks model quality, so this is the one pipeline stage where a frontier hosted model is usually worth it; a strong local model via Ollama/vLLM is the fallback for restricted data.

   Copy

   import anthropic
   from pydantic import BaseModel
   
   class ChunkContext(BaseModel):
       context: str  # 1-2 sentences situating the chunk in the document
   
   client = anthropic.Anthropic()
   
   def enrich(document: str, chunk: str) -> str:
       response = client.messages.parse(
           model="claude-opus-4-8",
           max_tokens=200,
           thinking={"type": "adaptive"},
           messages=[{
               "role": "user",
               "content": [
                   {"type": "text", "text": f"<document>\n{document}\n</document>",
                    "cache_control": {"type": "ephemeral"}},
                   {"type": "text", "text":
                    f"<chunk>\n{chunk}\n</chunk>\n"
                    "Write 1-2 sentences situating this chunk within the document "
                    "(what document, which section/party/product, what period). "
                    "Answer with only the context."},
               ],
           }],
           output_format=ChunkContext,
       )
       return response.parsed_output.context
   

   Expected output per chunk: something like "From the 2025 Master Services Agreement between Acme and Beta Corp, Section 7 (Late Delivery), covering penalty escalation terms." — short, factual, front-loadable.

4. 4.

   Run it as a batch job on the pb-30 rails. Enrichment is embedding-generation's twin: same work-queue table (add context and enriched_at columns), same idempotent re-runs, same DLQ. For full-corpus passes, submit through the provider's batch API (~50% discount, hours-scale turnaround) rather than synchronous calls. Pre-compute the cost from token counts before launching: docs × doc_tokens (cached rate) + chunks × (chunk_tokens + ~60 output tokens).

5. 5.

   Embed context + "\n" + chunk into a shadow collection. Store the original chunk text for display and the enriched text only as the embedding input (and in the BM25 leg too — contextual BM25 is part of the published gains). Never overwrite the live index; this is a versioned cutover (pb-31 model_version discipline, pb-36 mechanics).

6. 6.

   Measure the lift. The metric that proves this step worked: golden-set fail@20 (and recall@5 post-rerank), enriched versus baseline, same queries, same pipeline otherwise.

   Copy

   Expected shape:
   baseline hybrid+rerank : fail@20 = 6.0% · recall@5 = 0.84
   contextual  +  same    : fail@20 = 4.0% · recall@5 = 0.89
   

   Also spot-audit 30 generated contexts against their source documents for factual drift — a context that misdescribes the chunk poisons retrieval more efficiently than no context at all.

Verify before you trust it

• •
  Measured fail@20 improvement over baseline on the identical golden set; if the delta is under ~2 points, check whether your chunks were already self-contained (structure-aware chunking sometimes pre-empts the gain).
• •
  30-context human audit: ≥95% factually consistent with the source document; every failure classified before cutover.
• •
  Cost reconciliation: actual batch spend within 15% of the pre-launch estimate; cache hit rate confirmed in API usage stats (a cold cache doubles the bill).
• •
  Re-run safety: enrichment job re-executed on 100 chunks produces byte-identical work-queue state (idempotency inherited from pb-30 holds).

Pitfalls

• Paying full price by skipping caching or batch. Per-chunk document re-sends without caching inflate cost by 10-50x on long documents. Cache the document block; batch the corpus.
• Embedding the context but not indexing it for BM25. Half the published gain comes from contextual BM25; enrich both legs of hybrid search.
• Contexts that summarize instead of situate. A summary duplicates the chunk's content; a situating sentence adds document, section, entities, timeframe. Enforce with the prompt and length validation.
• Enriching with a weak model to save money. Bad contexts are worse than none. If budget-bound, enrich only the document types where the baseline eval shows contextual failures, not the whole corpus.
• Overwriting the live index in place. A mid-job failure leaves a half-enriched embedding space — mixed representations, unexplainable retrieval. Shadow collection, then cutover.

Tools

• Claude batch API + prompt cachingthe reference implementation of affordable enrichment; caching amortizes document tokens, batch halves the rate.
• LlamaIndex extractorspipeline scaffolding for metadata/context extraction stages if you are already on its ingestion framework.
• sentence-transformers + Chroma/pgvectorthe local embedding-and-shadow-index side of the cutover, unchanged from pb-28/pb-30.

Client-safe variant

Enrichment sends full documents to the LLM — the maximum-exposure operation in the entire RAG pipeline — so for client data it requires explicit written approval covering the provider, retention terms, and document classes, with ACL-restricted material excluded from the batch. Where approval isn't granted, run the same prompt through a client-approved or self-hosted model (vLLM/Ollama with a strong open-weights model) inside their boundary and accept a smaller lift, measured by the same shadow-index eval. Either way, the generated contexts become part of the client's index and inherit its access controls, and the pre-launch cost estimate should be signed off — a full-corpus enrichment pass is the largest single LLM spend most RAG projects ever make.