RAG Pipelines II — Vector Stores, Hybrid Search, Reranking

M5 turned messy documents into measured chunks and embeddings; this module is the serving side — choosing and tuning the store, and layering hybrid search, reranking, and contextual enrichment until retrieval failure rates drop by more than half. Every layer earns its place with a before/after number, because each one adds cost and latency that must be paid for in recall.

The scenario — the query that embeddings can't see

The plateau. Meena is the data platform engineer at a manufacturing-analytics company, and by June 8 her team's RAG retrieval layer — the M5-style pipeline over 2.1M chunks of service manuals, incident reports, and parts documentation — is stuck. The golden-set eval reads recall@10 of 0.82, and the failure bucket has a shape: queries like "error E-4402 on the VX-220 conveyor" return beautifully semantic results — chunks about conveyor faults in general, error-handling procedures, adjacent models — everything except the one chunk containing the literal string E-4402.

Dense embeddings compress meaning, and an alphanumeric error code has almost none; to the vector space, E-4402 and E-4419 are nearly the same point. Field engineers have noticed. The internal Slack channel has a pinned message: "if it has a part number, just use grep."

Meanwhile the p95 latency budget from the app team is 400 ms, the store is a single pgvector instance whose index build ate 9 GB of RAM at defaults, and the roadmap says three enterprise customers — each demanding their data be provably isolated, a requirement Divya, the platform lead, has already written into the SOW — onboard in Q3.

The layered fix. Meena runs this module's sequence, measuring at each step with the same golden set.

Choosing the store — a decision framework, then a schema

The store decision is four families, not twenty products. Evaluate against your actual constraints — scale, hybrid support, tenancy, governance, and what you already operate:

pgvector is this track's teaching store because it makes every concept inspectable in SQL, and it is a legitimate production choice below ~10M vectors — the honest mistake is not choosing it, but outgrowing it silently. Decide with numbers: vector count and growth rate, QPS, filter selectivity, and whether tenancy or governance constraints pin you to a family before performance even gets a vote.

Schema design is where data modeling instincts transfer directly. Every chunk row carries: a deterministic chunk ID (doc_id + content-hash — the M5 convention, which M7's incremental sync depends on), the vector, the chunk text, and a metadata payload you design as deliberately as any dimension table — doc_id, source_system, doc_type, section_path, language, updated_at, plus tenant_id and allowed_groups stamped at ingestion. Two rules with teeth:

• Anything you'll ever filter by must be in the payloadand, in dedicated stores, declared as an indexed payload field; filtered ANN over an unindexed field degrades to a scan.
• Metadata you didn't capture at ingestion doesn't exist at query timeadding a filterable field later means reprocessing the corpus.

Tenancy is a design-time decision you make now, even though operating it is M7's subject. The spectrum — shared collection with a mandatory tenant_id filter → store-enforced namespace/partition per tenant (Pinecone namespaces, Qdrant payload-partitioned multitenancy) → collection or index per tenant → dedicated instance — trades isolation strength against index sprawl and per-collection memory overhead. The namespace/partition tier is the SaaS default; per-tenant collections buy per-tenant tuning and SLAs for a handful of large customers; and for services companies, client MSAs and BFSI-sector rules often mandate the dedicated end of the spectrum regardless of the engineering trade-off. What you must get right now is stamping tenant_id and ACL metadata on every chunk — retrofitting tenancy onto an unstamped corpus is a full reprocess.

Index tuning — recall@k is a dial, and you should know where it's set

Approximate indexes trade recall for speed, and the defaults chose a trade-off for you without asking. First establish truth: exact search over a sample as the baseline, then measure your index against it —

-- Baseline: exact scan (no index) over a 5K-query sample
SET LOCAL enable_indexscan = off;
SELECT chunk_id FROM kb_chunks ORDER BY embedding <=> :query_vec LIMIT 10;

-- Candidate: HNSW at a given ef_search
SET LOCAL hnsw.ef_search = 120;
SELECT chunk_id FROM kb_chunks ORDER BY embedding <=> :query_vec LIMIT 10;

Overlap between the two result sets is recall@10 of the index — Meena's four missing points were exactly this gap, invisible until measured. HNSW has three knobs:

• mgraph connectivity, paid in memory and build time; 16 is the common default. Fixed at build time.
• ef_constructionbuild-time effort; higher = better graph, slower build. Also fixed at build time.
• ef_searchcandidates explored per query; the runtime dial you sweep to plot the recall-vs-latency curve and pick an operating point.

IVF-family indexes trade differently — cluster the space, probe nprobe cells at query time — cheaper to build and better suited to batch-heavy workloads, but recall degrades as inserts skew the clusters, which means periodic reclustering as an operational task.

Index builds are batch jobs with batch-job economics, and Meena's 9 GB build was the reminder. HNSW construction on millions of vectors takes real memory (pgvector's maintenance_work_mem governs it) and real hours, scaling with m and ef_construction — so treat a rebuild like any heavy backfill: schedule it, size it, and build concurrently or into a fresh index rather than locking the serving path. This is also the practical argument for getting the parameters right early: the runtime dial (ef_search, nprobe) is free to change, but the build-time knobs cost a rebuild, which at scale is an orchestrated event, not an ALTER statement.

Quantization is the memory lever, and the arithmetic is worth doing before you shard: 2.1M chunks at 1,024 float32 dimensions is ~8.6 GB of raw vectors; scalar int8 quantization cuts it ~4× and binary quantization ~32×, at a recall cost you measure — not guess — on the golden set. The standard production pattern (native in Qdrant, standard in FAISS) is quantized first-pass search with full-precision rescoring of the shortlist: most of the memory savings, a fraction of the recall cost. Product quantization compresses hardest and costs the most recall — reach for it only when the corpus genuinely doesn't fit otherwise.

Hybrid search — dense for meaning, sparse for the strings that mean nothing

Dense retrieval fails on exactly the tokens enterprise corpora are full of: part numbers, error codes, invoice IDs, statute references, people's names — high-information strings with no semantic neighborhood. Sparse lexical scoring (BM25, or learned-sparse models like SPLADE) treats them as what they are: rare exact terms worth their weight in gold. Production RAG over enterprise corpora defaults to hybrid — Anthropic's contextual-retrieval experiments measured adding BM25 alongside dense embeddings cutting top-20 retrieval failures by ~49%, and the effect concentrates precisely in the queries users care most about, because a user who types an exact code expects an exact hit.

Fusion is simpler than it sounds. Run both searches, then combine with Reciprocal Rank Fusion — score each result Σ 1/(k + rank) across the lists (k=60 by convention) — which needs no score normalization between incomparable scales. In pgvector it is two CTEs and a join:

WITH dense AS (
    SELECT chunk_id, ROW_NUMBER() OVER (ORDER BY embedding <=> :qvec) AS r
    FROM kb_chunks LIMIT 20
),
sparse AS (
    SELECT chunk_id, ROW_NUMBER() OVER (
        ORDER BY ts_rank_cd(tsv, websearch_to_tsquery(:qtext)) DESC) AS r
    FROM kb_chunks WHERE tsv @@ websearch_to_tsquery(:qtext) LIMIT 20
)
SELECT chunk_id, SUM(1.0 / (60 + r)) AS rrf
FROM (SELECT * FROM dense UNION ALL SELECT * FROM sparse) fused
GROUP BY chunk_id ORDER BY rrf DESC LIMIT 20;

Qdrant and Weaviate ship hybrid + fusion as a query option; OpenSearch pairs kNN with the BM25 engine it already is. The tuning surface is small — candidate depth per arm, the fusion constant, optionally a weighted variant if your eval shows one arm should dominate — and every change is judged by the same golden-set harness. One India-relevant note: for code-mixed corpora (Hinglish tickets, transliterated names), the sparse arm also backstops the embedding model's weakest inputs, which makes hybrid a robustness play, not just a part-number play.

Reranking — a second, smarter opinion on a shortlist

Embeddings compress every chunk to one point before seeing any query — that's what makes ANN search fast, and what caps its precision. A cross-encoder reranker reads the actual query and each candidate chunk together, which is far too slow for millions of chunks and exactly right for 40: retrieve wide (top 20–50 from hybrid), rerank the shortlist, send the top 3–5 to the LLM. In Anthropic's published numbers, adding reranking on top of contextual embeddings plus BM25 pushed the top-20 failure reduction to ~67% — the layers compound because each fixes a different failure class.

The diagnostic for whether you need it is one query against your eval data: is the answer-bearing chunk usually in the top-50 but not the top-5? If yes, reranking attacks your actual failure mode and will beat swapping embedding models — a much cheaper experiment than re-embedding the corpus. If the answer isn't in the top-50 at all, reranking can only reorder failure; go fix recall upstream (hybrid, chunking, enrichment) first.

Options: Cohere Rerank and Voyage rerank as managed APIs (per-query pricing, trial keys for labs), BGE-reranker-class cross-encoders self-hosted (free, GPU-friendly at shortlist sizes, and the data-boundary-safe choice for client-restricted corpora). Engineer it against an explicit latency budget, written down per stage the way Meena's was:

Rerank latency scales linearly with candidate count, so the depth knob is your main dial, and the failure posture is decided in advance: on reranker timeout, serve the hybrid order rather than the error. Cache identical query-candidate scores; measure NDCG or answer-in-top-k uplift on the golden set like every other layer.

Contextual enrichment — chunks that know where they came from

Chunking amputates context: a chunk reading "the same fault recurred after the bearing replacement" embeds terribly because the machine, the site, and the fault live two paragraphs away. Contextual retrieval fixes this at index time — for each chunk, an LLM writes a short situating context from the full document, prepended before embedding. In Anthropic's measurements, contextual embeddings alone cut top-20 retrieval failures ~35%; combined with BM25 and reranking, the stack reached the ~67% above.

from anthropic import Anthropic
from pydantic import BaseModel

class ChunkContext(BaseModel):
    context: str   # 1-2 sentences situating the chunk within the document

client = Anthropic()

def contextualize(doc_text: str, chunk_text: str) -> str:
    response = client.messages.parse(
        model="claude-opus-4-8",
        max_tokens=200,
        thinking={"type": "adaptive"},
        system=("Write 1-2 sentences situating the chunk within the document "
                "(which product/section/incident it concerns) to improve search retrieval. "
                "Output only the situating context."),
        messages=[{"role": "user",
                   "content": f"<document>{doc_text}</document>\n<chunk>{chunk_text}</chunk>"}],
        output_format=ChunkContext,
    )
    return response.parsed_output.context  # prepend to chunk_text before embedding

The economics only work because of two M4 levers applied together: the full document is the shared prompt prefix across all its chunks, so prompt caching prices the repeated document tokens at a small fraction of fresh input — and the whole job is index-time work, so it runs on the batch tier at ~50% off. Combined, enrichment lands around a dollar-and-change per thousand chunks, which is why Meena could afford it and why LAB-10 has you compute the exact figure on your own corpus. Spend it where the eval says context is the failure mode — cross-reference-heavy segments — rather than uniformly; enrichment is an index-time cost with zero query-time latency, which is the best kind of quality spend. LlamaIndex's extractors provide the framework-native version of the same pattern.

Two operational footnotes before you adopt it:

• Enrich both armsenriched text is what gets embedded and what BM25 indexes, so apply the enrichment to both arms of hybrid search, or the arms drift out of agreement about what a chunk says.
• Enrichment adds a document-level dependency to your chunkswhen a document is edited, its chunks' contexts may be stale even if the chunk text is unchanged: a re-enrichment trigger M7's sync pipeline needs to know about, and one more reason enrichment decisions belong in the pipeline-version metadata.

Warehouse-native retrieval — when the best new infrastructure is none

If the corpus already lives in the warehouse, you can stand up production retrieval without operating any new system. Snowflake Cortex Search is a managed hybrid (semantic + keyword) service over a table — ingestion, embedding, indexing, and fusion handled inside Snowflake. Databricks Mosaic AI Vector Search syncs a Delta table to a vector index automatically and ships built-in retrieval-quality evaluation. BigQuery pairs AI.GENERATE_EMBEDDING with VECTOR_SEARCH, so the M5 embedding stage and this module's serving stage are two SQL statements over the sandbox tier:

SELECT base.chunk_id, base.chunk_text, distance
FROM VECTOR_SEARCH(
       TABLE kb.chunk_embeddings,        -- embeddings from AI.GENERATE_EMBEDDING
       'embedding',
       (SELECT ml_generate_embedding_result
        FROM ML.GENERATE_EMBEDDING(MODEL kb.embed_model,
             (SELECT @user_query AS content))),
       top_k => 20)
ORDER BY distance;

Retrieval as a SQL statement means every skill in this module transfers — the hybrid pattern becomes this joined with a keyword search, filters become WHERE clauses under existing row-level policies, and the eval harness points at a view.

The trade is control for gravity and governance: you give up index-internals tuning and per-component choice, and in exchange retrieval inherits the platform's access control, lineage, audit logging, and residency — and the documents never leave the tenancy. For DPDP-sensitive personal data, RBI/SEBI-regulated BFSI clients, and services-company MSAs that prohibit shipping client text to external APIs, that boundary is frequently the deciding argument — warehouse-native retrieval is the compliance-friendly pattern, not the compromise. The decision mirrors the store table above: dedicated stores win on control, latency headroom, and stack independence; warehouse-native wins on governance, time-to-production, and not adding a stateful system to your on-call. LAB-13's retrieval half has you build the same search both ways and produce the per-million-queries cost comparison, which is the artifact that ends the meeting.

Check yourself: Meena's dense-embedding retrieval returns great semantic matches for "error E-4402 on the VX-220 conveyor" but never the chunk containing the literal E-4402. Why does this happen, and what is the fix?

Where this lands in your week

This module completes the retrieval layer and the track's differentiator sequence: a store chosen by framework and modeled deliberately, an index tuned against a measured recall baseline, hybrid search catching what embeddings can't see, a reranker spending latency exactly where it pays, enrichment restoring the context chunking destroyed, and a warehouse-native alternative for when governance outranks control. It anchors LAB-09 (hybrid search + reranking) — load pgvector and a dedicated vector DB, implement dense-only vs hybrid-RRF vs hybrid-plus-rerank, and measure failure rate, recall@k, p95 latency, and cost per query at each layer, reproducing the layered-gains pattern; artifact: the retrieval service repo plus the gains table. LAB-10 (contextual enrichment upgrade) — add batch-plus-cached chunk context to the LAB-09 index and quantify retrieval improvement against enrichment cost per 1K chunks; artifact: the before/after eval report. And the retrieval half of LAB-13 (in-SQL AI functions) — embed and VECTOR_SEARCH the same data warehouse-natively and build the cost model against the external stack. Chained onto M5's artifacts, the gains table is the strongest interview exhibit in this track: anyone can say "we used hybrid search and a reranker" — you can show what each layer bought, in recall points, milliseconds, and rupees. Do this at work this week → pb-33 Implement Hybrid Search with Reciprocal Rank Fusion, pb-34 Add a Reranking Layer to Lift Precision at the Top, and pb-35 Enrich Chunks with LLM-Generated Context Before Embedding. M7 takes this platform to production: keeping the index fresh, observed, sealed, and multi-tenant.