Tune Vector Index Parameters Against an Exact-Search Baseline

Tier: Advanced · Workflow area: RAG data pipelines

Defaults are tuned for vendor benchmarks, not your corpus — and an untuned index either burns memory you don't need or silently drops 10% of the right answers before your retrieval pipeline ever sees them. This playbook establishes the exact-search baseline and tunes against it with measured recall, latency, and memory.

When to reach for this

• •
  Moving a RAG index from prototype (thousands of vectors, defaults everywhere) to production scale (millions).
• •
  p95 retrieval latency or index memory is over budget and you need to know what recall each saving costs.
• •
  Retrieval quality dropped after a scale-up and you suspect ANN recall loss rather than embedding or chunking issues.
• •
  Evaluating quantization (PQ/SQ/binary) to fit a growing index into fixed memory.

The workflow

1. 1.

   Build the ground truth with exact search. ANN recall is measured against exact (brute-force) k-NN over the same vectors — no golden set needed for this layer, though you should sanity-check with it at the end. Sample 200–500 queries (real query logs beat random chunks):

   Copy

   import numpy as np
   
   # vectors: (N, d) float32, normalized; queries: (Q, d)
   def exact_topk(queries, vectors, k=10):
       sims = queries @ vectors.T          # cosine via dot product on normalized vecs
       return np.argsort(-sims, axis=1)[:, :k]
   
   truth = exact_topk(queries, vectors, k=10)
   

   Expected output: the true top-10 IDs per query. For corpora too big for in-memory numpy, FAISS IndexFlatIP is the exact baseline.

2. 2.

   Instrument the metric that matters: recall@10 versus exact. recall@10 = |ANN top-10 ∩ exact top-10| / 10, averaged over queries. This is the number every parameter change must report, alongside p50/p95 latency and index memory.

   Copy

   def ann_recall(ann_ids, truth, k=10):
       return np.mean([len(set(a[:k]) & set(t[:k])) / k for a, t in zip(ann_ids, truth)])
   

3. 3.

   Sweep ef_search first — it is free. Query-time ef_search (pgvector: hnsw.ef_search; Qdrant: hnsw_ef) needs no rebuild. Sweep 40 → 400:

   Copy

   -- pgvector
   SET hnsw.ef_search = 100;
   EXPLAIN ANALYZE SELECT chunk_id FROM rag_chunks
   ORDER BY embedding <=> $1 LIMIT 10;
   

   Expected output: a curve like ef=40: R 0.91/3ms · ef=100: 0.97/6ms · ef=200: 0.99/11ms. Pick the knee. If the knee can't reach your recall target (≥0.95 is a common bar; ≥0.98 when a reranker depends on candidates being present), move to build parameters.

4. 4.

   Rebuild with better construction parameters only if step 3 plateaus. Higher m (16 → 32 → 48) and ef_construction (64 → 200) raise the recall ceiling at the cost of memory (m roughly doubles graph edges) and build time. Rebuild on a shadow index, re-run the sweep, compare curves. Record build time — it is your future re-index cost (pb-36 cutovers).

5. 5.

   Add quantization last, and measure what it costs. Scalar quantization (SQ, ~4x memory saving) usually costs ≤1 recall point; product/binary quantization saves 8–32x but can cost real recall unless paired with rescoring (search quantized, rescore top-100 with full-precision vectors — Qdrant and FAISS support this natively). Re-measure recall@10 after enabling; never trust the vendor's "negligible loss" claim over your own curve.

6. 6.

   Close the loop with the end-to-end golden set. Re-run the pb-28 golden-set recall@10 on the tuned index. ANN-versus-exact recall of 0.97 should leave golden-set recall within a point of its exact-search value; a bigger gap means your query distribution differs from the tuning sample — re-sample from real logs.

Verify before you trust it

• •
  ANN recall@10 versus exact baseline meets target (≥0.95, or your documented bar) at the chosen settings, measured on ≥200 realistic queries.
• •
  p95 latency measured under concurrent load, not single-query — HNSW latency degrades under contention and with metadata filters applied; test with your actual tenant_id/ACL filters, which can silently force wider searches.
• •
  Memory reconciled: observed index size versus the vectors × dim × 4 bytes + graph overhead estimate; a 3x surprise usually means duplicated or unpruned segments.
• •
  Settings persisted in infrastructure config (not a one-off session SET), and the recall/latency/memory triple recorded next to them.

Pitfalls

• Tuning without an exact baseline. Comparing ANN setting A to ANN setting B tells you which is less bad, not whether either is good. The flat index is cheap to build once and is the only ground truth.
• Filtered search recall cliff. Recall measured unfiltered can collapse when tenant/ACL filters shrink the candidate pool — some engines traverse the graph then filter. Measure with production filters on; raise ef_search or use filter-aware index features if recall drops.
• Benchmarking latency cold. First queries page the index into memory; report steady-state p95, and note cold-start behavior separately for ops.
• Chasing recall 1.0. The last two points can triple latency and memory. If the right passage reliably reaches top-50, the reranker (pb-34) is a cheaper fix than a perfect ANN layer.
• Forgetting build-time cost.ef_construction=400 may add hours to every full rebuild; that cost lands on every index cutover and disaster recovery, not just today.

Tools

• pgvectorHNSW with hnsw.ef_search session control; tuning happens in plain SQL with EXPLAIN ANALYZE.
• Qdrantper-collection HNSW params, scalar/product quantization with rescoring, and a good filtered-search story.
• FAISSthe lab bench: exact baselines, IVF/PQ experiments, and parameter intuition before committing changes to the production engine.

Client-safe variant

Index tuning needs vectors and query samples, not raw documents — so on client engagements you can often run the entire exercise on embeddings alone, which are already one step removed from source text (though treat them as confidential regardless; inversion attacks exist). Run sweeps on a shadow index in the client environment, never on the serving index, and present the recall/latency/memory curves as the deliverable so the client's platform team can re-run the same harness after upgrades. If query logs are sensitive, generate tuning queries from the corpus itself (sampled chunk texts as pseudo-queries) and note the caveat that production query distribution may differ.