Add a Reranking Layer to Lift Precision at the Top

Tier: Core · Workflow area: RAG data pipelines

The rule of thumb: if the right passage is usually in your top 50 but not your top 5, reranking beats switching embedding models. In Anthropic's contextual-retrieval measurements, adding reranking pushed the reduction in top-20 retrieval failures to roughly 67%. This playbook drops a reranker between hybrid search (pb-33) and generation, with the latency and cost budget made explicit.

When to reach for this

• •
  Golden-set eval shows high recall@50 but weak recall@5 or MRR — right passage present, wrongly ranked.
• •
  The LLM context budget forces you to send few chunks, so top-of-list precision is everything.
• •
  Users report "the answer ignored the obvious document" while eval shows that document retrieved at rank 20.
• •
  Before re-embedding the corpus with a bigger model: rerank first, it is index-free and reversible.

The workflow

1. 1.

   Confirm the reranker will pay for itself. From your golden set (pb-28), compute recall@50 and recall@5 on the current pipeline. Expected decision signal: recall@50 = 0.92, recall@5 = 0.61 → large headroom, proceed. If recall@50 is itself low, fix retrieval first (pb-28/pb-29/pb-33) — a reranker cannot rank what never arrived.

2. 2.

   Stand up a local cross-encoder. BGE-reranker runs on CPU for lab scale and modest GPU for production; hosted equivalents are Cohere Rerank and Voyage rerank behind one HTTP call.

   Copy

   from sentence_transformers import CrossEncoder
   
   reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")  # local; hosted: Cohere Rerank / Voyage
   
   def rerank(query: str, candidates: list[dict], top_n: int = 5) -> list[dict]:
       scores = reranker.predict([(query, c["content"]) for c in candidates])
       ranked = sorted(zip(candidates, scores), key=lambda x: -x[1])
       return [{**c, "rerank_score": float(s)} for c, s in ranked[:top_n]]
   

3. 3.

   Wire it as a stage: retrieve wide, rerank narrow. Pull top 20–50 from hybrid search, rerank, pass 3–5 forward. Candidate depth is the main quality/latency knob — sweep 20/30/50.

   Copy

   candidates = hybrid_search(q, top=50)          # pb-33, filters applied
   context = rerank(q, candidates, top_n=5)
   

4. 4.

   Measure the lift with recall@5 and MRR@5 on the golden set. These are the metrics that prove the step worked, because the reranker's whole job is the top of the list:

   Copy

   Expected shape:
   before rerank: recall@5 = 0.61 · MRR@5 = 0.48
   after  rerank: recall@5 = 0.84 · MRR@5 = 0.74   (candidates from same top-50)
   

   A reranker that moves recall@5 by under ~5 points on real headroom is misapplied — check that candidate texts (not truncated stubs) are being scored, and that the model matches your language mix.

5. 5.

   Budget latency and cost explicitly. Measure per-query rerank time at each candidate depth: a cross-encoder scores query×passage pairs, so cost is linear in depth. Typical shape: 50 pairs ≈ 40–150 ms on GPU, 0.5–2 s on CPU; hosted rerank APIs price per 1K searches. Write the chosen point into the retrieval SLA: e.g. "top-30 rerank, p95 ≤ 120 ms, fits the 500 ms retrieval budget." If CPU-bound in production, drop depth to 20 before dropping the reranker.

6. 6.

   Use rerank scores as a quality gate, not just an ordering. Cross-encoder scores are roughly calibrated per model; a top result scoring far below the model's typical "relevant" range signals an unanswerable query. Log the top score per query and consider a threshold below which the application says "no good source found" instead of letting the LLM improvise from weak context.

Verify before you trust it

• •
  recall@5 and MRR@5 improve on the golden set with the same candidate lists feeding both arms of the comparison — reranker evaluation must hold retrieval fixed.
• •
  Slice check: lift holds on both lexical and semantic slices (pb-33 tags); multilingual corpora need a multilingual reranker (bge-reranker-v2-m3 class), verify the non-English slice explicitly.
• •
  p95 end-to-end retrieval latency re-measured under load with reranking enabled; the stage runs after hybrid search, so budgets add.
• •
  Failure-mode probe: 10 deliberately unanswerable queries — top rerank scores should be visibly lower than answerable ones, validating the quality-gate threshold.

Pitfalls

• Reranking to fix recall. The reranker only reorders candidates. If the gold chunk is not in the top 50, no reranker helps — that is a pb-28/pb-29/pb-33 problem.
• Truncated passages at scoring time. Cross-encoders have sequence limits (typically 512–8K tokens for query+passage). Silently truncated chunks score on their first paragraph; align chunk size (pb-28) with reranker limits or score parent summaries.
• Comparing scores across queries or models. Rerank scores are per-query orderings, not global relevance probabilities. Thresholds need per-model calibration on your data.
• Reranking 500 candidates "to be safe". Latency and cost scale linearly with depth while quality saturates; past ~50 candidates you pay real money for noise.
• Skipping the no-answer path. Without a score threshold, the reranker dutifully orders five bad chunks and the LLM hallucinates from them confidently.

Tools

• BGE-reranker (via sentence-transformers CrossEncoder)free, self-hosted, multilingual v2 variants; the default local path.
• Cohere Rerankmanaged cross-encoder behind one API call; trial keys make it the fastest hosted proof-of-value.
• Voyage rerankhosted alternative frequently paired with Voyage embeddings; same integration shape.

Client-safe variant

A hosted reranker sees the query and the full text of every candidate chunk — for sensitive corpora that is more exposure per query than the embedding API ever had, so client engagements should default to the self-hosted BGE-reranker inside the client boundary; CPU inference at top-20 depth is workable for internal-tool query volumes. If the client approves a hosted reranker, scope the approval explicitly to "queries plus retrieved passages", confirm retention terms, and exclude ACL-restricted document classes from rerank candidates as defense in depth. The eval harness and its recall@5 lift report contain golden-set questions, so they stay inside the client environment like all pb-28 artifacts.