RAG Feature, Part 2: Hybrid Search, Reranking, and Groundedness Citations

Tier: Advanced · Workflow area: Building LLM Features

The production-grade stack — hybrid retrieval, a reranker, and verifiable citations — is precisely the sub-skill list that appears verbatim in applied-LLM job descriptions. Each upgrade here is measured against the pb-47 baseline before it stays.

When to use

• •
  pb-47 is live and your eval set shows retrieval misses — especially on queries containing exact terms, codes, or names.
• •
  Users challenge answers and you cannot show where an answer came from at claim level.
• •
  You are about to raise k to "fix" quality — stop, and do this instead.

Prerequisites

• •
  The pb-47 pipeline, its 10+question eval set, and its recorded baseline numbers.
• •
  A reranker: Cohere Rerank or Voyage rerank API (simplest), or a local cross-encoder via sentence-transformers (free).
• •
  30 minutes of query-log reading: pull the 20 worst real queries — they drive every decision below.

The workflow

1. 1.

   Add keyword retrieval beside vector retrieval. Postgres full-text search is already in your database:

   Copy

   ALTER TABLE chunks ADD COLUMN tsv tsvector
     GENERATED ALWAYS AS (to_tsvector('english', content)) STORED;
   CREATE INDEX ON chunks USING gin (tsv);
   

   Copy

   def keyword_retrieve(query: str, k: int = 20) -> list[Chunk]:
       return db.execute(
           """SELECT id, title, section, url, content,
                     ts_rank(tsv, websearch_to_tsquery('english', %s)) AS score
              FROM chunks
              WHERE tsv @@ websearch_to_tsquery('english', %s)
              ORDER BY score DESC LIMIT %s""",
           (query, query, k)).fetchall()
   

2. 2.

   Fuse the two lists with Reciprocal Rank Fusion. RRF needs no score normalization, which is why it is the standard fusion choice:

   Copy

   def rrf_fuse(vector_hits, keyword_hits, k_const: int = 60) -> list[Chunk]:
       scores: dict[int, float] = {}
       for hits in (vector_hits, keyword_hits):
           for rank, chunk in enumerate(hits):
               scores[chunk.id] = scores.get(chunk.id, 0) + 1 / (k_const + rank + 1)
       by_id = {c.id: c for c in [*vector_hits, *keyword_hits]}
       return [by_id[cid] for cid, _ in
               sorted(scores.items(), key=lambda x: -x[1])]
   

3. 3.

   Rerank the fused candidates, then cut hard. Retrieve wide (top ~30 fused), rerank with a cross-encoder that reads query and chunk together, keep the top 5–6 for the context. This is the single biggest precision win in most RAG stacks:

   Copy

   def rerank(query: str, chunks: list[Chunk], top_n: int = 6) -> list[Chunk]:
       resp = rerank_client.rerank(
           query=query, documents=[c.content for c in chunks], top_n=top_n)
       return [chunks[r.index] for r in resp.results if r.relevance_score > 0.3]
   

   The reranker's score threshold replaces pb-47's raw-similarity floor as your refusal trigger — it is a far better relevance signal.

4. 4.

   Upgrade citations from decoration to verification. Require claim-level bracketed citations (pb-47's [n] scheme), then check them mechanically: for each cited passage, verify n-gram overlap or run a cheap entailment check between the claim sentence and the cited chunk. Strip or flag claims whose citation does not support them. This turns "sources listed below" into "every claim traceable to a passage".

5. 5.

   Add a groundedness gate for high-stakes surfaces. For answers going to customers (vs internal users), run a second cheap model call: "Given only these passages, is each claim in this answer supported? Answer per-claim: supported / unsupported." Block or soften answers with unsupported claims. This is your first LLM-as-judge — pb-50 industrializes it.

6. 6.

   Re-run the eval and publish the comparison table. The deliverable that makes this engineering:

   Config	Hit rate	Answer correct	p95 latency	Cost/query
   Vector only (pb-47 baseline)	8/10	7/10	900ms	$0.004
   Hybrid (RRF)	9/10	8/10	950ms	$0.004
   Hybrid + rerank	10/10	9/10	1250ms	$0.006

   Keep whichever row your latency/cost budget can afford — with numbers, that is now a decision, not a debate.

Tool variants

• Postgres FTS + RRF (shown): zero new infrastructure; websearch_to_tsquery handles user-typed queries safely.
• Qdrant/Weaviate/Elastic hybrid: native hybrid search with fusion built in — right call if you outgrew pgvector in pb-47.
• Rerankers: Cohere Rerank / Voyage (hosted, one API call); BAAI/bge-reranker via sentence-transformers (local, free, GPU-hungry at scale).
• Ragas / promptfoo: automate the eval table once it exists — feeds directly into pb-50's CI gate.
• Anthropic-side citations: the API's document citations: {enabled: true} feature returns span-level cited text natively — worth evaluating as a replacement for hand-rolled [n] parsing when your context fits its document-block model.

Verification

• •
  The comparison table exists with all four columns measured on the same eval set as pb-47's baseline.
• •
  Exact-identifier queries ("error E4021") that failed on vector-only now hit via the keyword leg — check at least three.
• •
  Citation audit: sample 10 answers, manually verify every bracketed citation supports its claim; the mechanical checker agrees with your manual judgment ≥90%.
• •
  Off-corpus questions still refuse — reranker thresholds must not have broken the refusal path.

Pitfalls

• Normalizing scores instead of fusing ranks. Vector similarity and ts_rank live on incomparable scales; min-max normalizing them is fragile. RRF sidesteps it — use ranks.
• Reranking a narrow candidate set. Reranking the top 8 barely helps; the wins come from retrieving wide (30+) and letting the reranker find precision.
• Citation theater. Rendering [3] without checking that passage 3 supports the claim is worse than no citations — it launders hallucinations with fake provenance. Step 4's check is the point.
• Latency creep. Each stage adds time; the eval table's latency column keeps you honest. Rerankers batch — one call, not one per chunk.
• Tuning on the eval set until it memorizes. Hold out a few questions; if held-out performance diverges from tuned performance, you overfit your thresholds.

Restricted-environment variant

• Client tenancyPostgres FTS and RRF are in-database; use a local cross-encoder for reranking (no data leaves) and Bedrock/Azure for generation — full hybrid stack inside the boundary.
• Fully localbge-reranker-class models run on CPU for modest traffic; the groundedness gate works with a local judge model, with the caveat that judge quality drops — keep the human-audit loop.
• Compliance-heavy surfacesthe citation checker's output is your audit artifact — log claim→passage mappings with retention matching the underlying documents.

Success criteria

• •
  A measured, defensible retrieval config chosen from a comparison table — not the default you started with.
• •
  Exact-term queries and paraphrase queries both retrieve; the reranker threshold drives refusals.
• •
  Every shipped claim carries a citation that a mechanical check (and a spot-checking human) confirms.
• •
  The eval set has grown with real production failures and is wired to run on every config change — you are one step from pb-50's CI gate.