Upgrade Lab 05's dense retriever with BM25 + vector fusion and a cross-encoder reranker, benchmark two embedding models on a labeled retrieval set, and — most importantly — discover with your own numbers that "hybrid" is not automatically better.
On this lab's benchmark, naive RRF fusion hurts recall@4, while the same fusion used as a candidate generator for a reranker produces the best system. That distinction is the production insight employers probe for.
~2.5 hours hands-on · Advanced tier · $0 API spend (everything runs on CPU with open models; optional Ragas re-check ≤ $0.20).
Run in the Lab 05 working directory (corpus/ and qa.jsonl must exist — re-run Lab 05's make_corpus.py if not). Install rank_bm25 alongside sentence-transformers. Lab 05's questions share vocabulary with the documents, which flatters every retriever — so first generate a harder query set with deliberate vocabulary drift (how real users actually ask):
"""make_hard_queries.py — paraphrased queries with vocabulary drift.
Run after make_corpus.py. Writes qa_hard.jsonl."""
import json
HARD = [
"How much can I spend on a hotel room per night in Mumbai or Delhi?",
"My airfare looks pricey - at what ticket price does a VP need to sign off?",
"How much annual PTO do people get here?",
"If I don't use all my vacation, how much rolls over to January?",
"What do we get paid extra for carrying the pager on a Saturday?",
"Is there a limit on how often one person can carry the pager each quarter?",
"Eating out with a customer - what is the most I can put on the company card per day?",
"How long after buying something do I have before finance rejects the claim?",
"My screen keeps locking fast - what is the idle timeout on work machines?",
"How frequently do admin account passwords have to change?",
"Once a client leaves us, how long do we keep their personal data?",
"When do old backups get purged automatically?",
"How many days a week can I skip the office without asking anyone?",
"Is there money for setting up a desk at home when you join?",
"Big purchases need multiple vendor bids - what is the cutoff amount?",
"How quickly do we pay supplier bills after they arrive?",
"How much can each person spend on courses and books a year?",
"If I clear a certification exam, how much of the fee comes back to me?",
"A production outage just started - how fast must someone respond?",
"After an outage is over, what is the deadline for the writeup?",
"Which date does my pay land in the bank?",
"What do I get for referring a senior dev who gets hired?",
"How old does my laptop have to be before IT swaps it?",
"I left my work laptop in a cab - how soon do I have to tell IT?",
]
qa = [json.loads(l) for l in open("qa.jsonl")]
assert len(qa) == len(HARD)
with open("qa_hard.jsonl", "w") as f:
for row, q in zip(qa, HARD):
f.write(json.dumps({**row, "question": q}) + "\n")
print(f"wrote {len(HARD)} hard queries")
Note the design: "pager" for on-call, "PTO" for paid leave, "purged" for deletion. Gold labels carry over unchanged because the paraphrases never mention the answer values.
"""l6_bench.py — BM25 vs dense vs hybrid-RRF vs hybrid+rerank on qa_hard.jsonl."""
import json, pathlib, re
import numpy as np
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer, CrossEncoder
def chunk_recursive(text, size=700):
out, buf, parts = [], "", []
for para in text.split("\n\n"):
parts.extend([para] if len(para) <= size else re.split(r"(?<=\.)\s+", para))
for p in parts:
if len(buf) + len(p) + 1 > size and buf:
out.append(buf); buf = buf[-int(size * 0.15):]
buf = (buf + " " + p).strip()
return out + ([buf] if buf else [])
def score(ranked_ids, row, k=4):
for rank, ci in enumerate(ranked_ids[:k], 1):
if SRC[ci] == row["gold_doc"] and row["ground_truth"] in CHUNKS[ci]:
return 1, 1.0 / rank
return 0, 0.0
def report(name, ranked_per_q):
hits, rrs = zip(*(score(r, row) for r, row in zip(ranked_per_q, qa)))
print(f"{name:<28} recall@4={sum(hits)/len(qa):.2f} mrr={sum(rrs)/len(qa):.2f}")
CHUNKS, SRC = [], []
for p in sorted(pathlib.Path("corpus").glob("*.md")):
for c in chunk_recursive(p.read_text()):
CHUNKS.append(c); SRC.append(p.name)
qa = [json.loads(l) for l in open("qa_hard.jsonl")]
queries = [r["question"] for r in qa]
tok = [re.findall(r"[a-z0-9]+", c.lower()) for c in CHUNKS]
bm25 = BM25Okapi(tok)
bm_ranked = [np.argsort(-bm25.get_scores(re.findall(r"[a-z0-9]+", q.lower())))
for q in queries]
report("bm25", bm_ranked)
dense_ranked = {}
for mname in ("all-MiniLM-L6-v2", "BAAI/bge-small-en-v1.5"):
model = SentenceTransformer(mname)
emb = model.encode(CHUNKS, normalize_embeddings=True, show_progress_bar=False)
qemb = model.encode(queries, normalize_embeddings=True, show_progress_bar=False)
dense_ranked[mname] = [np.argsort(-s) for s in (qemb @ emb.T)]
report(f"dense {mname.split('/')[-1]}", dense_ranked[mname])
def rrf(rank_lists, k=60):
scores = np.zeros(len(CHUNKS))
for ranks in rank_lists:
for pos, ci in enumerate(ranks):
scores[ci] += 1.0 / (k + pos + 1)
return np.argsort(-scores)
best_dense = dense_ranked["BAAI/bge-small-en-v1.5"]
hybrid_ranked = [rrf([bm_ranked[i], best_dense[i]]) for i in range(len(qa))]
report("hybrid rrf(bm25+bge)", hybrid_ranked)
ce = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
rerank_ranked = []
for i, q in enumerate(queries):
cand = hybrid_ranked[i][:20]
ce_scores = ce.predict([(q, CHUNKS[ci]) for ci in cand])
rerank_ranked.append(cand[np.argsort(-ce_scores)])
report("hybrid20 + ce-rerank", rerank_ranked)
bm25 recall@4=0.46 mrr=0.46
dense all-MiniLM-L6-v2 recall@4=0.83 mrr=0.63
dense bge-small-en-v1.5 recall@4=0.92 mrr=0.78
hybrid rrf(bm25+bge) recall@4=0.62 mrr=0.55
hybrid20 + ce-rerank recall@4=0.96 mrr=0.82
Retrieve wide (fusion, k=20), rank narrow (cross-encoder, top 4) — and never report a fusion win without showing what fusion did at k=4 alone.
retrieval-upgrade-report.md: the five-row table (plus recall@20 column), latency notes, the two-finding analysis, and the recommended production config ("retrieve 20 via BM25+bge fusion, rerank with ms-marco cross-encoder, serve top 4"). Rubric (pass = 8/10):
k (60 here) and that both rank lists cover all chunks; with k=1-style fusion or truncated lists you built a different (sometimes luckier) fusion.ce.predict (it accepts the full list) rather than looping pairs.Every component here is open-weights and runs on-prem — BM25, both embedding models, and the cross-encoder — so this exact benchmark is safe to run directly on client corpora with zero data egress. Deliver the four-arm table on the client's own hard-query set (mine paraphrases from their support tickets); it converts a religious hybrid-vs-dense debate into a 90-second experiment.