Benchmark Embedding Models on Your Own Corpus

Tier: Core · Workflow area: RAG data pipelines

Teams routinely pay 5x for a hosted model that a self-hosted BGE-class model matches on their domain — or worse, self-host a model that quietly loses 8 points of recall. This playbook builds a small harness that measures candidates on your corpus with your golden set, and factors in the dimensions-versus-storage trade-off that decides your vector-store bill.

When to reach for this

• •
  Standing up a new RAG pipeline and choosing the embedding model deliberately for the first time.
• •
  Evaluating a switch: hosted (text-embedding-3-large class) versus self-host (BGE-M3 class), or a new model release versus the incumbent.
• •
  Vector storage or embedding API cost is material and you want to test dimension truncation (Matryoshka) before paying for full-width vectors.
• •
  Your corpus is multilingual or jargon-heavy — exactly where leaderboard rank and domain reality diverge most.

The workflow

1. 1.

   Reuse the retrieval golden set from your chunking eval (pb-28): 30–50 questions mapped to gold documents. Same set, same chunks — the only variable this time is the model.

2. 2.

   Shortlist 3–4 candidates across the trade-off space. A sane 2026 bracket: one small local model (bge-small class, 384-dim, fast), one strong local multilingual model (BGE-M3 class, 1024-dim), one hosted model (text-embedding-3-large class or provider equivalent). Record dimension, max sequence length, license, and per-1M-token price for each.

3. 3.

   Embed the same chunk set with each candidate and index separately.

   Copy

   from sentence_transformers import SentenceTransformer
   import chromadb
   
   candidates = ["BAAI/bge-small-en-v1.5", "BAAI/bge-m3"]  # hosted APIs join via their SDKs
   client = chromadb.PersistentClient("embed_eval")
   
   for name in candidates:
       model = SentenceTransformer(name)
       vecs = model.encode([c["text"] for c in chunks], normalize_embeddings=True, batch_size=64)
       col = client.get_or_create_collection(name.replace("/", "_"))
       col.add(ids=[c["id"] for c in chunks], embeddings=vecs.tolist(),
               metadatas=[{"doc_id": c["doc_id"]} for c in chunks])
   

   Expected output: one collection per model over identical chunks. Embed queries with the same model as the collection — cross-model query/index mismatches score near zero and are a classic harness bug.

4. 4.

   Score recall@10 and MRR@10 per model. These are the metrics that prove the step worked: recall@10 for "did the right document surface at all", MRR for "how high did it rank".

   Copy

   def mrr_at_k(col, model, golden, k=10):
       total = 0.0
       for item in golden:
           res = col.query(query_embeddings=model.encode([item["q"]], normalize_embeddings=True).tolist(), n_results=k)
           for rank, m in enumerate(res["metadatas"][0], 1):
               if m["doc_id"] == item["doc_id"]:
                   total += 1 / rank
                   break
       return total / len(golden)
   

   Expected output: bge-small: R@10 0.78 / MRR 0.61 · bge-m3: 0.86 / 0.70 · hosted-large: 0.87 / 0.72. If the hosted premium buys one recall point, that is a finding worth money.

5. 5.

   Test Matryoshka truncation before buying full-width storage. Many modern models are trained so the first N dimensions carry most of the signal. Truncate, re-normalize, re-score:

   Copy

   import numpy as np
   short = vecs[:, :256]
   short = short / np.linalg.norm(short, axis=1, keepdims=True)
   

   Measure recall@10 at 256-dim versus full-dim. A common production pattern: coarse ANN search at 256-dim, rescore the top 100 at full dimension — near-full recall at a quarter of the index memory. Your harness now proves or disproves that for your corpus.

6. 6.

   Decide on a scorecard, not a single number. Columns: recall@10, MRR, dimension, index size for your corpus, embed throughput (chunks/sec measured in step 3), unit cost, license, multilingual coverage. Freeze the choice, record the scorecard, and note the model version — hosted models change under stable names.

Verify before you trust it

• •
  Identical chunks, identical golden set, identical k across all candidates — diff the collection sizes to confirm.
• •
  Margins beat noise: on a 40-question set, differences under ~5 points of recall are a tie; prefer the cheaper/simpler model on a tie.
• •
  Slice by language and document type if the corpus is mixed — an aggregate tie can hide a 15-point multilingual gap, which is where BGE-M3-class models earn their keep.
• •
  Re-run one model twice end to end; recall must reproduce exactly. If not, fix the harness (usually non-determinism in chunk IDs) before trusting any number.

Pitfalls

• Choosing by MTEB rank alone. Leaderboard position predicts your-domain performance weakly; a 10-place gap on MTEB regularly inverts on niche corpora. The golden set is the tiebreaker.
• Ignoring max sequence length. A 512-token model silently truncates your 800-token chunks and recall degrades mysteriously. Match chunking (pb-28) to model limits.
• Forgetting the re-embedding tax. Switching models later means re-embedding the entire corpus (see pb-30) and a versioned index cutover (pb-36). Weigh that against a one-point win today.
• Benchmarking on unnormalized vectors with cosine assumptions. Normalize consistently at index and query time; mixed normalization skews similarity scores and can flip rankings.
• Treating hosted model names as stable. Providers revise weights behind the same name. Pin versions where offered, and keep the harness ready to re-run on provider change notices.

Tools

• sentence-transformerslocal embedding and evaluation workhorse; runs the entire harness offline on CPU for small corpora.
• MTEB harnessfor shortlisting candidates and adding your corpus as a custom task if you want standardized reporting.
• Chroma / pgvectoreval-time store; the scorecard transfers directly to the production store decision (pb-31).

Client-safe variant

Model benchmarking sends every chunk of the corpus through the embedding model, so with client data the local candidates (bge-small, BGE-M3 via sentence-transformers) are the default: the full harness runs inside the client boundary with no tokens leaving. Include hosted candidates only after written approval naming the provider, and note that embedding APIs typically retain nothing but still constitute data transfer under most DPAs. A useful compromise: run the local-versus-local comparison on real client data, run the local-versus-hosted comparison on a public look-alike corpus, and present both scorecards so the client can decide whether the hosted delta justifies a data-processing amendment.