Embedding-Model Selection: MTEB Shortlist to Domain Benchmark

Tier: Core · Workflow area: RAG & Retrieval Engineering

This playbook builds a small labeled retrieval benchmark from your corpus, evaluates an MTEB-informed shortlist on it, and folds in the dimensions everyone forgets: cost per million tokens, vector storage footprint, and latency.

When to use

• •
  After pb-22 fixed chunking — embedding choice is the next variable in the retrieval stack.
• •
  Your corpus is domain-heavy (medical, legal, fintech, code, Hinglish/multilingual) where general-purpose rankings are least predictive.
• •
  You're weighing hosted embeddings vs self-hosted, or considering re-embedding an existing index and need to justify the migration cost.

Prerequisites

• •
  The pb-21/22 harness and the winning chunk configuration (frozen).
• •
  sentence-transformers, mteb (optional, for public-task sanity checks), FAISS/Qdrant for per-model indexes.
• •
  100–300 labeled (query → relevant chunk) pairs. You have ~50 from pb-21; mine the rest (below).

The workflow

1. 1.

   Expand the labeled set cheaply. Three sources: (a) pb-21's grounding set; (b) mined pairs — take section headings/FAQ questions that naturally point at passages; (c) LLM-generated queries per chunk, filtered by a round-trip check (generate query from chunk → does BM25 rank that chunk top-3 for the query? keep : discard). Target ≥100 pairs, with ≥30% human-written; mark provenance so you can slice by it.

2. 2.

   Shortlist 4–6 models spanning the real trade-off axes. From current MTEB retrieval standings, pick across: small-local (bge-small-en-v1.5, 384d), mid-local (bge-base/gte-base, 768d), large-local (bge-m3 for multilingual/long-doc, 1024d), hosted (text-embedding-3-large, voyage-3-class), and — if latency-sensitive — one Matryoshka/truncatable model to test dimension cuts.

3. 3.

   Index and evaluate identically. One index per model over the same chunks; same top-k:

   Copy

   for name, model in shortlist.items():
       emb = model.encode(chunks, normalize_embeddings=True)
       q_emb = model.encode(queries, normalize_embeddings=True)   # use query prompts/prefixes where the model requires them!
       scores[name] = recall_mrr_ndcg(q_emb, emb, gold, ks=(1, 5, 10))
   

   Respect each model's query/passage prefix conventions ("query: ...", instruction prompts) — omitting them is the most common way benchmarks silently misrank models.

4. 4.

   Add the economics columns. For each model: $/1M tokens to embed (0 for local + GPU-hour cost), vector bytes × chunk count (index RAM/storage), query encode latency p95, and re-embedding cost for your full corpus. A model 1 point better at 2.7× the dimension count is often a net loss at scale.

5. 5.

   Test dimension truncation where supported. Matryoshka-trained models let you cut 1024→512→256 dims; measure recall at each cut. A 2× storage/latency saving for <1 point of recall is frequently the right trade.

6. 6.

   Decide per-slice, then commit. Slice recall by query provenance and document type. Ship the winner; record model name and revision hash in the index metadata (pb-25 will enforce this), because "which embedder built this index" must never be archaeology.

Tool variants

• Local serving: sentence-transformers on CPU is fine to ~1M chunks of batch embedding; a T4 raises throughput ~20×. TEI (text-embeddings-inference) for production self-hosted serving.
• Hosted: OpenAI/Voyage/Cohere/Gemini embeddings — fold rate limits into the latency column; batch endpoints for the initial index build.
• mteb library: run your shortlist on 2–3 public retrieval tasks nearest your domain as a sanity check that your harness reproduces published orderings — if it doesn't, suspect the prefix handling.
• Fine-tuned embeddings (pointer): if no off-the-shelf model clears your bar, contrastive fine-tuning on mined triples is the next escalation — a later-track topic; this benchmark becomes its training/eval data.

Verification

• •
  Public-task sanity check reproduces expected model ordering (harness validated).
• •
  Winner beats the incumbent on recall@5 outside the bootstrap CI on the human-written query slice — the slice that predicts production.
• •
  Economics table complete; the decision cites cost and latency, not just quality.
• •
  Round-trip-filtered synthetic queries comprise ≤70% of the set, and results hold when scored on human queries alone.

Pitfalls

• Missing query prefixes/instructions. bge/gte/e5-family models trained with prefixes lose several recall points without them — enough to invert rankings.
• Synthetic-query echo. LLM queries generated from chunks share vocabulary with those chunks, flattering dense retrieval; the human-written slice is your guard.
• Benchmarking on the wrong chunks. Chunk config changes move embedding rankings; always benchmark on the pb-22 winner, and re-check if chunking later changes.
• Ignoring index growth. 1024d float32 × 5M chunks ≈ 20 GB of vectors before HNSW overhead; quantization (int8/binary) is available in Qdrant/pgvector but re-benchmark recall after enabling it.
• Upgrading for the leaderboard. A new #1 on MTEB is a reason to rerun this playbook (an hour, now that it exists) — not a reason to migrate.

Restricted-environment variant

• •
  Local-only shortlists (bge/gte/e5/m3 families) keep client text fully on-prem; hosted contenders can be benchmarked on a public stand-in corpus to bound the gap without exposing data.
• •
  The labeled pair set derived from client documents stays in the engagement environment; the harness code is reusable across clients.
• •
  GPU-free path: bge-small on CPU handles the full benchmark for a 10k-chunk corpus in well under an hour — no spend at all.

Success criteria

• •
  A labeled domain retrieval benchmark (≥100 pairs, provenance-tagged) that outlives this decision and powers future ones (reranker choice, fine-tuning).
• •
  A model decision backed by per-slice quality + cost + latency + storage, committed with model name and revision in index metadata.
• •
  Measured, documented effect of dimension truncation (or a note that the chosen model doesn't support it).
• •
  Rerunning the whole selection on a new candidate model takes under one hour.