Make the two embedding decisions every RAG data platform forces — which model and how to run it at scale — with evidence instead of leaderboard rank.
You will benchmark two open-source embedding models on the Lab 07 corpus golden set and watch the lower-MTEB model win on your data (it does, measurably, on this corpus); test Matryoshka-style dimension truncation and find the point where shrinking vectors is free versus where recall collapses; then build the production-shaped batch embedding pipeline: content-hash idempotency, retry with backoff, a dead-letter table, and a per-run cost log — proven by running it twice and seeing the second run embed zero chunks. Employers name "embedding selection" verbatim in data-engineering postings, and what they mean is exactly this: a benchmark harness on your own corpus plus a pipeline that can re-embed a changing corpus without re-paying for the unchanged 99%.
~120 minutes hands-on · Advanced tier (maps to module M5 · RAG Pipelines I — Ingestion, Chunking, Embeddings).
Fully local, $0. Requires Lab 07's folder (the handbook corpus and bakeoff.py are imported directly — run Lab 07 first). First run downloads two small models (~90 MB and ~130 MB) once.
mkdir embed-lab && cd embed-lab # sibling of chunk-lab
python3 -m venv .venv && source .venv/bin/activate
pip install sentence-transformers duckdb numpy
The contenders: all-MiniLM-L6-v2 (384-dim, the ubiquitous baseline) versus BAAI bge-small-en-v1.5 (384-dim, ranked well above MiniLM on MTEB, and requiring a model-specific query prefix — a real-world gotcha the harness must handle). The vector store is DuckDB with fixed-size FLOAT arrays and array_cosine_similarity — zero extra infrastructure; in production you would point the same pipeline at pgvector, Qdrant, or Chroma by swapping the two storage functions. The hosted-API slot (Voyage, OpenAI, or Cohere embeddings) is discussed in step 6; the offline lab compares two local models, which is also the compliance-preferred configuration.
Build the benchmark harness. Save benchmark.py — it reuses Lab 07's winning corpus, golden set, and structure-aware chunks so the only variable is the model:
"""Benchmark two open-source embedding models on YOUR golden set, plus a truncation test."""
import os, sys, time
import numpy as np
from sentence_transformers import SentenceTransformer
LAB07 = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "chunk-lab")
sys.path.insert(0, LAB07)
from bakeoff import chunk_structure, TEXT, GOLD # reuse corpus + golden set
CHUNKS = [c["text"] for c in chunk_structure(TEXT)]
MODELS = {
"all-MiniLM-L6-v2": {"name": "sentence-transformers/all-MiniLM-L6-v2", "query_prefix": ""},
"bge-small-en-v1.5": {"name": "BAAI/bge-small-en-v1.5",
"query_prefix": "Represent this sentence for searching relevant passages: "},
}
def recall_at(k, doc_emb, q_emb):
hits = 0
for qi, g in enumerate(GOLD):
top = np.argsort(-(doc_emb @ q_emb[qi]))[:k]
if any(g["anchor"].lower() in CHUNKS[i].lower() for i in top):
hits += 1
return hits / len(GOLD)
print(f"{'model':20s} {'dims':>5s} {'encode_s':>9s} {'recall@1':>9s} {'recall@3':>9s}")
for label, cfg in MODELS.items():
model = SentenceTransformer(cfg["name"])
t0 = time.perf_counter()
doc_emb = model.encode(CHUNKS, normalize_embeddings=True)
dt = time.perf_counter() - t0
q_emb = model.encode([cfg["query_prefix"] + g["question"] for g in GOLD], normalize_embeddings=True)
print(f"{label:20s} {doc_emb.shape[1]:>5d} {dt:>9.2f} {recall_at(1, doc_emb, q_emb):>9.2f} {recall_at(3, doc_emb, q_emb):>9.2f}")
for dims in (128, 64): # Matryoshka-style truncation: slice, renormalize, re-measure
d = doc_emb[:, :dims]; d = d / np.linalg.norm(d, axis=1, keepdims=True)
q = q_emb[:, :dims]; q = q / np.linalg.norm(q, axis=1, keepdims=True)
print(f"{' -> truncated':20s} {dims:>5d} {'-':>9s} {recall_at(1, d, q):>9.2f} {recall_at(3, d, q):>9.2f}")
Run it and confront the leaderboard. python benchmark.py. Reference results on this corpus:
model dims encode_s recall@1 recall@3
all-MiniLM-L6-v2 384 0.20 0.92 1.00
-> truncated 128 - 0.92 1.00
-> truncated 64 - 0.67 1.00
bge-small-en-v1.5 384 0.40 0.83 0.92
-> truncated 128 - 0.92 0.92
-> truncated 64 - 0.83 0.83
The result to sit with: bge-small ranks clearly higher than MiniLM on MTEB, yet loses on this corpus (0.83 vs 0.92 recall@1, and it fails to reach 1.00 even at recall@3) while taking twice the encode time. MTEB averages fifty-plus tasks; your corpus is one task, and short factoid-question retrieval over filler-heavy policy prose is simply a distribution MiniLM handles well. Neither result generalizes — that is the entire argument for benchmarking on your own golden set before committing a platform to a model.
Read the truncation table like a capacity planner. MiniLM at 128 dims: recall unchanged, vectors one-third the size — memory, disk, and ANN-search cost all drop ~3x for free. At 64 dims recall@1 collapses to 0.67 while recall@3 holds at 1.00 — which suggests the production pattern: coarse search at low dimension over millions of vectors, then rescore the top candidates at full dimension. Note the small-sample caveat honestly in your report: with 12 questions, one flip moves recall by 0.083 (bge appearing to improve at 128 dims is exactly that noise — a real evaluation wants 50+ questions before trusting single-point differences). Also note these two models are not Matryoshka-trained; truncation still works as a measurement here, but models trained with Matryoshka representation learning (nomic-embed-text, mxbai-embed-large class) are explicitly optimized to degrade gracefully and are the right choice when truncation is the plan.
Build the batch pipeline. Save embed_pipeline.py — the production shape, independent of which model won:
"""Production-shaped batch embedding pipeline: idempotent, retrying, dead-lettering, cost-logged.
Store: DuckDB with FLOAT[384] arrays (swap for pgvector/Qdrant/Chroma in production)."""
import hashlib, os, sys, time
import duckdb
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, os.path.join(HERE, "..", "chunk-lab"))
from bakeoff import chunk_structure, TEXT
MODEL_NAME = "sentence-transformers/all-MiniLM-L6-v2"
DIMS, BATCH, MAX_RETRIES = 384, 16, 3
def content_hash(text):
return hashlib.sha256(text.encode()).hexdigest()[:16]
def embed_batch(model, texts):
return model.encode(texts, normalize_embeddings=True).tolist()
def main():
con = duckdb.connect(os.path.join(HERE, "vectors.duckdb"))
con.sql(f"""create table if not exists embeddings (
chunk_id varchar primary key, content_hash varchar, model varchar,
vec FLOAT[{DIMS}], embedded_at timestamp default now())""")
con.sql("""create table if not exists dead_letter (
chunk_id varchar, error varchar, failed_at timestamp default now())""")
con.sql("""create table if not exists cost_log (
run_id varchar, model varchar, chunks_total int, chunks_embedded int,
chunks_skipped int, wall_ms int, est_tokens int, est_cost_usd double,
run_at timestamp default now())""")
chunks = [{"chunk_id": f"handbook-{i:03d}", "text": c["text"], "hash": content_hash(c["text"])}
for i, c in enumerate(chunk_structure(TEXT))]
known = dict(con.sql("select chunk_id, content_hash from embeddings").fetchall())
todo = [c for c in chunks if known.get(c["chunk_id"]) != c["hash"]] # idempotency gate
print(f"{len(chunks)} chunks, {len(todo)} need embedding, {len(chunks)-len(todo)} unchanged (skipped)")
from sentence_transformers import SentenceTransformer
model = SentenceTransformer(MODEL_NAME)
t0, embedded = time.perf_counter(), 0
for i in range(0, len(todo), BATCH):
batch = todo[i:i+BATCH]
for attempt in range(1, MAX_RETRIES + 1):
try:
vecs = embed_batch(model, [c["text"] for c in batch])
break
except Exception as e:
if attempt == MAX_RETRIES:
for c in batch:
con.execute("insert into dead_letter (chunk_id, error) values (?, ?)",
[c["chunk_id"], str(e)])
vecs = None
else:
time.sleep(2 ** attempt) # backoff before retry
if vecs is None:
continue
for c, v in zip(batch, vecs):
con.execute("delete from embeddings where chunk_id = ?", [c["chunk_id"]])
con.execute("insert into embeddings values (?, ?, ?, ?, now())",
[c["chunk_id"], c["hash"], MODEL_NAME, v])
embedded += len(batch)
wall_ms = int((time.perf_counter() - t0) * 1000)
est_tokens = sum(len(c["text"]) // 4 for c in todo)
est_cost = est_tokens / 1e6 * 0.02 # what a hosted small-embedding API would have billed (~$0.02/M tokens)
run_id = f"run-{int(time.time())}"
con.execute("insert into cost_log values (?, ?, ?, ?, ?, ?, ?, ?, now())",
[run_id, MODEL_NAME, len(chunks), embedded, len(chunks) - len(todo),
wall_ms, est_tokens, est_cost])
print(f"embedded={embedded} wall_ms={wall_ms} est_tokens={est_tokens} hosted-equivalent=${est_cost:.6f}")
print(con.sql("select run_id, chunks_embedded, chunks_skipped, wall_ms from cost_log order by run_at"))
q = model.encode(["how often are vector indexes refreshed?"], normalize_embeddings=True)[0].tolist()
print(con.sql(f"""select chunk_id, array_cosine_similarity(vec, cast(? as FLOAT[{DIMS}])) as sim
from embeddings order by sim desc limit 3""", params=[q]))
if __name__ == "__main__":
main()
Run it twice — idempotency is the test. First run: 24 chunks, 24 need embedding, 0 unchanged, then embedded=24 with wall time and the hosted-equivalent cost (~6,265 estimated tokens, ~$0.000125 at a $0.02/M-token hosted rate — trivially small here, decidedly not small at 50 million chunks, which is the point of logging it per run). The retrieval sanity check should return handbook-003 (the vector-index-refresh section) on top with similarity ~0.52. Second run, changing nothing: 24 chunks, 0 need embedding, 24 unchanged (skipped) and embedded=0 wall_ms=0. That second line is the difference between a pipeline and a script — re-runs cost nothing when nothing changed. Now edit one word inside one handbook section, re-run make_handbook and the pipeline, and confirm exactly the changed chunks re-embed (the content hash, not the filename or timestamp, is the change detector — the same mechanism that will drive CDC-to-vector sync later in the track).
Wire the hosted-API slot without using it. Add (or review) the seam that makes the pipeline provider-agnostic: embed_batch is the only function that touches a model, so a hosted implementation — Voyage, OpenAI, or Cohere embeddings over HTTPS — slots in behind the same retries, dead-letter, and cost log, with est_cost becoming a real invoice number from the provider's per-token price. Write into your report the decision table this enables: self-hosted wins on data boundary, per-token cost at scale, and offline capability; hosted wins on zero GPU ops, stronger frontier models, and elastic throughput; rate-limit-aware batching and retry budgets — which you already built — are what make the hosted path survivable.
Write the selection report. embedding-report.md: the benchmark table with the MTEB-vs-your-corpus finding stated plainly; the truncation recommendation (128-dim serving with full-dim rescore, plus the Matryoshka-trained-model caveat and the small-sample honesty note); the pipeline evidence (both cost_log rows proving idempotency, the dead-letter design, the one-word-edit experiment); and the standing decision rule — re-run this benchmark when the corpus, chunking, or candidate models change, because the golden set plus harness is now a 30-second regression test.
MTEB rank predicts your corpus about as well as a stranger's shoe size — benchmark on your golden set, and it costs one script. Then scale out behind a content-hash gate: an embedding pipeline that cannot skip unchanged work will eventually re-bill you for your entire corpus on every deploy.
benchmark.py runs offline (post-download) and reproduces the pattern: MiniLM ≥ 0.9 recall@1, bge-small below it on this corpus, bge query-prefix applied.cost_log.handbook-003) at rank 1.embedding-report.md contains the benchmark table, truncation recommendation with caveats, idempotency evidence, and the hosted-vs-self-hosted decision table.The embed-lab/ folder: benchmark.py, embed_pipeline.py, vectors.duckdb, and embedding-report.md. Self-grading rubric (pass = 8/10):
from bakeoff import ... fails: the path insert expects Lab 07 at ../chunk-lab — adjust LAB07/the sys.path line to wherever your Lab 07 folder lives.array_cosine_similarity unknown: upgrade DuckDB (pip install -U duckdb); fixed-size FLOAT arrays need a recent version.Embedding is where client text leaves the boundary wholesale — every chunk of every document goes to whatever model you chose, which is why regulated engagements default to exactly this lab's configuration: open-source models (sentence-transformers, bge, nomic-embed via Ollama) running inside the client's environment, benchmarked with this harness to prove the local model is adequate on the client's own golden set. If a hosted embedding API is proposed, the decision needs the client's data-protection sign-off, and warehouse-native embedding functions (Snowflake Cortex, BigQuery, Databricks) are the middle path — managed models inside the platform's existing governance boundary. Redact or synthesize the golden-set documents if even evaluation traffic is restricted; the benchmark table, cost log, and idempotency evidence are all client-presentable artifacts with no raw text in them.