Fine-Tune an Embedding Model for Domain Retrieval

Tier: Advanced · Workflow area: Fine-tuning & Model Adaptation

Contrastive fine-tuning of a small open embedder on mined (query, positive, negative) triples is a few hours of CPU/free-GPU work and routinely buys double-digit recall lift — this playbook does it end to end and proves the lift on your own benchmark.

When to use

• •
  Your retrieval benchmark shows recall@k failures concentrated on domain-specific queries, and a reranker did not close the gap.
• •
  Query and document vocabularies mismatch systematically (users say "customer churn", documents say "attrition/logo loss").
• •
  You control the RAG stack end to end, so you can re-embed the corpus when the model changes.

Skip if you have no retrieval benchmark yet — build the labeled (query → relevant chunks) set first; tuning an embedder without recall numbers is flying blind. Also skip if your corpus re-embeds are prohibitively expensive and hybrid search + reranking has not been tried.

Prerequisites

• •
  A retrieval benchmark: 100+ real queries with labeled relevant chunks (from logs, or built during your RAG-evaluation work), split into train-mining and held-out test.
• •
  sentence-transformers (v3+) and a small open base embedder — BAAI/bge-base-en-v1.5 or intfloat/e5-base-v2 class; base size trains comfortably on a Kaggle T4 and even on CPU overnight.
• •
  Your corpus in chunked form, identical chunking before and after (change one variable).

The workflow

1. 1.

   Freeze the baseline. Embed the corpus with the off-the-shelf model, run the held-out queries, record recall@5, recall@20, and MRR@10. This table is the "vs off-the-shelf" claim your report hinges on.

2. 2.

   Mine training triples from your own data. Positives: the labeled relevant chunks from the train split, plus (query, chunk) pairs users clicked or cited. Hard negatives: for each query, retrieve top-50 with the baseline model and take high-ranked chunks that are not labeled relevant — these near-misses are where the learning signal lives. Random negatives teach almost nothing.

3. 3.

   Audit for false negatives. Hard-negative mining's classic failure: a "negative" that is actually relevant but unlabeled. Skim 30 mined negatives; if more than ~10% look actually relevant, tighten mining (skip ranks 1–5, take ranks 10–50) or LLM-screen negatives with a relevance prompt and human-check a sample.

4. 4.

   Train with MultipleNegativesRankingLoss.

   Copy

   from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer, losses
   
   model = SentenceTransformer("BAAI/bge-base-en-v1.5")
   loss = losses.MultipleNegativesRankingLoss(model)
   # dataset columns: anchor (query), positive (chunk), negative (hard negative)
   

   Key settings: the largest batch size that fits (MNRL uses in-batch negatives — bigger batches = harder task = better model; 32–64 on a T4 for base-size), learning_rate=2e-5, 1–2 epochs, warmup 10%. Keep the query/passage prefix conventions of the base model (e5 needs query:/passage:; bge uses a query instruction) — dropping them silently degrades both baseline and tune.

5. 5.

   Evaluate during and after training. Wire an InformationRetrievalEvaluator on a validation slice so you can early-stop, then run the full held-out benchmark: same corpus, re-embedded with the tuned model, same queries, same metrics. Report deltas per query category — domain-jargon queries should jump, generic queries should hold.

6. 6.

   Check for general-capability collapse. A domain-tuned embedder can forget general English similarity. Run 20 generic sanity queries (paraphrase pairs, unrelated pairs) and confirm similarity orderings are still sane; if not, lower LR/epochs or mix 10–20% general triples (e.g., from MS MARCO) into training.

7. 7.

   Ship with a re-embedding plan. Version the model, re-embed the corpus, and record the pairing (embedder version ↔ index version) in config — mixed-version indexes produce silent garbage. Feed the tuned retriever back through your full RAG evaluation (pb-35) to confirm end-to-end answer-quality lift.

Tool variants

• sentence-transformers: the default shown above; trains base-size models on free GPUs or CPU.
• FAISS or pgvector benchmark rig: local, free index for the before/after recall runs; identical index parameters both runs.
• Managed alternatives (Cohere Rerank, Voyage): if fine-tuning is off the table, a strong reranker over the off-the-shelf retriever is the comparison point your report should include — sometimes it wins on ops-simplicity even when the tuned embedder wins on recall.

Verification

• •
  Recall@5 / recall@20 / MRR@10 reported before vs after on held-out queries that supplied no training triples.
• •
  Hard-negative audit note: false-negative rate in the mined set and the mitigation used.
• •
  Generic-similarity sanity check passed (no capability collapse).
• •
  Index/model version pairing recorded; end-to-end RAG metrics re-run with the tuned retriever.

Pitfalls

• Testing on training queries. Query leakage inflates recall dramatically; the held-out split must be by query, and near-duplicate queries deduplicated across splits.
• Random negatives. MNRL with only easy negatives converges fast and changes nothing in production ranking.
• Dropping instruction prefixes. e5/bge-family models are trained with prefixes; benchmarking either model without them corrupts the comparison in whichever direction you least expect.
• Re-chunking at the same time. Change embedder or chunking per experiment, never both — otherwise the recall delta is unattributable.
• Forgetting the corpus re-embed cost. Budget it: tuned embedder value includes paying re-embedding on every corpus refresh.

Restricted-environment variant

• •
  Everything here runs self-hosted by default — open base model, local training, FAISS/pgvector index — so client corpora never leave the boundary; this is the standard India services-delivery posture.
• •
  If even query logs are restricted, mine synthetic queries from documents with a self-hosted LLM ("write 3 questions this chunk answers"), then human-review a sample for realism.
• •
  Store triples with the same classification as the corpus: queries frequently contain client identifiers.

Success criteria

• •
  Measured recall lift on held-out domain queries (typical good outcome: +10–25 points recall@5 on jargon-heavy slices) with no generic-query regression.
• •
  End-to-end RAG faithfulness/answer-quality improvement confirmed, not just retrieval metrics.
• •
  A reproducible pipeline: mining script, training config, benchmark harness, all versioned.
• •
  A clear cost line: training time, re-embed time, and when the team should re-tune (corpus drift trigger).