Resolve Duplicate Entities with Embeddings and LLM Adjudication

Tier: Advanced · Workflow area: Unstructured data processing

The modern pattern is two-stage: embeddings do the cheap, wide work of blocking (finding candidate pairs worth comparing), and an LLM does the expensive, careful work of adjudicating each candidate pair with a structured verdict and confidence. A human review band sits between auto-merge and auto-reject. Done right, this reaches accuracy hand-tuned rule systems struggle to match, at a fraction of the rule-maintenance burden.

When to reach for this

• •
  Customer/vendor/product masters with duplicates that fuzzy matching alone can't safely merge.
• •
  Merging entity data across systems (CRM + billing + support) with no shared keys.
• •
  An existing rules-based MDM match engine has become unmaintainable and misses obvious matches.

Don't LLM-adjudicate what deterministic keys resolve — if a GSTIN/PAN/tax-ID match exists, that's the verdict. The LLM is for the residue.

The workflow

1. 1.

   Normalize, then embed every entity record. Concatenate the identity-bearing fields into a canonical string ("name | address | city | phone-last-4") and embed once per record, storing vectors alongside IDs:

   Copy

   text = f"{r.name} | {r.address} | {r.city} | {r.phone[-4:] if r.phone else ''}"
   

   Any solid embedding model works; store in pgvector so blocking is a SQL query.

2. 2.

   Block via similarity search. For each record, retrieve its top-k neighbors above a loose similarity floor:

   Copy

   SELECT b.entity_id, 1 - (a.emb <=> b.emb) AS sim
   FROM entities a JOIN entities b ON a.entity_id < b.entity_id
   WHERE a.entity_id = :id
   ORDER BY a.emb <=> b.emb
   LIMIT 10;
   

   Tune the floor on a labeled sample for recall, not precision — a true duplicate pair missed at blocking is unrecoverable, while a false candidate just costs one LLM call. Blocking should collapse an O(n²) comparison space into a candidate list ~10–50x the entity count.

3. 3.

   Define the verdict schema. Structured output with an explicit decision, confidence, and cited evidence:

   Copy

   from pydantic import BaseModel
   from typing import Literal
   
   class MatchVerdict(BaseModel):
       decision: Literal["same_entity", "different_entity", "unsure"]
       confidence: float
       evidence: str          # which fields drove the decision
   

4. 4.

   Adjudicate candidate pairs in a loop over the API. Give the model both records verbatim and domain rules that matter (abbreviation conventions, city synonyms, what a legal-suffix difference means):

   Copy

   import anthropic
   
   client = anthropic.Anthropic()
   for pair in candidate_pairs:            # checkpointed loop; skip already-judged pairs
       response = client.messages.parse(
           model="claude-opus-4-8",
           max_tokens=1024,
           thinking={"type": "adaptive"},
           messages=[{
               "role": "user",
               "content": (
                   "Are these two records the same real-world company?\n"
                   f"A: {pair.a.to_json()}\nB: {pair.b.to_json()}\n"
                   "Rules: legal suffixes (Pvt Ltd / Private Limited) are equivalent; "
                   "'Vizag' and 'Visakhapatnam' are the same city; different GSTINs "
                   "mean different entities regardless of name similarity. "
                   "If evidence is thin, answer 'unsure' rather than guessing."
               ),
           }],
           output_format=MatchVerdict,
       )
       save_verdict(pair, response.parsed_output)
   

   Expected output per pair: a MatchVerdict like {"decision": "same_entity", "confidence": 0.94, "evidence": "identical address and phone; name differs only in legal suffix"}.

5. 5.

   Apply the gate: thresholds and the review band. Schema is enforced by parse(); decisions route three ways — same_entity with confidence ≥ 0.9 auto-merges, different_entity ≥ 0.9 auto-rejects, and everything else (including all unsure) lands in the human-review queue. Reviewer decisions append to a labeled-pairs table that becomes your calibration set and few-shot pool.

6. 6.

   Cluster and survive. Pairwise merges compose into clusters via connected components (watch for chains: A=B, B=C, but A≠C — flag such clusters for review rather than transitively merging). Survivorship — which record's values win in the golden record — stays a deterministic, documented rule set, not an LLM decision.

7. 7.

   Measure against the labeled sample. Hold out 200 human-labeled pairs (stratified across easy/hard). Report precision and recall of auto-decisions and the review-band rate. Tune the two confidence thresholds on this set — precision of auto-merge is the number that protects you from wrongly merging two customers' billing.

Verify before you trust it

• Auto-merge precision≥ 99% on the held-out labeled pairs; report recall alongside.
• Blocking recall auditedfor a labeled duplicate set, what fraction of true pairs survived blocking? Below ~98%, loosen the floor or raise k.
• Chain-merge checkno cluster auto-merged with internal contradictory verdicts.
• Idempotencyre-running adjudication on the same pairs produces the same decisions table.

Pitfalls

• Tuning blocking for precision. Cheap false candidates are fine; missed true pairs are permanent. Blocking optimizes recall, adjudication optimizes precision.
• No "unsure" option. Forced binary verdicts convert genuine ambiguity into confident coin-flips that pass your thresholds.
• Transitive merges without checks. Chains through borderline pairs quietly merge distinct entities; inspect clusters, not just pairs.
• Letting the LLM decide survivorship. Merge/no-merge is judgment; which address wins is policy. Keep policy deterministic.
• Re-adjudicating unchanged pairs. Cache verdicts by pair-hash; only re-run when a record's identity fields change or the prompt version bumps.

Tools

• pgvectorembedding storage and similarity blocking in plain SQL; free and everywhere.
• Anthropic/OpenAI batch APIs + structured outputsthe adjudication loop with schema-enforced verdicts.
• Qdrantdedicated vector store alternative when the entity set outgrows Postgres-side search.

Client-safe variant

Entity masters are concentrated PII — names, addresses, phone numbers — so this workflow needs explicit data-processing approval before any external API sees a record pair. Prefer running adjudication inside the client's perimeter: warehouse-native AI functions, or a self-hosted model via vLLM for the verdict step (the schema, thresholds, and review band are identical; budget for lower accuracy and a wider review band). Minimize what each prompt carries — identity fields only, never full account history. The labeled-pairs set and review queue stay client-side, and the deliverable includes the threshold calibration report and survivorship rules doc so the client's MDM team can re-tune as their data evolves.