"Acme Corp", "ACME Corporation Pvt Ltd" and "Acme Corp. (do not use)" are one customer wearing three numbers. This playbook finds duplicate customers, vendors and materials with embedding similarity, adjudicates the borderline pairs with LLM reason codes — and keeps every merge decision human, because merges rewrite history.
Tier: Advanced · Workflow area: ERP Data & Close · ~45 minutes
Classic dedupe tooling matches strings; duplicates are made of meaning. Fuzzy matching catches "Acme Corp"/"Acme Corp." and misses "IBM"/"International Business Machines", transliterations, and the vendor rekeyed after a typo in the tax ID. Embeddings compare records in meaning-space, which finds the semantic pairs — and produces a gray zone in the middle that neither threshold can settle. That gray zone is where the LLM earns its seat: reading both records and issuing a recommendation with reason codes, so the human who owns the merge decides fast and on evidence.
Extract and canonicalize a matching view. Pull the candidate object (say vendors) into Postgres with the fields that carry identity — name, street, city, country, tax IDs, bank account (hashed), contact email domain — and a matching_text column concatenating the meaningful ones in a fixed order, lowercased, legal-form noise ("pvt ltd", "gmbh", "inc") normalized but not removed. Keep the raw fields; the LLM step needs them intact.
Embed and index with pgvector.
CREATE EXTENSION IF NOT EXISTS vector;
ALTER TABLE vendor_match ADD COLUMN emb vector(1024);
-- Populate emb via your embedding model over matching_text
-- (batch script; tenant-approved endpoint for real party data).
CREATE INDEX ON vendor_match USING hnsw (emb vector_cosine_ops);
Then generate candidate pairs — nearest neighbors above a loose floor, blocked by country to keep the pair count sane:
SELECT a.vendor_id, b.vendor_id,
1 - (a.emb <=> b.emb) AS sim
FROM vendor_match a
JOIN LATERAL (
SELECT vendor_id, emb FROM vendor_match b
WHERE b.vendor_id > a.vendor_id AND b.country = a.country
ORDER BY b.emb <=> a.emb LIMIT 5
) b ON TRUE
WHERE 1 - (a.emb <=> b.emb) > 0.60;
Calibrate thresholds on a labeled sample. Hand-label 100–200 pairs across the similarity range as dup / not-dup / unsure. Plot the labels against the score and choose two cut lines: above the upper (say 0.92) pairs are near-certain duplicates that still get human confirmation but no LLM step; below the lower (say 0.72), discard. The band between is the adjudication queue. Calibration on your data is mandatory — thresholds do not transfer between datasets or embedding models.
Adjudicate the gray zone with reason codes. Per pair, the LLM sees the full raw records and answers in a fixed schema:
Compare these two vendor records and recommend one of:
MERGE / DISTINCT / ESCALATE (cannot determine).
Justify using ONLY these reason codes (any that apply):
RC1 name variation of same legal entity RC2 same tax ID
RC3 address match/adjacent RC4 same bank account
RC5 tax IDs conflict RC6 different legal form
RC7 successor/renamed entity RC8 franchise or group
sibling suspected
Output JSON: {recommendation, reason_codes[], evidence: one quote
from each record per code, confidence: high|medium|low}.
Rule: RC5 present => never MERGE; recommend ESCALATE.
[record A] [record B]
Hard business rules (RC5's veto, bank-account matches always escalate to fraud-aware review) live in your post-processing, not in model goodwill — parse the JSON and enforce them in code.
Route to human deciders in worklist form. Build the review queue sorted by confidence and business weight (open PO value, open items, transaction recency): recommendation, reason codes, side-by-side records, evidence quotes. A reviewer with reason codes processes a pair in under a minute; the same pair from a bare similarity score takes five. Record every human verdict — agree/override — because that log recalibrates thresholds and audits the model's judgment over time.
Execute merges through the ERP's own machinery, then measure. Approved merges run through sanctioned channels — SAP MDG consolidation or your governed merge process, never direct table surgery — with the survivor record chosen by your data-quality rules and the loser blocked-and-referenced, not deleted. Close the loop with numbers: duplicates confirmed, false-positive rate from reviewer overrides, credit exposure re-consolidated. Then schedule the rerun — dedupe is a cycle, not an event, until intake validation (search-before-create on the same embedding index) closes the tap.
Party master data is dense PII and commercially sensitive. Embeddings and adjudication prompts must run on tenant-approved endpoints under your data-processing agreement — a vendor list with bank details in a consumer chat tool is a reportable incident, not a workflow.
Embeddings find what string-matching cannot, the LLM makes the gray zone reviewable at speed, and the human merge decision keeps the one irreversible step accountable. Score, reason codes with quoted evidence, human verdict — that funnel is the whole design.