Semantic Dedup, Clustering, and Topic Discovery with Embeddings

Tier: Core · Workflow area: Embeddings & Unsupervised Text Analysis

This playbook is the embed → reduce → cluster → LLM-label pipeline, plus semantic near-duplicate detection, that turns 50k free-text rows into a defensible topic taxonomy in an afternoon.

When to use

• •
  "What are people actually saying?" questions over 5k+ free-text records with no existing taxonomy.
• •
  Deduplicating semantically-identical records (paraphrased tickets, reposted listings, near-identical survey answers) where exact/fuzzy string matching fails.
• •
  Building a labeling taxonomy before training a classifier (pb-15) — clusters make better label sets than brainstorms.

Prerequisites

• •
  sentence-transformers, umap-learn, hdbscan, faiss-cpu (all free, laptop-runnable; GPU optional).
• •
  The corpus as a dataframe; light cleaning done (drop empties, strip boilerplate signatures/templates).
• •
  An LLM API key for cluster labeling only — a few hundred calls, not one per row.

The workflow

1. 1.

   Embed once, persist forever. Use a strong open model and normalize:

   Copy

   from sentence_transformers import SentenceTransformer
   model = SentenceTransformer("BAAI/bge-small-en-v1.5")   # or bge-m3 for multilingual
   emb = model.encode(texts, normalize_embeddings=True, batch_size=256, show_progress_bar=True)
   np.save("emb.npy", emb)  # embeddings are an asset; never recompute casually
   

2. 2.

   Semantic dedup with FAISS. Cosine similarity over normalized vectors; pairs above threshold are duplicate candidates:

   Copy

   import faiss
   index = faiss.IndexFlatIP(emb.shape[1]); index.add(emb)
   D, I = index.search(emb, k=5)         # top-5 neighbors per row
   pairs = [(i, j, d) for i, row in enumerate(I) for j, d in zip(row, D[i]) if j > i and d > 0.92]
   

   Calibrate the threshold by hand-reviewing 30 pairs at 0.88 / 0.92 / 0.96 — the right value is corpus-specific. Collapse duplicates via union-find, keeping the longest exemplar.

3. 3.

   Reduce, then cluster. HDBSCAN on raw 384–1024-dim vectors is slow and noisy; reduce with UMAP first:

   Copy

   import umap, hdbscan
   red = umap.UMAP(n_neighbors=15, n_components=5, metric="cosine", random_state=42).fit_transform(emb)
   labels = hdbscan.HDBSCAN(min_cluster_size=max(20, len(texts)//200),
                            min_samples=5).fit_predict(red)
   

   Sweep min_cluster_size over 3–4 values; prefer the granularity where clusters survive a change of random seed. Expect 10–30% noise (-1) — that is HDBSCAN being honest, not failing.

4. 4.

   LLM-label each cluster from exemplars. For each cluster, take the 10 members closest to the centroid + 5 random members, and request a structured label:

   Copy

   class ClusterLabel(BaseModel):
       name: str = Field(description="3-6 word topic name")
       description: str
       is_actionable: bool
       suggested_merge_with: str | None
   

   One call per cluster (50–300 calls total). Feed suggested_merge_with hints into a manual merge pass — LLMs are good at spotting that "billing error" and "wrong invoice amount" are one topic.

5. 5.

   Quantify and report. Topic share, topic-over-time trend if you have dates, dedup rate, and a 2-D UMAP scatter colored by cluster for the appendix. Report noise share explicitly rather than force-assigning it.

Tool variants

• Embeddings:bge-small/base-en-v1.5 (local, free) is the default; hosted embedding APIs (Voyage, OpenAI text-embedding-3-*, Gemini) trade cost for a few points of quality — check MTEB and your wallet (pb-23 covers rigorous selection).
• BERTopic: packages steps 1–4 (embed + UMAP + HDBSCAN + c-TF-IDF labeling) into one library; great for speed, use the manual pipeline when you need control over each stage.
• Cluster labeling model: a default-tier model (e.g. claude-opus-4-8) for labeling reads exemplars carefully; volume is low so tier cost barely matters here.
• Scale-out: past ~1M rows, swap FAISS flat for HNSW/IVF indexes and sample for UMAP fitting, transforming the remainder.

Verification

• •
  Dedup precision: ≥90% of 30 hand-reviewed candidate pairs at your chosen threshold are true duplicates.
• •
  Cluster stability: ≥70% of clusters re-appear (by member overlap >0.6) under a different random seed.
• •
  Label fidelity: for 10 random clusters, read 10 raw members each — the LLM label describes ≥8/10 members.
• •
  The topic taxonomy is mutually exclusive enough that a colleague assigns 20 fresh documents to topics with ≥80% agreement with you.

Pitfalls

• Clustering the reduced 2-D projection used for plotting. Reduce to ~5 dims for clustering and 2 dims for visualization separately; 2-D UMAP distorts density and manufactures clusters.
• Trusting LLM labels without reading members. The labeler only sees exemplars; a bad exemplar draw yields a plausible label for an incoherent cluster. Spot-read every cluster you will present.
• One giant "misc" cluster ignored. A dominant noise/misc share usually means mixed languages, templated text, or a min_cluster_size set too high — investigate before reporting.
• Embedding model mismatch across runs. Trend analyses comparing this quarter's clusters to last quarter's require the identical embedding model and version; pin it like a dependency.
• Dedup destroying signal. Duplicate count is data (the same complaint 400 times matters); keep counts as a weight column when you collapse.

Restricted-environment variant

• •
  The entire pipeline runs locally — embeddings, UMAP, HDBSCAN, FAISS are all on-prem by default; only step 4 (labeling) touches an API.
• •
  For fully air-gapped work, label clusters with a local 7–8B instruct model, or fall back to c-TF-IDF keyword labels (BERTopic default) and name topics manually — 300 clusters is an hour of human naming.
• •
  Free-compute path: bge-small on CPU embeds ~1k docs/min; Kaggle T4 does 50k docs in minutes. Zero API spend required except optional labeling (<₹100).

Success criteria

• •
  A topic taxonomy with names, descriptions, shares, and an explicit noise percentage, delivered from a corpus that had none.
• •
  Dedup pass with a calibrated threshold, measured precision, and duplicate counts preserved as weights.
• •
  Stability check performed and unstable topics either dropped or flagged.
• •
  Pipeline scripted end-to-end: new month's data → refreshed topics in under an hour of wall-clock time.