Regression suites grow by accretion — every incident adds tests, none retire — until runtime, not risk, decides what gets tested. Embeddings make suite overlap measurable: semantically near-duplicate tests cluster together, tests map to features for change-based selection, and cuts happen against an explicit risk budget instead of a hunch.
Tier: Advanced · Workflow area: Triage, flakes & defect intelligence
This playbook embeds your test corpus, finds redundancy candidates and coverage gaps, builds a test-to-feature map for change-based selection, and retires tests through a coverage-guarded process — never on similarity score alone.
text-embedding-3-small, Voyage, or local via Ollama) and a few lines of Python — no vector database required at this scale.Extract test units. One record per test: identifier, source text, file path, tags, runtime, last-5-runs status. A 50-line script over your test tree; keep it in the repo — you will rerun this quarterly.
Embed and cluster.
# embed_suite.py — sketch
embs = client.embeddings.create(model="text-embedding-3-small",
input=[t["text"] for t in tests])
# cosine similarity matrix; pairs above 0.90 -> duplicate candidates
# agglomerative clustering at 0.80 -> semantic groups per feature area
Output two artifacts: a ranked list of high-similarity pairs (duplicate candidates) and cluster membership (the raw material for the feature map).
Review duplicate candidates with an assistant — semantically, not numerically. Similarity flags candidates; a language model plus a human decide:
These two tests scored 0.94 cosine similarity (sources attached).
Answer: (a) do they exercise the same behavior path, or same setup with
DIFFERENT assertions? (b) what does each catch that the other would miss?
(c) verdict: TRUE-DUPLICATE / OVERLAPPING-KEEP-BOTH / DISTINCT.
Quote the assertion lines that justify the verdict.
Expect the humbling case: 0.94 similarity, but one asserts the happy total and the other asserts the rounding edge — keep both. Verdicts are drafted by the model, confirmed by a reviewer who reads the quoted assertions.
Guard every retirement with coverage evidence. A test dies only if the survivors cover its ground:
pytest --cov=src --cov-report=json tests/keep_set/ # coverage without the candidate
python scripts/coverage_diff.py candidate=tests/checkout/test_totals_v2.py
# -> "0 lines uniquely covered by candidate" = safe; anything else = keep or port asserts
Retire in a dedicated PR listing each test, its duplicate-verdict, and its coverage-diff result. Mutation-test the survivor set on affected areas if you want the stronger guarantee that assertion power, not just line coverage, survived the cut.
Build the test-to-feature map for change-based selection. Embed feature descriptions (or module READMEs/paths) in the same space and assign each test cluster to its nearest features. Store the map as data:
{ "tests/checkout/test_totals.py::test_vat_rounding":
["pricing-engine", "checkout-flow"] }
Wire selection into CI: diff touches src/pricing/** → run mapped clusters plus the always-on smoke tier. Uncovered-by-map diffs run the full suite — the map must fail safe.
Find the inverse signal: gaps. Clusters of app features with zero mapped tests, and incident postmortems whose embedded descriptions land far from every test cluster, are your coverage holes — feed them to the test-design playbooks. The same matrix that finds redundancy finds absence.
Set the risk budget explicitly and re-run quarterly. Decide, in writing, the target: e.g. "PR gate ≤15 min at ≥95% historical-defect recall; nightly runs everything." Validate selection against history — replay the last quarter's defect-introducing commits and confirm the selected subset would have caught them. Re-embed as the suite and app evolve.
Embeddings measure semantic distance, not importance. They are superb at surfacing candidates — duplicates, mappings, gaps — and blind to why a strange-looking test exists (that regression test for the 2023 Diwali-sale incident looks like an outlier and is untouchable). Coverage diffs, defect-history replay, and a human verdict stand between similarity scores and the delete key.
Test source often embeds client business logic in its scenario names and data. For client-owned suites, use client-approved embedding endpoints or run local embedding models; never ship the corpus to a personal account. The artifacts this playbook produces — duplicate ledger, feature map, risk-budget document — are exactly what client audits ask for when you propose cutting their regression suite: keep them in the engagement's document store, with the coverage diffs attached to every retirement PR.
Publish the duplicate ledger internally before deleting anything. Authors defending "their" flagged tests will hand you exactly the missing context — the incident it guards, the subtle assert difference — that no similarity score contains. The argument is the review.