Scope Regression by Risk — Mine Defect and Change History Instead of Guessing

Tier: Core · Workflow area: Test design & planning

This playbook correlates two datasets every team has and almost none uses — where bugs have historically clustered, and what changed in this release — into a risk-ranked regression scope. The model does the correlation and drafting; you own every inclusion and, more importantly, every exclusion.

When to use

• Time-boxed regression before a releaseyou have two days and a suite that takes five; something must be cut, defensibly.
• Recurring escape patternbugs keep shipping from areas your regression scope keeps skipping; the guessing is measurably failing.
• Building or rebalancing a suitedeciding where automation investment goes next deserves data, not folklore.

When NOT to use:

• Sparse or dirty historya young product or a tracker full of "Misc" components gives the model nothing real to mine; fix the taxonomy first.
• Safety-critical or contractual full-regression mandatesscope is not yours to cut; use the analysis to order execution instead.

Prerequisites

• Twelve months of resolved defects exportable from Jira with component/severity fields populated — the analysis inherits your taxonomy's quality.
• Repo access with historygit log over the release window; a monorepo needs path filters agreed up front.
• An approved assistantdefect data frequently contains customer references; scrub before pasting or use an internally approved endpoint.
• A release owner willing to sign the exclusion list — without that signature this playbook has no customer.

The recipe

1. 1.

   Export the two histories. Defects from Jira (12 months, resolved, with component/label and severity) and change data from git for the release window:

   Copy

   git log --since="6 weeks ago" --numstat --pretty="COMMIT|%h|%an|%s" \
     > changes.txt
   git diff --stat v2.14.0..HEAD -- 'src/**' > release-diff.txt
   

   From Jira, export the defect list as CSV with key, summary, component, severity, and linked area. Strip customer names and incident details before anything leaves approved tooling.

2. 2.

   Ask for a defect-density map first — one dataset at a time:

   Copy

   Here is 12 months of resolved defects (CSV). Produce a table:
   Component | Defect count | P1/P2 count | Trend (rising/flat/falling by
   quarter) | Recurring themes (2-3 words each).
   Note data-quality problems you see (miscategorized items, dumping-ground
   components) rather than silently working around them.
   

3. 3.

   Overlay the change history. Now the correlation prompt:

   Copy

   Here is the change log for release 2.15 (git numstat + commit subjects).
   Cross-reference against the defect-density table you produced.
   Output a regression-risk ranking: Area | Changed in 2.15? (which commits) |
   Historical defect density | Coupling risk (files changed together with
   historically buggy files) | Risk score 1-5 | One-line rationale.
   Flag areas that are HIGH history but NOT changed — candidates for a
   thin smoke pass, and say why they still carry risk (shared dependencies).
   

   A good ranking row is auditable back to its sources:

   Copy

   Payments/refunds | YES — 9 commits incl. a1b2c3 "rework refund state
   machine" | 31 defects/12mo, 6 P1s, rising | High: refund_service.py
   co-changes with ledger.py (14 P1-linked commits) | 5 | State machine
   rework in the historically worst component.
   

4. 4.

   Verify the load-bearing claims. Spot-check the commits it cites (git show a1b2c3 --stat), confirm the defect counts against a Jira filter, and check the co-change claim (git log --oneline -- src/refund_service.py src/ledger.py). Models occasionally attribute commits to the wrong area or double-count moved files — two minutes of verification protects the whole ranking's credibility.

5. 5.

   Cut the scope and write the exclusions down. Map risk scores to depth: 5 → full manual + automated pass; 4 → automated suite + targeted exploratory charter (pb-07); 3 → automated only; 1–2 → smoke or skip. The critical artifact is the exclusion list — every skipped area, its risk score, and a one-line acceptance rationale, signed by whoever owns the release.

6. 6.

   Close the loop after release. Any escape gets mapped against the scoping decision: was its area scored low, or scored high and tested anyway? Feed the answer into next release's prompt as calibration.

Make it yours

• CI-data enrichmentadd test-failure history (JUnit XML archives or ReportPortal exports) as a third dataset; areas where tests flake historically correlate with areas that break, and the model can weave all three into one ranking.
• Scheduled risk radarrun steps 1–3 weekly via API against a dashboard instead of per-release; risk scoping becomes a standing chart the release meeting already trusts by the time you need to cut scope.
• Test-to-feature mapping follow-uppair this playbook with embedding-based suite optimization (pb-21's territory) so a "risk 5" verdict translates automatically into the concrete list of test IDs to run.

Quality check

• Every risk-5 and risk-4 claim was source-verifiedcommits opened, Jira filters re-run; no headline number taken on faith.
• The exclusion list exists and is signedscope cutting without written risk acceptance is how this playbook becomes a blame machine.
• Unchanged-but-historically-hot areas got a smoke pass, not silence — shared-dependency regressions are the classic blind spot of change-based scoping.
• Data-quality flags were actionedif the model reported a dumping-ground component, a ticket exists to fix the taxonomy, or the next run inherits the same blur.
• Escape retrospective happenedeach post-release bug annotated with what the ranking said about its area; three releases of this and your prompt is measurably calibrated.