Flaky-Test Triage and Quarantine Proposals

Tier: Core · Workflow area: CI/CD & release engineering

When to use

• •
  Your team reruns failed CI "to see if it passes" more than a few times a week.
• •
  You have (or can export) per-test pass/fail history across runs.
• •
  You want a defensible quarantine list rather than ad-hoc @Skip annotations nobody revisits.

Skip this if failures are all genuine — flake triage on a healthy suite is procrastination.

Prerequisites

• •
  Test result history: JUnit XML artifacts from the last 2–4 weeks of runs, or a CI test-visibility export (Datadog CI Visibility, BuildPulse-class tools, GitLab test reports).
• •
  An AI assistant with a decent context window, or an LLM API for scripting.
• •
  A team-agreed quarantine mechanism (tag, skip-list file, separate suite) — agree on it before generating proposals.

The workflow

1. 1.

   Aggregate the evidence. Produce a per-test table: name, runs, failures, failure-on-retry-pass count (the flake signature), first-seen, last-seen. A short script over JUnit XML is enough:

   Copy

   # collect all junit xml from recent runs into ./results/, then:
   python3 flake_stats.py results/ --since 28d --out flake_report.csv
   

2. 2.

   Ask for classification with a structured output contract.

   Copy

   Below is 28 days of per-test CI history (CSV) plus sample failure messages
   per test. Classify each failing test as:
   - FLAKY (fails intermittently on unchanged code; passes on retry)
   - REGRESSION (started failing at a specific commit and stays failing)
   - ENV (failures correlate with runner/infra messages, not test logic)
   Output JSON: [{test, class, confidence, evidence, first_failing_commit?}].
   Evidence must quote the data — no unsupported claims. Mark confidence LOW
   when history is under 10 runs.
   

3. 3.

   Audit the borderline calls. Sort by confidence ascending. For every LOW item, check the raw history yourself — misclassifying a real regression as flaky is the expensive failure of this whole workflow.

4. 4.

   Generate quarantine PRs, one per test area. Have the assistant draft the PR: the skip/tag change, plus a mandatory issue link, owner, and un-quarantine criterion ("passes 50 consecutive runs on main"). No orphan skips.

   Copy

   Draft a quarantine PR for the 4 confirmed-FLAKY tests in payments-service.
   Include: the quarantine tag change, a tracking issue body per test with the
   flake evidence table, suggested owner from CODEOWNERS, and an explicit
   exit criterion. Keep the diff minimal.
   

5. 5.

   Human merges; pipeline enforces the exit. Add a scheduled job that runs quarantined tests nightly and reports — quarantine without a path back is deletion in slow motion.

Tool variants

• CI test-visibility products (Datadog CI Visibility, BuildPulse): already compute flake scores; use AI only for the evidence narrative and PR drafting on top.
• LLM API + scheduled job: script steps 1–2 to run weekly and post the report to your team channel; humans take it from there. This is the sustainable form.
• IDE assistant ad hoc: fine for a one-off cleanup sprint; paste the CSV and iterate interactively.

Verification

Apply the gate chain to the quarantine change like any infra change:

• Validate: the JSON classification parses and every test named actually exists in the suite.
• Plan: the PR diff touches only quarantine tags/lists — no test logic edits smuggled in.
• Policy: every quarantined test has an issue, an owner, and an exit criterion; regressions found in step 3 are filed as bugs, never quarantined.
• Cost/impact: measure retry-rate and pipeline duration for two weeks after; the quarantine should visibly reduce reruns or it was misdiagnosed.

Pitfalls

• Thin history. Ten runs cannot distinguish flaky from unlucky. Demand LOW confidence flags and defer those.
• Flake-labeling a regression. The single worst outcome. Cross-check first_failing_commit against the actual git log before accepting any FLAKY verdict that began at a specific commit.
• Quarantine graveyard. Without the nightly re-run job and exit criteria, quarantine lists only grow. Automate the exit check from day one.
• Env failures treated as test problems. Runner OOMs and registry timeouts need infra fixes; quarantining their victims hides the real signal.

Restricted-environment variant

Test names and failure messages can leak client domain details (endpoints, schema names). For client codebases under AI-tool restrictions: run the aggregation script locally, replace test identifiers with stable hashes before sending history to a hosted model, and map back locally — classification only needs the pattern, not the name. Or run the classification on a local model; the task is tabular pattern-reading, which mid-size local models handle acceptably when history is clean.

Success criteria

• •
  A ranked, evidence-backed flake report exists and refreshes automatically.
• •
  Retry rate and CI wall-clock time drop within two weeks of the first quarantine batch.
• •
  Zero regressions were quarantined (audited against subsequent bug reports).
• •
  Every quarantined test has an owner and at least one has already exited quarantine via the criterion.