Diagnose Flaky Tests by Failure Signature

Tier: Core · Workflow area: Triage, flakes & defect intelligence

This playbook takes a set of intermittently failing tests, classifies each by failure signature with an assistant, verifies the classification with a designed rerun experiment, and routes each class to its standard fix — with retries-as-medication explicitly banned.

Prerequisites

• •
  Failure evidence per flaky test: CI logs, stack traces, JUnit XML, and (for UI tests) Playwright traces or screenshots across multiple failing runs.
• •
  The ability to rerun tests with controlled variations: repeat counts, serial vs parallel, fresh vs reused environment.
• •
  An assistant with a large context window (Claude/ChatGPT) for multi-log analysis.
• •
  A quarantine mechanism and a tracker for flake tickets.

The workflow

1. 1.

   Assemble evidence per suspect — multiple failures, not one. A single failure is an anecdote; three failing runs plus one passing run per test is a dataset. Pull the JUnit XML and logs for each.

2. 2.

   Ask for classification against the standard taxonomy. Constrain the model to the categories and demand evidence quotes:

   Copy

   Here are 4 runs (3 fail, 1 pass) of "order-history pagination" — JUnit XML,
   stack traces, and test source attached.
   Classify the flake into exactly one primary category:
   ASYNC/WAIT (races, missing waits, animation timing)
   DATA (shared state, leftover records, order-dependent data)
   ENVIRONMENT (ports, containers, clock, resources, third-party deps)
   ORDER (passes alone, fails after specific tests)
   CONCURRENCY (parallel workers colliding)
   For your classification: quote the exact log/trace lines that support it,
   state confidence (high/med/low), name the most likely alternative, and
   propose ONE rerun experiment that would distinguish the two.
   

   Sample output worth expecting:

   Copy

   CLASSIFICATION: DATA (confidence: medium)
   EVIDENCE: fail runs show "expected 10 rows, found 12" with ids 4491, 4492
   not present in seed set; pass run shows exactly seeded ids.
   ALTERNATIVE: ORDER — extra rows may be written by a test earlier in the shard.
   EXPERIMENT: run this test alone 10x on a fresh DB (expect: 10/10 pass),
   then run after checkout.spec 10x (fail rate returning = ORDER/DATA-pollution).
   

3. 3.

   Run the discriminating experiment — this is the actual QA. The model proposed it; you execute it:

   Copy

   npx playwright test order-history --repeat-each=10 --workers=1   # isolation baseline
   npx playwright test checkout order-history --repeat-each=10     # pollution probe
   

   Record the numbers. A classification without a confirming rerun is a plausible story, and plausible stories are how flakes survive for years.

4. 4.

   Apply the standard fix for the confirmed class. Route by diagnosis, not by creativity:

   • •
     ASYNC/WAIT — replace sleeps/fixed timeouts with condition-based waits (expect(...).toPass, web-first assertions); await the response, not the spinner.
   • •
     DATA — unique per-test data (UUID suffixes), transactional rollback or truncate-on-setup; never rely on cleanup from other tests.
   • •
     ORDER — remove hidden coupling (shared login state, module-level caches); enforce independence by running shuffled in CI.
   • •
     ENVIRONMENT — pin versions, dedicate ports, mock the third party (see the mock-generation playbook); fix the clock with fake timers.
   • •
     CONCURRENCY — shard-safe resources, per-worker databases/accounts.

   The assistant can draft the patch; review it like any AI test edit — the review-clinic rubric applies.

5. 5.

   Prove the fix statistically, then release from quarantine. The exit bar is numeric:

   Copy

   npx playwright test order-history --repeat-each=25 --workers=4
   # 25/25 green, parallel, shuffled -> release from quarantine
   

6. 6.

   Log the diagnosis. Test, class, evidence, experiment result, fix, proof run — one row per flake. This log is what turns next quarter's flake wave into pattern-matching instead of re-derivation, and it is the input the clustering playbook uses to spot systemic causes.

Variations

• ReportPortal / CI analytics in place: use their ML failure grouping as a pre-sort, then run this protocol per group — the taxonomy and experiments stay the same.
• Playwright trace-first: for UI flakes, attach the trace viewer's action timeline to the prompt; race conditions are often visible as action-vs-network ordering, and the model spots them readily in the timeline text.
• Batch mode: feed 20 flaky tests' evidence in one session and ask for a classification table with per-test confidence, then spend your rerun budget on the low-confidence rows first.
• JVM/pytest suites: same taxonomy; swap repeat mechanics (pytest-repeat, -p no:randomly vs shuffled) and watch for order flakes hidden by fixed execution order.

Quality check

• Every classification has quoted evidence from real logs — a category name without quoted lines is vibes, send it back.
• Every diagnosis has a rerun experiment result with counts (n/N pass rates), not adjectives.
• Fixes match the classno sleeps added under an ASYNC diagnosis, no retries added anywhere.
• Quarantine exits are statistical: ≥25 consecutive green runs under parallel, shuffled conditions.
• The flake log grows: each diagnosed flake adds a row; recurrences reference prior rows.

Pitfalls

• Accepting confident misclassification. Models pattern-match "timeout" to ASYNC even when the timeout is a symptom of a dead container (ENVIRONMENT). The discriminating experiment exists because confident-and-wrong is the default failure mode.
• Fixing the symptom line. Bumping one timeout "fixes" the race locally and re-breaks on slower CI runners. The class fix (condition-based waiting) is the fix.
• Retries as policy. A retried pass records no evidence and rots the suite's meaning; reserve retries for known-infra blips, tagged and counted, never as a blanket setting.
• Diagnosing from one failure. One log cannot separate DATA from ORDER; insist on multiple runs before classification.
• Quarantine amnesia. Quarantine without a ticket and an exit bar is deletion with sentimentality. Every quarantined test carries its diagnosis row and a deadline.

Restricted-environment variant

CI logs routinely leak internal hostnames, tokens in URLs, and customer identifiers in test data — scrub before pasting into any external assistant (the PII-masking playbook's pipeline applies to logs, not just datasets). On client engagements, run classification through approved tools or a local model (flake-signature classification works acceptably on strong local models since the taxonomy constrains the answer), and keep the flake log itself in the client's tracker if their audit rules require it.

Success criteria

• •
  Flake backlog classified with evidence, and ≥80% of diagnoses confirmed by their discriminating experiment on first pass.
• •
  Fix PRs shipped per class — waits replaced, data isolated, mocks installed — with 25× proof runs attached.
• •
  Suite-level flake rate measurably down within a month, without any new retries or timeout inflation.
• •
  A living flake log the next engineer actually consults before re-diagnosing.