A flaky test is a debugging problem wearing a testing costume, and teams handle it with the two worst tools available: retry-until-green and @skip.
Tier: Core · Workflow area: Testing & Quality
AI changes the economics because flake diagnosis is fundamentally a correlation task — test source, CI history, timing patterns, and ordering effects considered together — and that is exactly what a model with all four in context does well. The discipline this playbook adds: classify before fixing, and prove determinism after.
Do not use for a test that fails 100% of the time under specific conditions — that is a plain bug; debug it directly (see the debugging playbooks in batch 1).
pytest -p randomly, pytest-repeat, jest --runInBand).Assemble the evidence bundle. One file: the test + its fixtures, 5–10 failure logs with timestamps, the pass/fail history, and notes on where it fails (shard? time of day? parallel only?). Diagnosis quality is bundle quality.
Demand classification before any fix. Prompt:
This test fails ~8% of CI runs. Evidence bundle below (source, fixtures,
10 failure logs, run history with shard + duration).
Classify the flake into: (a) timing/async race, (b) test-order dependency /
shared state, (c) isolation failure (DB/files/ports not cleaned), (d) infra
(runner resources, network), (e) concurrency in the code under test,
(f) time/UTC/clock dependence.
Rank the top 2 hypotheses WITH the specific evidence lines supporting each,
state what evidence would distinguish them, and give a local reproduction
command for the leading hypothesis. Do not propose a fix yet.
Reproduce deterministically. Run the discriminating experiment: order dependency → pytest -p randomly --randomly-seed=<failing seed> or run the suspected pair alone; race → add load (stress-ng) or run pytest-repeat -n 200; clock → freeze time near midnight/DST. No reproduction, no fix — you would be patching a hypothesis.
Fix the cause, not the symptom. Now ask for the repair, constrained: Hypothesis (b) confirmed: test_export reuses the module-scoped tmp dir mutated by test_import. Propose a fix that removes the shared state. Forbidden: sleeps, retries, skip markers, widening timeouts. Typical honest fixes: function-scoped fixtures, unique temp dirs/ports per test, awaiting the actual condition instead of sleeping, injecting a clock.
Prove determinism. Re-run the reproduction command that previously failed — 200 repeats, the failing seed, the parallel mode — and record the green streak in the PR: "Failed 11/200 before, 0/500 after, seed 1847233."
Institutionalize. Have the AI draft a one-paragraph entry for your team's flake ledger: symptom, class, root cause, fix pattern. Ten entries later, that ledger is context you paste into the next forensics prompt, and diagnosis gets faster every quarter.
Retries convert a flaky test into a slow, dishonest one. Classification-first forensics converts it into a fixed one — and the fix pattern compounds across the suite.
sleep|retry|skip and make the AI justify any hit.CI logs routinely leak hostnames, internal URLs, tokens, and customer identifiers in stack traces. Scrub before pasting: strip env dumps, mask hostnames/IPs, redact bearer tokens (CI secrets masking is not sufficient for copy-paste). On client engagements route the bundle through the sanctioned tenant (Copilot Business / Azure OpenAI in-tenancy). Fully offline: the classification taxonomy in step 2 works as a human checklist, and a local Qwen-Coder-class model can still do log-pattern triage across your scrubbed bundle — weaker at multi-source correlation, so lean harder on the discriminating experiments.