Healer agents will get your red test green — that is the problem. A retry here, a longer timeout there, a looser assertion, and the flake is hidden, waiting to return as a production bug. Classify the root cause first — timing, selector, data, or infra — and accept only fixes that remove that class.
Tier: Advanced · Workflow area: Testing & quality
Flaky tests are evidence. Sometimes evidence of a bad test; alarmingly often, evidence of a real race in the app that users hit too. The healing workflow must preserve that evidence long enough to read it.
trace: 'on-first-retry'), screenshots, videos, and CI logs retained.Reproduce the flake on purpose before touching anything. A flake you cannot reproduce, you cannot verify you fixed:
npx playwright test cart.spec.ts --repeat-each=25 --workers=4
Note the failure rate. Parallel workers plus repetition is the flake amplifier — many "random" failures are deterministic under contention. If 25 repeats stay green locally, the flake is environment-conditioned (CI resources, env data) — that finding redirects the whole investigation.
Make the agent classify before it fixes. This prompt structure is the heart of the playbook:
Attached: the failing test, the Playwright trace and screenshot from
3 failed runs, and the passing-run trace. Do NOT propose a fix yet.
First classify the root cause as exactly one of:
- TIMING: the test races the app (assertion before render settles,
navigation race, animation, debounce) — cite the trace timestamps.
- SELECTOR: locator matches unstably (multiple matches, virtualized
list, portal timing, changed accessible name) — cite the DOM state.
- DATA/STATE: shared or leaking state (test order dependence, seeded
data mutated by another test, leftover session) — cite the evidence.
- INFRA: CI environment (cold starts, resource contention, network to
test env) — cite the variance between CI and local.
- APP BUG: the app itself races (double-submit possible, hydration
race, out-of-order responses) — this is the one we WANT to find.
State your confidence and the single strongest piece of evidence.
Demanding evidence citations from the trace is what separates diagnosis from vibes — and APP BUG must always be on the menu, or the agent will never look for it.
Match the fix to the class — and reject class mismatches. The acceptance rules, per class. TIMING: replace the race with a deterministic wait on the actual condition — a web-first assertion on the element or response the test really depends on. Reject any added waitForTimeout: sleeps do not remove the timing class, they just move the finish line. SELECTOR: fix the locator to a unique role/name, or fix the app markup that made it ambiguous. Reject nth-child, XPath-by-position, or "click the first match" resolutions. DATA/STATE: isolate — per-test fixtures, fresh context, seeded reset. Reject test reordering or serial-mode-as-fix; serialization hides the coupling, it does not remove it. INFRA: fix the environment or quarantine the test with a ticket; do not loosen the test to tolerate a broken environment. APP BUG: stop healing entirely — file it, fix the app, keep the test that caught it. That test just earned its keep.
Audit healer output for the four paper-over patterns. Whatever produced the patch — a commercial self-healing runner, a Copilot suggestion, your own agent loop — grep the diff for: added retries or raised retry counts; raised or added timeouts; assertions weakened (toHaveText → toContainText → gone); and try/catch or conditionals around the flaky step ("dismiss the dialog if it appears"). Each is an automatic reject without a class-based justification in the PR description. These four patterns are how suites rot into green-but-meaningless.
Verify by re-running the amplifier, not by one green run. The fix must survive the same conditions that reproduced the flake:
npx playwright test cart.spec.ts --repeat-each=25 --workers=4
Same repeats, same parallelism, zero failures — plus a normal full-suite run to confirm the fix did not shift contention onto a neighbor. Record before/after failure rates in the PR; "5/25 → 0/25 under contention" is the sentence that makes the review trivial.
Track flake debt like defect debt. Keep a small ledger (a JSON in the repo or your CI dashboard): test, class, fix, date, recurrence. Two payoffs — recurring TIMING entries in one feature area are a flashing arrow at a real app race, and the ledger is the dataset that tells you whether your healing process removes classes or just cycles them.
The most expensive outcome of automated healing is silently converting APP BUG into TIMING — adding a wait that makes the test tolerate a race your users still lose. Any timing fix in code paths involving submits, payments, or navigation deserves the explicit question: "could a fast user hit this same race?"
npx playwright show-trace for your own reading; the zip contents for the agent) and run the classify-then-fix loop with the amplifier as the verification command.A flaky test is a measurement whose noise you have not yet explained. Healing that deletes the noise without explaining it deletes the measurement too. Classify first, fix the class, prove it under the same contention that exposed it — and treat every APP BUG the process uncovers as the flake that paid for all the others.