A healed test is an unreviewed edit to your safety net. The audit queue makes that edit visible: every heal becomes a queued diff with a required accept/reject decision, so a heal that masks a real regression is caught in review — not discovered in production a month later.
Tier: Core · Workflow area: Agentic test stack & self-healing
This playbook builds the operational workflow around self-healing: collecting heals into a queue, reviewing them with a three-question protocol, committing accepted heals so the baseline stays versioned, and turning rejected heals into filed bugs.
Make heals visible in one place. Configure the mechanism to emit heals as artifacts, not just runtime behavior. Healenium: enable the report UI/plugin and export healed selectors per run. Playwright healer: run it so patches arrive as diffs. Vendor tools: pull the heal log via API. Land everything in a queue — a heals/ directory of JSON entries, a dashboard, or simply draft PRs. A queue entry carries:
{
"test": "checkout.spec.ts > applies discount code",
"old_locator": "#discount-apply",
"new_locator": "button[data-testid='promo-apply']",
"run": "nightly-2031",
"confidence": 0.94,
"status": "PENDING_REVIEW"
}
Review with the three questions — in order. For each queued heal:
Q1 Is the new locator pointing at the SAME logical control?
(Open the app or the DOM snapshot; don't trust the similarity score.)
Q2 Did the app's INTENDED behavior change in this area?
(Check the release notes / recent PRs for this UI. Renamed button: fine.
Removed discount step: not fine.)
Q3 Does the test still assert the outcome that matters, and did it truly
exercise it in the healed run? (Read the healed run's assertion evidence.)
Three yes/no/no-change answers in the right pattern → accept. Anything else → reject.
Accept = commit, so the heal count returns to zero. An accepted heal is a locator improvement; put it in the suite:
git checkout -b heal/checkout-discount-locator
# apply the new locator in source, delete the queue entry
git commit -m "Accept heal: #discount-apply -> data-testid promo-apply (UI refactor PR#412)"
This step is what keeps the versioned suite identical to the executed suite. Runtime-healing tools that keep healing the same selector every night are telling you this step is being skipped.
Reject = file the bug and pin the test red. If Q2 revealed a behavior change, the heal was masking a regression. Revert/block the heal, let the test fail honestly, and file the defect with the heal entry as evidence — "the healer re-pointed this test at X to keep it green" is a compelling repro narrative. Record it:
REJECTED heal nightly-2031/checkout-discount
Reason: discount no longer applied after redesign; heal re-targeted a decorative button
Action: bug FE-2214 filed, heal reverted, test intentionally red until fix
Use AI to pre-screen, never to decide. An assistant can batch-summarize the queue and flag suspicious entries — locator changes that jump to a different DOM region, heals coinciding with feature PRs, repeated heals on one test:
Here are 12 healed-locator entries (JSON) and the list of app PRs merged this week.
For each heal, flag LIKELY-BENIGN or NEEDS-SCRUTINY with one line of reasoning —
e.g. heal coincides with PR touching that component's behavior, or new locator
is in a different page region. Do not approve anything; this is triage ordering only.
The human still answers the three questions for every entry; the model only orders the work.
Report queue health weekly. Track: heals queued, accepted, rejected, median time-in-queue, and repeat-healers (tests healed more than twice — candidates for locator rewrite). Rejected-heal count is your caught-regression count; celebrate it.
Accept-and-commit or reject-and-file — those are the only two exits from the queue. The moment a heal can take a third path (silently becoming tomorrow's baseline), the suite stops being evidence of anything.
The queue itself is a governance asset for regulated and client-audited environments: it converts "AI modified our tests" into "every AI-proposed change carries a named approver and a timestamp." Keep queue entries free of client-sensitive data (URLs and selectors are usually fine; embedded test data may not be). If policy bars model-based pre-screening, skip step 5 — the queue works identically with human-only triage, just order by confidence score and PR-coincidence manually.
Put the weekly heal report next to your flake report in the quality dashboard. Heals and flakes are the two ways a suite lies to you — one by changing itself, one by changing its mind — and leaders who see both numbers weekly make better calls on agentic-tooling trust than any policy document achieves alone.