Testing & Accessibility in the AI Era

Kavya's green suite is the signature failure of AI-era frontend quality: verification theater. Models produce tests that exercise code without constraining behavior, because "tests that pass" is the visible success criterion and "tests that would fail if the feature broke" is not. The same generative fluency produces markup that renders beautifully and excludes anyone not using a mouse. Both failures are invisible in review and expensive in production — and both have mechanical countermeasures, which this module installs: mutation testing to score assertion quality, accessible queries to force behavior-first tests, ARIA snapshots to make E2E deterministic, root-cause discipline for flake healing, and a three-layer accessibility audit that knows exactly what machines cannot see.

What You'll Learn

• Behavior-first RTL generationaccessible queries, proven by mutation score
• Agent-authored Playwright E2Evia MCP, with ARIA-snapshot assertions and determinism review
• Planner/generator/healer agentsand the flake-fix acceptance bar
• AI triage of visual-regression diffshumans review only the ambiguous minority
• The three-layer a11y auditautomated, AI-guided, manual — and each layer's blind spots
• Generation constraints and CI gatesthe setup that makes accessibility unskippable
• The legal realitywhy "AI made it" is never an accessibility defense

Behavior-First Component Tests, Mutation-Proven

The fix for verification theater starts with what generated tests are allowed to say. Constrain generation to RTL's accessible-query doctrine — getByRole, getByLabelText, user-visible assertions — and forbid the two theater staples: mock-call assertions as primary verification, and wholesale DOM snapshots. The prompt shape:

Generate RTL tests for [component]. Rules:
- Query only via accessible queries (getByRole, getByLabelText).
  No test-ids, no container.querySelector.
- Each test: simulate a user action, assert user-visible outcome
  (visible text, role state, enabled/disabled, navigation).
- No snapshot tests. Mock assertions only as secondary checks.
- Cover: happy path, each validation error, submit failure,
  disabled-while-pending, empty state.

Accessible queries do double duty: they make tests resemble real usage and they fail on inaccessible markup — a div-button has no role to query, so the test can't be written against it. The doctrine quietly enforces a11y at test-authoring time.

Then prove the suite with mutation testing. StrykerJS mutates your component code — inverts conditions, drops calls, swaps operators — and reruns the tests; assertions that never fail are decorative, and the mutation score is the number coverage pretends to be. Kavya's 41% first-run kill-rate gap is typical for unconstrained generated suites. The loop: generate under constraints, run Stryker, feed surviving mutants back to the model ("these mutations survived — strengthen the assertions that should have caught them"), iterate to threshold. Lab 05 runs exactly this loop against a form component with a stated score to clear.

Agent-Authored E2E with Playwright MCP

Playwright MCP lets an agent drive the real app — navigate, click, fill, observe — and then emit the spec for what it just did. This kills the blank-page cost of E2E, but raw agent output has two habitual defects: selector fragility and nondeterminism. The countermeasures are part of the workflow, not afterthoughts:

• Demand role-based locatorsgetByRole('button', { name: 'Cancel appointment' }), never CSS-path selectors that die on the next refactor. Same doctrine as RTL, same a11y side benefit.
• Assert with ARIA snapshotsPlaywright's toMatchAriaSnapshot captures the accessibility tree, not the pixel or DOM soup. It is resilient to cosmetic change, sensitive to structural and semantic change, and readable in review — the right default assertion for generated E2E.
• Review for determinism before commithunt for hard waits (waitForTimeout), ordering assumptions on parallel data, and time/locale dependencies. Generated specs record what happened once; your review makes them specify what must always happen.
• Run the flake gauntletten consecutive green runs before a generated spec earns a place in CI (Lab 06's bar). A spec that can't pass ten in a row is a flake you're about to institutionalize.

The full test-agent trio — planner (proposes what to test), generator (writes specs), healer (repairs breaks) — is production-real in Playwright's ecosystem. The healer deserves its own discipline: when it proposes a fix for a red test, classify the root cause first — timing, selector, test data, environment, or actual regression. Accept fixes that eliminate the flake class (replace the hard wait with a web-first assertion); reject fixes that paper over it (bump the timeout, retry harder) — and never let a healer "heal" a genuine regression into a passing test. That last case is the healer's most dangerous failure mode, and only a human who reads the diff can catch it.

Visual Regression at Scale: AI Pre-Triage

Screenshot suites (Chromatic, Percy, Argos — or the Playwright setup from M4) generate review fatigue: hundreds of diffs per PR, most of them intentional restyles or anti-aliasing noise. The AI-era workflow adds a pre-classification pass — a vision model sorts each diff into intentional (matches the PR's stated styling change), regression (unrelated visual damage), or flake (rendering noise) — and humans review only the regression pile plus a sample of the rest. Treat vendor "AI review" claims with the standard skepticism: audit the classifier monthly by hand-reviewing a random slice of what it auto-approved. The efficiency is real; unaudited, so is the drift.

The Three-Layer Accessibility Audit

Accessibility verification has a hard structural fact at its center: automated scanners catch roughly a third of WCAG failures by volume, AI-guided testing extends coverage substantially further, and a stubborn remainder is machine-invisible — detectable only by a human at a keyboard or behind a screen reader. The three-layer audit is built on that fact, and knowing which layer catches what is the skill:

Lab 07 seeds a page with ten defects — four automatable, four guided-detectable, two manual-only — and passing requires not just finding all ten but correctly identifying which two no machine could have found. That classification is the audit skill in miniature.

Constrain Generation, Gate the Pipeline

Auditing catches; prevention is cheaper. Both halves from M2 apply with a11y-specific content — and they're the difference between accessibility as heroics and accessibility as infrastructure:

• Generation constraintsinstruction files mandate semantic elements, associated labels, focus-visible states, APG-pattern compliance for composite widgets, and aria-live for dynamic regions. Generated div-buttons stop at the source.
• The unskippable gateaxe-core in CI (jest-axe for components, Playwright-axe for pages) on every PR, no bypass label. Kavya's dialog shipped because the gate existed but wasn't wired — an unwired gate is a story you tell auditors, not a control.
• APG-referenced repairwhen keyboard/focus behavior needs fixing (roving tabindex, focus traps, combobox semantics), point the model at the specific ARIA Authoring Practices pattern and generate against it — then verify with a real screen reader, never on the model's assurance. The APG is ground truth; the model is a fast typist with a good memory of it.

"AI Made It" Is Not a Defense

One paragraph of legal clarity, because it changes team behavior: accessibility obligations — ADA, the European Accessibility Act, EN 301 549, India's RPwD Act — attach to the published product, not to its authorship process. A lawsuit, a procurement bar, or a regulator does not care that a model generated the markup; the operator ships it, the operator answers for it. Provenance never transfers liability. Teams that internalize this stop treating the a11y gate as friction on AI velocity and start treating it as the thing that makes AI velocity safe to use.

Try It Now

1. 1Mutation-check one real suite. Run StrykerJS on one component's existing tests (generated or not). Read every surviving mutant and classify it: decorative assertion, missing case, or acceptable survivor. The first run's score is your baseline honesty metric.
2. 2Generate one E2E spec via Playwright MCP. Point the agent at a flow you know well, review its emitted spec against the determinism list (locators, waits, data assumptions), fix what you find, then run it ten times consecutively.
3. 3Do a fifteen-minute Layer 3 pass. Pick your most important flow. Complete it keyboard-only, then once with VoiceOver or NVDA. Log every point of friction. Most engineers' first pass on a "clean-scanning" page finds something within five minutes — that something is what the machines were never going to tell you.

Your playbooks & labs

This module's playbooks: pb-22 (Behavior-First Component Tests), pb-23 (Mutation-Check Your Generated Tests), pb-24 (Agent-Authored E2E via Playwright MCP), pb-25 (Flaky E2E Healing Without Papering Over the Cause), pb-26 (Visual-Regression Triage with AI Pre-Classification), pb-27 (NL Mobile Journeys as Exploration, Deterministic Suites as Gates), plus the accessibility set: pb-29 (Three-Layer Audit), pb-30 (A11y Constraints and CI Gates), pb-31 (Alt-Text Pipeline), pb-32 (Keyboard and Focus Repair Against APG), and pb-33 (Accessible Streaming AI UI). Labs: 05 (mutation-proofed tests), 06 (agent E2E plus flake hardening), 07 (the ten-seed three-layer audit), and 09 (the provenance PR review, where several of these defects hide).

Further Reading

• •
  Testing Library — Query priority doctrine — the accessible-query rules this module's generation constraints encode.
• •
  Stryker Mutator — StrykerJS — mutation testing for JS/TS suites.
• •
  Playwright — Test agents and ARIA snapshots — planner/generator/healer, from the source.
• •
  Deque — Intelligent Guided Tests — the AI-guided middle layer.
• •
  W3C — ARIA Authoring Practices Guide — ground truth for every composite-widget repair.