Agentic IDEs write Playwright suites that compile, run and pass — and passing is exactly the problem. This module teaches spec-first authoring under team conventions, API suites from OpenAPI, ARIA-snapshot assertions, and the anti-pattern catalog for AI-generated tests, with mutation testing as the tiebreaker that proves your assertions actually assert something.
The tooling context matters: Playwright, Selenium and Cypress appear in roughly 78% of QA postings, and AI coding assistants — Claude Code, Cursor, GitHub Copilot — are now the default way test code gets written in product companies. But there is a documented catch. Research on LLM-generated tests (arXiv 2601.08998) finds them measurably more flaky than human-written ones, and Stack Overflow 2025 has 66% of developers frustrated by AI code that is almost right. Almost-right application code fails visibly. Almost-right test code passes silently — which is why this module treats assertion strength, not generation speed, as the deliverable.
Ananya — SDET, six years, at a Bengaluru D2C commerce company — inherits checkout coverage for the new promotions engine. She points Cursor at the feature: "Write Playwright tests for the coupon flow." Twenty minutes later: 38 tests, all green. She merges and moves on.
A developer PR breaks stacked-coupon math — two discounts now apply where policy allows one. The suite stays green. The bug ships to staging and is caught by a support escalation from UAT, not by automation.
Ananya reads the generated suite properly for the first time. await page.waitForTimeout(3000) before every assertion. Tests that assert the coupon input field exists rather than that the discount math is correct. One test literally asserts the response equals whatever the response was — captured at generation time from the buggy build. Three tests depend on execution order.
She runs Stryker mutation testing against the suite. Mutation score: 31% — two-thirds of deliberate code breakages survive her tests. The suite was not thin; it was decorative.
Same feature, new discipline: a spec-first prompt with the team's conventions file loaded, a review pass against the anti-pattern rubric, ARIA-snapshot assertions instead of brittle selectors, and a mutation gate in CI. Mutation score: 84%. The next promotions bug is caught by the suite, in the PR, before merge.
Nothing about Ananya's Thursday was unusual. Generated tests optimize for plausibility — they look like tests, run like tests, and report like tests. Whether they can fail when the product breaks is a property you have to engineer in, and this module is that engineering.
"Write tests for the coupon flow" delegates the two decisions that were your job: what to test and how your team writes tests. The professional pattern inverts this. You write a short test specification — scenarios, boundaries, the assertions that matter, in plain language — and the agent turns spec into code under your conventions. The conventions travel in a rules file (CLAUDE.md, .cursorrules, Copilot instructions) committed to the repo:
await expect(locator).toHaveText(...)), no waitForTimeout, every test asserts an outcome the user cares about, not an implementation detail.The payoff compounds: with conventions in context, review shifts from "rewrite this to match our style" to "is this assertion strong enough?" — which is the review that matters. Ananya's spec for the rewrite was eleven lines; the agent produced code she could review in one pass because it was shaped like her team's code.
Rules files are shared team infrastructure, not personal settings. Treat them like lint config — versioned, reviewed, and improved every time a review catches the agent doing something the rules should have prevented.
These five patterns account for most of the gap between "suite is green" and "suite protects the product." Each is common in generated code because each is common in the public code the models learned from:
| Anti-pattern | What it looks like | Why generation produces it | The fix |
|---|---|---|---|
| Sleep-based waits | waitForTimeout(3000) before actions | Timeouts "work" in demos and dominate training data | Web-first assertions that auto-retry until the condition holds |
| Tautological asserts | Assert element exists; assert response equals captured response | Existence is the easiest postcondition to infer from a DOM | Assert the business outcome: the math, the state change, the message |
| Over-mocking | Mock so much that the test exercises the mock | Mocks make generated tests pass reliably | Mock at system boundaries only; keep one integration path real |
| Order dependence | Test B needs the cart state test A created | Generated tests share setup implicitly | Independent fixtures per test; run the suite shuffled in CI |
| Selector brittleness | Deep CSS chains, positional XPath | Models mimic recorded-selector style | Role-based locators and ARIA snapshots (next section) |
The review clinic is a fifteen-minute pass with this table as the rubric — and mutation testing is the tiebreaker when reviewers disagree about whether an assertion is strong enough. Mutation tools (Stryker for JS/TS, mutmut for Python) plant small deliberate bugs — flip a comparison, delete a statement — and rerun the suite. Every mutant that survives is a bug class your tests would miss. Coverage tells you code was executed; mutation score tells you code was checked. For AI-generated suites the distinction is the whole game, because generated tests are systematically strong on execution and weak on checking.
Shown a generated test and asked what is wrong with it, look for the assertion first. "The test asserts the element is visible but the requirement is about the computed total" is the expected shape of answer — tautological and outcome-free assertions are the most-planted defect in this domain's items.
When AI tools regenerate frontend code freely, the DOM becomes a moving target and CSS/XPath assertions become maintenance debt. Playwright's ARIA snapshots assert against the accessibility tree — the semantic structure of roles, names and states that assistive technology sees:
await expect(page.locator('.checkout-summary')).toMatchAriaSnapshot(`
- heading "Order summary" [level=2]
- list:
- listitem: /Espresso Grinder .* ₹12,499/
- listitem: /Coupon FESTIVE10 .* −₹1,250/
- button "Place order"
`);
Three properties make this the AI-era default. It is resilient: a refactor that changes divs and classes but preserves semantics passes, while a change that breaks what users perceive fails. It is readable: the assertion is a human-legible statement of what the page should be, reviewable by someone who has never seen the implementation. And it double-covers: a page whose accessibility tree is assertable is a page with an accessibility tree — regressions in semantic structure fail functional tests. The discipline it demands: snapshot the regions that carry the requirement, not whole pages, and review snapshot updates like code — an approved-but-wrong snapshot update is the visual-baseline problem all over again.
API testing is the strongest generation story in this module because the ground truth is machine-readable: an OpenAPI or GraphQL schema declares endpoints, types, auth and status codes, so the model is grounded rather than guessing. The workflow:
Alongside the suites, the same convention-grounded generation scaffolds the boring 60% of framework ownership — auth-state fixtures, data seeding, MSW/WireMock network stubs, custom matchers — with one rule: you must be able to explain every generated line of harness code, because harness magic is where debugging goes to die. And for legacy estates, agentic migration (Selenium/Java to Playwright/TypeScript) is now a realistic engagement — with behavior parity as the verification spine: old and new suites run against the same build, and every converted spec must catch the same planted bugs the original caught, before the original is deleted (UC-13).
mutation-allowlist.mdCoverage says the code ran. This number says the tests were watching.
AI makes test code cheap and assertion strength expensive — the suite that matters is the one that fails when the product breaks. Prompt spec-first under versioned conventions, review against the anti-pattern catalog, assert on ARIA semantics and business outcomes, and let mutation testing — not the green check — tell you whether your tests are watching.
CLAUDE.md or .cursorrules, and generate one test with and without it loaded. Diff the results.Take this into Lab 04 (AI-authored suite quality clinic) — planted anti-patterns, mutation scores before and after — and Lab 06 (API suite from OpenAPI with an assertion audit) against a server with seeded contract bugs. Companion playbooks: pb-08 through pb-14.