Lab 06 · Agent-Generated E2E with Playwright MCP

Lena's team adopted agent-authored E2E tests with enthusiasm: point Claude at the app, get specs for free. Three weeks later CI is a coin flip and everyone has learned to click "re-run." The problem isn't that agents write bad tests — it's that agents write tests that pass once, on the run where they were authored, and nobody applies a determinism review before committing. In this lab you generate specs the right way, then take one seeded flaky spec and fix it the way a senior engineer would: classify the flake first, then remove the class, not the symptom.

Time & tier

~90 minutes hands-on · Verify stage (maps to module M5, playbooks PB-24/PB-25) · $0 — Playwright + Playwright MCP, local app, Claude Code or any MCP-capable agent.

Setup

npm create vite@latest e2e-lab -- --template react-ts
cd e2e-lab && npm install
npm install -D @playwright/test && npx playwright install chromium
mkdir -p tests

The app under test, src/App.tsx — a small bookshelf with a deliberately realistic wrinkle: saves are async with variable latency:

import { useState } from "react";

type Book = { id: number; title: string; author: string };
const seed: Book[] = [
  { id: 1, title: "Refactoring UI", author: "Wathan" },
  { id: 2, title: "Design Systems", author: "Kholmatova" },
];

export default function App() {
  const [books, setBooks] = useState(seed);
  const [title, setTitle] = useState("");
  const [author, setAuthor] = useState("");
  const [status, setStatus] = useState<"idle" | "saving" | "saved">("idle");

  const add = async () => {
    setStatus("saving");
    await new Promise((r) => setTimeout(r, 200 + Math.random() * 900)); // fake API
    setBooks((b) => [...b, { id: Date.now(), title, author }]);
    setTitle(""); setAuthor(""); setStatus("saved");
  };

  return (
    <main>
      <h1>Bookshelf</h1>
      <form onSubmit={(e) => { e.preventDefault(); add(); }} aria-label="Add book">
        <label htmlFor="t">Title</label>
        <input id="t" value={title} onChange={(e) => setTitle(e.target.value)} />
        <label htmlFor="a">Author</label>
        <input id="a" value={author} onChange={(e) => setAuthor(e.target.value)} />
        <button disabled={!title || !author || status === "saving"}>Add book</button>
      </form>
      <p role="status">{status === "saving" ? "Saving…" : status === "saved" ? "Saved" : ""}</p>
      <ul aria-label="Books">
        {books.map((b) => (
          <li key={b.id}>
            {b.title} — {b.author}{" "}
            <button onClick={() => setBooks(books.filter((x) => x.id !== b.id))}>
              Delete {b.title}
            </button>
          </li>
        ))}
      </ul>
    </main>
  );
}

Wire Playwright MCP for the agent (Claude Code shown; Cursor equivalent works), .mcp.json:

{
  "mcpServers": {
    "playwright": { "command": "npx", "args": ["@playwright/mcp@latest"] }
  }
}

playwright.config.ts:

import { defineConfig } from "@playwright/test";
export default defineConfig({
  testDir: "tests",
  use: { baseURL: "http://localhost:5173" },
  webServer: { command: "npm run dev", url: "http://localhost:5173", reuseExistingServer: true },
});

And the seeded artifact: a spec "the agent generated last sprint," tests/add-book.flaky.spec.ts. Commit it verbatim:

import { test, expect } from "@playwright/test";

test("user can add a book", async ({ page }) => {
  await page.goto("/");
  await page.locator("input").nth(0).fill("Clean Code");
  await page.locator("input").nth(1).fill("Martin");
  await page.locator("button").first().click();
  await page.waitForTimeout(400);
  const items = page.locator("li");
  await expect(items).toHaveCount(3);
  expect(await page.locator("li").nth(2).innerText()).toContain("Clean Code");
});

Guided steps

1. 1Let the agent explore, then author. Start the dev server. In your agent: "Using the playwright MCP, open http://localhost:5173, explore the Bookshelf app, then write three Playwright specs to tests/: (1) adding a book, (2) deleting a book, (3) the add form's disabled/saving behavior. Use role-based locators only and include an expect(...).toMatchAriaSnapshot() assertion on the book list in at least one spec. Wait on visible state changes, never on time." Watch the MCP calls — the agent should navigate, snapshot the accessibility tree, and interact before writing anything. Expected: three spec files that reference roles and accessible names it observed, not guessed.
2. 2Review the generated specs for determinism before running them. Apply this checklist to each: any waitForTimeout? any positional selector (nth, first) where an accessible name exists? any assertion racing an async state change? any dependence on ordering between tests? Fix violations with the agent ("replace the timeout with an assertion on the status region") — you are the determinism gate the agent doesn't have. Log each violation found in flake-report.md.
3. 3Run the suite ten times.npx playwright test --repeat-each=10 --workers=1 (exclude the flaky file for now: add --grep-invert flaky). Expected: 30/30 green. If anything fails even once, treat it as a real defect in the spec and fix the wait condition, not the timeout number.
4. 4Now run the seeded flaky spec ten times.npx playwright test add-book.flaky --repeat-each=10 --workers=1. Expected: intermittent failures — some runs green, some red on the count assertion. Record the observed failure rate in flake-report.md. If all ten pass, run ten more; the latency is randomized up to 1.1s against a 400ms wait, so it fails often.
5. 5Classify before fixing. In flake-report.md, write the before analysis: which flake class is this — timing, selector, data, or infra? What exactly races what? What secondary fragilities exist even when it passes (look at every locator)? Name the class in one sentence a teammate could act on. Do not touch the code until this paragraph exists.
6. 6Harden it — remove the class, not the symptom. Rewrite the spec: role/name-based locators (getByLabel("Title"), getByRole("button", { name: "Add book" })), await the observable completion signal (the role="status" region showing "Saved", or expect(list).toContainText, which auto-retries), and drop every positional index and every waitForTimeout. The forbidden fix: raising 400 to 2000 — that converts a frequent flake into a rare one and adds 2s to every run forever.
7. 7Prove it: ten consecutive runs.npx playwright test add-book.flaky --repeat-each=10 --workers=1 → 10/10. Then stress it: temporarily change the app's latency to 200 + Math.random() * 2500 and run five more. A class-level fix survives; a tuned timeout dies. Revert the latency after.
8. 8Write the after-analysis. Complete flake-report.md: root-cause class, the mechanism, the fix principle ("await observable state, address elements by accessible name"), and the review rule your team should apply to every agent-generated spec before commit.

Verify

• •
  Three agent-authored specs committed, each using role-based locators, at least one containing an ARIA-snapshot assertion on the list.
• •
  Step-2 determinism review logged with at least the violations you found and fixed (if the agent produced zero, say so — and check harder).
• •
  Main suite: 30/30 across --repeat-each=10.
• •
  Flaky spec: documented failure rate before; 10/10 after; survived the raised-latency stress run.
• •
  flake-report.md names the root-cause class before the fix and states the team review rule after.

Self-grading rubric (pass = 7/10)

• Flake classification (0–3): correct class named, mechanism explained before any fix = 3; fixed first, explained after = 1.
• Fix quality (0–3): all timing and positional fragilities removed, survives the stress run = 3; timeout raised or only partially de-flaked = 0–1.
• Generated-spec review (0–2): determinism checklist applied and logged for all three agent specs = 2.
• Assertion strategy (0–2): ARIA snapshot present and a written note on what it pins vs. what a pixel/DOM assertion would = 2.