Lab 03 · Playwright Planner → Generator → Healer Pipeline

Sara runs automation for a small product team and has been told the Playwright test agents will "write the suite for you." Partly true. The loop is planner → human edit → generator → human review → healer → human review, and the humans are load-bearing. In this lab you build a tiny task-manager app, run all three agents against it, and then ship a "release" that changes the UI in two places — one an innocent rename, one hiding a real regression. If the healer's second patch lands unreviewed, your suite goes green while the feature stays broken. Catching that is the pass condition.

Time, tier & cost

~120 minutes hands-on · Core tier (maps to module M3 and playbook PB-15). Cost: $0–2 (agent runs inside Claude Code / VS Code with Copilot; app is local).

Setup

Node 18+, VS Code or Claude Code, Playwright 1.55+ (the agents shipped behind init-agents).

mkdir -p qa-lab-03 && cd qa-lab-03
npm init -y && npm install express@4
npm init playwright@latest -- --quiet
npx playwright install chromium
npx playwright init-agents --loop=vscode   # or --loop=claude

app.js — version 1 of the app under test:

const express = require("express");
const app = express();
app.use(express.urlencoded({ extended: true }));
let tasks = [];
let nextId = 1;
app.get("/", (req, res) => {
  res.send(`<!doctype html><html><head><title>TaskBoard</title></head><body>
  <h1>TaskBoard</h1>
  <form method="POST" action="/tasks">
    <input name="title" id="task-title" placeholder="Task title">
    <button id="add-btn">Add task</button>
  </form>
  <ul id="task-list">${tasks
    .map(
      (t) => `<li>${t.title}
      <form method="POST" action="/tasks/${t.id}/done" style="display:inline">
        <button class="done-btn">Done</button></form>
      <form method="POST" action="/tasks/${t.id}/delete" style="display:inline">
        <button class="delete-btn">Delete</button></form>
      ${t.done ? "<em>(done)</em>" : ""}</li>`
    )
    .join("")}</ul>
  <p id="count">${tasks.filter((t) => !t.done).length} open</p>
  </body></html>`);
});
app.post("/tasks", (req, res) => {
  if (req.body.title && req.body.title.trim())
    tasks.push({ id: nextId++, title: req.body.title.trim(), done: false });
  res.redirect("/");
});
app.post("/tasks/:id/done", (req, res) => {
  const t = tasks.find((x) => x.id == req.params.id);
  if (t) t.done = true;
  res.redirect("/");
});
app.post("/tasks/:id/delete", (req, res) => {
  tasks = tasks.filter((x) => x.id != req.params.id);
  res.redirect("/");
});
app.listen(3000);

Start it with node app.js. Keep this terminal running for the whole lab.

Steps

1. 1Run the planner and read the plan like a reviewer. Invoke the 🎭 planner agent (from the generated .github/ or agent config) with: "Explore http://localhost:3000 and produce a test plan for the task management features." It writes specs/ Markdown. Read every scenario before anything else happens.
2. 2Edit out the over-scope. Planners pad. Expect scenarios this app cannot support or that no one asked for — login/authentication flows, task editing, persistence across restart, mobile viewports, sorting. Delete them, and record each deletion in plan-review.md with one line of justification. An unedited plan is an automatic fail on the rubric.
3. 3Edit in the under-scope. Planners also miss. Check the plan for: adding a task with an empty or whitespace-only title (the app rejects it — silently), the open-task counter updating on done/delete, deleting a done task, and adding two tasks with identical titles. Add whichever are missing to specs/ yourself, in the same Markdown style. Log the additions in plan-review.md.
4. 4Run the generator. Invoke the generator agent on your edited plan: it executes steps against the live app as it writes, so the output should be genuinely verified, not guessed. Review the emitted tests/*.spec.ts for locator quality — prefer getByRole/getByLabel; flag any brittle CSS it chose. Run npx playwright test and get a green baseline. Commit: git init && git add -A && git commit -m "green baseline".
5. 5Ship the "release." Apply both UI changes to app.js. Change A (innocent rename): id="add-btn" becomes id="create-btn" and the button text "Add task" becomes "Create task". Change B (the seeded regression): in the /tasks/:id/done handler, change t.done = true to t.done = t.done — the button now silently does nothing — and in the list template change class="done-btn" to class="complete-btn". Restart the app. Run the suite: tests covering both areas now fail.
6. 6Run the healer on the add-task failures. Invoke the healer agent on the failing add-task specs. It should replay, inspect the live page, and propose a locator update to the new button name. Read the diff before accepting: it should touch only the locator/text, not the assertions. Save the diff into healer-audit.md with verdict ACCEPT and your reasoning.
7. 7Run the healer on the done-task failures — and audit hard. The healer will find the renamed complete-btn, fix the locator, click it… and then face an assertion that the task shows "(done)" and the counter decremented, which is still false because the feature is broken. Watch what it does: a correct healer outcome is "test still fails, likely product bug." A dangerous outcome is the healer weakening the assertion (removing the "(done)" check, asserting only that the click succeeded) to get green. If it proposes any assertion change, that is your wrong heal. Verdict: REJECT, with the evidence in healer-audit.md. If your healer run correctly refused to weaken anything, simulate the wrong heal: apply the assertion-weakening diff from the answer key yourself and write the rejection you would post on that PR.
8. 8Verify the bug by hand and file it. Reproduce manually: click Done, observe the task never marks done and the counter never moves. Write bug-report.md with repro steps and the exact line (t.done = t.done). Then fix app.js properly, restore the healed suite's original assertions, and run to green.
9. 9Close the loop in writing. Finish healer-audit.md with the governance rule you'd adopt: every heal is a proposed change; locator-only diffs are fast-tracked with review, any diff touching an assertion or adding a wait is escalated as a suspected product bug.

Verify

• •
  plan-review.md shows at least two deleted over-scope scenarios and at least two added under-scope scenarios, each justified.
• •
  The generated suite ran green at baseline and is committed.
• •
  healer-audit.md contains two diffs with opposite verdicts: one ACCEPT (locator-only) and one REJECT (assertion-weakening or bug-masking), each with reasoning.
• •
  bug-report.md reproduces the done-button regression and cites the offending line.
• •
  Final state: bug fixed, original assertion strength restored, suite green.

Self-grading rubric (pass = 8/12):