Lab 04 · AI-Authored Suite Quality Clinic

Ananya inherited a repo where a coding agent generated 400 tests in a week. Coverage went up; confidence didn't. This lab is that situation in miniature: a small cart module, a unit suite and an E2E spec that look thorough and pass reliably, but were written the way LLMs actually write tests when nobody reviews them. Research on LLM-generated suites keeps finding the same defect families, so the suite below plants one of each. Mutation testing is your objective referee — before/after mutation scores are the artifact, because "the tests feel better now" is not evidence.

Time, tier & cost

~90 minutes hands-on · Core tier (maps to module M2 and playbooks PB-09/PB-13). Cost: $0 (Jest, Stryker and Playwright are free; the AI reviewer is optional).

Setup

mkdir -p qa-lab-04 && cd qa-lab-04 && npm init -y
npm install --save-dev jest @stryker-mutator/core @stryker-mutator/jest-runner
npm install --save-dev @playwright/test && npx playwright install chromium

cart.js — the module under test (correct code; the defects are in the tests):

function addItem(cart, item, qty) {
  if (qty <= 0) throw new Error("qty must be positive");
  const found = cart.items.find((i) => i.sku === item.sku);
  if (found) found.qty += qty;
  else cart.items.push({ ...item, qty });
  return cart;
}
function cartTotal(cart) {
  const subtotal = cart.items.reduce((s, i) => s + i.price * i.qty, 0);
  const discount = subtotal >= 1000 ? subtotal * 0.1 : 0;
  return Math.round((subtotal - discount) * 100) / 100;
}
function applyCoupon(cart, code) {
  if (code === "SAVE50" && cartTotal(cart) >= 500) cart.coupon = 50;
  else cart.coupon = 0;
  return cartTotal(cart) - cart.coupon;
}
module.exports = { addItem, cartTotal, applyCoupon };

cart.test.js — the "AI-generated" unit suite, verbatim. Do not fix it yet:

const { addItem, cartTotal, applyCoupon } = require("./cart");
let sharedCart = { items: [] };
test("addItem adds an item", () => {
  const result = addItem(sharedCart, { sku: "A1", price: 300 }, 2);
  expect(result).toBe(sharedCart);
  expect(sharedCart.items.length).toBeGreaterThan(0);
});
test("cartTotal returns the total", () => {
  const total = cartTotal(sharedCart);
  expect(total).toEqual(cartTotal(sharedCart));
  expect(typeof total).toBe("number");
});
test("discount applies for big carts", () => {
  addItem(sharedCart, { sku: "B2", price: 400 }, 1);
  const total = cartTotal(sharedCart);
  expect(total).toBeTruthy();
});
test("applyCoupon works", () => {
  const price = applyCoupon(sharedCart, "SAVE50");
  expect(price).toBeLessThanOrEqual(cartTotal(sharedCart));
});
test("rejects bad quantity", () => {
  try {
    addItem({ items: [] }, { sku: "C3", price: 10 }, -1);
  } catch (e) {
    expect(e).toBeDefined();
  }
});

checkout.spec.ts — the "AI-generated" E2E spec (uses any demo page; the anti-patterns are the point, you will not run this one green):

import { test, expect } from "@playwright/test";
test("checkout flow", async ({ page }) => {
  await page.goto("http://localhost:3000");
  await page.waitForTimeout(3000);
  await page.click(".btn-primary");
  await page.waitForTimeout(5000);
  expect(await page.title()).toBeDefined();
});

stryker.conf.json:

{
  "testRunner": "jest",
  "mutate": ["cart.js"],
  "reporters": ["clear-text", "progress"],
  "coverageAnalysis": "perTest"
}

Steps

1. 1Get a green, deceptive baseline. Run npx jest. All five tests pass. Record that in clinic-notes.md — a fully green suite is the starting condition, which is exactly why this defect class survives in real repos.
2. 2Run the mutation baseline. Run npx stryker run and record the mutation score. Expect it to be dismal — most mutants of cart.js survive because the assertions barely constrain behavior. This number is your "before."
3. 3Audit the unit suite against the anti-pattern catalog. Go test by test through cart.test.js and classify every defect you find into: tautological assertion, vacuous assertion, order dependence / shared state, swallowed failure, missing boundary. Write each finding in clinic-notes.md as: test name, line, anti-pattern, why it cannot fail when the code breaks. There are at least six findings; the answer key enumerates them.
4. 4Use AI as review partner — with the rubric, not vibes. Paste cart.test.js and cart.js into Claude/ChatGPT: "Review this test suite against this rubric: sleep-based waits, tautological asserts, assertions that can never fail, shared state between tests, exception tests that pass when no exception is thrown, missing boundary values. For each finding cite the line." Compare its findings to yours; log anything it caught that you missed and anything it missed or invented.
5. 5Refactor the unit suite. Rewrite cart.test.js: fresh cart per test (no module-level shared state), exact-value assertions (expect(cartTotal(cart)).toBe(540)), expect(() => addItem(...)).toThrow("qty must be positive") for the error path, and explicit boundary tests — subtotal 999 vs 1000 for the discount threshold, coupon at total exactly 500, and the merge-quantity path for a repeated SKU. Every assertion should be able to name the mutant it would kill.
6. 6Prove it with mutation testing. Run npx stryker run again. The score should jump substantially (typical honest outcome: from roughly 30–50% to 85%+). For each surviving mutant, decide: add a test to kill it, or write one line in clinic-notes.md on why it is equivalent/not worth killing. No silent survivors.
7. 7Fix the E2E spec on paper. Rewrite checkout.spec.ts in your editor: replace both waitForTimeout sleeps with web-first assertions (await expect(page.getByRole(...)).toBeVisible() auto-waits), replace the class selector with a role-based locator, and replace the vacuous title assertion with a real outcome (await expect(page).toHaveURL(/confirmation/) or an on-page assertion). Annotate each change with the anti-pattern it removes.
8. 8Write the before/after summary. Close clinic-notes.md with a table: mutation score before/after, count of findings by anti-pattern, which the AI reviewer caught vs missed, and the one-paragraph policy you'd propose for AI-generated tests entering your team's repo.

Verify

• •
  clinic-notes.md lists ≥6 unit-suite findings and 3 E2E findings, each with line and anti-pattern class.
• •
  Before and after Stryker scores recorded; after-score ≥ 85% or every survivor justified in writing.
• •
  Refactored suite: no shared mutable state, no toBeTruthy/toBeDefined as primary assertions, error path uses toThrow, boundaries 999/1000 and 500 covered.
• •
  Rewritten E2E spec has zero waitForTimeout calls and at least one real outcome assertion.
• •
  AI-reviewer comparison logged: caught, missed, invented.

Self-grading rubric (pass = 7/10):