Lab 05 · Component Tests That Survive Mutation

Sara's team lets Copilot write most component tests now. Coverage is up, the dashboards are green, and last month a one-line change inverted the signup form's validation — every test still passed. Coverage measures what code ran, not what behavior was checked; an assertion-free test executes plenty and verifies nothing. Mutation testing is the instrument that tells these apart: it deliberately breaks the component in dozens of small ways and counts how many breaks your tests notice. In this lab, the AI-generated suite you start from is seeded with exactly that pathology — decorative assertions that mutants survive.

The reason this matters for AI-generated tests specifically is that generators optimize for the reward they can see. A test that renders a component and asserts it exists is short, plausible, and turns CI green — so an assistant produces it happily, and a reviewer skimming a green diff waves it through. Coverage tools reinforce the illusion: a test that executes every line while asserting nothing meaningful shows 100% coverage. Mutation testing is the only cheap, automatable signal that grades assertion strength rather than line execution, which is precisely the axis on which AI-drafted suites are weakest. You are about to make that weakness visible as a number, then drive the number up by rewriting around behavior.

Time & tier

~75 minutes hands-on · Verify stage (maps to module M5, playbooks PB-22/PB-23) · $0 — Vitest, React Testing Library, StrykerJS, all local.

Setup

npm create vite@latest mutation-lab -- --template react-ts
cd mutation-lab && npm install
npm install -D vitest @testing-library/react @testing-library/user-event \
  @testing-library/jest-dom jsdom @stryker-mutator/core @stryker-mutator/vitest-runner
mkdir -p src/components

vitest.config.ts:

import { defineConfig } from "vitest/config";
import react from "@vitejs/plugin-react";

export default defineConfig({
  plugins: [react()],
  test: { environment: "jsdom", globals: true, setupFiles: "./src/setup.ts" },
});

src/setup.ts contains one line: import "@testing-library/jest-dom";

The component under test, src/components/SignupForm.tsx:

import { useState, type FormEvent } from "react";

export function SignupForm({ onSubmit }: { onSubmit: (email: string, pw: string) => void }) {
  const [email, setEmail] = useState("");
  const [pw, setPw] = useState("");
  const [errors, setErrors] = useState<{ email?: string; pw?: string }>({});

  const validate = () => {
    const next: { email?: string; pw?: string } = {};
    if (!/^[^@\s]+@[^@\s]+\.[^@\s]+$/.test(email)) next.email = "Enter a valid email";
    if (pw.length < 10) next.pw = "Password must be at least 10 characters";
    else if (!/\d/.test(pw)) next.pw = "Password must contain a number";
    return next;
  };

  const handleSubmit = (e: FormEvent) => {
    e.preventDefault();
    const next = validate();
    setErrors(next);
    if (Object.keys(next).length === 0) onSubmit(email, pw);
  };

  const disabled = email === "" || pw === "";

  return (
    <form onSubmit={handleSubmit} noValidate>
      <label htmlFor="email">Email</label>
      <input id="email" type="email" value={email} aria-invalid={!!errors.email}
             onChange={(e) => setEmail(e.target.value)} />
      {errors.email && <p role="alert">{errors.email}</p>}
      <label htmlFor="pw">Password</label>
      <input id="pw" type="password" value={pw} aria-invalid={!!errors.pw}
             onChange={(e) => setPw(e.target.value)} />
      {errors.pw && <p role="alert">{errors.pw}</p>}
      <button type="submit" disabled={disabled}>Create account</button>
    </form>
  );
}

The AI-generated suite you inherit, src/components/SignupForm.ai.test.tsx — commit it exactly as written:

import { render, screen } from "@testing-library/react";
import userEvent from "@testing-library/user-event";
import { describe, it, expect, vi } from "vitest";
import { SignupForm } from "./SignupForm";

describe("SignupForm", () => {
  it("renders the form", () => {
    render(<SignupForm onSubmit={vi.fn()} />);
    expect(screen.getByLabelText("Email")).toBeInTheDocument();
    expect(screen.getByLabelText("Password")).toBeInTheDocument();
    expect(screen.getByRole("button")).toBeInTheDocument();
  });

  it("lets the user type", async () => {
    render(<SignupForm onSubmit={vi.fn()} />);
    await userEvent.type(screen.getByLabelText("Email"), "a@b.com");
    await userEvent.type(screen.getByLabelText("Password"), "hunter2hunter2");
    expect(screen.getByLabelText("Email")).toHaveValue("a@b.com");
  });

  it("submits the form", async () => {
    const onSubmit = vi.fn();
    render(<SignupForm onSubmit={onSubmit} />);
    await userEvent.type(screen.getByLabelText("Email"), "a@b.com");
    await userEvent.type(screen.getByLabelText("Password"), "hunter2hunter2");
    await userEvent.click(screen.getByRole("button"));
    expect(screen.getByRole("button")).toBeInTheDocument();
  });

  it("shows validation", async () => {
    render(<SignupForm onSubmit={vi.fn()} />);
    await userEvent.type(screen.getByLabelText("Email"), "not-an-email");
    await userEvent.type(screen.getByLabelText("Password"), "short");
    await userEvent.click(screen.getByRole("button"));
    expect(document.querySelectorAll("p").length).toBeGreaterThanOrEqual(0);
  });
});

Stryker config, stryker.config.json:

{
  "testRunner": "vitest",
  "mutate": ["src/components/SignupForm.tsx"],
  "thresholds": { "high": 90, "low": 80, "break": 80 },
  "reporters": ["clear-text", "html"]
}

Guided steps

1. 1Green-check the inherited suite.npx vitest run — expected: 4/4 pass. Note your gut feeling about suite quality before measuring; you'll calibrate against it later.
2. 2Run the mutation gate.npx stryker run. Expected: the run fails the 80% break threshold, typically landing a score in the 30–55% range. Open reports/mutation/mutation.html — every surviving mutant is listed with the exact mutation applied (e.g., pw.length < 10 → pw.length <= 10).
3. 3Triage ten survivors into classes. In mutation-log.md, pick ten surviving mutants and classify each: which behavior is untested (boundary, error message, callback contract, disabled logic)? Map each survivor back to the test that should have killed it and say why it didn't. Expected pattern: the assertions are about existence (toBeInTheDocument), not behavior.
4. 4Rewrite the suite as behavior-first, accessible-query tests. Create SignupForm.test.tsx (delete the .ai. file from the Vitest include or keep both and let both run). Rules: query only via getByRole/getByLabelText; assert outcomes — onSubmit called with exact arguments on valid input, onSubmitnot called on invalid input, specific role="alert" messages, button disabled/enabled transitions. You may draft these with your assistant, but every assertion must name a user-observable behavior.
5. 5Target the boundaries explicitly. The component has decision points: email regex pass/fail, length < 10 boundary (test 9 and 10 chars), digit-required rule (test a 10+ char password with no digit), empty-fields disabled state (each field alone and both). Write one test per boundary side. This is where most mutants live.
6. 6Re-run Stryker and iterate.npx stryker run. For each remaining survivor: kill it with a sharper assertion, or write one line in the log justifying it as equivalent/acceptable (e.g., a mutant only reachable through an impossible state). Iterate until the score clears 80% and the run passes the break threshold.
7. 7Compare the two suites in the log. Test count, line coverage (roughly equal!), mutation score, and the three most instructive killed-vs-survived examples. Finish with a two-sentence team policy: what mutation threshold gates AI-generated tests, and who pays the iteration cost.

A word on the mindset shift this lab is training. When a test suite passes, the instinct is relief; when Stryker reports a 40% score against that passing suite, the instinct is to argue with the tool. Resist it. Every surviving mutant is a concrete, reproducible claim: "I changed the component's behavior in this specific way and your tests did not object." That is not a false alarm — it is a test you forgot to write, expressed as the smallest possible failing example. The engineers who get the most from mutation testing treat the survivor list the way they treat a bug tracker: each entry is a task with a clear definition of done (write the assertion that kills it, or justify in one sentence why the mutant is genuinely equivalent). Do that mechanically and the score climbs on its own; the 80% threshold is not a mountain, it's the natural result of taking each survivor seriously.

Verify

• •
  Initial .ai. suite: 4/4 green under Vitest and a failing Stryker score, both recorded with numbers.
• •
  mutation-log.md classifies ten survivors with the missing behavior named for each.
• •
  Final suite: mutation score ≥ 80%, break threshold passes, and every remaining survivor has a written justification.
• •
  Every query in the final suite is getByRole/getByLabelText — grep -n "querySelector\|getByTestId" src/components/SignupForm.test.tsx returns nothing.
• •
  The final suite still passes when run alone (npx vitest run) — you improved assertions, not the component.

Self-grading rubric (pass = 7/10)

• Diagnosis (0–3): ten survivors classified with the untested behavior named, not just the mutated line quoted = 3.
• Score (0–3): ≥ 80% honestly (no mutate scope-shrinking, no threshold edits) = 3; 70–79% with justified survivors = 2.
• Assertion quality (0–2):onSubmit contract tested both ways (called with exact args; not called on invalid) = 2.
• Boundary discipline (0–2): 9/10-char password boundary and no-digit rule each have a dedicated test = 2.