Run a Test-Code Review Clinic on AI-Generated Suites

Tier: Core · Workflow area: Automation engineering

Treat this as a clinic you run on any suite with significant AI authorship: your own pb-08/pb-09 output, a teammate's agent-built PR, or an inherited repo full of generated specs nobody trusts. The rubric turns "this looks off" into named findings; the grep gates automate the cheap half; mutation testing settles the arguments.

When to use

• Reviewing an agent-authored test PRyours or anyone's; generation volume has outrun hand-review instincts.
• Auditing an inherited AI-heavy suitethe repo full of tests nobody understands, before you build on it.
• Setting the team's merge baryou want one written standard for what generated test code must clear.

When NOT to use:

• Reviewing application codethis rubric targets test-specific defects; app-code review (including AI-authored app code, pb-49) has its own checklist.
• Three-line test tweaksproportionality matters; run the greps, skip the ceremony.

The recipe

1. 1.

   Run the mechanical gates first — free findings before human minutes:

   Copy

   grep -rn "waitForTimeout\|sleep(\|time\.sleep\|cy\.wait([0-9]" tests/
   grep -rn "expect(true)\|toBeTruthy()\s*$\|assert True" tests/
   grep -rn "\.only(\|\.skip(" tests/
   npx playwright test --repeat-each=3 --workers=4
   

   The repeat run with parallel workers flushes both nondeterminism and order dependence before anyone reads a line.

2. 2.

   Apply the anti-pattern rubric. Review each file against the named catalog — findings cite the pattern, not taste:

   • •
     Sleep-based waits — fixed delays instead of condition waits; the top generated-flake source. Verdict: replace with web-first assertions or explicit condition polling.
   • •
     Tautological asserts — the test asserts what it arranged: a mock returns X, the test asserts X; or expect(result).toEqual(result) in disguise via shared helpers. Verdict: assert a consequence the code under test computed, not an input restated.
   • •
     Over-mocking — so many collaborators stubbed that the subject can't fail; integration tests that mock the integration. Verdict: mock at the boundary you don't own (network, clock), keep everything inside real.
   • •
     Order dependence — tests passing only in file order, sharing module state or leaked data. Verdict: the repeat/parallel run from step 1 is the detector; fixtures with real teardown (pb-12) are the fix.
   • •
     Assertion-free passages — long arrange-act sections with one weak assert at the end, or coverage-only tests with no assert at all.
   • •
     Copied-context asserts — assertions hallucinated from another project's conventions (wrong error text, invented status codes) that pass only because the assert is loose.
3. 3.

   Use an assistant as first-pass reviewer — with adversarial framing:

   Copy

   Review these test files against this anti-pattern catalog: [paste step-2
   list]. For each finding: file:line, pattern name, why it can mask a
   real failure, and a minimal rewrite. Then answer: which three tests in
   this suite would still pass if the feature they test was completely
   broken? Explain how.
   

   That final question is the highest-yield prompt in test review — it forces the model to reason about detection power, and its answers are checkable.

4. 4.

   Check the checkable claims. For each "would still pass if broken" claim, verify one by hand: stub the feature out or flip its logic and run the named test. Confirmed claims become PR findings with proof attached; unconfirmed ones are discarded — the human verdict is the finding, the model only drafted suspicion.

5. 5.

   Settle disputes with mutation testing. When the author insists an assert is strong enough:

   Copy

   npx stryker run --mutate "src/checkout/**/*.ts"
   

   Copy

   mutmut run --paths-to-mutate src/checkout/ && mutmut results
   

   Surviving mutants in the code a disputed test covers end the argument with data (the full loop is pb-09). Record the score in the PR.

6. 6.

   Close the loop into prevention. Every accepted finding becomes a line in the pb-08 rules file or a new grep gate in CI — the clinic's output is a stricter generator next week, not just a cleaner PR today.

Make it yours

• Python/pytest clinicssame rubric; the greps target time.sleep, bare assert on truthy mocks, and fixture-scope leaks (scope="module" state mutated by tests); mutmut is the tiebreaker.
• CI-enforced rubriccodify the greps plus --repeat-each into a test-quality CI job so the mechanical half of the clinic runs on every PR without a human; reserve human review for assertion strength.
• Team calibration workshoprun the clinic on a deliberately seeded suite (plant one of each anti-pattern) as a training exercise; a team that has hunted planted defects reviews live PRs measurably faster and agrees more.

Quality check

• Every finding names a pattern from the catalog with file:line — no "this feels fragile" comments in the final review.
• The would-pass-if-broken list was empirically checkedat least one claim verified by breaking the feature; the proof is linked in the PR.
• Repeat/parallel runs are green post-fix — --repeat-each=3 --workers=4 (or pytest -p no:randomly off, randomization on) with zero flakes.
• No mock-echo asserts surviveeach remaining mock has a one-line justification for why the boundary is stubbed.
• Disputes were settled by mutation score, not seniorityand the score appears in the PR description.
• The rules file changeda clinic that fixed a PR but didn't tighten the generator will re-review the same defects next sprint.