Validate AI-Generated Dev Code as a QA Function

Tier: Core · Workflow area: Quality leadership & governance

This playbook gives QA a repeatable protocol for the PR streams AI is now producing: triage by risk, review against an AI-specific defect checklist, and answer weak spots with tests — because the scalable response to suspicious code is not longer arguments, it is a failing test.

When to use

Prerequisites

• •
  Visibility into which PRs are substantially AI-generated — a disclosure convention (see the policy playbook) or a working agreement with dev leads.
• •
  Coverage tooling on the repos in question, and mutation testing (Stryker, mutmut) available for the high-risk paths.
• •
  An agreed lane: QA validates behavior and risk; style, architecture and idiom remain the dev reviewer's lane. Written down, or this becomes a turf war.

The workflow

1. 1.

   Triage incoming AI-heavy PRs by risk, not by order. A two-minute score per PR: touches money, auth, PII, or data migration (+3 each); modifies error handling or concurrency (+2); >400 changed lines (+2); area with weak test coverage (+2); pure refactor with strong existing tests (−2). QA deep-validates the top of the list; the rest get the checklist pass only. Publishing the scoring rule matters more than the exact weights — it makes QA's attention allocation legible and negotiable.

2. 2.

   Review against the AI-specific defect checklist. AI code fails differently from human code — plausible, confident, and wrong at the seams. The checklist encodes the known hotspots:

   • •
     Edge and boundary handling — empty collections, None/null, zero, negative, max-size: AI code handles the happy path it inferred and invents nothing for the boundary the prompt never mentioned.
   • •
     Error-path realism — catches that swallow, retries without backoff, error messages referencing nonexistent context; ask "what happens when the dependency is down?" of every new call.
   • •
     Hallucinated interfaces — methods, config keys, or library flags that do not exist in your versions; verify any API you do not recognize against the installed package, not documentation memory.
   • •
     Silent behavior drift in refactors — "equivalent" rewrites that reorder side effects, change rounding, or drop a filter clause; diff behavior, not text.
   • •
     Concurrency plausibility — locks acquired but not released on error paths, non-atomic check-then-act patterns.
   • •
     Security seams — string-built queries, unvalidated input at trust boundaries, secrets in new config surface.
3. 3.

   Interrogate the tests that came with the PR. AI PRs arrive with AI tests, and generated tests systematically flatter generated code — they assert what the code does, not what it should do. Check for tautological asserts, mocks of the very behavior under test, and missing negative cases. Then let mutation testing arbitrate on the high-risk paths:

   Copy

   npx stryker run --mutate "src/billing/**"
   

   Copy

   Mutation score src/billing: 41% (23/56 mutants killed)
   Survived: boundary-condition mutants in proration.ts (7),
             error-path mutants in refund.ts (9)
   

   A 41% score on new billing code converts "these tests look thin" from an opinion into a number a PR conversation can act on.

4. 4.

   Answer weak spots with targeted tests, not commentary. For each surviving-mutant cluster or checklist finding, write (or generate-and-review) the test that pins the correct behavior:

   Copy

   Prompt to your coding agent:
   Write pytest cases for proration.calculate() covering: month-end
   anchor dates (Jan 31 -> Feb), zero-day proration, negative line
   items, and the documented rounding rule (banker's rounding per
   ADR-114). Assert exact expected values computed independently
   from the spec — do not derive expectations by calling the
   function under test.
   

   That last constraint is the anti-flattery clause: expectations must come from the spec, or the test enshrines the bug. Review the generated tests against the same checklist — turtles all the way down, gates at every layer.

5. 5.

   Verify behavior parity on AI refactors. For "no functional change" PRs, run the old and new implementations against the same input corpus (golden inputs from production traces where possible) and diff outputs. Any divergence is either a bug or an undocumented improvement — both need a human decision recorded in the PR.

6. 6.

   Feed findings back as prevention. Escapes and review catches aggregate into two artifacts: an updated checklist (new hotspot classes earn a bullet) and prompt guidance for dev teams ("always specify rounding rules and boundary behavior in codegen prompts"). QA reviewing AI code without feeding patterns back is a filter; with the loop closed it is a control system.

Variations

• Pairing model: for the highest-risk PRs, QA joins the dev review synchronously rather than gating afterward — 30 minutes of joint checklist walking beats a week of comment ping-pong.
• Sampling model: where volume swamps even triage, validate a random 20% of AI-heavy PRs deeply and publish the escape-rate trend — the sample keeps everyone honest and sizes the real risk.
• Tooling assist: run your own review agent over the diff with the checklist as its rubric to pre-flag candidates — treat its output as a draft finding list to verify, never as the verdict.

Quality check

• Risk triage rule is written and publishedanyone can predict which PRs get deep validation.
• Checklist findings phrased as behaviors ("fails on empty cart") with a test attached or requested, not as code-style opinions.
• Mutation scores recorded for high-risk paths before and after QA's added tests.
• Parity runs on refactor PRs with the input corpus and diff results linked in the PR.
• A feedback artifact existsthe checklist's git history shows it learning from escapes.

Pitfalls

• Becoming the style police. Fastest way to lose the lane and the goodwill. Behavior, risk, evidence.
• Trusting the PR's own green tests. Coverage percentage from tests generated alongside the code measures agreement, not correctness. Mutation testing or independent expectations, always.
• Line-by-line reviewing everything. Volume is the problem; undifferentiated effort is not the answer. Triage is the playbook's load-bearing step.
• Accepting "the AI wrote it" as a defect explanation. Ownership rules from the policy playbook apply: the merging engineer owns the code; QA's findings route to a human, every time.