Draft the team's review standards for AI-generated code — provenance labels, risk-tiered depth, the author-explains-every-line rule, coverage floors, no-AI zones — then apply them to three AI-generated PRs. One hides a hallucinated dependency and a plausible-but-wrong edge case that generic review waves through. Your standard must catch both.
Elena Vasquez leads two backend teams at Solvantra, a fictional 90-engineer payments-adjacent SaaS. Her PR volume is up 40% since AI tools rolled out; review time per PR is down; and last month a subtly wrong AI-generated branch condition refunded 212 customers twice. The reviewer had approved it in four minutes with "LGTM, nice tests." Elena's diagnosis: the team is reviewing AI code with human-code instincts — trusting fluency as a proxy for correctness — and fluent is exactly what AI-generated defects look like. You will write the standard that fixes this, then prove it works on three sample PRs.
Condensed exports: description, key changes, and metrics as your review tool would show them.
PR-1041 | "Add retry-with-backoff to webhook dispatcher"
Author note: "Generated with AI assistance, reviewed by me."
Files: dispatcher.ts (+84/-12), dispatcher.test.ts (+120)
Key changes (summarized):
- Wraps webhook POST in retry loop: 3 attempts, exponential
backoff 1s/2s/4s, retries on 5xx and network errors only
- Adds jitter; logs each retry with attempt count
- 8 new tests incl. 429 handling (NOT retried - documented)
Metrics: coverage 94% on changed lines; CI green; 6 min review
PR-1042 | "Normalize customer address input on signup"
Author note: none. (Commit messages match AI-tool default style.)
Files: address.ts (+141/-8), address.test.ts (+96), package.json (+1)
Key changes (summarized):
- Adds dependency "addr-normalize-intl" for street/locality
parsing (package.json: "addr-normalize-intl": "^2.3.1")
- Trims, title-cases, expands abbreviations (St -> Street)
- Edge case per description: "Addresses with a unit designator
but no street number (e.g. 'Unit 4, Elm Street') are treated
as PO-box type and routed to manual verification"
- 11 new tests, all passing locally per author
Metrics: coverage 91%; CI pending; review not started
PR-1043 | "Refactor invoice PDF service for testability"
Author note: "AI-assisted refactor, no behavior change intended."
Files: 9 files (+610/-540), no test changes
Key changes (summarized):
- Extracts rendering into injectable interface
- Moves currency formatting to shared util (existing tests pass)
- Deletes two "unused" private methods flagged by the AI
Metrics: coverage unchanged 71%; CI green; large-diff warning
SPEC EXCERPT | signup address handling (from product spec v3.2)
"Any address that cannot be fully parsed into street number,
street, and locality MUST be rejected back to the user with an
inline correction prompt. Manual-verification routing exists ONLY
for validated addresses flagged by the fraud model. PO-box
detection applies solely to explicit PO Box / postal-box formats."
Review all three PRs cold — fifteen minutes, no standard, no AI. Read like a busy reviewer: would you approve each as shown? Write your verdicts down. Most people approve at least two. Keep this; comparing it to your post-standard review is the point of the lab.
Draft the provenance rule. Define required AI-origin labeling: a PR-level declaration (none / assisted / substantially generated), applied by the author, with no penalty attached — the label routes review depth, it does not judge the author. Note which of the three PRs would have required a label it does not have.
Define risk tiers and review depth. Three tiers by blast radius: Tier 1 (money movement, auth, PII, data deletion) — line-by-line review, author walkthrough required, two reviewers; Tier 2 (business logic, integrations) — line-by-line, one reviewer, verification evidence in the PR; Tier 3 (internal tooling, docs) — standard review. Classify each sample PR into a tier and write down why.
Write the author-explains-every-line rule. The core clause: the author must be able to explain any line on request, and "the AI wrote it" is never an accepted answer — submitting code asserts you understand it. Add the walkthrough trigger: Tier 1 always; any tier when the reviewer asks. Add the dependency clause: every new dependency must be justified in the PR — name, license, maintenance status, and why no existing dependency suffices — verified by the author against the real registry, not the AI's description.
Set coverage floors and no-AI zones. Coverage floor per tier for changed lines (e.g., Tier 1: 95% with mutation-spot-checks; Tier 2: 90%; Tier 3: team default) — and the anti-gaming note: AI-generated tests asserting wrong behavior satisfy coverage while cementing bugs, so Tier 1–2 review includes reading the tests against the spec, not against the code. Declare no-AI zones if any (e.g., crypto, secrets handling) or explicitly declare none and require Tier 1 process instead.
Apply the standard to PR-1041. Annotate: tier, label check, line-by-line concerns, tests-vs-spec check, verdict. This PR is broadly competent — the graded skill here is not inventing problems: a standard that fails clean PRs gets abandoned in a month.
Apply the standard to PR-1042 — slowly. Follow your own clauses in order. The dependency clause: verify addr-normalize-intl exists — search the registry, check downloads, license, last publish. Log what you find. The tests-vs-spec check: compare the stated edge-case behavior against the spec excerpt line by line. Write the verdict your standard forces, and name the specific clause that forced each finding.
Apply the standard to PR-1043. Large "no behavior change" refactors with deleted "unused" methods and no test changes: what does your standard require? (Reasonable answers: characterization-test evidence before approval, usage-search proof for deletions, splitting the diff.) Annotate and rule.
Compare against your cold review and finalize. Write the delta memo: what the cold review approved that the standard caught, which clause did the work, and the one-page team rollout note. If the standard caught nothing your cold review missed, your clauses are decorative — tighten the dependency and tests-vs-spec clauses and redo step 7.
AI-generated defects do not look like sloppy code — they look like clean, well-tested, confidently documented code that is wrong about the world outside the diff.
A complete lab has all of the following:
Review standards for AI code exist to replace trust in fluency with demands for evidence — a real registry entry, a spec the tests actually assert, proof behind "no behavior change." The author explains every line; the reviewer verifies the world outside the diff.