This module installs provenance-aware review: reading the ticket before the diff, aiming your scrutiny at the defects AI produces rather than the ones humans do, and re-layering the gate so machines catch the mechanical and human attention lands only where judgment is required.
Sana's idempotency PR lands — disclosed as substantially AI-generated, 340 lines, tests green, CI passing. A reviewer starting at line 1 would approve it in ten minutes; it reads clean.
Vikram has a reputation on the team: the reviewer who reads the ticket before the diff. Intent: idempotency for POST /orders, four header cases, nothing else.
The change also "tidied" error handling in an adjacent handler the ticket never mentioned. Scope drift — flagged for revert. ITEM 2 · HALLUCINATED SURFACE | A call to ResponseStore.get_or_set(key, ttl=...) — except orders-api's ResponseStore has no get_or_set. The AI invented a convenient method that compiled only because a stub happened to exist in the tests. ITEM 3 · CONTRACT VS BODY | A docstring promising "raises on duplicate key" sitting over a body that returns silently.
None of these were visible to a reader who trusted the green check. All three were visible to a reader applying the provenance-aware checklist in order.
Three specific asks. The merge will still carry Sana's name, and the accountability with it.
Code review was designed for a world where writing code was the expensive step. That world is gone: DORA 2025 measured individual developers merging +98% more pull requests with AI assistance while organizational delivery stayed flat — which means the constraint moved from generation to review, and the flat delivery numbers are largely the review gate either eroding or gridlocking.
At the same time, the PRs arriving at your review queue increasingly weren't written by a human at all, and AI-generated code fails in patterns human code doesn't. This module is about operating on both sides of that gate as a senior engineer: using AI to raise the quality of what you submit, configuring AI to carry the mechanical share of what your team reviews, and reserving scarce human attention for the judgments only humans can make — because if you sign the merge, the defects are yours, whoever or whatever wrote the code.
At 2x PR volume, review capacity is the delivery bottleneck. The teams that win don't review less — they re-layer review so machines catch the mechanical, authors pre-clear the obvious, and humans spend their attention exclusively where judgment is required.
Every nit a human reviewer writes on your PR is a round-trip measured in hours. Every nit an AI reviewer catches before you open the PR costs seconds. The pre-review pass is the cheapest review-capacity win available, and it is standard practice at engineering organizations that have institutionalized AI review — Shopify's Roast framework exists precisely to run structured AI passes before human eyes get involved, and GitHub ships Copilot code review for the same pre-human slot.
A disciplined pre-review pass has three parts:
1. The self-review prompt. Run your own diff through an AI reviewer with your team's standards in context — before pushing:
Review this diff as a strict senior reviewer on our team.
Standards: [paste or reference your conventions file].
Report: (a) correctness risks with the specific line and mechanism,
(b) convention violations, (c) missing test coverage for changed
branches, (d) anything in the diff not required by the stated
intent. Do not comment on style the formatter already enforces.
Intent of this change: <one-sentence summary>.
The intent line is load-bearing — it lets the AI flag scope drift in your own work, which is exactly the thing you're blind to after two hours in the change. Fix what's real, discard what's noise, and note the discards (they'll calibrate your instruction files later).
2. The description that saves the reviewer ten minutes. Generate the PR description from the diff, then edit it for honesty: what changed, why, what you verified, what you're unsure about. The "what I'm unsure about" section is the highest-value sentence in the description — it aims the human reviewer at the exact spot where their judgment is needed, and it's the disclosure norm from the Common Core applied at PR granularity.
3. Test evidence, attached. Paste the test run output (or CI link) covering the changed behavior, including the new test that fails without your change. A PR that arrives with proof it works redirects review attention from "does this work?" to "should this exist, and is it designed right?" — the questions humans are actually for.
Run the pre-review pass with a different model or tool than the one that helped write the code. An AI reviewing its own generation shares its own blind spots; a second model breaks the correlation cheaply. Copilot-generated code reviewed by Claude (or vice versa) catches more than either reviewing itself.
An individual pre-review pass is a habit; a team review bot is infrastructure. GitHub's Copilot code review, CodeRabbit, and self-hosted patterns like Shopify's Roast all converge on the same architecture: an AI reviewer that runs on every PR, configured by instruction files checked into the repo — copilot-instructions.md and path-scoped *.instructions.md for Copilot, equivalent config for CodeRabbit, prompt files for Roast-style setups. As of GitHub's July 2026 changelog, Copilot reads review instructions from the head branch and supports firewalled review configurations — the instruction file is now a real, versioned engineering surface, and tuning it is a senior-engineer responsibility, not an install-and-forget.
Instruction files fail in two symmetric ways: too vague ("ensure code quality") produces generic nits; too voluminous produces instruction-following decay. What works is a small number of specific, checkable rules with examples:
# Review instructions
- Flag any new endpoint missing an authorization check. Our
pattern: `@require_permission(...)` decorator on the handler.
- Flag raw SQL string interpolation. Required: parameterized
queries via the repository layer.
- Flag `except Exception` without re-raise or structured logging.
- Do NOT comment on: import ordering, line length, docstring
style (enforced by linters), or test file naming.
- For files under /payments: flag any float arithmetic on
amounts; we use integer cents.
Note the explicit "do NOT comment on" block — suppressing what deterministic tools already enforce is the single biggest noise reduction available, and it embodies the layering principle: linters and type-checkers catch what they can catch deterministically; the AI reviewer covers what needs pattern judgment; humans cover what needs context.
Treat the review bot as a system with a measurable signal-to-noise ratio. The tuning loop: collect two weeks of bot comments, label each acted-on, dismissed-correct-but-trivial, or wrong; then edit instructions to suppress the trivial and wrong classes and re-measure. A practical target: if fewer than half of bot comments are acted on, the bot is training your team to scroll past it — and a review bot the team has learned to ignore is worse than none, because it provides the feeling of a review layer without the layer. Seed the evaluation deliberately: keep a small set of PRs with known planted violations of your standards, and check the bot still catches them after every instructions change — a regression test for your review config, which is the evals mindset from the Common Core applied to review tooling.
Review-bot comments are suggestions from an unauthenticated colleague, not gates. The bot approving a PR means nothing; the bot's comments being addressed means nothing by itself. The merge decision remains a human's, with a human's accountability — every mature policy, Coinbase's most explicitly, holds this line.
When a PR is substantially AI-generated — disclosed by the author, or evident from its texture — the reviewer's priors must change. Human authors make human errors: typos, forgotten edge cases they'd recognize instantly if asked. AI authors make taxonomy errors: fabricated APIs formatted perfectly, tests that assert the code's misunderstanding, confident docstrings over noncompliant bodies, and diffs that wander beyond the ticket. Reviewing AI-generated code with human-error priors is how "almost right" reaches production. This checklist re-aims the review; it is also, nearly verbatim, the rubric AI-enabled interview formats at Meta and Google score candidates against.
The provenance-aware review checklist:
Time-box the checklist by risk tier (Common Core trust tiers): a T2 internal utility gets items 1–6 briskly; anything T3 — production paths, money, auth, data integrity — gets all ten, unhurried.
Assessment items present a realistic AI-generated PR and score which checklist items you apply and in what order. Intent-match before line-reading is the discriminating behavior — reviewers who start at line 1 of the diff miss scope drift almost every time.
Your first 15 minutes reviewing any AI-generated PR: read the ticket and write the intent down before you open the diff, then check intent-match first — the scope drift and the missing requirement are invisible to a reviewer who starts at line 1, which is exactly how Vikram caught the tidy-up Sana's agent slipped into an adjacent handler.
The checklist's unifying question is never "does this look right?" — AI output always looks right. It is "what would prove this wrong, and did anyone run that check?"
Security review is the sharpest case for a separate AI pass, because general review prompts reliably under-weight security: asked to "review this code," models allocate attention to correctness and style and mention security only when it's flagrant. The fix is a dedicated pass with an explicit checklist and a triage discipline — the pattern Meta engineers report using AI for pre-merge security catches.
The prompt structure:
Security-focused review of this diff ONLY. Check against:
- Injection: SQL/command/template/log injection via any
user-influenced value reaching an interpreter
- AuthZ: every new/changed endpoint or handler — is authorization
checked? At the right granularity (object-level, not just role)?
- Secrets: hardcoded credentials, tokens in logs, keys in config
- Deserialization and parsing of untrusted input
- SSRF/path traversal on any URL or filepath built from input
- Sensitive-data exposure in logs, errors, or responses
For each finding: CWE class, the exact line, the attack scenario
(who sends what, from where), and exploitability given our
context: [internal-only service / internet-facing / handles PII].
Rank by exploitability, not severity-in-the-abstract.
Two disciplines make this useful rather than noisy:
../ in report_name to read arbitrary files as the service account") rather than a category name. You then triage: exploitable-now gets fixed pre-merge, defense-in-depth gets a ticket, not-applicable gets a written reason (which feeds your instructions file).For AI-generated diffs the security pass is non-optional, because generated code inherits the average security posture of its training data — which is to say, mediocre: string-built SQL, broad exception handling that hides failed security checks, and example-grade crypto appear at meaningful rates in generated code and are precisely what checklist item 8 exists to catch.
Everything above is mechanism; this section is the operating model. The DORA finding bears repeating as a design constraint: individual throughput up (+21% tasks, +98% PRs merged), organizational delivery flat. The gap is the gate. Teams respond in one of three ways — two of them wrong:
Re-layering has concrete parts:
Review is also where team AI policy becomes real. Disclosure norms, provenance labeling, and tier agreements (which artifacts demand full-checklist review) live or die at the PR gate — a policy the review process doesn't enforce is a wiki page, not a policy.
Check yourself: Why is reviewing an AI-generated PR with the same instincts you'd use on a human's PR a mistake, and what is the checklist's single unifying question?
This module's playbooks package the operational assets: the pre-review prompt and PR-description template, a starter review-instructions file with the tuning-loop worksheet, the ten-item provenance-aware checklist as a printable review card, and the security-lens prompt with the exploitability-triage rubric. Do this at work this week → pb-17 Run a Pre-Review AI Pass on Your Own PR (arrive review-ready before Vikram ever sees it), pb-19 Review Large and AI-Generated PRs Without Drowning (the provenance checklist plus large-diff comprehension tooling), and pb-20 Security-Focused AI Review Pass: Checklist-Driven, Exploitability-Triaged (the dedicated pass general review under-weights). The paired labs are the review-pipeline lab (configure and tune an instructions file against seeded violations until signal beats noise) and the reviewing-the-machine lab (the timed 1,500-line AI-generated PR — the closest rehearsal this track offers for the AI-enabled interview formats now spreading through product companies and GCC pipelines). Run the pipeline lab first; the checklist lab assumes you've felt what bot noise looks like.