Review Large and AI-Generated PRs Without Drowning

Tier: Core · Workflow area: Code Review

This playbook combines AI-assisted comprehension of oversized diffs (summaries, risk hotspots, review order) with a provenance-aware checklist for the defects AI authorship specifically introduces: hallucinated APIs, phantom tests, over-broad diffs, and license smells.

When to use

• •
  A 1,000+ line PR where "Files changed" alphabetical order guarantees you review the trivial files first and tire before the risky ones.
• •
  Any PR you know or suspect was substantially AI-generated (agent-authored PRs, teammates using coding agents, external contributions).
• •
  Time-boxed review situations — you have 45 minutes for a diff that deserves three hours.

Human-authored 50-line PRs need neither the harness nor the checklist; just read them.

Prerequisites

• •
  The PR checked out locally or the full diff available: gh pr diff <number> > /tmp/pr.diff.
• •
  An assistant with context for the whole diff (Claude, Copilot Chat) — or an agent with repo access for cross-file checks.
• •
  Authority to say "split this PR" — comprehension tooling is a mitigation, not an endorsement of 3,000-line PRs.

The workflow

1. 1.

   Build a review map before reading a single hunk.

   Copy

   Here is a PR diff. Produce a review map:
   1. One-paragraph summary of what this PR actually does (from the code, not
      the description — I will diff your summary against the author's).
   2. Group changed files into: core logic / tests / config & wiring /
      generated or mechanical.
   3. Risk-rank the core files: state mutation, auth/permissions, money,
      concurrency, data migration, external I/O rank high; renames rank low.
   4. Recommend a reading order and flag any file whose changes do not fit
      the PR's stated purpose.
   

   Diff the model's summary against the PR description — a mismatch means either a misleading description or smuggled changes; both are findings.

2. 2.

   Read the top-risk files yourself, fully. The map buys focus, not exemption. Spend your best hour on the 3–5 risky files; skim mechanical groups with AI-generated per-group summaries ("all 30 files: same rename, plus these 2 exceptions" — verify the exceptions).

3. 3.

   Run the provenance checklist (for AI-generated or unknown-provenance PRs). Four defect classes that human review habits miss because human authors rarely produce them:

   • •
     Hallucinated APIs and versions: every import and unfamiliar call must resolve. Fast check: does CI actually build; do dependency versions in the lockfile exist upstream? Agents have invented package versions and even packages (a supply-chain risk — squatters publish malicious packages under commonly hallucinated names).
   • •
     Phantom or weakened tests: tests that cannot fail (assert on their own setup, mock the function under test, snapshot-everything), or existing assertions quietly loosened to make the suite pass. Diff test files with extra care; ask: "For each test in this diff, what code change would make it fail?"
   • •
     Over-broad diffs: unrequested refactors, formatting churn, and drive-by "improvements" mixed into the feature. Every hunk outside the stated scope is a revert candidate by default.
   • •
     License/IP smells: verbatim-looking blocks with alien style, embedded attributions, or suspiciously complete algorithm implementations. Spot-check the most complete-looking block with a code search before merging into proprietary code.
4. 4.

   Interrogate instead of line-comment. For a suspect area, ask the assistant targeted questions with the diff in context — "Under what input does retry_with_backoff retry forever?"; "Which callers of normalize() see behavior change from this signature edit?" — then verify the answer in the code. This is faster than deriving everything yourself and safer than trusting the summary.

5. 5.

   Verdict with teeth. Approve only what you actually reviewed. Legitimate outcomes include: "split this into 4 PRs along the review map's groups", "remove the out-of-scope hunks, then re-request", and "provide failing-test evidence for the bug this claims to fix". For agent-authored PRs, require the author (human sponsor) to attest they reviewed it first — an unreviewed agent PR is a draft, not a review request.

Tool variants

• GitHub PR summaries + Copilot: built-in summary for step 1's first pass; Copilot Chat on the PR for interrogation; assign Copilot code review as an extra nit-catcher, not a substitute for your pass.
• Claude Code with the branch checked out: strongest for cross-file verification — it can jump to definitions to kill hallucinated-API doubts and run the test suite with individual tests reverted to expose phantom tests.
• CodeRabbit: auto-generates walkthrough + file-grouping on PR open, close to step 1's map out of the box; still run the provenance checklist yourself.

Verification

• •
  Your review verdict cites specific files/lines for every blocking finding — no "generally looks fine".
• •
  Model summary vs PR description mismatches were resolved (description fixed or smuggled changes removed).
• •
  All new dependencies verified to exist upstream at the pinned version; CI green on a clean re-run.
• •
  For each new/changed test, you can name the code change that would make it fail.

Pitfalls

• Letting the summary replace reading. The map tells you where to look, not what is true; hotspot files get full human reads.
• Reviewing 2,000 lines in one sitting. Defect detection collapses after ~400 lines/hour; schedule two passes or demand a split.
• Assuming green CI means real tests. Phantom tests are green by construction; that is what makes them phantom.
• Politeness toward machine effort. "The agent worked hard on this" is not a review consideration. Request changes as freely as on a first-week intern's PR — that is the correct trust level.

Restricted-environment variant

• •
  If the diff cannot go to a hosted model: GitHub's native PR summary under an enterprise agreement may still be sanctioned; otherwise generate the review map with a local model over the diff file — grouping and ordering work fine at local-model quality.
• •
  The provenance checklist is fully manual-capable: lockfile verification, "what makes this test fail" reads, scope audit, and license spot-checks need no LLM.
• •
  For client engagements, log AI-authored-PR review verdicts (what was checked, what evidence) — services contracts increasingly require demonstrable human review of machine-generated code, and your notes are the audit trail.

Success criteria

• •
  A 1,500-line PR reaches a defensible verdict in a 45–60 minute time box, with your effort concentrated on the risk-ranked files.
• •
  Seeded-defect drills (hallucinated API + phantom test + scope creep) score 3/3 caught using the checklist.
• •
  Your review comments shift toward design and correctness; scope-creep hunks and unreviewable mega-PRs get pushed back instead of absorbed.