Frontend Provenance Review of AI-Generated PRs

Tier: Core · Workflow area: Testing & quality

This playbook gives you a provenance-aware review protocol: a fixed sequence of frontend-specific checks that target where generators actually fail, plus a way to use a second AI as a reader without letting it approve anything.

When to use

• •
  Your team ships PRs that are substantially agent-authored — Copilot agent tasks, Claude Code sessions, Cursor composer output.
• •
  You are the designated reviewer for a large generated diff and your instinct is to skim because "it's mostly boilerplate."
• •
  A previously merged AI PR caused a production UI defect and you are rebuilding review discipline.
• •
  You are interviewing or being interviewed — this review skill is now a hiring signal.

Skip it for trivial generated diffs (a story file, a copy change) where ordinary review suffices.

Prerequisites

• •
  PR metadata that marks AI-assisted changes — a label, a PR-template checkbox, or commit trailers. You cannot apply provenance review if you cannot see provenance.
• •
  CI that already runs typecheck, unit tests, and ideally an axe-core scan and a bundle-size report. Review verifies judgment; CI verifies mechanics.
• •
  The design system's component and token documentation open in a tab — you will check the diff against it, not against memory.
• •
  A time budget. Provenance review of a big generated PR is 30–45 focused minutes, not a scroll-through.

The workflow

1. 1.

   Read the prompt before the diff. If the PR description includes the task given to the agent, read it first and ask: what would a model plausibly get wrong here? Missing constraints in the prompt (no a11y mention, no design-system mention) predict the defect classes you should hunt hardest.

2. 2.

   Run the six-lane frontend checklist, one lane at a time. Do not interleave — each lane is a separate pass over the diff:

   • •
     Hallucinated APIs — every imported component, prop, and hook exists and is spelled correctly. Generators invent plausible props (variant="subtle", onDismiss) on real components; typecheck catches some, but untyped props, string unions widened to string, and spread props slip through.
   • •
     Off-system styling — hardcoded hex colors, px spacing, ad-hoc font sizes, one-off shadows. Diff-search for #, px, rgb(, style={{. Every hit either maps to a token or gets flagged.
   • •
     State completeness — for each new surface: loading, empty, error, and success all rendered? Generators write the happy path; the error state is the most commonly missing artifact in generated UI.
   • •
     Accessibility — interactive elements are real <button>/<a>, inputs have labels, focus order follows the visual order, icons have accessible names, custom widgets follow an APG pattern rather than a click-handler-on-div.
   • •
     Render performance — new effects with object/array deps, inline object props into memoized children, state lifted higher than needed, missing keys or index keys on reordering lists.
   • •
     Weight — new dependencies in package.json, duplicate utility libraries, a chart library imported for one sparkline. Check the CI bundle diff if you have one.
3. 3.

   Deploy a second reader, scoped and adversarial. Paste the diff into a different assistant than the one that wrote it:

   Copy

   You are reviewing a PR written by another AI. Do not summarize it.
   Find only: (1) props or APIs that may not exist on the imported
   components, (2) missing loading/error/empty states, (3) accessibility
   violations, (4) hardcoded style values. Cite file and line for each.
   Say "none found" per category if clean.
   

   Treat its output as leads to verify, never as findings to forward.

4. 4.

   Exercise one unhappy path by hand. Pull the branch, run the app, and break something: kill the network mid-fetch, submit the form empty, tab through the new screen with the keyboard. Five minutes of hostile usage catches what diff-reading cannot.

5. 5.

   Write findings by defect class. Comment with the lane name ("hallucinated prop", "missing error state") so the team accumulates a shared vocabulary — and so instruction files (see the constraints playbooks) can be updated to prevent the class, not just the instance.

Tool variants

• GitHub PR UI + CI gates: the baseline. Wire axe-core and Lighthouse CI/bundle-diff as required checks so mechanical lanes are pre-verified before human review.
• Claude / ChatGPT as second reader: best for the hallucination and state-completeness lanes; always a different model or session than the author.
• Cursor / IDE review mode: useful for jumping from diff hunk to component definition fast when checking props against reality.

Verification

• •
  Every lane of the checklist has an explicit outcome in the review — "clean" is a finding too.
• •
  At least one unhappy path was exercised in a running build, noted in the review comment.
• •
  Findings are labeled by defect class, and recurring classes have a corresponding instruction-file or CI-gate update within a sprint.
• •
  The PR's approval comes from a human; no second-reader output was pasted unverified.

Pitfalls

• Skimming because it compiles. Type-safe garbage is still garbage; the worst generated defects (missing states, fake buttons) are type-clean.
• Reviewing all lanes at once. Interleaved attention misses systematically; the lane-by-lane pass exists because each defect class needs a different kind of looking.
• Letting the author model review itself. Same model, same blind spots. A second reader must be a different model or at minimum a fresh, adversarially-prompted session.
• Treating the second reader as an approver. It generates leads with confident false positives; every lead gets human verification before it becomes a review comment.
• No provenance signal at all. If AI-assisted PRs are not labeled, you will apply this expensive review to the wrong PRs and skip it on the right ones.

Restricted-environment variant

When client code cannot leave the tenant, the second-reader step changes; the human protocol does not.

• In-tenant assistant (Copilot Business, Bedrock/Azure-backed chat): same adversarial review prompt, zero data movement.
• Local model via Ollama: competent at the hallucination lane if you paste the component's type definitions alongside the diff hunk; run lanes one at a time with narrow context.
• No model available: the six-lane checklist is fully executable by a human with grep and the design-system docs — the AI reader is an accelerator, not a dependency.

Success criteria

• •
  Post-merge UI defect rate from AI-authored PRs drops and stays below the human-authored baseline.
• •
  Review time per generated PR stabilizes around the budget — thoroughness without stalling throughput.
• •
  The team's instruction files visibly grow from review findings, and repeat defect classes trend to zero.
• •
  New reviewers can run the six lanes from the checklist doc without shadowing you first.