An Advisory AI Review Gate for Infra PRs

Tier: Core · Workflow area: CI/CD & release engineering

Humans stay the gate; the AI makes the human review better-informed. This playbook builds that bot.

When to use

• •
  Your team reviews Terraform/K8s/pipeline PRs and routinely misses cross-cutting concerns (cost, IAM scope, blast radius) under time pressure.
• •
  Junior-heavy rotations where a consistent review checklist would raise the floor.
• •
  You want AI review but your team (rightly) refuses to let it block merges.

Skip this if your PRs are tiny and reviewed within the hour — the bot adds noise, not signal.

Prerequisites

• •
  CI that can run on pull_request and post comments (GitHub Actions/GitLab CI with an API token scoped to comments only).
• •
  An LLM API key stored as a CI secret.
• •
  A written rubric — start from the one below, then make it yours.

The workflow

1. 1.

   Write the rubric as a file in the repo (.ai-review/rubric.md) so it is versioned and reviewable itself:

   Copy

   Review this infrastructure diff ONLY against these criteria:
   1. BLAST RADIUS — resources deleted/replaced/renamed; changes affecting
      shared infra (VPC, IAM, DNS, security groups) vs single-service scope.
   2. SECURITY — IAM wildcards, 0.0.0.0/0 ingress, unencrypted storage,
      secrets in plaintext, public buckets, disabled logging.
   3. COST — instance sizes, counts, storage classes, NAT/data-transfer
      implications. Flag anything that plausibly adds >10% to service cost.
   4. IRREVERSIBILITY — destroy/replace of stateful resources (databases,
      volumes, queues with data).
   Output: max 5 findings, each with severity (INFO/WARN/CRITICAL), the file
   and line, and a one-sentence rationale. If the diff is clean, say exactly:
   "No findings against the rubric." Do not comment on style. Do not invent
   context not present in the diff.
   

2. 2.

   Wire the CI job. Keep it advisory by construction — the job always exits 0:

   Copy

   ai-review:
     if: github.event_name == 'pull_request'
     runs-on: ubuntu-latest
     permissions: { contents: read, pull-requests: write }
     steps:
       - uses: actions/checkout@<pinned-sha>
         with: { fetch-depth: 0 }
       - name: Review diff
         run: |
           git diff origin/${{ github.base_ref }}...HEAD -- '*.tf' '*.yaml' \
             | python3 .ai-review/review.py --rubric .ai-review/rubric.md \
             --post-comment "$PR_URL" || true   # advisory: never fail the build
   

3. 3.

   Feed the diff plus minimal context — the diff alone, plus terraform plan summary if available. Resist sending the whole repo; more context means more hallucinated "issues" about code that is not changing.

4. 4.

   Calibrate on historical PRs before going live. Run the bot on 10–20 merged PRs where you know the ground truth (including a couple that caused incidents). Count true/false positives per rubric line. Cut or rewrite any rubric line with worse than ~50% precision.

5. 5.

   Ship with a feedback loop. Add 👍/👎 reactions convention or a weekly 10-minute review of bot comments; prune what the team ignores. An ignored bot is worse than no bot — it trains people to skim past all review comments.

Tool variants

• Native review products (Copilot code review, GitLab Duo, Claude Code /review): fastest to enable; you trade rubric control for zero maintenance. Layer your rubric as custom instructions where supported.
• Danger + LLM API: Danger handles the PR plumbing (comments, dedupe on force-push); the LLM call slots into a Dangerfile — good middle path.
• Custom script (as above): full rubric control, ~100 lines of Python; you own prompt updates and comment dedupe.

Verification

• Validate: bot output parses; every finding cites a file/line that exists in the diff (auto-check this — hallucinated locations are common and instantly destroy trust).
• Plan: the bot job has read-only repo access and comment-only PR permission; it cannot approve, request changes, or merge.
• Policy: measured precision from step 4 is recorded in the repo; rubric changes go through PR review like code.
• Cost: per-PR token spend is logged; cap diff size (e.g., truncate beyond 3,000 changed lines with a notice) so a vendored-dependency PR does not cost dollars.

Pitfalls

• Blocking mode by accident. One exit 1 on a CRITICAL finding and you have built the thing your team rejected. Keep advisory in the job design, not just in policy.
• Rubric creep. Every added criterion lowers precision on the others. Five sharp criteria beat fifteen vague ones.
• Reviewing generated/vendored files. Exclude lockfiles, .terraform.lock.hcl, vendored modules — findings there are pure noise.
• Treating bot silence as approval. "No findings" means the rubric found nothing, not that the PR is safe. Human review requirements are unchanged.

Restricted-environment variant

Client repos: the diff itself is client IP. Options, in order of preference:

1. 1Client tenancy. Run the bot with the client's approved enterprise AI tenancy (Bedrock/Azure OpenAI inside the client account) so the diff never leaves their boundary.
2. 2Self-hosted model. Self-host a capable open model behind an internal endpoint and point review.py at it.
3. 3Internal repos only. Restrict the bot to your own company's internal repos and keep client PRs human-only.

Never route client diffs through a personal AI account.

Success criteria

• •
  Bot comments on infra PRs within 2 minutes of open, always non-blocking.
• •
  Measured precision ≥70% on your calibration set for shipped rubric lines.
• •
  At least one real WARN/CRITICAL caught pre-merge in the first month that reviewers confirm they had missed.
• •
  Reviewers report reading the bot's comments — verified by the feedback loop, not assumed.