Lab 05 · Advisory AI Review Bot for Infra PRs

Stack Overflow 2025 found 59% of developers reject AI code review — usually because someone deployed it as a gate. This lab teaches the placement that works: advisory comments, measured signal-to-noise, human merge authority untouched.

Time & tier

~90 minutes hands-on · Advanced tier (maps to module M2). $0–$2: any free-tier chat model works via the manual path; an API key makes it scriptable.

Setup

Create reviewbot/ with rubric.md, reviewer.py, score.py, and prs/pr-1.diff … prs/pr-6.diff.

# rubric.md — Infra PR review rubric (advisory)
You are reviewing an infrastructure diff. Evaluate ONLY these dimensions:
1. SECURITY: exposure widened (CIDRs, ports, IAM actions/resources, encryption off)?
2. BLAST RADIUS: does this remove a safety net (prevent_destroy, deletion
   protection, backups) or touch state shared by other systems?
3. COST: step-change in instance size/count/storage class? Estimate direction.
4. AVAILABILITY: reduced redundancy, tightened health checks, riskier rollout?
Rules: cite the exact diff line for every finding; severity critical/warning/nit;
if the change is safe say so in one line; you are advisory — never demand.

# reviewer.py
#!/usr/bin/env python3
import json, os, sys, urllib.request

diff = open(sys.argv[1]).read()
rubric = open("rubric.md").read()
prompt = (rubric + "\n\n--- DIFF UNDER REVIEW ---\n" + diff +
          "\n--- END DIFF ---\nRespond with a JSON list of findings: "
          '[{"severity":"critical|warning|nit","category":"security|cost|blast-radius|availability|other",'
          '"line":"<quoted diff line>","finding":"<one sentence>","suggestion":"<one sentence>"}] '
          "Return [] if the change is safe. Advisory tone; never invent lines not present in the diff.")

key = os.environ.get("OPENAI_API_KEY")
if not key:                      # $0 path: paste output into any chat UI
    print(prompt); sys.exit(0)
base = os.environ.get("OPENAI_BASE_URL", "https://api.openai.com/v1")
model = os.environ.get("OPENAI_MODEL", "gpt-4o-mini")
req = urllib.request.Request(base + "/chat/completions",
    data=json.dumps({"model": model, "messages": [{"role": "user", "content": prompt}]}).encode(),
    headers={"Authorization": "Bearer " + key, "Content-Type": "application/json"})
print(json.load(urllib.request.urlopen(req))["choices"][0]["message"]["content"])

The six diffs. Five contain one seeded issue each; one is clean (your false-positive probe). Save each fenced block as prs/pr-N.diff.

--- a/network.tf  (pr-1: "open db access for analytics team")
   ingress {
     from_port   = 5432
     to_port     = 5432
     protocol    = "tcp"
-    cidr_blocks = ["10.42.0.0/16"]
+    cidr_blocks = ["0.0.0.0/0"]
   }

--- a/compute.tf  (pr-2: "bump worker size for month-end load")
 resource "aws_instance" "batch" {
-  instance_type = "t3.medium"
+  instance_type = "m5.8xlarge"
   count         = 6
 }

--- a/rds.tf  (pr-3: "cleanup: remove noisy lifecycle block")
 resource "aws_db_instance" "orders" {
-  deletion_protection = true
-  lifecycle {
-    prevent_destroy = true
-  }
+  deletion_protection = false
 }

--- a/tags.tf  (pr-4: "add cost-allocation tags")
 resource "aws_instance" "app" {
   tags = {
     Name = "app"
+    CostCenter = "retail-372"
+    Owner      = "platform-team"
   }
 }

--- a/iam.tf  (pr-5: "unblock the ETL job permissions")
 resource "aws_iam_role_policy" "etl" {
   policy = jsonencode({
     Statement = [{
-      Action   = ["s3:GetObject"]
-      Resource = ["arn:aws:s3:::etl-input/*"]
+      Action   = ["*"]
+      Resource = ["*"]
       Effect   = "Allow"
     }]
   })
 }

--- a/asg.tf  (pr-6: "faster deploys: tighten health checks")
 resource "aws_autoscaling_group" "web" {
-  min_size                  = 2
+  min_size                  = 1
-  health_check_grace_period = 300
+  health_check_grace_period = 30
 }

Guided steps

1. 1.
   Run python3 reviewer.py prs/pr-1.diff. No API key? It prints the full prompt — paste into any free chat model and save the JSON reply as results/pr-1.json. Repeat for all six.
2. 2.
   Judge every finding yourself. Build findings.csv with rows pr,category,verdict — one row per bot finding, verdict tp (matches a real problem) or fp (noise, invented line, or wrong category). Be strict: a security finding on pr-4's tags is an fp even if phrased plausibly.
3. 3.
   Build the ground truth truth.csv: five rows — pr-1,security · pr-2,cost · pr-3,blast-radius · pr-5,security · pr-6,availability (pr-4 is clean).
4. 4.
   Score it:

# score.py
#!/usr/bin/env python3
import csv
truth = {(r["pr"], r["category"]) for r in csv.DictReader(open("truth.csv"))}
tp_set, fp = set(), 0
for r in csv.DictReader(open("findings.csv")):
    if r["verdict"] == "tp" and (r["pr"], r["category"]) in truth:
        tp_set.add((r["pr"], r["category"]))
    else:
        fp += 1
tp, fn = len(tp_set), len(truth - tp_set)
prec = tp / (tp + fp) if tp + fp else 0.0
rec = tp / (tp + fn) if tp + fn else 0.0
print(f"TP={tp} FP={fp} FN={fn}  precision={prec:.2f} recall={rec:.2f}")
print("missed:", sorted(truth - tp_set) or "none")

5. 1Tune one variable at a time. If precision is low (chatty bot), tighten the rubric ("max 2 findings per PR; nits forbidden"). If recall is low, check which category it missed — pr-6 is the usual casualty because "tighten health checks" sounds like an improvement. Re-run all six after each rubric change; keep a run log of prompt-version → precision/recall.
6. 2Write the placement memo (5 sentences max): given your measured numbers, should this bot comment on every PR, only on PRs touching certain paths, or not at all — and why is it advisory rather than a required check?

Verification

• •
  Six result files exist; every bot finding has a hand-assigned tp/fp verdict.
• •
  python3 score.py runs and prints precision, recall, and the missed list.
• •
  At least two rubric iterations logged with before/after scores.
• •
  pr-4 produced zero findings in your final run (or your memo explains the residual noise).

Completion artifact

reviewbot/ repo with rubric versions, results, findings.csv, truth.csv, score log, and the placement memo. Self-grading rubric (pass = 8/10):

• Measurement discipline (0–3): scores computed per rubric version, one variable changed at a time = 3.
• Judging honesty (0–2): fp verdicts assigned even to plausible-sounding findings = 2.
• Final detector quality (0–2): precision ≥ 0.7 AND recall ≥ 0.8 on the six PRs = 2.
• Placement judgment (0–2): memo ties deployment decision to your measured numbers = 2.
• Citation integrity (0–1): every tp finding quotes a line actually present in the diff = 1.

Stuck?

• •
  Model returns prose instead of JSON: append "Output MUST be a single JSON array, no markdown fences."
• •
  Everything flagged critical: add severity definitions with examples to the rubric — models anchor on examples.
• •
  pr-6 keeps getting missed: add an availability example to the rubric, but then verify you didn't break pr-4's clean verdict — that trade-off is the whole game.
• •
  API path fails: check OPENAI_BASE_URL; any OpenAI-compatible endpoint works (Groq, Together, local Ollama at http://localhost:11434/v1).

Client-safe variant

Client infra diffs are among the most sensitive artifacts you touch — they map the estate. Run the bot only through client-approved tenancy (Bedrock/Azure OpenAI) or fully local (Ollama + an instruct model; expect lower recall, re-measure — never assume). Redaction alternative: review module/variable diffs with account IDs, CIDRs and bucket names templated out. Your precision/recall log is the client-facing evidence that the bot earns its comment space.

Answer key (don't read before attempting)

Seeded issues:

• pr-1 · securityDB port open to the internet.
• pr-2 · costt3.medium→m5.8xlarge across count = 6 is roughly a 40x step in instance spend, and the count multiplier is what weak reviews miss.
• pr-3 · blast radiusremoves both deletion protection layers, making a one-line terraform destroy of production data possible.
• pr-4 · cleanany finding here is a false positive.
• pr-5 · securityAction:* on Resource:* is full-account admin.
• pr-6 · availabilitymin_size 1 removes redundancy and a 30s grace period recycles slow-booting instances before they pass health checks, a classic cascade trigger.