Lab 05 · AI Review Pipeline

This is the tech-lead skill of 2026: GitHub's Copilot code review reads copilot-instructions.md from the branch it reviews, CodeRabbit and Roast-style bots take equivalent config, and a badly tuned reviewer trains your team to ignore all reviewers. Signal-to-noise is the deliverable.

Time & tier

~75 minutes hands-on · Core.

Setup

Inline scaffold. Base file notify.py in a fresh git repo:

import logging
import requests
from config import WEBHOOK_URL, API_TOKEN

logger = logging.getLogger(__name__)

def send_alert(channel: str, message: str) -> bool:
    resp = requests.post(WEBHOOK_URL,
                         json={"channel": channel, "text": message},
                         headers={"Authorization": f"Bearer {API_TOKEN}"},
                         timeout=5)
    return resp.status_code == 200

The five team standards you must encode (treat as your team's agreed policy):

1. 1.
   Public functions carry type hints on parameters and return.
2. 2.
   No print — use the module logger.
3. 3.
   Every external HTTP call sets an explicit timeout.
4. 4.
   No hardcoded URLs, tokens, or secrets — config module only.
5. 5.
   Handler-level user input is validated before use (no raw dict trust).

Save these two seeded diffs as pr-a.diff and pr-b.diff. PR-A (adds an escalation path):

--- a/notify.py
+++ b/notify.py
@@
+def escalate(channel, incident):
+    print(f"escalating {incident['id']} to {channel}")
+    resp = requests.post("https://hooks.pager.example.com/v2/escalate",
+                         json={"channel": channel,
+                               "sev": incident["sev"]},
+                         headers={"Authorization": "Bearer tok_live_9f3a"})
+    return resp.status_code == 202

PR-B (adds a digest helper — mostly fine, with nit bait):

--- a/notify.py
+++ b/notify.py
@@
+def build_digest(alerts):
+    lines = []
+    for a in alerts:
+        x = a["channel"]
+        lines.append("[" + x + "] " + a["text"])  # TODO tidy someday
+    digest_text_for_the_daily_channel_summary = "\n".join(lines)
+    return digest_text_for_the_daily_channel_summary

Verification checklist: PR-A violates ≥3 of the five standards; PR-B violates exactly one standard while containing style bait (ugly names, string concatenation, a TODO) that is not covered by any standard. Confirm you can list which before running any reviewer — you seeded them; the answer key is your backstop.

Guided steps

1. 1Write v1 instructions in .github/copilot-instructions.md (works verbatim for Copilot code review; the same text drives a chat reviewer on free tiers). Naive first draft is fine — one line per standard.
2. 2Run the review. Two routes, same lab. Copilot Pro/Business route: push the repo to GitHub, open each diff as a PR, request Copilot review. Free route: in Copilot Free chat, Gemini Code Assist, or Windsurf, prompt: "You are our team's PR reviewer. Apply ONLY the standards in the pasted instructions file. Review this diff. Output: one finding per line as SEVERITY | standard # | line | comment. If a concern maps to no listed standard, do not raise it." Paste instructions + pr-a.diff. Repeat for PR-B. Save both transcripts as review-v1-a.md, review-v1-b.md.
3. 3Score the run in tuning-log.md: for each PR, count true positives (seeded violations caught), misses, and noise (comments not tied to a listed standard — the PR-B bait measures this). Decision point: misses mean your instruction for that standard is too vague; noise means your instructions lack an explicit scope limit.
4. 4Tune v2. Sharpen with the three levers that work on real review bots: make each rule checkable ("every requests.* call must pass timeout=" beats "handle timeouts properly"); add an explicit suppression clause ("do not comment on naming, formatting, TODOs, or style unless a listed standard is violated"); add severity mapping (secrets = blocking, type hints = suggestion).
5. 5Re-run both PRs with v2 → review-v2-a.md, review-v2-b.md. Target: all seeds caught, zero noise comments on PR-B's bait.
6. 6Adversarial check. Add a sixth micro-diff of your own with a subtle standard-4 violation (a hardcoded URL split across two string literals). Does v2 still catch it? If not, one more tuning round — rules keyed to exact tokens are brittle; say what the rule protects, then give the checkable pattern as an example.

Verification

• •
  v2 transcripts show every seeded violation caught, each tagged to the right standard number (check against the answer key now).
• •
  PR-B's v2 review contains zero comments about naming, concatenation, or the TODO.
• •
  Before/after is measurable: tuning-log.md shows v1 vs v2 counts (e.g., "A: 2/4 caught, 3 noise → 4/4, 0 noise").
• •
  The step-6 adversarial diff is caught by v2 without instruction changes.

Completion artifact

Keep the final instructions file, all four transcripts, and tuning-log.md. Self-grading rubric:

• Recall (0–3): all seeded violations caught in v2 = 3; one miss = 1.
• Precision (0–2): zero noise on PR-B bait = 2; one noise comment = 1.
• Checkability (0–2): every instruction has a mechanically checkable pattern = 2.
• Severity design (0–2): findings ranked (blocking vs suggestion) and rank matches risk = 2.
• Robustness (0–1): adversarial diff caught without retuning = 1.

Pass = 7/10.

Stuck?

• The chat reviewer moralises beyond the standards. Your suppression clause is missing or buried — put "comment ONLY on the numbered standards" as the first line of the instructions, and repeat the output format contract in the prompt.
• It keeps missing the missing-timeout seed. Vague rule. Replace with the exact predicate: "flag any requests.get/post/put/delete call whose arguments do not include timeout=."
• Copilot code review ignores your file. The instructions file must exist on the PR's head branch (behaviour per GitHub's 2026-07 changelog), and repo-level custom instructions must be enabled in settings.

Client-safe variant

Instruction-file tuning is one of the safest AI skills to run inside a client environment: the file contains team policy, not client code. On restricted accounts, tune against sanitized synthetic diffs exactly like these, then deploy the tuned file to the client's Copilot Business / CodeRabbit setup through the normal PR process. Keep the tuning log — it is your evidence when a client asks "why should we trust the bot?": show recall/precision numbers, not adjectives.

Answer key (don't read before attempting)

PR-A seeds (4): missing type hints on escalate (standard 1); print instead of logger (standard 2); requests.post without timeout (standard 3); hardcoded URL and a live-looking bearer token (standard 4 — the token is the blocking-severity find; if your reviewer flagged the URL but not tok_live_9f3a, count standard 4 as half-caught). Arguably incident["id"]/incident["sev"] trusted raw is a standard-5 smell — credit yourself if your reviewer raised it, but it is not a counted seed.

PR-B seed (1): build_digest lacks type hints (standard 1). Everything else — the horrible variable name, string concatenation, the TODO — is deliberate bait covered by no standard; any comment on it counts as noise.