Lab 06 · Reviewing the Machine

With AI roughly doubling merged-PR volume (DORA 2025), reviewing machine output fast without rubber-stamping it is now a core senior competency and an explicit interview subject.

Time & tier

~90 minutes total (45-minute timed review inside it) · Advanced.

Setup

Two-session pattern: Session A generates the PR and the hidden answer key; Session B (you, later) reviews blind.

Base scaffold — create notes-api/ with pip install fastapi pytest httpx, and this app.py:

from fastapi import FastAPI, HTTPException, Header

app = FastAPI()
NOTES, USERS = {}, {"tok_alice": "alice", "tok_bob": "bob"}

def auth(token: str | None):
    if token not in USERS:
        raise HTTPException(401, "bad token")
    return USERS[token]

@app.post("/notes")
def create(payload: dict, x_token: str | None = Header(None)):
    user = auth(x_token)
    nid = len(NOTES) + 1
    NOTES[nid] = {"id": nid, "owner": user, "text": payload.get("text", "")}
    return NOTES[nid]

@app.get("/notes/{nid}")
def read(nid: int, x_token: str | None = Header(None)):
    user = auth(x_token)
    note = NOTES.get(nid) or {}
    if note.get("owner") != user:
        raise HTTPException(404, "not found")
    return note

Add a minimal test_app.py (two happy-path tests you write yourself — 10 lines), commit, and confirm pytest -q passes.

Now paste into Session A (any capable free model):

Here is a FastAPI notes service [paste app.py + tests]. Generate a single
unified diff implementing "shared folders": folder CRUD, notes belong to
folders, folders shared with other users grant read access. Target
1,200-1,500 diff lines across at least 8 new/modified files (routers,
service layer, permissions module, tests, README). Style: competent but
verbose, like an eager agent. Deliberately seed EXACTLY these 5 defects,
unmarked: (1) one call to a method that does not exist on a real library
or on our own code; (2) one test that always passes regardless of the code
under test; (3) one unrelated drive-by change to existing behaviour nobody
asked for; (4) one copy-pasted block where a stale variable from the
original block survives; (5) one route where the permission check is
missing or short-circuited. Then output SEEDS.md listing each defect with
file, approximate line, and a one-line explanation.

Save the diff as feature.diff and SEEDS.md unread. Scaffold verification (all blind-safe):

• •
  wc -l feature.diff ≥ 1,000 (regenerate if short; models under-count).
• •
  grep -c "^+++" feature.diff ≥ 8 files touched.
• •
  grep -c "^- " SEEDS.md || grep -c "^[0-9]" SEEDS.md indicates 5 entries.
• •
  The diff contains no words like "seeded", "deliberate", "defect" (grep -i — if found, have Session A re-emit clean).

Guided steps

1. 1Set a 45-minute timer. Open review.md with the provenance-aware checklist as headings: Hallucinated APIs · Phantom/weak tests · Over-broad diff · Copy-paste residue · Security/permissions · License/IP smells.
2. 2Triage pass (10 min). Don't read linearly. List touched files from the diff headers, sort by risk: permissions module first, then routes, then service layer, then tests, README last. Note anything touching existing code — in an additive feature PR, edits to existing behaviour are guilty until explained.
3. 3Existence pass (10 min). For every API call on a library or on the base code, verify it exists: python -c "import inspect, fastapi; ...", open the base file, or check official docs. Machines invent plausible methods; humans typo real ones — this pass is the provenance difference.
4. 4Test forensics (10 min). For each new test ask: what code change would make this fail? Red flags: assertions on constants, try/except around the assertion, mocks that mock the function under test, tests that never call the new route. Decision point: any test you can't imagine failing gets marked phantom until proven otherwise.
5. 5Permission trace (10 min). For every new route, trace token → auth → ownership/share check. Write the chain out; a missing link is a finding even if you can't yet build the exploit.
6. 6Verdict (5 min). End review.md with: findings table (file, line, checklist category, severity), overall verdict (approve / request changes / reject-and-regenerate), and what you'd demand before merge. Under time pressure, an honest "did not review: X" list is part of a professional verdict.
7. 7Timer stops. Now open SEEDS.md and score.

Verification

• •
  Found ≥4 of the 5 seeds within the 45 minutes (5/5 = distinction; the security seed — number 5 — is mandatory: missing it caps you at fail regardless of the others).
• •
  Zero false certainty: nothing you labelled "verified correct" appears in SEEDS.md.
• •
  Your verdict is request-changes or reject (any seeded PR that gets "approve" is an automatic fail).
• •
  Bonus check: Session A models often introduce accidental defects beyond the five — any real extra defect you found and can demonstrate counts as +1.

Completion artifact

Keep review.md (timed, with the findings table and verdict) and your SEEDS.md score. Self-grading rubric:

• Recall under budget (0–3): 5/5 seeds = 3; 4/5 = 2; 3/5 = 1.
• Security catch (0–2): permission seed found with the broken chain written out = 2.
• Method (0–2): review shows triage order and existence checks, not linear reading = 2.
• Verdict quality (0–2): verdict states severity-ranked demands and an honest not-reviewed list = 2.
• No rubber stamp (0–1): nothing wrong marked as verified-correct = 1.

Pass = 7/10.

Stuck?

• The generated diff is only 400 lines. Tell Session A "expand: add the service layer, a permissions module with docstrings, README, and full test files; keep the same 5 seeds" — length comes from breadth, not padding.
• 45 minutes is not enough. Correct — that's the constraint being trained. The pass bar rewards risk-ordering: if you spent your budget on permissions and tests and missed the copy-paste seed in the README-adjacent code, you made the right trade.
• You can't tell if a method is hallucinated. One-line rule: if inspect/docs/grep of the base code can't produce it in 60 seconds, flag it as unverified-existence and move on — flagging is the job; proving is the author's.

Client-safe variant

On client work you rarely may generate a fake PR, but you review real AI-authored PRs constantly. Bring only the checklist and the timed method; run existence checks with the client's sanctioned tooling or plain grep/inspect (no model needed for the review itself — steps 2–6 are fully offline). File your review.md in the client's PR thread; the checklist headings double as the provenance-review record that IP-cautious clients increasingly require before merging machine-generated code.

Answer key (don't read before attempting)

The key is the SEEDS.md written by Session A — five entries, one per defect class: (1) nonexistent API call, (2) always-passing test, (3) unrelated behaviour change, (4) copy-paste with stale variable, (5) missing/short-circuited permission check. Score against it, with two adjudication rules: if a documented seed is not actually defective in the emitted diff (generators occasionally fumble), drop it from the denominator; if you found a genuine defect SEEDS.md doesn't list, add it to your numerator with a one-line proof. The source code, not the generator's notes, is always the ground truth — noticing where the two diverge is the skill.