Lab 04 · "Almost Right" Triage

Two-thirds of developers report spending more time than expected debugging AI code, and Meta- and Google-style AI-enabled interviews now score exactly this judgment; this lab is direct rehearsal.

Time & tier

~60 minutes hands-on · Core.

Setup

Inline scaffold. pip install pytest requests. Two files, then a diff to review.

refunds.py (the current, trusted code):

import requests

GATEWAY = "https://pay.example.internal/v1"

class RefundError(Exception):
    pass

def issue_refund(charge_id, amount_cents, already_refunded_cents):
    """Refund amount_cents from a charge. Cumulative refunds must never
    exceed the original charge. Full refunds are allowed."""
    charge = requests.get(f"{GATEWAY}/charges/{charge_id}", timeout=5).json()
    if charge["status"] != "settled":
        raise RefundError("charge not settled")
    if amount_cents <= 0:
        raise RefundError("amount must be positive")
    if already_refunded_cents + amount_cents > charge["amount_cents"]:
        raise RefundError("refund exceeds charge")
    resp = requests.post(f"{GATEWAY}/refunds",
                         json={"charge_id": charge_id,
                               "amount_cents": amount_cents},
                         timeout=5)
    resp.raise_for_status()
    return resp.json()["refund_id"]

test_refunds.py (current tests):

from unittest.mock import patch, MagicMock
import pytest
from refunds import issue_refund, RefundError

def _charge(amount=1000, status="settled"):
    m = MagicMock()
    m.json.return_value = {"amount_cents": amount, "status": status}
    return m

@patch("refunds.requests.post")
@patch("refunds.requests.get")
def test_full_refund_allowed(get, post):
    get.return_value = _charge(1000)
    post.return_value.json.return_value = {"refund_id": "r_1"}
    assert issue_refund("c_1", 1000, 0) == "r_1"

@patch("refunds.requests.get")
def test_over_refund_rejected(get):
    get.return_value = _charge(1000)
    with pytest.raises(RefundError):
        issue_refund("c_1", 600, 500)

The ticket: "Add retry with backoff to the gateway POST (3 attempts), and reject refunds on charges older than 180 days (charge payload has created_days_ago)." An AI agent produced this PR — copy it into pr.diff and review it before applying it:

diff --git a/refunds.py b/refunds.py
--- a/refunds.py
+++ b/refunds.py
@@ -11,13 +11,15 @@ def issue_refund(charge_id, amount_cents, already_refunded_cents):
     charge = requests.get(f"{GATEWAY}/charges/{charge_id}", timeout=5).json()
     if charge["status"] != "settled":
         raise RefundError("charge not settled")
+    if charge.get("created_days_ago", 0) > 180:
+        raise RefundError("charge too old to refund")
     if amount_cents <= 0:
         raise RefundError("amount must be positive")
-    if already_refunded_cents + amount_cents > charge["amount_cents"]:
+    if already_refunded_cents + amount_cents >= charge["amount_cents"]:
         raise RefundError("refund exceeds charge")
     resp = requests.post(f"{GATEWAY}/refunds",
                          json={"charge_id": charge_id,
                                "amount_cents": amount_cents},
-                         timeout=5)
+                         timeout=5, retries=3, backoff_factor=0.5)
     resp.raise_for_status()
     return resp.json()["refund_id"]
diff --git a/test_refunds.py b/test_refunds.py
--- a/test_refunds.py
+++ b/test_refunds.py
@@ -12,7 +12,11 @@ def _charge(amount=1000, status="settled"):
 def test_full_refund_allowed(get, post):
     get.return_value = _charge(1000)
     post.return_value.json.return_value = {"refund_id": "r_1"}
-    assert issue_refund("c_1", 1000, 0) == "r_1"
+    try:
+        result = issue_refund("c_1", 1000, 0)
+        assert result
+    except RefundError:
+        pass  # boundary behavior may vary by gateway config
 
 @patch("refunds.requests.get")
 def test_over_refund_rejected(get):

Verification checklist: baseline pytest -q (before the diff) shows 2 passed; the diff applies cleanly with git apply pr.diff (do this only when the steps say so).

Guided steps

1. 1Diff-vs-intent read (15 min, no AI). In review.md, map every hunk to a ticket requirement. Anything that changes behaviour the ticket didn't ask for gets flagged. Write down your three findings before any tool confirms them.
2. 2Hallucination check. For each new API usage in the diff, verify against the real library — python -c "import requests, inspect; print(inspect.signature(requests.post))" or the official docs. Decision point: an argument that doesn't exist in the signature is not a style problem; classify its blast radius (silent no-op, immediate crash, crash only on the untested path?).
3. 3Boundary interrogation. For the changed comparison, write the truth table: charge 1000, already refunded 0, refund 1000 — allowed before? allowed after? The docstring is the contract.
4. 4Test-diff review. Ask the hardest reviewer question: does the modified test still fail if the code is wrong? Mentally revert each code hunk and check whether any test would notice.
5. 5Second-reader pass. Now paste refunds.py, the diff, and the ticket into your assistant (Copilot Chat, Gemini, Windsurf): "Review this diff. List defects with file/line, classify each as hallucinated-API / wrong-boundary / weakened-test / other, and say which tests would catch each." Compare with your step-1 findings; investigate disagreements manually — the second reader is a colleague, not an oracle.
6. 6Repair or regenerate — decide per defect and record one sentence of reasoning: repair when the intent is right and the edit is a token-level slip; regenerate when the approach itself is wrong or the code was written to make a bad test pass. Then produce the corrected diff: apply pr.diff, fix per your decisions, restore a strict full-refund test, and add a test for the 180-day rule and one for the retry behaviour (assert the POST is attempted 3 times on failure — requests alone can't do this; implementing it honestly is part of the fix).
7. 7Green run. Run pytest -q until green with the strict tests.

Verification

• •
  Your review.md from step 1 names all three seeded defects with line references (check against the answer key only now).
• •
  python -c "import inspect, requests; print(inspect.signature(requests.post))" confirms your hallucination finding.
• •
  Final pytest -q passes with: strict equality restored in the full-refund test, an exact-full-refund case, a >180-day rejection case, and a retry-count assertion; temporarily re-introducing >= makes at least one test fail.

Completion artifact

Keep review.md (findings, classifications, repair-vs-regenerate decisions with reasoning) and the corrected final diff. Self-grading rubric:

• Detection (0–3): one point per seeded defect found before AI assistance.
• Classification (0–2): each defect correctly labelled by kind = 2.
• Judgment (0–2): repair/regenerate decisions defensible and stated per defect = 2.
• Test restoration (0–2): final suite fails on each re-introduced defect = 2.
• Honest retry (0–1): retry implemented for real (session adapter or loop), not a copied hallucination = 1.

Pass = 7/10.

Stuck?

• Everything looks fine on read-through. That is the design. Stop reading for smell and start executing the checks: signature inspection, boundary truth table, revert-the-hunk test thought experiment.
• retries=3 doesn't crash your tests. Correct — the POST is mocked, so the bogus kwarg is swallowed by MagicMock. That is the lesson: mocks hide hallucinated APIs. Add one unmocked-signature check or a spec=-strict mock.
• Retry test is hard to write. Mock requests.post to raise twice then succeed, and assert post.call_count == 3 against your loop/adapter implementation.

Client-safe variant

Run identically — the scaffold is synthetic, so any free-tier tool is fine for practice. On real client work, apply the same protocol with the sanctioned reviewer only, and keep review.md as the provenance record many client contracts now require for AI-authored changes ("AI-generated, human-reviewed by X on date Y, defects found: N"). If no AI second reader is permitted, steps 1–4 and 6–7 are fully manual and lose nothing essential.

Answer key (don't read before attempting)

• Hallucinated API:requests.post(..., retries=3, backoff_factor=0.5) — requests.post accepts neither kwarg (retries belong to urllib3.util.Retry mounted on a Session via HTTPAdapter). In production this raises TypeError on the first real call; in the tests it is invisible because the mock accepts anything. Verdict: regenerate the retry approach (the design, not just the token, is wrong).
• Wrong boundary:> became >=, so a refund that exactly equals the remaining charge amount — a full refund, explicitly allowed by the docstring — is now rejected. Verdict: repair (one-character fix), plus a pinned exact-boundary test.
• Weakened test:test_full_refund_allowed now swallows RefundError and asserts only truthiness — it passes both with and without the boundary bug, and its comment ("may vary") is the classic tell of a test edited to make bad code pass. Verdict: regenerate the test from the contract, never repair a test to fit code.