Lab 03 · Hypothesis-Driven Debugging

Engineers who debug AI-first without this discipline ship the model's first confident guess; ~68% of developers now use AI first when debugging, and this worksheet is what separates the ones it helps from the ones it misleads.

Time & tier

~75 minutes hands-on · Core track, Foundation difficulty.

Setup

Inline scaffold — copy exactly, including the oddly named root file. pip install pytest. Layout: billing/ package + two root files.

billing/__init__.py: empty file.

billing/config.py:

import os

TAX_RATE = float(os.environ.get("TAX_RATE", "0.18"))
# Feature flag: never send receipts from dev machines.
SEND_RECEIPTS = os.environ.get("SEND_RECEIPTS", "False")

billing/discounts.py:

def discount_rate(order_total):
    """Spec: >=10000 -> 15%; >=5000 -> 10%; >=1000 -> 5%; else 0."""
    if order_total > 10000:
        return 0.15
    if order_total > 5000:
        return 0.10
    if order_total > 1000:
        return 0.05
    return 0.0

billing/invoice.py:

import json
from billing.config import TAX_RATE, SEND_RECEIPTS
from billing.discounts import discount_rate

SENT = []

def build_invoice(order_total):
    rate = discount_rate(order_total)
    subtotal = order_total * (1 - rate)
    total = round(subtotal * (1 + TAX_RATE), 2)
    invoice = {"order_total": order_total, "discount_rate": rate,
               "total": total}
    if SEND_RECEIPTS:
        SENT.append(invoice)
    return json.dumps(invoice)

json.py (at repo root — copy as-is):

# scratch from a pricing spike -- TODO move somewhere sensible
data = {"trial": True}

run_invoice.py (repo root):

import sys
from billing.invoice import build_invoice
print(build_invoice(float(sys.argv[1])))

test_billing.py (repo root):

import billing.invoice as inv
from billing.discounts import discount_rate

def setup_function(f):
    inv.SENT.clear()

def test_boundary_discount():
    assert discount_rate(1000) == 0.05   # spec: 1000 qualifies

def test_no_receipts_in_dev(monkeypatch):
    # SEND_RECEIPTS defaults to off; building an invoice must not send
    inv.build_invoice(2000)
    assert inv.SENT == []

Verification checklist: the symptoms reproduce — (1) pytest -q fails test_boundary_discount with an assertion error; (2) test_no_receipts_in_dev fails too, but with an AttributeError mentioning json, not an assertion; (3) python run_invoice.py 2000 crashes with the same AttributeError. If all three fire, the scaffold is correct. (One bug is masking another — that layering is deliberate.)

Guided steps

1. 1Start the debug worksheet. Create debug-worksheet.md with one section per symptom and this table per section: hypothesis | rank | disconfirming test | result | verdict.
2. 2Symptom 1 (failing boundary test). Paste the test failure, discounts.py, and the docstring spec into your assistant with this prompt: "Do not propose a fix. Give me your three most likely causes, ranked, and for each one a concrete check that would DISPROVE it if it's wrong." Decision point: if any hypothesis has no possible disproving check, reject it as unfalsifiable and demand a replacement.
3. 3Run the disconfirming checks yourself (a one-line python -c probe or a temporary test each). Record results in the worksheet. Only when exactly one hypothesis survives, ask for the minimal fix, apply it, and re-run pytest -q.
4. 4Symptom 2 (the AttributeError — hits both the CLI and test_no_receipts_in_dev). Paste the full traceback plus the repo file listing (ls output matters here — include it). Same protocol: three ranked hypotheses, each with a disconfirming check. A killer disconfirming probe: python -c "import json; print(json.__file__)" run from the repo root. Decision point: if the AI's hypotheses are all about invoice.py's code, it has missed the environment class of bug — push with "what module does import json actually resolve to in this process, and why?"
5. 5Symptom 3 (revealed only after the step-4 fix). Re-run pytest -q: test_no_receipts_in_dev now fails on its assertion — receipts are being sent even though the flag defaults to off. Same protocol: paste config.py, invoice.py, and the failing test. Expect the AI to spot it fast — you must still write the disconfirming test for the top two hypotheses before fixing. A good probe: python -c "from billing.config import SEND_RECEIPTS; print(repr(SEND_RECEIPTS), bool(SEND_RECEIPTS))". Note in your worksheet that this bug was masked by symptom 2 — layered failures are the norm in real debugging.
6. 6Fix all three.pytest -q green and python run_invoice.py 2000 prints an invoice.
7. 7Prevention pass. For each bug, add one line to the worksheet: "Prevention:" — the lint rule, test, or convention that would have stopped it (e.g., a naming ban, strict boolean parsing helper, boundary tests in specs).

Verification

• •
  pytest -q: all tests pass; python run_invoice.py 2000 prints valid JSON.
• •
  Worksheet shows ≥3 hypotheses per symptom, each with a disconfirming test and a recorded result — nine rows minimum.
• •
  At least two hypotheses across the lab were disproved by your tests (if every first hypothesis survived, your AI got lucky or your tests were rubber stamps — note which).
• •
  Fix diffs are minimal: three bugs, roughly three lines changed (plus one file moved/renamed).

Completion artifact

Keep debug-worksheet.md plus the final git diff. Self-grading rubric:

• Process (0–2): every fix preceded by a completed hypothesis table = 2; any fix-first shortcut = 0 for that bug.
• Falsifiability (0–2): every recorded hypothesis has a genuinely disconfirming check = 2.
• Diagnosis (0–3): one point per bug correctly classified (logic / config / environment-shadowing) before reading the answer key.
• Fix minimality (0–2): no drive-by refactors bundled with fixes = 2.
• Prevention (0–1): all three prevention entries are concrete and adoptable = 1.

Pass = 7/10.

Stuck?

• The AI jumps straight to a patch. Restate the contract: "No fixes. Hypotheses with disconfirming tests only. I will run the tests." If it persists, shorten the pasted context to symptoms only.
• Symptom 2 won't reproduce (run_invoice.py works). You likely placed json.py inside billing/ or your shell's cwd differs — the file must sit in the same directory as run_invoice.py, and you must run from the repo root.
• Symptom 3's test passes unexpectedly after the shadowing fix. Check you copied the config default exactly as "False" (a string), and that no SEND_RECEIPTS variable is set in your shell (unset SEND_RECEIPTS).

Client-safe variant

The protocol is designed for restricted environments: nothing forces you to paste whole files. Practice symptom-only prompting — traceback, failing assertion, and signatures of the suspect functions, with client identifiers redacted. Local models via Ollama handle ranked-hypothesis prompting adequately for bugs of this size. The worksheet doubles as the evidence trail client leads increasingly ask for when AI touches their code.

Answer key (don't read before attempting)

• Bug 1 — logic (boundary).discounts.py uses > where the spec says >=: an order of exactly 1000 (and exactly 5000/10000) lands in the lower tier. Fix: >= on all three comparisons.
• Bug 2 — config (truthy string).SEND_RECEIPTS is read as the string"False", and any non-empty string is truthy, so if SEND_RECEIPTS: always sends. Fix: parse explicitly, e.g., SEND_RECEIPTS = os.environ.get("SEND_RECEIPTS", "false").lower() in ("1", "true", "yes").
• Bug 3 — environment (stdlib shadowing). Both the CLI run and pytest put the repo root on sys.path, so the scratch json.py shadows the standard library; json.dumps doesn't exist on it, hence the AttributeError everywhere invoice.py is imported — which is also why it masks bug 2 until fixed. Fix: delete or rename the scratch file. Prevention: never name project files after stdlib modules; a CI grep or flake8 plugin catches it. (Python 3.12+ tracebacks helpfully suggest the rename — if yours did, the diagnosis was free, so grade yourself on the worksheet discipline, not the find.)