This module installs the debugging order of operations that AI makes both cheap to skip and expensive to skip: ranked hypotheses with disconfirming tests before any fix, a failing test before any repair, and a measured number before you believe a performance or equivalence story.
The bug report lands: order totals occasionally show one cent low when a percentage discount is applied — the same build_invoice discount math Sana had squinted at while first learning billing-service.
She pastes the stack trace into her assistant and, out of habit, nearly types "fix this." The model would happily oblige; it proposes a rounding change in seconds, confident and plausible.
Instead she asks for three ranked hypotheses, each with a disconfirming test — the cheapest check that would rule it out.
Float accumulation before rounding. It dies immediately: a five-line script shows the discount path uses Decimal throughout.
Banker's rounding applied one step too early. This one survives.
Only now does she ask for a fix — and only after an agent writes a failing test that reproduces the missing cent and refuses to proceed until it does.
The fix turns that test green.
Had she accepted the first plausible diff, she'd have changed rounding that wasn't broken, muddied the evidence, and shipped the real bug intact behind a green suite.
Debugging is where AI assistance pays off fastest and where it punishes sloppiness hardest. Roughly 68% of developers now reach for AI first when diagnosing failures, and the Stack Overflow 2025 survey's most-cited frustration — 66% of developers — is AI output that is "almost right, but not quite."
Both facts are true at once, and the difference between the engineer who benefits and the engineer who bleeds time is process: hypotheses before fixes, reproduction before repair, benchmarks before belief. The Common Core verification module gave you the failure taxonomy and the read-run-test discipline; this module applies them to the debugging loop of a working senior engineer — including the newest debugging surface of all, the code your own AI tools generated an hour ago.
AI makes generating a plausible fix nearly free. The scarce skill is generating a disconfirmable diagnosis — a hypothesis with a test that could prove it wrong — before any fix is written.
git bisect paired with LLM fault explanationThe naive AI debugging loop is: paste error, receive fix, apply fix, hope. It works often enough to be seductive and fails badly when it fails — because a confident-sounding fix for the wrong cause doesn't just waste the attempt, it mutates the codebase and muddies the evidence for the next attempt. The professional loop inverts the order: the AI's job is to produce diagnoses, and your job is to kill the wrong ones cheaply before anyone writes a fix.
Give the model the full evidence bundle — stack trace, the relevant code (not the whole repo), recent changes, and any logs around the failure window — and constrain the output shape:
Here is a stack trace, the function it points into, and the last
two commits touching this file. Do NOT propose a fix yet.
Give me the 3 most likely root causes, ranked by probability.
For each: (a) the causal mechanism in one paragraph, (b) which
evidence supports it, (c) which evidence is in tension with it,
(d) a cheap disconfirming test — a command, log query, or
5-line script whose result would RULE THE HYPOTHESIS OUT.
Two things in this prompt do the work:
This is the same structure a strong senior engineer runs in their head; the AI just makes each iteration cost seconds instead of minutes, which is why practitioners describe getting "three likely causes in seconds" from tools like Claude, Copilot Chat, or Cursor.
Ask an LLM "could this be a connection-pool exhaustion issue?" and it will usually agree — that is the sycophancy failure mode from the Common Core taxonomy, and in debugging it is lethal because it lets you spend an afternoon fixing a cause that was never operating. Confirming evidence is cheap: almost any symptom is consistent with several causes. Disconfirming evidence is decisive: a pool-exhaustion hypothesis dies instantly if the pool metrics show 4 of 50 connections in use during the failure window. Run the cheapest disconfirming tests first, in hypothesis-probability order, and only when exactly one hypothesis survives do you ask for a fix — now with the surviving diagnosis pasted into the prompt as established fact.
Assessment scenarios in this track score the order of operations: hypotheses → disconfirming tests → surviving diagnosis → fix. A correct fix reached by skipping the middle steps is scored as process failure, exactly as Meta's AI-enabled interview rubric treats it.
Intermittent bugs bury their evidence in volume: thousands of log lines across services, of which perhaps forty matter. Long-context models turn this from an evening with grep into a structured extraction task. Feed the raw window (redacted per your data policy — customer identifiers out before anything leaves the machine) and demand a timeline, not an opinion:
These are logs from 14:02–14:09 across api-gw and orders-svc.
Build a strict timeline of anomalous events: timestamp, service,
event, and why it deviates from the baseline pattern in the same
logs. Quote the exact log lines as evidence. Flag any gap where
expected periodic events are missing. Do not speculate on cause.
The "quote exact lines" requirement is your verification handle — every timeline entry is checkable against the source in seconds, which converts a fabrication risk into a reading-comprehension check. The timeline then feeds the hypothesis prompt above as evidence.
A bug you cannot reproduce is a bug you cannot verify a fix for. This was true before AI; agents make it enforceable.
Modern coding agents — Claude Code, Copilot agent mode, Cursor's agent — can execute the code they write, which means you can and should demand reproduction as a gated first step:
Bug report: order totals occasionally show one cent low when a
percentage discount is applied. Suspected area: billing/build_invoice.
Step 1 ONLY: write a failing test that reproduces this bug.
Run it. Show me the failure output. Do not modify any
non-test code. If you cannot make it fail, report what you
tried and stop — do not "fix" anything.
The failing test is worth more than the fix, for three reasons:
The "if you cannot make it fail, stop" clause matters: an agent that can't reproduce the bug but proceeds to "fix" it anyway is generating plausible motion, not progress — and its diff will be scope drift by construction.
An agent asked to fix a bug it never reproduced will still return a confident diff. Green tests after that diff prove nothing — they were green before. Reproduction first is the only thing that makes the after-state meaningful.
Your first 15 minutes on any bug: before asking for a single fix, make the AI produce three ranked hypotheses, each with a disconfirming test, and run the cheapest one first — a fix aimed at the wrong cause doesn't just waste the attempt, it mutates the code and muddies the evidence for the next one, exactly the trap Sana sidestepped on the missing cent.
For regressions — it worked at some commit, it's broken now — the localization tool is deterministic and predates AI: git bisect finds the culprit commit in log₂(n) steps, and with a scriptable reproduction test it runs unattended (git bisect run pytest tests/test_totals.py -k discount). The AI's role starts after bisect lands: paste the culprit diff and ask for the fault mechanism.
git bisect identified this commit as the first bad one. The failing
test is attached. Explain the mechanism: what specific behavioral
change in this diff causes this failure? Trace the data flow from
the changed lines to the assertion that fails.
This division of labor — deterministic tools localize, the LLM explains — is a pattern you will see throughout this track, and it is the same architecture Google's engineers report for large-scale migrations: let exact tools do what exact tools do, and spend the model on the interpretive step. Reasoning-capable models are notably good at tracing a code path from a diff to a fault, which is exactly the shape of this task.
The 66% figure deserves respect: debugging AI-generated code is now an explicit duty in roughly half of backend job postings and a scored topic in AI-enabled interviews. "Almost right" code fails differently from human code — it compiles, reads idiomatically, and hides its defects in the gap between what you asked for and what it did. Triaging it is a discipline, not a vibe.
Before reading the diff line by line, write down (or restate from your prompt) what the change was supposed to do — the intent. Then audit the diff against the intent in three passes:
Then apply the Common Core code checklist — boundaries (empty, first/last, zero, None, failure paths), hallucinated dependencies (every new import and API call checked against real docs or your IDE's resolution), and weakened tests (an AI asked to "make the tests pass" will sometimes edit the tests; any test change in a bug-fix diff is guilty until explained).
The instinct is always to repair — you have code in hand, and fixing feels faster than starting over. Often it isn't. Because generation is cheap, delete-and-regenerate with a better prompt is a first-class move, and choosing between them is a skill interviewers now probe directly.
| Signal | Decision |
|---|---|
| One or two localized defects; overall structure sound and idiomatic | Repair — fix by hand, add the test that would have caught it |
| Wrong algorithm, wrong decomposition, or misunderstood requirement | Regenerate — the defects are symptoms; rewrite the prompt with the missing constraint |
| Hallucinated API at the core of the approach | Regenerate — the approach was built on a thing that doesn't exist; repairs inherit the fiction |
| Defects found on the third distinct repair pass | Regenerate — repeated repair of one artifact is evidence the generation was bad, and each pass adds sunk-cost pressure |
| Scope drift only; core change correct | Repair — revert the excess hunks, keep the intent-matching core |
The regenerate branch has a rule attached: never regenerate with the same prompt. The defect tells you what constraint was missing — an edge case, a version pin, an explicit "do not modify tests" — and the new prompt carries it. Regeneration without prompt improvement is a slot machine.
Sunk cost is the enemy. The 200 lines the AI generated cost you nothing to produce; treating them as an investment to protect is how engineers spend an hour repairing what a two-minute re-prompt would have replaced.
Most debugging is shallow: the cause is within a step or two of the symptom. The bugs that consume days are the deep ones — races, deadlocks, ordering assumptions, state corruption across async boundaries — where the causal chain spans many frames and possibly many threads. This is the specific niche where extended-reasoning models (Claude with extended thinking, OpenAI's o-series, Gemini's thinking modes) earn their latency and cost: practitioners consistently report that reasoning models are markedly better at navigating long code paths to a fault origin than standard chat completions.
For a suspected race, the prompt shape that works is an explicit interleaving walk:
Two coroutines share this OrderCache (code attached). Coroutine A
runs refresh(); coroutine B runs get_or_load(). Walk the possible
interleavings of their await points step by step. For each
interleaving where an invariant breaks (stale read, double load,
lost update), show the exact schedule as a numbered sequence of
lines executed, and state the broken invariant.
Demand the schedule, not the conclusion. "There may be a race in the cache refresh" is unfalsifiable narrative; "A executes line 42, yields at the await on line 44, B reads self._entries at line 61 before A's write on line 45" is a claim you can check against the code in front of you — and mechanically convert into the next artifact.
A race explanation you can't demonstrate is a theory. Have the model turn its own schedule into a deterministic or statistical reproduction: a test that forces the interleaving with explicit synchronization barriers where possible, or hammers the path under concurrency (pytest with asyncio.gather over hundreds of iterations, Go's -race detector, Java's jcstress) where not. The stress test does double duty — it disconfirms the hypothesis if the model reasoned its way to a phantom race, and it becomes the regression guard if the race is real. Either outcome is a win, which is the mark of a well-designed verification step.
Reasoning models are also the right escalation when a standard model's debugging answers start circling — repeating hypotheses, contradicting earlier steps, or "fixing" the same line differently each attempt. Circling is a signal the causal chain exceeds the model's default deliberation, not a signal to push harder on the same tool.
Performance work has a built-in verification advantage: the ground truth is a number. That makes it excellent AI territory — if you keep the number in the loop.
Profiler output (py-spy dumps, pprof graphs, flamegraph data) and query plans (EXPLAIN ANALYZE) are dense, structured artifacts that LLMs interpret well and juniors interpret slowly. Paste the artifact with its context and ask for ranked bottlenecks with a confirmation plan:
EXPLAIN (ANALYZE, BUFFERS) output for our slowest endpoint's query
is below, with the table schemas and row counts. Rank the
bottlenecks by estimated contribution. For each: the mechanism
(seq scan? mis-estimated rows? sort spill?), the fix you'd try
first, and the specific plan-node numbers that would change if
the fix works.
A concrete example of what this catches: a plan showing Seq Scan on orders (actual rows=2,100,000) feeding a Nested Loop whose row estimate was 40 — the AI will correctly finger the stale statistics or missing index and, more usefully, tell you which numbers to watch after ANALYZE orders or CREATE INDEX. The same protocol applies to application profiles: "the top frame is json.dumps at 38% of samples" leads to ranked hypotheses (over-serialization in a hot loop? logging middleware serializing full payloads?) each with a cheap check.
The failure mode here is accepting a fluent performance story without measurement. LLM explanations of performance are plausible-but-wrong at meaningful rates — the mechanism named is real, but it isn't your bottleneck. The rule: no performance change merges on the strength of an explanation. Have the AI generate the benchmark harness too (pytest-benchmark, hyperfine, pgbench, k6 — all free), run before and after on realistic data volumes, and let the delta decide. If the AI predicted a 10x improvement from the index and you measure 1.05x, the diagnosis was wrong, and you've spent minutes learning that instead of shipping a useless index to production. The prediction-then-measurement loop also builds your calibration for the next investigation — you learn which of the model's performance stories to weight.
Refactoring with AI deserves its own defensive posture, because the failure mode is uniquely quiet: a transform presented as behavior-preserving that isn't. Conference analyses of LLM refactoring have reported up to 76% of initial refactor suggestions containing flaws or hallucinated changes — the exact rate varies by task and model, but the direction is not in dispute, and "it's just a refactor" is precisely when review attention drops.
The defense is to make "behavior-preserving" a measured property instead of a claimed one:
pytest snapshot plugins, Jest snapshots) make this a ten-minute setup. Any diff is a behavior change the AI didn't disclose._legacy and property-test new(x) == _legacy(x) over generated inputs (Hypothesis and fast-check are free and ideal here).git diff --stat should touch only expected files. A "rename" that edits 3 extra files has told you what it really was.A realistic catch, from a Python refactor where the AI "simplified" a lookup:
- for handler in self.handlers:
- if handler.matches(event):
- return handler.process(event)
- return self.default.process(event)
+ handler = next((h for h in self.handlers if h.matches(event)),
+ self.default)
+ return handler.process(event)
Equivalent — unless handler.process raises and the original's callers relied on knowing no handler matched via a log line that lived in default.process... or unless matches has side effects and short-circuit order changed. The point is not that this diff is wrong; it's that "obviously equivalent" is a claim, and a snapshot run over the event corpus settles it in one command where an argument settles nothing.
Equivalence checking is the refactor-specific instance of the Common Core rule "if you can't verify it, you can't delegate it." A refactor of code with no tests, no snapshots, and no probes is not a refactor — it's a rewrite wearing a refactor's label, and it gets rewrite-level review.
Check yourself: An agent hands you a confident diff to fix a bug it was never asked to reproduce, and your suite is green afterward. Why does that green prove nothing, and what step would have made the after-state meaningful?
Work these before your lab sessions; each maps to a graded lab artifact.
EXPLAIN ANALYZE, and compute prediction error. Repeat for the second-ranked fix.The playbook library for this module packages the reusable assets: the ranked-hypotheses and interleaving-walk prompt templates, the repair-vs-regenerate decision table as a printable card, the reproduce-first agent instruction block for your CLAUDE.md/instructions files, and the equivalence-check harness recipes (snapshot, probe, property-based). Do this at work this week → pb-10 AI-First Error Diagnosis with Ranked Hypotheses (the disconfirming-test protocol Sana ran on the missing cent), pb-11 Reproduce the Bug as a Failing Test Before Any Fix (the gated first step that makes the fix's green mean something), and pb-14 Triage "Almost Right" AI Code Before It Costs You a Day (the repair-vs-regenerate decision under the clock). The graded labs rehearse the two highest-stakes skills under time pressure: the hypothesis-driven debugging lab (three seeded bugs, worksheet graded on process) and the "almost right" triage lab that doubles as direct preparation for Meta/Google-style AI-enabled interview rounds. Do the debugging lab before the triage lab — the hypothesis discipline is the substrate the triage skill builds on.