AI-First Error Diagnosis with Ranked Hypotheses

Tier: Foundation · Workflow area: Debugging

A single confident answer to a stack trace is often plausible and wrong; a ranked differential diagnosis with cheap ways to falsify each candidate turns the model into a genuinely fast debugging partner instead of a confident guesser.

When to use

• •
  Any nontrivial error: unfamiliar exception, failure in code you did not write, an error that survived your first five minutes of looking.
• •
  Triage under pressure — you need the likely causes ordered by probability and cost-to-check.

Skip it for errors whose message literally tells you the fix (missing import, obvious typo) — read before you paste.

Prerequisites

• •
  The full artifact set: complete stack trace (not the last line), the code of the top 2-3 frames you own, relevant log lines around the failure, and what changed recently.
• •
  Any chat assistant; IDE assistants save the copy-paste.
• •
  Discipline to sanitize: strip customer emails, tokens, and internal hostnames before pasting.

The workflow

1. 1.

   Assemble context before prompting. A trace alone invites guessing. Gather: trace, the code of the frame where it raised plus its caller, one config fact ("runs in 4 gunicorn workers"), and recency ("started after yesterday's deploy of #482").

2. 2.

   Demand the differential. The prompt shape that changes everything:

   Copy

   Production error, Python/FastAPI, started after deploy #482 (diff summary
   below). Full trace, the two relevant functions, and surrounding logs
   attached.
   
   psycopg.OperationalError: connection pool exhausted [trace...]
   [code of get_report() and its caller...]
   [logs: 12:01:03 pool size 20/20 ... 12:01:04 ERROR ...]
   
   Do NOT jump to a fix. Give me:
   1. The 3-5 most likely root causes, ranked, with a probability guess.
   2. For EACH: the specific evidence in what I pasted that supports or
      undercuts it.
   3. For EACH: the cheapest disconfirming test - a command, log query, or
      code probe that would rule it out in under 5 minutes.
   4. What additional single piece of evidence would most change the ranking.
   

3. 3.

   Run the disconfirming tests, cheapest first. This is the human half. For "leaked connections in the new report endpoint": grep the diff for connection usage outside the context manager. For "pool too small for new traffic": check request-rate dashboards. Kill hypotheses; report results back:

   Copy

   Test results: (1) confirmed - get_report() acquires a connection in a
   branch that returns early without releasing when format=csv. Hypothesis 2
   ruled out - traffic flat. Now: minimal fix, and a regression test that
   fails on the current code.
   

4. 4.

   Fix with a failing test first. Even at Foundation tier, insist the fix comes with a test that reproduces the failure (early-return with format=csv leaks) and passes after. (Deep version: pb-11.)

5. 5.

   Bank the diagnosis. Paste the confirmed root cause + fix into the ticket/postmortem in two sentences. Future-you searching "pool exhausted" will find it.

Tool variants

• Chat assistant (any major vendor): the default; strongest models give noticeably better differentials on gnarly traces.
• IDE assistant (Copilot Chat, Cursor): attach files instead of pasting; good when the relevant code spans several files.
• Terminal agent: can run the disconfirming tests itself (grep the diff, run the probe script) — powerful, but keep it read-only against production-adjacent systems.

Verification

• •
  At least two hypotheses were actually tested — if you only ever check hypothesis #1 and it always "confirms", you are rubber-stamping, not diagnosing.
• •
  The confirmed cause explains all observed evidence, including timing ("why after deploy #482?" and "why only csv exports?"). Unexplained evidence means the diagnosis is incomplete.
• •
  A regression test fails on the old code and passes on the fix.
• •
  The error rate actually went to zero after deploy — watch the dashboard, don't assume.

Pitfalls

• Accepting the single confident answer. The model's first guess fits the trace's surface; the ranked-differential format is precisely the countermeasure.
• Pasting the last line of the trace. The root frame and the middle "caused by" sections carry the signal. Full trace, always.
• Letting the model "confirm" its own hypotheses. Disconfirming tests run in your terminal, not in its imagination — it will happily hallucinate log lines that support its theory.
• Context contamination. After several wrong turns, models anchor on their earlier theories. Fresh conversation, re-paste evidence, no mention of the abandoned hypotheses.

Restricted-environment variant

Stack traces and logs are among the leakiest artifacts in engineering: they carry customer identifiers, internal hostnames, query fragments, sometimes tokens. Before any hosted tool: replace customer values (user 8841 / rohit@... → user_A), internal hosts (db-prod-3.corp → DB_HOST), and cut auth headers entirely. Under DPDP-style rules, log lines with personal data must not leave approved systems. Client work: use the sanctioned enterprise assistant, or run the identical differential prompt on a local model — mid-size local models are respectable at ranked diagnosis when you provide rich context, and the disconfirming tests are yours to run either way.

Success criteria

• •
  Time-to-root-cause measurably below your solo baseline on comparable bugs.
• •
  Zero incidents of shipping a fix for a hypothesis that was never actually confirmed.
• •
  Your last three diagnosis chats all contain the phrase "disconfirming test" — the habit is installed.
• •
  Nothing sensitive left your machine unredacted.