Reproduce the Bug as a Failing Test Before Any Fix

Tier: Core · Workflow area: Debugging

Coding agents make this affordable for the first time: an agent can read the bug report, explore the code, and iterate until it has written a test that fails for the reported reason — and that test then becomes the fix's acceptance criterion and the permanent regression guard.

When to use

• •
  Any reported bug in code with a test harness — especially "works on my machine", timezone/boundary bugs, and bugs in code you did not write.
• •
  Bug reports from support/QA that describe symptoms, not causes.
• •
  Before delegating a fix to an agent or a junior: repro-test first, fix second, is the safest delegation contract there is.

Skip only when a repro is genuinely impractical to automate (hardware faults, third-party outages) — then document the manual repro steps instead.

Prerequisites

• •
  A runnable test suite and knowledge of how to run one test file fast.
• •
  The bug report with concrete inputs ("invoice for subscription ending 31 Mar shows April charge") — vague reports go back for details first.
• •
  A coding agent that can run tests (or a chat assistant plus you running them).

The workflow

1. 1.

   Brief the agent: repro only, no fixing. The separation is the whole method:

   Copy

   Bug report: "Customer 'annual' subscriptions that renew on the last day
   of a month are charged twice when the renewal crosses a DST change in
   the customer's timezone (report: renewal 2026-03-29, Europe/Berlin)."
   
   Your ONLY job right now is a failing test. Do not fix anything.
   1. Locate the renewal/billing rollover code and its existing tests.
   2. Write a test in tests/billing/ reproducing this exact scenario:
      fixed clock, timezone Europe/Berlin, renewal date 2026-03-29.
   3. Run it. It must FAIL, and fail with the double-charge symptom
      (two charge records), not with a setup error.
   4. Show me the test, the failure output, and one sentence on why the
      code produces it.
   

2. 2.

   Audit the failure, not just the test. Read the failure output yourself. The test must fail for the reported reason. A test failing with fixture 'customer' not found reproduces nothing; a test asserting the wrong expectation "fails" uselessly. Ask: Explain why this failure output demonstrates the reported bug and not a test bug.

3. 3.

   Harden the repro. Two quick checks: run it three times (must fail deterministically — flaky repro means the test still depends on real time or ordering), and confirm the assertion is on behavior a user described (charge count), not on internals that a refactor would break.

4. 4.

   Now — and only now — fix. Same session or new one:

   Copy

   The failing test tests/billing/test_dst_renewal.py is the contract.
   Fix the underlying bug. Constraints: the new test must pass, the whole
   billing suite must stay green, and the fix must handle the general case
   (any DST transition, any tz), not special-case 2026-03-29.
   

5. 5.

   Cross-check with a sibling case. Ask for one more test at an adjacent boundary (southern-hemisphere DST, non-DST timezone control case) to confirm the fix generalizes and the control still passes. Overfit fixes die here.

6. 6.

   Ship test + fix together. The PR contains both; the description quotes the original failure output — reviewers can see the bug existed and is gone.

Tool variants

• Terminal agent (Claude Code, etc.): the natural fit — explores, writes, runs, and iterates the test loop itself.
• Copilot agent mode: same two-phase instruction works; keep phase 1 and phase 2 as separate requests so "no fixing yet" is enforceable.
• Chat assistant + you at the keyboard: the model drafts the test from pasted code; you run it and paste failure output back. Slower, same rigor.

Verification

• •
  The test fails on unfixed code with the reported symptom, deterministically (3/3 runs).
• •
  After the fix: new test green, sibling/control tests green, full affected suite green.
• •
  Revert check: git stash the fix, run the test, watch it fail again — proves the test still guards the behavior.
• •
  The fix handles the general case; grep it for suspicious literals from the bug report (a hardcoded 2026-03-29 is an alarm bell).

Pitfalls

• Letting the agent fix "while it's there". Merged repro-and-fix in one step gives you a green test you never saw fail — which proves nothing. Phase discipline is the method.
• Setup-error "failures". The most common agent shortcut: a test that errors before reaching the buggy path. Step 2 exists for this.
• Asserting internals. Tests pinned to private call sequences break on refactor and outlive their usefulness; assert observable behavior.
• Accepting a flaky repro. Intermittent failure means the trigger condition is not yet controlled (real clock, shared state) — keep tightening until deterministic.

Restricted-environment variant

The workflow runs entirely inside the repo, so with a client-sanctioned agent it transfers unchanged. Chat-only sanction: paste the billing function and existing test shapes (sanitized — fake customer IDs, no real amounts if contractually sensitive) and have the model draft the test for you to run locally. Local model route: mid-size coder models draft serviceable unit tests when you paste the target function plus one exemplar test; the run-audit-harden loop (steps 2-3) is yours regardless of which model drafted. Bug reports themselves often contain customer data — sanitize before pasting anywhere hosted.

Success criteria

• •
  Zero fixes shipped this quarter without a first-failing regression test.
• •
  Reopened-bug rate on your tickets drops — the regression guard is doing its job.
• •
  Repro time for a typical report: under 30 minutes with the agent doing the exploration.
• •
  Reviewers stop asking "how do we know this fixes it?" — the PR shows the before/after.