Triage "Almost Right" AI Code Before It Costs You a Day

Tier: Core · Workflow area: Verification & Trust

The skill is not avoiding AI output — it is a fast triage discipline: diff-vs-intent review, boundary tests at the edges, and a deliberate repair-vs-regenerate decision instead of an open-ended debugging spiral.

When to use

• •
  An AI assistant or agent produced a diff that compiles and mostly works, but you have not yet verified it.
• •
  An AI-suggested refactor claims to be behavior-preserving ("extracted method", "simplified conditional", "modernized loop").
• •
  You have already spent 15+ minutes "fixing" AI output — that is the signal you skipped triage.

Skip this playbook for throwaway scripts and prototypes where wrong behavior is cheap and visible.

Prerequisites

• •
  The original prompt or task description you gave the AI (intent), and the generated diff side by side.
• •
  A runnable test suite, or at minimum the ability to execute the changed code path locally.
• •
  Any coding assistant (Copilot Chat, Claude Code, Cursor, Continue.dev) for the audit steps — a different session than the one that generated the code.

The workflow

1. 1.

   Diff-vs-intent pass (5 minutes, no execution). Read the diff against what you asked for, hunting three specific defect classes that dominate AI output:

   • •
     Hallucinated APIs: calls that look plausible but do not exist (pd.read_json(orient="records_strict"), redis.expire_at_ms(...)). Hover/jump-to-definition on every unfamiliar symbol.
   • •
     Silently changed contracts: a return type widened to Optional, an exception swallowed into a log line, a default parameter added.
   • •
     Scope creep: hunks touching files or functions your request never mentioned. Every unrequested hunk is guilty until explained.
2. 2.

   Boundary probe. Generated code is usually right in the middle and wrong at the edges. Write (or generate, then read) tests for: empty input, single element, exact boundary values, unicode/None/zero, and the largest realistic input. For example, for a generated pagination helper:

   Copy

   def test_last_page_partial():
       assert paginate(items=list(range(25)), page=3, size=10) == list(range(20, 25))
   
   def test_page_past_end_returns_empty():
       assert paginate(items=list(range(25)), page=9, size=10) == []
   
   def test_empty_items():
       assert paginate(items=[], page=1, size=10) == []
   

   Off-by-one at the last partial page is the single most common seeded defect in AI pagination/chunking/windowing code.

3. 3.

   Equivalence check for "behavior-preserving" refactors. Never accept the claim on inspection alone. In a fresh session (so the model cannot defend its own work), run:

   Copy

   Here is a before/after diff of a refactor claimed to be behavior-preserving.
   List every input for which the two versions could differ: exceptions raised,
   ordering, mutation of arguments, laziness/eagerness, floating-point, locale,
   timezone, and concurrency behavior. Output a table: input scenario | before |
   after | same or different. Do not assess style.
   

   Then pin the suspicious rows with tests, or use snapshot/approval tests over recorded production-like inputs when the surface is wide.

4. 4.

   Decide: repair or regenerate. Apply the two-strike rule:

   • •
     Repair when defects are localized (one wrong condition, one hallucinated call) and you understand the fix fully.
   • •
     Regenerate when the approach is wrong (wrong algorithm, wrong layer, misunderstood requirement), when a second repair attempt fails, or when you cannot explain what the code does. Rewrite the prompt with the failure as an explicit constraint: "Previous attempt used expire_at_ms, which does not exist in redis-py. Use only methods from redis-py 5.x Redis class."
5. 5.

   Record the failure mode. Add one line to your team's assistant instructions file (CLAUDE.md, .github/copilot-instructions.md) so the same class of defect is caught earlier next time, e.g. "Always paginate with explicit min(end, len(items)) bounds; never assume full pages."

Tool variants

• Copilot Chat / Cursor: use /explain on the diff first, then paste the equivalence-check prompt; jump-to-definition catches hallucinated APIs instantly in-IDE.
• Claude Code: ask it to run the boundary tests itself (run the new tests and show failures), then use a fresh session for the adversarial equivalence pass so it is not grading its own homework.
• CI-side: wire mutation testing (mutmut, Stryker) on changed files so weakened tests — an AI specialty — surface as surviving mutants.

Verification

• •
  Every unfamiliar API symbol in the diff resolves via jump-to-definition or official docs.
• •
  Boundary tests exist for empty, single, exact-boundary, and oversized inputs — and they pass.
• •
  For refactors: the equivalence table has no unexplained "different" rows, and each "different" row is either fixed or documented as intended.
• •
  You can explain every hunk of the final diff to a reviewer without referring to the AI.

Pitfalls

• Debugging line-by-line from the top. Triage classes of defect (hallucination, contract change, boundary) instead of stepping through; the diff-vs-intent pass finds most issues without a debugger.
• Letting the generating session verify itself. Models defend their own output; run audits in a fresh session or a different tool.
• Trusting the AI's own tests. Generated tests often assert current (wrong) behavior or weaken assertions (assertAlmostEqual, broad pytest.raises(Exception)). Read every generated assertion.
• Sunk-cost repair. The 45-minute "almost fixed" session is the exact overrun the survey data describes. Two strikes, then regenerate.

Restricted-environment variant

• •
  If client policy forbids sending proprietary code to hosted models, do the diff-vs-intent and boundary passes fully offline — they need your eyes and a test runner, not a model.
• •
  For the equivalence-check prompt, extract a sanitized before/after of just the changed function with renamed identifiers, or run a local model (Ollama + a Qwen-Coder-class model) — adequate for enumerating divergence scenarios even if below frontier quality.
• •
  Mutation testing and approval tests are entirely local and policy-safe; lean on them harder when LLM access is limited.

Success criteria

• •
  Triage of a typical AI diff takes under 15 minutes and reaches an explicit repair/regenerate decision.
• •
  Zero hallucinated-API or boundary defects from AI code reach code review over your next five AI-assisted PRs.
• •
  Your regeneration prompts include the previous failure as a constraint, and second attempts land noticeably closer.
• •
  At least one recurring defect class is encoded into your instructions file.