A multi-step agent finishes cleanly — no crash, no cap, a fluent FINAL — and the answer is wrong. Often it is plausibly wrong: a reorder bill for an item that did not need reordering, a total computed from a price the shop never had. Nothing in the log looks broken, and every late step reads as sensible. The error only surfaces when a human compares the answer against the real data, sometimes days later.
Any agent whose runs take more than two or three steps — module 3.9's run_agent
onward. The trigger is one early observation being misread or mishandled: a tool
error treated as data ("item not found" read as quantity zero), a lookup on a
mistyped item name, or a stale value carried forward after a later step changed
it. Small local models misread more often, but long chains make every model
vulnerable. Linked module: 3.9 (Agents).
Agents compound errors. Each turn plans from everything already in the context, so a wrong belief entering at step 2 becomes the foundation for steps 3 through 7 — the model is not lying at the end, it is faithfully reasoning from a poisoned premise. The arithmetic is brutal: a 95 percent per-step accuracy across seven steps yields roughly a 70 percent clean run, and nothing inside the loop notices, because every step after the bad one is locally consistent. Diagnose by reading the trace backwards. Take each claim in the FINAL and find the observation that proves it; the claim with no supporting observation — or one resting on an error message — marks the poisoned step. Everything downstream of it is contaminated, however reasonable it reads.
Two harness changes, both mechanical. First, observation checks: when a tool
returns an error or an empty result, wrap it so it cannot be mistaken for data —
{"error": ...} with wording that says what to do next, never a bare zero or
empty string. Reject FINALs produced in the same turn an error was received.
Second, verify before FINAL: when the model declares FINAL, do not accept it
immediately — send one more message asking it to list each factual claim with the
step number of the observation that proves it. Claims that cite no step, or cite
a step whose result does not contain the number, send the run back with a named
correction. For high-stakes outputs, move this check into a second model call
with fresh context — the reviewer pattern — so the verifier does not inherit the
poisoned premise.
Scope small: five checked steps beat twelve unchecked ones, because compounding punishes length. Keep a set of test goals with known correct answers and run them after every prompt or tool change. Review one real trace end-to-end weekly, tracing FINAL claims to observations even when the answer looks right — near misses show up long before users do. And never let error messages resemble data: an error that can be read as a quantity will be, eventually.
Q: "Your agent's answer was fluent and wrong, and the log shows no errors. Walk me through your debugging." Strong answer: name error compounding, explain that late steps inherit early beliefs so the trace must be read backwards from the final claims to their supporting observations, and give both fixes — informative error shapes and claim-by-claim verification before FINAL is accepted. Bonus for the arithmetic: per-step accuracy to the power of step count, which is why scoping small is a reliability decision, not a style preference.