An agent that answers correctly after looping through eleven redundant tool calls, or after reading data it had no business touching, passes every final-answer test you own. Trajectory evaluation asserts on the path — the sequence of steps — and catches the false positives that final-response testing structurally cannot see.
Tier: Advanced · Workflow area: Testing AI systems — judges, agents & robustness
This playbook adds a trajectory layer to an agent test bench: expected-path specs, loop detection, and step-efficiency budgets, evaluated over traces from Langfuse or Arize Phoenix.
Multi-step agents where extra steps cost real money or latency. Compliance contexts where how the answer was produced matters (which sources were consulted, whether a scoped tool was bypassed). Final-answer suites are green but users report slow, expensive or erratic runs. Loop incidents have already occurred.
Single-call or fixed-pipeline features with no branching — tool-call asserts (previous playbook) cover it. You have no trace capture yet — instrument first; trajectory testing without step-level traces is archaeology without a dig site.
Choose the evaluation mode per scenario — write it down. Three modes, in increasing strictness, and most suites need all three somewhere:
Write expected-path specs as data, not code. A YAML spec per scenario keeps them reviewable by non-programmers:
scenario: refund-request-existing-order
mode: in-order-subsequence
must_include:
- order_lookup
- refund_policy_check
- refund_create
must_not_include:
- customer_pii_export
max_steps: 6
max_repeats_per_tool: 2
The must_not_include line is where compliance requirements become executable — tools the agent may have access to but must not touch on this path.
Pull trajectories from traces and assert against specs.
def tool_sequence(trace):
return [s.name for s in trace.spans if s.type == "TOOL"]
def test_trajectory(spec, trace):
seq = tool_sequence(trace)
assert is_subsequence(spec.must_include, seq), f"path was {seq}"
assert not set(spec.must_not_include) & set(seq)
assert len(seq) <= spec.max_steps
for tool, n in Counter(seq).items():
assert n <= spec.max_repeats_per_tool, f"{tool} called {n}x"
The repeat-count assert is your loop detector: an agent stuck re-calling order_lookup with the same failing arguments is the classic runaway, and max_repeats_per_tool: 2 catches it long before a token-budget blowout.
Add a step-efficiency budget and track it as a metric, not just a gate. Median and p95 steps per scenario, recorded per run:
refund-request-existing-order (n=10 runs)
pass rate: 10/10 final answer correct
steps: median 4, p95 9 <- was p95 5 last release
FLAG: 2 runs took 9 steps — order_lookup retried 4x on
malformed date arg before self-correcting
This run is the poster child for the playbook: every final answer correct, and a real regression (a flapping self-correction loop) visible only in the path.
Score path quality with a judge only where rules cannot reach. Whether interleaved extra steps were reasonable is a judgment call — a calibrated LLM judge over the trajectory ("were any steps redundant given the results already retrieved?") is legitimate here. Keep it advisory or calibrate it before it gates (see the judge playbook); the deterministic spec asserts remain the gate.
Run under repetition and gate with pass^k thinking. Paths vary run to run at nonzero temperature. Repeat each scenario 5–10 times; gate on "all runs satisfy the spec" for compliance scenarios (must_not_include violations are one-strike), and on rates (≥ 90%) for efficiency budgets.
Feed incidents back as specs. Every production loop or forbidden-tool incident becomes a scenario with a spec that would have caught it. The spec file grows into your agent's behavioral contract.
A trajectory violation with a correct final answer is still a failing test. The one-in-ten run that bypasses refund_policy_check and happens to get it right is exactly the run that will burn you at scale — do not let green answers argue you out of red paths.
Final-answer testing grades the destination; trajectory testing grades the driving. Agents are judged in production on both — a suite that only checks arrivals will certify an agent that speeds through every red light it meets.