Single-turn evals grade an answer; agents produce trajectories — sequences of reasoning, tool calls, observations, and retries — and two trajectories can reach the same final answer where one took 4 sensible steps and the other took 19 steps, called the payments API twice "to check", and got lucky.
Tier: Advanced · Workflow area: Evaluation Engineering
Final-answer grading alone systematically overrates lucky agents and underrates unlucky-but-sound ones. This playbook builds trajectory-level evaluation over traced runs: tool-choice correctness, step efficiency, task completion, and safety-violation detection — the eval layer that agentic QA roles are now hired to build.
Prerequisite mindset: this extends pb-34→pb-37 to agents; if you have no single-turn eval discipline yet, build that first — trajectory evals inherit all of it and add structure.
Do trajectory error analysis first (pb-34, agent edition). Read 30 full trajectories — not summaries, the actual step sequences. Open-code what you see; agent-specific failure modes cluster reliably: wrong tool for the step, correct tool with malformed arguments, ignoring a tool error and proceeding, retry loops, hallucinating a tool result that never came back, goal drift mid-task, unnecessary steps, and stopping early with a confident wrong answer. Build the taxonomy; it defines your graders.
Define the four metric families per task.
Implement graders in the assertions-before-judges hierarchy. Programmatic: tool-call sequence checks against the allowed-phase map, argument schema validation, budget counters, PII regexes on arguments, terminal-state checks — all pure code over the trace JSON. Judge-graded (pb-36 discipline: rubric, dual-order where pairwise, calibrated on human-labeled trajectories): reasoning coherence, whether a deviation from the reference path was legitimate creativity or drift.
Score trajectories, not just endpoints. Emit per-trajectory records: {task_id, completed, first_divergence_step, tool_choice_accuracy, efficiency_ratio, violations[], cost_tokens, wall_clock}. The first-divergence-step field is the debugging gold: aggregate it to find which state transitions break agents most (it is usually one or two specific tool results the model misreads).
Run the suite with repetition. Agents are high-variance: run each task 3–5× and report completion as mean ± spread, plus a flakiness stat (tasks with mixed outcomes across identical runs). A 70%-completion agent that is deterministic beats an 80% agent that is a coin flip on a third of tasks — flakiness is a first-class result, not noise to average away.
Gate autonomy on trajectory evidence. Wire the suite into pb-37's CI (agents have prompts, tool definitions, and model pins too — all gate-worthy paths). Set promotion thresholds per capability tier: e.g., read-only tools → write tools requires completion ≥ X, zero safety violations across N×reps, flakiness ≤ Y. Demotion is automatic on any production safety violation.
Close the loop from production. Sample production trajectories (with the pb-36 5–10% human-review pattern applied to trajectory judges), feed new failure modes into the taxonomy, and convert incidents into new suite tasks — the pb-34 growth rule, at trajectory scale.
Grade the process, not just the outcome — an agent that reached the right answer through the wrong process is a failure you have not observed yet.
An agent trace is a recording of everything the agent saw and did — including secrets a tool accidentally returned; scrub tool-result payloads in traces before anyone outside the boundary views them.