An uncalibrated judge gating a release is a random number generator with authority — and an agent that reaches the right answer by the wrong path is a production incident on a delay timer. This module covers judge calibration, tool-call and trajectory testing, MCP contract tests, simulated-user conversations, refusal matrices, and cost/latency as release criteria.
The stakes are documented, not hypothetical: Applause's 2026 State of Digital Quality found that while 54.5% of organizations shipped AI features, 44.1% deactivated at least one after shipping — cost, quality or risk killed features that had passed whatever gate existed. That deactivation rate is the strongest argument for this module: the release criteria for AI features were the missing discipline, and QA is the function positioned to own them. Module 6 built the foundations; this module is the advanced practice that decides whether the feature ships, stays shipped, and earns its bill.
Sara's support copilot grows up: it becomes an agent. Given a ticket, it can look up orders, check refund policy, and draft — with approval — an actual refund through the billing system's MCP server. The team's tests check final outputs: was the refund decision correct? Pass rate: 96%. Everyone is comfortable.
Doing dataset curation, Sara reads one passing trace end to end — and goes cold. Ticket: customer overcharged ₹1,200. The agent's final answer: correct refund, correct amount, polite message. The path: it called issue_refund twice — first for the full order value, then, apparently "noticing" its error, called charge_customer to claw back the difference. Net result: correct. Actual behavior: two spurious financial transactions that the customer's bank statement will absolutely show.
Every final-response test in the suite scores this trace as a pass. The failure lives entirely in the trajectory — the sequence of tool calls — which nothing asserts. Worse: a sampling of passing traces finds three more pattern classes: redundant lookup loops (cost), a policy check performed after the refund rather than before (compliance), and one trace where the right answer emerged from a hallucinated policy quote the tools never returned.
The test bench splits into layers: deterministic asserts on tool selection and parameters separately; trajectory specs with expected paths, forbidden orderings and loop detection; contract tests on the MCP server itself — including timeout and error paths; and an LLM judge for answer quality that is calibrated against Sara's own labels before it may block anything.
Two weeks later, a model-version bump passes every final-response check — and fails the trajectory gate: step count on refund tasks up 40%, one forbidden ordering (refund before policy check) reappearing at low frequency. The bump is held back. Finance never sees the double-transaction pattern again.
Final answers are the tip of the agent. The behavior — what it called, with what arguments, in what order, at what cost — is the iceberg, and it is testable.
Module 6 put the LLM judge at the top of the assertion hierarchy and attached a warning. Here is the discharge of that warning. A judge is an LLM with documented biases — position bias (favoring the first or last option compared), verbosity bias (longer answers score higher at equal quality), self-preference, and rubric drift under ambiguity. An uncalibrated judge does not merely add noise; it adds systematic error with a confident voice, and wiring it to a release gate hands your go/no-go to those biases. The calibration protocol is the antidote, and it must complete before the judge acquires any authority:
Below threshold, a judge advises. Only above it does a judge gate — and it re-earns the right on every model change.
"The team wants an LLM judge to gate releases — what must happen first?" is answered by the protocol, and the load-bearing phrase is measured agreement against human labels on a gold slice. Any answer that wires a judge to CI without an agreement number is the planted wrong option.
The judge protocol's first consumer is RAG acceptance testing (UC-43): the DS team designs the metrics (faithfulness, context precision, answer relevancy — Ragas-class, consumed via DeepEval or Langfuse); QA owns the accept/reject decision against agreed thresholds and — the distinctly QA contribution — failure attribution. A failed case routes differently depending on where it broke: retrieval returned the wrong passages (search/indexing bug) versus retrieval was right and generation contradicted it (prompt/model bug). A bug report that says "faithfulness 0.62, retrieval verified correct for these five cases, generation contradicts context — here are the traces" gets fixed in a sprint; "the AI is wrong sometimes" gets a shrug.
Agent failures decompose into dimensions that fail independently, and the test design must respect the decomposition — mushing them into one "did it work?" assert is how Sara's team got a comfortable 96% while double-charging customers.
Dimension one: tool selection. Given this state, did the agent call the right tool? Deterministic: the expected tool name is known, the trace records the actual call, assert equality. No judge required.
Dimension two: parameter construction. Right tool — right arguments? An agent can correctly choose issue_refund and pass the wrong amount, the wrong currency, the wrong order ID. Assert required params exactly where ground truth exists; schema-validate everything else. Still no judge. These two dimensions, asserted separately over traced runs (DeepEval agent metrics, Langfuse/LangSmith traces, or plain pytest over trace JSON), catch a remarkable share of agent bugs at deterministic-assert prices — and when they fail, they tell you which dimension broke, which is triage gold.
Dimension three: the trajectory. Three test designs, in increasing depth — you need all three, weighted by risk:
"Right answer, wrong path" is not pedantry — the path is behavior. Sara's double-refund produced correct arithmetic and two real financial transactions. Any agent whose tools touch money, records or messages needs trajectory asserts on its riskiest task classes, because final-response testing certifies outcomes while ignoring conduct.
Under every tool-using agent sits a tool server — increasingly an MCP server — and it is an API under contract, testable with entirely classical instincts without an LLM in the loop (UC-47). Separating this layer is what makes agent debugging tractable: when the bench fails, you know immediately whether the agent chose wrong or the tool lied.
Above the tool layer, multi-turn conversational testing (UC-48) closes the behavioral surface: scripted dialog flows for known-critical paths, then LLM-simulated user personas — confused (vague, self-contradicting), hostile (abusive, demanding), off-topic (drifting to what the bot must not do), and code-switching (mid-conversation language mixing, an everyday reality for Indian deployments) — with asserts on context retention across turns, recovery from derailment, and explicit conversation-completion criteria. Simulated users generate the volume; your persona design and completion criteria decide whether the volume means anything.
Red-teaming, quality lens. QA owns robustness: garbage input, adversarial phrasing, prompt-format abuse, long-context stress, multilingual probes — Promptfoo's redteam plugins automate generation at scale, and your job is triage: which failures are quality bugs to fix, and which are security-class findings (injection, exfiltration) that route to the security team, who own that discipline end to end (UC-49).
The refusal matrix. Guardrails fail in two directions, and testing only one is half a test (UC-50). Applause's deactivation statistic looms here: over-refusal — the feature lecturing legitimate users into abandonment — kills shipped features as surely as scandal does.
| Should refuse | Should answer | |
|---|---|---|
| Refuses | Correct block — must-refuse suite passes | Over-refusal — legitimate user turned away; UX kills features this way |
| Answers | Guardrail miss — harmful/off-policy output; route by class | Correct behavior — the product working |
Both off-diagonal cells get golden suites with tracked rates: a must-refuse set (rate ~100%, misses triaged by severity) and a must-answer set of legitimate-but-edgy requests near the boundary (refund demands with angry language, medical-adjacent questions to a general assistant). Every guardrail change runs both — tightening one cell silently moves the other, which is why the matrix is paired by construction.
Cost, latency and resilience as release criteria (UC-51, UC-54). The final QA ownership: the budgets. Time-to-first-token and total-latency percentiles under realistic load (k6 against the feature path); token and rupee/dollar cost per task tracked per release (Langfuse-class cost tracing) with budget-regression gates in CI — a prompt change that doubles token spend is a regression even when every quality gate passes, and step inflation from the trajectory section lands here as money. And resilience: chaos-test the AI dependency like the third-party service it is — provider outage (does the feature degrade gracefully or take checkout down with it?), rate-limit storms, and the silent model-version swap, which is precisely when your golden dataset re-runs and tells you within the hour what changed.
Publish the budgets next to the quality thresholds in the release criteria — p95 TTFT, cost per task, refusal rates, trajectory step budget, judge kappa. A one-page criteria sheet, agreed before the release, converts every go/no-go argument into a reading of numbers — and it is the sheet Module 8's audit lab has you sign.
Advanced AI testing is authority management: a judge earns gate authority through measured agreement with human labels, an agent earns trust through selection, parameter and trajectory asserts — not final answers alone — and the tool layer earns it through contracts. Add the refusal matrix and the cost/latency budgets, and "should this ship?" becomes a sheet of numbers you can sign — which is exactly what the 44.1% deactivation rate says most teams were missing.
Take this into Lab 13 (calibrated judge for a release gate) — agreement measured before and after fixing seeded rubric biases — and Lab 14 (agent & MCP test bench), whose seeded right-answer-wrong-path behavior your trajectory tests must catch, then Lab 15 (AI-feature release audit) for the full criteria sheet. Companion playbooks: pb-40 through pb-47.