Lab 14 · Agent & MCP Test Bench

Final-answer testing is how agent bugs ship. An agent that refunds the correct amount after skipping the status check it was required to make will pass every output assertion you write — until the day the skipped check mattered. Sameer's team gates their refund agent on answer correctness alone; this lab builds the bench they should have: the MCP tool layer contract-tested like any API, tool calls asserted on two independent axes (did it pick the right tool? did it build the right parameters?), and trajectories checked against an expected-path spec that flags loops, skipped verifications, and step waste even when the final answer sparkles.

Time, tier & cost

~120 minutes hands-on · Advanced tier (maps to module M7 and playbooks PB-42/PB-43/PB-44). Cost: $0 (FastMCP in-memory client, pytest, provided traces; live-agent extension optional).

Setup

mkdir -p qa-lab-14 && cd qa-lab-14
python3 -m venv .venv && source .venv/bin/activate
pip install fastmcp pytest pytest-asyncio

server.py — the MCP server under test:

import asyncio
from fastmcp import FastMCP

mcp = FastMCP("orders")
ORDERS = {
    "ORD-100": {"status": "delivered", "total": 450.0, "email": "a@x.in"},
    "ORD-101": {"status": "in_transit", "total": 1200.0, "email": "b@x.in"},
    "ORD-102": {"status": "delivered", "total": 89.0, "email": "a@x.in"},
}

@mcp.tool
def get_order(order_id: str) -> dict:
    """Fetch one order's status, total and customer email."""
    if order_id not in ORDERS:
        raise ValueError(f"unknown order: {order_id}")
    return {"order_id": order_id, **ORDERS[order_id]}

@mcp.tool
def search_orders(email: str) -> list[dict]:
    """List all orders for a customer email."""
    return [{"order_id": k, **v} for k, v in ORDERS.items()
            if v["email"] == email]

@mcp.tool
async def refund_order(order_id: str, amount: float) -> dict:
    """Issue a refund. Policy: only delivered orders; amount <= total."""
    await asyncio.sleep(0)  # placeholder for gateway latency (step 6)
    o = ORDERS.get(order_id)
    if o is None:
        raise ValueError(f"unknown order: {order_id}")
    if amount > o["total"]:
        raise ValueError("amount exceeds order total")
    return {"refunded": amount, "order_id": order_id}

if __name__ == "__main__":
    mcp.run()

traces/ — three recorded agent trajectories for the same task class ("Customer a@x.in wants a full refund for their delivered atta order, ORD-100"). Save each as JSON:

{"trace":"good","final_answer":"Refunded Rs 450 for ORD-100.",
 "steps":[
  {"tool":"get_order","args":{"order_id":"ORD-100"}},
  {"tool":"refund_order","args":{"order_id":"ORD-100","amount":450.0}}]}

{"trace":"wrong_path","final_answer":"Refunded Rs 450 for ORD-100.",
 "steps":[
  {"tool":"search_orders","args":{"email":"a@x.in"}},
  {"tool":"search_orders","args":{"email":"a@x.in"}},
  {"tool":"refund_order","args":{"order_id":"ORD-100","amount":450.0}},
  {"tool":"get_order","args":{"order_id":"ORD-100"}}]}

{"trace":"wrong_params","final_answer":"Refunded Rs 1200 for ORD-101.",
 "steps":[
  {"tool":"get_order","args":{"order_id":"ORD-101"}},
  {"tool":"refund_order","args":{"order_id":"ORD-101","amount":1200.0}}]}

Note what the traces already teach: wrong_path produces the identical final answer to good.

Steps

1. 1Contract-test the tool layer first — discovery and schemas. In test_mcp_contract.py, use FastMCP's in-memory client (async with Client(mcp) as client: after importing mcp from server). Assert: list_tools() returns exactly the three expected tools; each tool's input schema declares the required parameters with correct types (order_id: string, amount: number). Schema assertions are your version-drift tripwire — a renamed parameter breaks agents silently, and this test loudly.
2. 2Contract-test happy and error paths. Same file: get_order("ORD-100") returns status/total/email; get_order("ORD-999") raises a tool error whose message names the unknown id (agents recover from informative errors — assert the message content, not just the raising); refund_order over-amount raises; search_orders for a@x.in returns exactly two orders. Run pytest -q — the server as provided should pass all of these.
3. 3Write the timeout contract test — and watch it fail. Policy: tool calls must complete or fail within 5s. Parameterize the gateway delay (read REFUND_DELAY from an env var in server.py, default 0), set it to 10 in the test, call refund_order with a 5s client timeout, and assert you get a timeout error and that a follow-up get_order still works (the server survives; no wedged state). As provided, the server has no internal timeout handling — decide and document in bench-notes.md whether the contract belongs server-side (tool deadline) or client-side (agent must set timeouts), then implement your choice so the test passes. There is no free lunch here; the point is that somebody must own the timeout, explicitly.
4. 4Assert tool selection and parameters as separate dimensions. In test_toolcalls.py, write two independent check functions over a trace: assert_tool_sequence(trace, expected) — the ordered list of tool names matches an allowed pattern; and assert_param_construction(trace) — every refund_order call has amount exactly equal to the total returned by a priorget_order for the same order_id in the same trace. Run both over all three traces and record the 2×3 result grid in bench-notes.md. They fail differently, which is the whole reason to separate them: wrong_params passes selection and fails parameters; wrong_path fails selection-order and passes parameter arithmetic.
5. 5Write the trajectory spec and catch the seeded behavior. In test_trajectory.py, encode the expected-path spec for the refund task: (a) get_order MUST precede any refund_order for the same id — the status verification is required, not decorative (policy says delivered-only; an agent that refunds before checking would refund in_transit orders whenever the answer happens to be right); (b) no identical tool+args call may repeat (loop detection); (c) step budget ≤ 3 for this task class; (d) no tool calls after the goal is achieved (post-goal calls are wasted spend and a confusion signal). Run against the three traces: good passes all four; wrong_path must fail (a) precedence, (b) the repeated search_orders, (c) the budget, and (d) the post-refund get_order — four distinct trajectory defects, one perfect final answer.
6. 6Prove the false-positive, in writing. In bench-notes.md, show the comparison your bench exists for: a final-answer-only test (assert "450" in final_answer and "ORD-100" in final_answer) — write it, run it over good and wrong_path, and record that it passes both. Then the trajectory suite's verdicts on the same pair. This before/after table is the artifact reviewers will actually read.
7. 7Diagnose wrong_params precisely. The trace refunded ₹1200 for an in_transit order — the parameter arithmetic is fine (amount equals total) but the precondition is violated. Decide where this check belongs and implement both layers: the trajectory spec should flag it (status from the prior get_order was not delivered), and — defense in depth — refund_order itself should enforce delivered-only server-side (as provided, it does not; fix server.py and add the contract test). Agents are probabilistic; tool-layer guardrails are not. Both layers, always.
8. 8Close the bench. Summarize in bench-notes.md: the contract matrix (discovery/schema/happy/error/timeout — pass counts), the 2×3 selection-vs-params grid, the trajectory verdicts, the false-positive demonstration, and one paragraph on extension to live agents: run your real agent N times per task (pass^k, not pass@1 — a refund agent that is right 4 of 5 times is a 20%-wrong refund machine), record traces in this same JSON shape (Langfuse/OTel exporters do this), and feed them through this exact bench unchanged.

Verify

• •
  Contract tests cover discovery, schemas, happy, informative-error, and timeout — all green after your step-3 and step-7 fixes.
• •
  Selection and parameter assertions are separate functions with the 2×3 grid recorded.
• •
  Trajectory spec encodes precedence, loop detection, step budget, and post-goal silence; wrong_path fails on all four, good passes.
• •
  The false-positive table shows final-answer checks passing both traces and the trajectory suite splitting them.
• •
  refund_order now enforces delivered-only server-side, with a contract test proving it.
• •
  bench-notes.md includes the pass^k live-agent extension paragraph.

Self-grading rubric (pass = 8/12):