An MCP server is an API consumed by a model — and it deserves the discipline you give any API under contract: schema validity, discovery, happy and error paths, timeouts, auth scoping, and drift detection. Test the tool layer deterministically so agent-level failures can be blamed on the model, not the plumbing.
Tier: Advanced · Workflow area: Testing AI systems — judges, agents & robustness
This playbook builds a pytest contract suite for an MCP server using an in-memory FastMCP client, with MCP Inspector for exploration and a schema-snapshot regression that catches version drift before the agents that depend on the server do.
Your team ships or consumes an MCP server that agents depend on. Agent tests are failing and you cannot tell model bugs from server bugs. A server update broke agents downstream and nobody noticed until production. Auth-scoped tools (read vs write) need proof that scoping holds.
Tools are in-process functions with no protocol boundary — plain unit tests suffice. You need to test the model's use of tools — that is the tool-call and trajectory playbooks; this one tests the server those playbooks assume is correct.
npx @modelcontextprotocol/inspector) for interactive exploration.Explore with Inspector and write the contract down. Connect Inspector to the server, list tools, and record for each: name, description, input schema, expected result shape, error behavior, and which auth scope it requires. This table is the contract — most MCP servers have never had one written.
npx @modelcontextprotocol/inspector uv run my_server.py
Test discovery first. Agents choose tools by what discovery returns, so discovery is behavior, not metadata:
import pytest
from fastmcp import Client
from my_server import mcp
@pytest.fixture
async def client():
async with Client(mcp) as c:
yield c
async def test_discovery(client):
tools = {t.name: t for t in await client.list_tools()}
assert set(tools) == {"order_lookup", "order_cancel", "refund_create"}
for t in tools.values():
assert t.description and len(t.description) > 20
assert t.inputSchema.get("type") == "object"
assert "required" in t.inputSchema
The description-length assert looks fussy until you remember the model routes on descriptions: an empty description is a routing bug shipped as documentation.
Happy path per tool, asserting result structure.
async def test_order_lookup_happy(client, seeded_order):
res = await client.call_tool("order_lookup",
{"order_id": seeded_order.id})
assert res.data["status"] == "confirmed"
assert res.data["total"] == "129.00"
Error paths: invalid input must produce protocol-clean errors, not crashes. How a server fails is what the agent sees — a raw stack trace in a result string sends the model off reasoning about Python internals:
async def test_unknown_order_is_clean_error(client):
res = await client.call_tool("order_lookup",
{"order_id": "NOPE-404"},
raise_on_error=False)
assert res.is_error
assert "not found" in res.content[0].text.lower()
assert "Traceback" not in res.content[0].text
async def test_missing_required_param_rejected(client):
res = await client.call_tool("order_lookup", {},
raise_on_error=False)
assert res.is_error
Timeout and slow-backend behavior. Wrap the backend fixture with an injected delay and assert the server times out cleanly within its declared budget instead of hanging the session:
async def test_slow_backend_times_out(client, slow_backend):
slow_backend.delay("orders_db", seconds=30)
res = await client.call_tool("order_lookup",
{"order_id": "X1"},
raise_on_error=False)
assert res.is_error and "timeout" in res.content[0].text.lower()
Auth scoping: prove the read-only token cannot write. Run the suite twice, parameterized by credential, and assert write tools are absent from discovery (or refused on call) under the read scope:
async def test_readonly_scope_hides_writes(readonly_client):
tools = {t.name for t in await readonly_client.list_tools()}
assert "refund_create" not in tools
assert "order_cancel" not in tools
Snapshot the schemas for drift regression. Serialize list_tools output to a committed JSON file; the test fails when any name, description, or input schema changes, forcing a deliberate snapshot update in the same PR as the change:
pytest tests/mcp_contract -q
FAILED test_schema_drift - tool 'order_lookup' inputSchema changed:
required: ['order_id'] -> ['order_id', 'region']
Update snapshot deliberately: pytest --snapshot-update
A new required parameter is a breaking change for every agent prompt-tuned against the old schema — exactly the drift this catches.
Top with a thin LLM-in-the-loop smoke. Two or three end-to-end tasks through a real model against the contract-tested server, run nightly rather than per-commit. It exists to catch integration seams; the deterministic layers below carry the regression load.
Keep the contract suite model-free. Everything through step 7 runs without an LLM key — fast, deterministic, and cheap enough for every commit. The moment a model enters the loop you are testing the model, and that belongs in the agent suites.
Every hour of deterministic contract testing on the tool layer buys you back days of agent debugging — because once the server is provably correct, an agent failure has exactly one remaining suspect.