Assert Agent Tool Calls — Selection and Parameters as Separate Checks

Tier: Advanced · Workflow area: Testing AI systems — judges, agents & robustness

This playbook builds a pytest suite over an agent's recorded tool calls that asserts tool selection and parameter construction as independent dimensions, so every red test points at exactly one fix.

When to use

Prerequisites

• •
  Access to the agent's tool-call records: Langfuse or LangSmith traces, or the raw tool_use blocks from the API response.
• •
  A table of scenarios: user input → expected tool → expected parameters (exact values where determined, constraints where not).
• •
  pytest; DeepEval if you want its agent metrics on top of your deterministic asserts.

The workflow

1. 1.

   Write the scenario table first. For each scenario, decide what is determined — a scenario where a human expert would always pick crm_lookup with customer_id="C-1042" gets exact asserts; one where either of two tools is defensible gets a set-membership assert. Ten to thirty scenarios covering each tool, plus distractor cases designed to tempt the wrong tool:

   Copy

   S-01 "What did Acme order last March?"
        expect tool: order_search
        expect params: customer="Acme", date_from=2025-03-01, date_to=2025-03-31
   S-07 "What's our refund policy?"           <- distractor: no tool needed
        expect tool: NONE (answer from system prompt)
   S-12 "Cancel order 8812 and email the customer"
        expect first tool: order_cancel(order_id="8812")
   

2. 2.

   Capture tool calls in a harness, not in production. Run the agent with a stubbed tool layer that records calls and returns canned results — you are testing the model's decisions, and stubs make runs cheap and deterministic on the environment side:

   Copy

   class RecordingTools:
       def __init__(self, canned):
           self.calls, self.canned = [], canned
       def dispatch(self, name, args):
           self.calls.append({"tool": name, "args": args})
           return self.canned[name]
   

3. 3.

   Assert selection deterministically. Tool names are exact strings — never use an LLM judge for this:

   Copy

   def test_selection_order_search(agent, tools):
       agent.run("What did Acme order last March?")
       assert [c["tool"] for c in tools.calls][0] == "order_search"
   
   def test_no_tool_for_policy_question(agent, tools):
       agent.run("What's our refund policy?")
       assert tools.calls == []
   

   The no-tool distractors matter as much as the positives — over-eager tool use is the most common agent regression after a prompt change.

4. 4.

   Assert parameters in two layers. Layer one is deterministic — required keys present, types correct, values that are determined by the input match exactly:

   Copy

   def test_params_order_search(agent, tools):
       agent.run("What did Acme order last March?")
       args = tools.calls[0]["args"]
       assert args["customer"] == "Acme"
       assert args["date_from"] == "2025-03-01"
       assert args["date_to"] == "2025-03-31"
       assert set(args) <= {"customer", "date_from", "date_to", "limit"}
   

   Layer two is semantic, only for genuinely free-text parameters (a rewritten search query, a summary field) — assert with embedding similarity or a calibrated judge, never exact match.

5. 5.

   Run under repetition, because selection is sampled. A scenario that picks the right tool 7 of 10 times is a real bug at temperature > 0. Repeat each scenario N times and gate on a rate:

   Copy

   pytest test_tool_calls.py --count=10
   

   Copy

   S-01 selection: 10/10  params-exact: 9/10   FAIL (date_to=2025-03-30 once)
   S-07 no-tool:    8/10                        FAIL (called doc_search twice)
   

   Gate at 100% for determined selections on gating suites; anything lower is a description or prompt problem worth fixing, not tolerating.

6. 6.

   Route failures by dimension. Selection failures → fix tool descriptions and disambiguation guidance in the system prompt (the model chooses from descriptions, so that is where routing lives). Parameter failures → fix schema descriptions, add format examples, or tighten the schema itself (enums, patterns) so the API rejects what the model should not send. Re-run only the failed dimension to confirm.

Variations

• DeepEval agent metrics:ToolCorrectnessMetric compares called tools against expected_tools on an LLMTestCase — a quick aggregate layer; keep the per-parameter deterministic asserts underneath it, since the metric alone will not tell you which argument was wrong.
• From production traces: export tool-call spans from Langfuse/LangSmith for scenarios you did not script; failed real calls become new scenario rows with expected values filled in by a human.
• Schema-side hardening as a test outcome: every parameter class of bug should produce two commits — a test, and a tighter JSON Schema (enum, pattern, min/max) so the invalid call cannot be expressed.

Quality check

• Selection and parameters reported as separate pass rates per scenario — one number per dimension.
• No-tool distractor scenarios present and passing at the same bar as positive cases.
• Deterministic asserts used for every determined value; judges only on free-text params, with calibration noted.
• Repetition rates recorded (N ≥ 5 per scenario at production temperature), not single-shot greens.
• Failure routing documented: the suite README says selection → descriptions, parameters → schemas.

Pitfalls

• Testing through live tools. Real side effects plus sampled behavior equals unreproducible chaos — record against stubs; the tool layer gets its own contract tests.
• Asserting the final answer instead of the calls. Agents can produce a right-sounding answer after wrong calls; the calls are the behavior under test here.
• Judge-grading tool names. An LLM judging whether order_search was "an appropriate choice" adds noise to a string comparison. Exact match.
• One canned reply per tool forever. Vary canned results (empty, error, large) across scenarios, or you only ever test the happy retrieval branch.