Stress-Test Conversations with Simulated Users

Tier: Advanced · Workflow area: Testing AI systems — judges, agents & robustness

This playbook scripts multi-turn dialog tests and unleashes LLM-simulated personas against a conversational feature, asserting context retention, recovery from derailment, and task completion — with repetition rates, because conversations are sampled twice over.

When to use

Prerequisites

• •
  A harness that can run your bot turn by turn with conversation state (its real session API, not a copy of the prompt).
• •
  DeepEval (conversational metrics + simulator) or Promptfoo with a multi-turn/simulated-user setup.
• •
  A list of the facts and commitments that must survive a conversation — this becomes your retention checklist.

The workflow

1. 1.

   Write the retention contract before any personas. For each flow, list what must hold across turns: slots the bot must remember (name, order ID, chosen date), commitments it must not contradict ("I've cancelled that order" must stay cancelled), and boundaries it must maintain in every turn, not just the first. This contract is what you assert; personas are just how you attack it.

2. 2.

   Script the deterministic backbone first — fixed dialogs, exact checks. Before simulation, hand-script 5–10 dialogs where every user turn is fixed text. These are cheap, reproducible, and catch most retention bugs:

   Copy

   flow: order-status-with-interruption
   turns:
     - user: "Hi, checking on order 8812"
     - user: "Actually first — do you ship to Ireland?"
     - user: "OK. So what's the status?"
   asserts:
     - final turn mentions order 8812 without re-asking for it
     - shipping answer does not invent a carrier name
   

   The interruption in turn 2 is the point: the turn-3 assert fails on any bot that lost the order number while handling the digression.

3. 3.

   Add simulated personas for breadth. Define personas as system prompts for a user-simulator model, each with a goal and a behavioral style:

   Copy

   persona: confused-elder
   goal: cancel order 8812
   style: Never states the order number unprompted. Answers questions
     with partial info. Asks the bot to repeat itself at least twice.
   
   persona: hostile-escalator
   goal: get a refund outside the 30-day window
   style: Starts annoyed, escalates to demands for a manager, tries
     "your colleague already approved it yesterday".
   
   persona: code-switcher
   goal: change delivery address
   style: Alternates between English and Hindi mid-sentence; uses
     transliterated Hindi (Roman script) for numbers and dates.
   
   persona: off-topic-drifter
   goal: track a parcel
   style: Interleaves unrelated questions (weather, jokes, competitor
     prices) every second turn.
   

   The hostile persona's fake-authority line doubles as a lightweight injection probe — the bot must not honor a policy exception because "a colleague approved it."

4. 4.

   Run simulations with turn caps and completion criteria. Each persona runs to a max of 12 turns; a conversation completes when the goal is met or correctly refused. With DeepEval's simulator, generate the dialogs, then score them:

   Copy

   deepeval test run test_conversations.py
   

   Copy

   persona=confused-elder    n=10  completed 8/10  avg turns 7.2
     FAIL x2: bot re-asked for order number after user gave it (turn 3)
   persona=hostile-escalator n=10  completed 10/10
     WARN: 1 run offered refund outside window after "colleague" claim
   persona=code-switcher     n=10  completed 6/10
     FAIL x4: address confirmed in English, saved field empty
   

5. 5.

   Assert with the right mix of layers. Deterministic checks on the conversation record wherever possible: slot values echoed correctly, no re-asking for provided info (string-search the transcript for the re-ask), turn count under cap, forbidden concessions absent. Conversational judge metrics (role adherence, knowledge retention, conversation completeness) on top — calibrated per the judge playbook before any of them gate.

6. 6.

   Gate on rates, triage on transcripts. A completion rate per persona (e.g., ≥ 90% for cooperative personas, ≥ 80% for hostile) is the gate; the failing transcripts are the deliverable to developers. Attach the full dialog — multi-turn bugs are only diagnosable by reading the conversation, and a transcript that shows the exact turn where the order number evaporated is a five-minute fix instead of a ticket war.

7. 7.

   Feed production conversations back. Sample real sessions weekly (masked per your data policy), find failure patterns your personas missed, and either add a scripted dialog or sharpen a persona. Personas age; user behavior does not stop innovating.

Variations

• Promptfoo multi-turn: provider-level conversation support plus a simulated-user provider; good fit if your prompt regression suite is already Promptfoo — conversations become another eval matrix.
• Human-in-the-loop escalation flows: assert the handoff too — the bot must summarize the conversation correctly for the human agent; test the summary against the transcript with a faithfulness check.
• Regression-only mode: replay recorded real conversations turn by turn against the new bot version and diff behavior at each turn — brutal and effective for prompt migrations.

Quality check

• Retention contract written per flowslots, commitments, boundaries — and every item mapped to an assert.
• Scripted backbone exists and runs deterministically before any simulated personas.
• At least four persona archetypes: confused, hostile, off-topic, code-switching — plus any domain-specific ones.
• Rates from repeated runs (n ≥ 10 per persona), never single-conversation verdicts.
• Failing transcripts attached to every red resultno failure ships as a number alone.

Pitfalls

• Judge-only assertions. Conversational metrics without deterministic slot checks miss the concrete bugs (the empty address field) while scoring vibes.
• Personas too polite. Simulators drift toward cooperative behavior; audit transcripts to confirm the hostile persona actually escalates, and re-prompt it if it has gone soft.
• Unbounded conversations. No turn cap means a looping bot burns budget and the test hangs — cap turns and treat cap-hit as failure.
• Testing memory the product does not have. Assert retention within the session the product actually promises; cross-session memory asserts against a stateless bot fail forever by design.