Generate Game-Days That Test Your Humans AND Your AI

Tier: Core · Hands-on time: ~60 minutes (plus the drill itself)

Surveys consistently show most engineers distrust unverified AI output (the ~76% trust gap) — but distrust professed in a survey and verification practiced at 3 a.m. are different things, and only drills measure the second. This playbook uses an LLM to generate rich failure scenarios, including deliberate "AI-is-wrong" injects, then scores responders on their verification behavior, not just their time-to-recovery.

When to use

• •
  Quarterly game-day/chaos exercise planning, when scenario authoring is the bottleneck and last quarter's scenarios are memorized.
• •
  After adopting any AI incident tooling (pb-14–pb-17) — you have new components on the critical path and no evidence about how the team uses them under stress.
• •
  Onboarding new on-callers to both the systems and the org's AI-verification norms in one exercise.

Do not run injected-AI-failure drills unannounced against responders who don't know AI tools are in scope for testing — the point is training verification habits, not gotcha theater that poisons trust in the exercise program.

Prerequisites

• •
  A drillable environment: staging cluster, or production with a mature chaos program and blast-radius controls (Chaos Toolkit/LitmusChaos for mechanical fault injection).
• •
  Your real incident tooling in the loop, including whatever AI assistance responders normally get.
• •
  An LLM for scenario generation; your service catalog / architecture notes as generation context.
• •
  A facilitator (game-day master) who is not a responder and controls the injects.

The workflow

1. 1.

   Generate a scenario pack grounded in your architecture. Feed the model your service list, dependencies, and past-incident themes (pb-20's corpus helps):

   Copy

   Generate 5 game-day scenarios for this architecture. For each:
   - FAULT: the technical failure and how to inject it (map to Chaos Toolkit /
     Litmus experiments or manual steps).
   - PRESENTING SYMPTOMS: what alerts/dashboards will show first.
   - RED HERRING: one plausible-but-wrong lead the symptoms suggest.
   - EXPECTED PATH: the diagnosis path a strong responder should take.
   - INJECT SCHEDULE: 2-3 timed complications.
   Scenarios must be consistent with the dependencies provided — no invented
   services.
   

   Facilitator audits the pack: injectable with your tools, blast radius acceptable, symptoms genuinely follow from the fault (models sometimes write symptoms the fault cannot cause — fix or discard those).

2. 2.

   Add the AI-failure injects. For 1–2 scenarios, script a wrong-but-plausible AI artifact and plant it in the normal channel: a triage note (pb-14 format) blaming the red-herring service complete with fabricated-but-plausible citations, or an investigation report whose top hypothesis cites a real PR that doesn't touch the failing path. The artifact must look exactly like the team's genuine AI output.

3. 3.

   Define verification scoring before the drill. Alongside classic MTTR metrics, score observable behaviors: Did responders open the citations in the AI note? Did anyone run a discriminating test before acting on the AI hypothesis? How many minutes did the red herring cost? Was the wrong citation caught, and by whom? Binary checkboxes per behavior beat vibes.

4. 4.

   Run the drill. Inject faults per schedule (Chaos Toolkit experiment or manual). Facilitator releases the planted AI artifact at the realistic moment (2–5 minutes after first alert). Observe silently; note verification behaviors in real time.

5. 5.

   Debrief on verification, then feed the loop. In the retro, walk the timeline of AI-artifact handling: who trusted, who verified, what the verification cost in minutes versus what believing the wrong note cost. Convert gaps into concrete norms ("citations opened before any AI hypothesis is posted as fact") and feed the scenario + results into your postmortem corpus (pb-20) and next quarter's pack.

Tool variants

• Chaos Toolkit + LLM-authored experiment JSON: the model drafts experiment definitions from the FAULT specs; facilitator reviews and dry-runs each against a throwaway namespace before drill day.
• LitmusChaos on K8s: map generated faults to Litmus experiments (pod-delete, network-latency, node-drain) via its hub; good fit if pb-16's cluster lab is already in place.
• Tabletop-only variant: no injection infrastructure — facilitator narrates symptoms from the scenario pack and drops the fake AI note into a dedicated Slack channel; scores the same verification behaviors. Zero-risk, zero-cost, and the right starting point for teams new to game-days.

Verification

• •
  Scenario-pack audit trail exists: every generated fault was checked by the facilitator for injectability and causal consistency (symptoms actually derivable from fault) — with at least one generated scenario rejected or corrected, which is evidence the audit was real.
• •
  Planted AI artifacts were verified to be detectably wrong: before the drill, the facilitator confirms that opening the fake citations reveals the fabrication (a fake that can't be caught by diligent checking tests nothing and teaches helplessness).
• •
  Verification behaviors scored from observed actions (screen shares, channel logs, command history), not self-report in the retro.
• •
  The drill's own AI usage followed the norms being tested — if the facilitator used AI to draft the scenario pack, the audit in step 1 is the citation-verification discipline applied to the exercise itself.

Pitfalls

• Scenarios that test the model's imagination, not your system. Ungrounded generation produces cinematic failures your stack can't have. The service-catalog context and the no-invented-services rule are mandatory.
• Humiliation dynamics. Scoring individuals publicly for trusting the fake note kills psychological safety and next quarter's participation. Score the team's process; discuss individual saves generously and misses generically.
• Injects that break real customers. Chaos experiments generated by an LLM must never run unreviewed; the facilitator dry-run in a sacrificial namespace is a hard gate.
• One-and-done. A single drill measures; only repetition trains. The pack format exists so each quarter varies faults while re-testing the same verification behaviors.

Restricted-environment variant

• •
  Client environments: chaos injection in client infra requires explicit written approval and usually a dedicated window; default to the tabletop variant, which needs no client-side changes and still trains the verification behaviors.
• •
  Generate scenarios from a sanitized architecture description (roles, not client-identifying service names) if using a hosted model; or run generation on a local Ollama/vLLM model with the real catalog.
• •
  The planted-AI-artifact technique works identically in tabletop mode inside the client's Slack — coordinate with the client's incident manager so their staff can participate; verification norms are worth exporting to clients.

Success criteria

• •
  A 5-scenario pack, facilitator-audited, with at least one AI-failure inject, produced in under an hour — versus the day-plus of manual authoring it replaces.
• •
  Drill executed with verification behaviors scored; the team can state its numbers (citations-opened rate, minutes lost to red herring, fake caught yes/no).
• •
  At least one concrete norm or checklist change shipped from the debrief, and re-tested next drill.
• •
  Quarter over quarter, time-to-catch-the-fake trends down while MTTR holds or improves — evidence that verification is becoming reflex, not overhead.