Extend Your Incident Process for AI Failures — Before the First One Finds You

Workflow area: Policy & Governance · Tier: Core · Hands-on time: ~30 minutes

The first AI incident at most companies is handled badly not because responders are unskilled but because the taxonomy has no slot for it. A leaked prompt containing customer data is not an outage, so nobody pages; an agent that force-pushed over a release branch is "a tooling issue," so nobody writes a postmortem; and the engineer involved, sensing blame in the air, stops mentioning AI in incident channels entirely. Thirty minutes of taxonomy work now buys you honest reporting later.

When to use — and when not to

• Use it before AI tools reach production-relevant workthe taxonomy must exist before the first incident, because the first incident sets the cultural precedent for reporting.
• Use it when agents reach rung 2 or higher on the pb-29 ladder — agent actions gone wrong are the class your current process is least prepared for.
• Use it after any AI near-miss handled informallya quiet revert with no writeup is a free lesson your org just declined to learn.
• Skip creating a parallel AI-incident processthis playbook extends your existing severity levels, paging, and postmortem template; a second process nobody rehearses is worse than a gap in the first.
• Skip it if you have no incident process at allbuild the basic one first; AI classes bolt onto a process, not onto a vacuum.

The recipe

1. 1.

   Add three incident classes to the taxonomy. Keep your existing severity ladder; these are causes, not new severities:

   • •
     AI-caused defect or outage — AI-generated code, config, or content caused user-facing impact; includes hallucinated dependencies that shipped and subtly wrong logic that passed review.
   • •
     Prompt or context data leakage — sensitive data (customer PII, credentials, client code under NDA, unreleased financials) entered an unapproved AI tool, or an approved tool outside its permitted scope.
   • •
     Agent action gone wrong — an autonomous or semi-autonomous agent took an action causing damage or risk: wrong branch deleted, wrong environment touched, allowlist exceeded, runaway spend.
2. 2.

   Map each class to severity with pre-agreed floors. Leakage of regulated data or client-confidential material starts at your equivalent of Sev-2 regardless of user impact — the impact is contractual and legal, and it does not show on dashboards. An agent acting outside its allowlist is at least Sev-3 even if nothing broke, because the control failed. Ask Claude to pressure-test your mapping:

   Copy

   Here is our severity ladder: [PASTE]. Here are three AI incident
   classes and our proposed severity floors: [PASTE].
   
   Generate 10 realistic scenarios across the three classes, assign
   each a severity under our mapping, and flag any scenario where
   the mapping produces an absurd result (too high or too low).
   For each flag, propose the adjustment. Do not invent new
   severity levels.
   

3. 3.

   Pre-write the comms templates. Two additions to your incident comms pack, written now while nobody is panicking: an internal notice for data leakage (what entered which tool, containment status, who must be informed — security, legal, DPO, the client owner) and a client/external variant your legal team has pre-reviewed. Mid-incident is the wrong time to draft the sentence "your data was pasted into a chatbot."

4. 4.

   Extend the postmortem with blameless-but-accountable questions. Add these to your template for AI-class incidents:

   • •
     What did the human believe the AI had done, and what had it actually done?
   • •
     What rung was the agent on, and did the pb-29 criteria for that rung actually hold?
   • •
     Which guardrail (pb-30) or gate (pb-28) should have caught this, and why didn't it?
   • •
     Was the AI's output reviewable — did the pb-27 standard apply, and was it followed or waived?
   • •
     What made the unsafe path the easy path that day?

   Blameless means the engineer who pasted the data is a witness, not a defendant; accountable means the review names the process owner who fixes the gap.

5. 5.

   Run one tabletop. Forty-five minutes, on-call crew plus one lead: walk the agent-gone-wrong scenario end to end — detection (would you even know? check the pb-30 trail), kill switch, severity call, comms, postmortem. The drill always finds the same two gaps: nobody knew who could stop the agent, and nobody knew whether legal needed to hear about the leak. Fix both on paper the same day.

Make it yours

• Small team or startup: the taxonomy is three bullet points in your incident doc, the comms template is one paragraph, and the tabletop is 20 minutes over lunch — but do all three; small teams have the least slack to improvise during a real one.
• Large org or GCC: wire the three classes into your incident tooling as cause tags so trends are queryable; route leakage incidents to a pre-named group (security, legal, DPO, account owner for client data); feed postmortem actions into the pb-29 demotion triggers so incidents automatically adjust autonomy.
• Services context with client data: every leakage incident involving client material triggers the client-notification clause in that client's contract — index those clauses now (pb-33), because the notification window is often 24–72 hours and starts at discovery.

Quality check

• Responders can classify in under a minutequiz an on-call engineer with two scenarios; hesitation means the classes are written for auditors, not responders.
• AI-class incidents actually appear in the logzero reports two quarters into agent adoption means underreporting, not excellence; say so out loud and re-earn the blameless credential.
• Postmortems change controls, not behavior-lectureseach AI-class review should end with a gate, guardrail, or rung change; "be more careful with AI" as an action item is the process punting.
• The comms templates have legal's fingerprints on thempre-review beats mid-incident review by exactly the length of your worst notification deadline.
• The tabletop repeats yearlyagents, tools, and rungs change; last year's drill validated last year's process.