The postmortem is the highest-leverage document in operations and the most reliably late: written a week after the incident by a tired responder, from a chat scroll nobody wants to reread.
Tier: Core · Workflow area: Incidents & Operations
AI collapses that to an hour — timeline extraction, blameless drafting, action-item derivation — while you supply the two things it cannot: causal judgment and organizational honesty. The second half of this playbook closes the loop: incidents become runbooks, and runbooks get periodically diffed against infra reality so they stop being historical fiction with command-line formatting.
Skip full postmortems for trivial blips your process does not require them for; a three-line incident note is honest, a skipped-but-required postmortem is how repeat incidents stay repeats.
runbooks/ in a repo, not a wiki graveyard — so drift-checking has something to diff.Extract the timeline from the channel export.
Below is the incident channel export plus alert/deploy timestamps.
Build a timeline: every event with time, what happened, and source
(alert/human action/system event/deploy). Separate OBSERVATIONS ("p99
spiked") from ACTIONS ("raised pool limit") from HYPOTHESES discussed.
Flag gaps longer than 10 minutes with no recorded activity — I will fill
those from memory. Note contradictions between messages verbatim rather
than resolving them.
Fill the flagged gaps now, while you still remember what happened in them. This is why the postmortem happens today, not Friday.
Draft blameless, with the discipline enforced in the prompt. Draft the postmortem from the verified timeline using our template. Rules: name systems and roles, never individuals ("the deploy" not "Rahul's deploy"). Distinguish the trigger, the root cause, and contributing factors — a config change triggering an outage usually reveals a missing guardrail as the deeper cause; find it. Write "what went poorly" about process and tooling gaps. Include a "what we got lucky on" section — near-miss factors that saved us this time. Mark any causal claim the timeline does not support with [UNSUPPORTED]. Resolve every [UNSUPPORTED] yourself: the model will happily assert a clean causal chain the evidence does not carry, and a postmortem with a wrong root cause is a map to the wrong fix.
Derive action items that survive contact with a backlog. Ask for each proposed action as: specific change, which contributing factor it addresses, detection-vs-prevention-vs-mitigation classification, and a falsifiable done-condition ("alert fires in staging chaos test", not "improve monitoring"). Then you cut the list to the 3–5 the team will actually do — twelve open action items is how postmortems lose credibility. Assign owners in the review meeting, not in the doc.
Mint the runbook from the incident. The resolved incident is a verified diagnostic path — capture it while it is true:
From this postmortem, draft runbooks/payment-webhook-saturation.md:
symptoms as the responder sees them (alert names, dashboard signatures),
a check-ordered diagnosis flow (fastest discriminator first, exact
queries/commands from the timeline), mitigation steps with expected
observable effect after each, escalation contacts by role, and "verified
against incident INC-2417, 2026-07-24" at the top.
Dry-run the commands against staging or read-check them against current infra before merging — the timeline's commands were true at 3 a.m. that night; the runbook claims they are true in general.
Schedule the rot-repair diff. Quarterly (or on infra-change triggers), run an agent over runbooks/ against reality: For each runbook: verify every command, dashboard link, service name, and escalation route against the current repo/infra definitions. Report per runbook: VALID / DRIFTED (with the specific stale lines and suggested fix) / OBSOLETE (system decommissioned). Open one issue per drifted runbook. Deterministic checks do the heavy lifting where possible (do the linked dashboards 200? do the service names exist in the compose/Terraform files?); the model reads what tools cannot — whether the procedure still matches the architecture.
Close the loop in review. The postmortem meeting reviews the AI-drafted doc as a draft, not a verdict — the humans in the room hold context no export contains (the hallway decision, the thing nobody typed). Their corrections go into the final; the final's lessons go into the runbook; the runbook enters the diff rotation. That circuit is the whole game.
AI makes the postmortem cheap enough to write while context is hot, and the runbook diff makes it durable enough to matter next quarter. Blamelessness and causal honesty remain human work — a model cannot know what your org needs to hear, only what the export says.
Incident channel exports are among the most sensitive texts in the company: customer impact numbers, internal candor, personal names, sometimes production data pasted mid-firefight. Scrub before any hosted tool: strip @-mentions to roles (@priya → [on-call SRE]), redact customer names and figures, drop pasted data blobs. Client incidents stay inside the client's sanctioned tenant, full stop. Local path: extraction-and-drafting is reorganization work that Ollama-hosted models do respectably — run steps 1–4 fully offline; the runbook diff (step 5) is mostly deterministic local checks anyway. The blameless-review meeting needs no model and remains the step you never skip.