Draft Postmortems with a Claim-Verification Gate

Tier: Core · Hands-on time: ~50 minutes

The catch is that a postmortem is an organizational record — timelines get cited in audits, action items drive quarters of work, and a fabricated detail becomes institutional "truth." So the deliverable of this playbook is not the draft; it is the draft plus a completed claim-verification checklist in which a reviewer has traced every factual statement back to an artifact before publication.

When to use

• •
  Any SEV with a required postmortem where the raw material (incident channel, alert timeline, monitor snapshots, PR links) exists but nobody wants to spend the afternoon assembling it.
• •
  Backfilling postmortem debt — incidents closed weeks ago with artifacts still in Slack.
• •
  Standardizing postmortem quality across teams with wildly different writing habits.

Do not use for incidents involving legal exposure, security breaches, or personnel issues without your incident commander's sign-off on where the artifacts may be processed.

Prerequisites

• •
  Artifact exports as text: incident channel transcript (with timestamps), alert/monitor timeline, deploy log, the verified investigation trail from pb-15 if one exists.
• •
  Your org's postmortem template (sections, blameless-language norms, action-item format).
• •
  A reviewer who was present at the incident — the checklist requires someone who can smell a wrong detail.
• •
  Any capable LLM with enough context for the transcript (chunk by hour if needed).

The workflow

1. 1.

   Assemble artifacts with provenance labels. Concatenate under headers: === SLACK TRANSCRIPT ===, === ALERT TIMELINE ===, === DEPLOYS ===, === MONITOR DATA ===. Keep native timestamps; normalize timezone to UTC first (mixed IST/UTC transcripts produce impossible timelines that read fluently).

2. 2.

   Generate the draft against your template, with per-claim sourcing.

   Copy

   Write a blameless postmortem using our template sections. Rules:
   - TIMELINE: every entry carries [source: SLACK 14:02 / ALERT #3 / DEPLOY d41f].
   - Only facts present in the artifacts. Contributing factors must each cite
     the artifact showing them. Label anything inferred as INFERENCE.
   - IMPACT: numbers only from MONITOR DATA or explicit statements; no estimates.
   - ACTION ITEMS: derive only from problems evidenced above; phrase as
     verifiable outcomes, each linked to the timeline entry that motivates it.
   - Blameless language: systems and processes, not names, as causes.
   - End with OPEN QUESTIONS: anything the artifacts do not answer.
   

3. 3.

   Run the claim-verification checklist (the gate). The reviewer works through the draft with the artifacts open:

   • •
     Every timeline entry: source exists, timestamp matches, actor/action correctly attributed.
   • •
     Every impact number: found in monitor data verbatim; duration math re-done by hand (start/end arithmetic is a frequent silent error).
   • •
     Every contributing factor: cited artifact actually demonstrates it — not merely mentions the component.
   • •
     Every quoted human statement: exists in the transcript verbatim; paraphrases marked as paraphrase.
   • •
     Every INFERENCE: either promoted to fact with newly attached evidence, rewritten as an open question, or deleted.
   • •
     Action items: each traces to an evidenced problem; none are generic hygiene the model padded in. Tick each line; the checklist is stored with the postmortem.
4. 4.

   Fix through the model, verify through the artifacts. Feed failed checks back ("Timeline says rollback at 14:31; deploy log shows 14:38 — correct and re-emit the timeline"). Re-verify the corrected entries; never accept a correction unchecked just because you prompted the correction.

5. 5.

   Human owns the narrative. The reviewer writes or heavily edits the "lessons learned" section personally — the judgment-heavy part is where the human adds value the review data says readers notice — then publishes with a footer: Drafted with AI assistance; all claims verified against artifacts by <name>, <date>.

Tool variants

• Plain LLM chat/CLI: paste artifacts, run the prompt; the universal path and the lab default.
• incident.io / Rootly postmortem generation: auto-assemble from their own timeline data, saving step 1 entirely. The gate does not shrink: their draft cites their timeline, and the reviewer still verifies entries against raw sources, because ingestion gaps (missed bot messages, uncaptured huddles) are invisible in the polished output.
• Scripted pipeline (postmortem factory): a script that pulls Slack export + Alertmanager history for an incident ID, runs the drafting prompt via API, and opens a PR with draft + empty checklist in your postmortem repo. PR review = verification gate; merge = publication.

Verification

• •
  The completed checklist exists as an artifact alongside the postmortem, with each claim category ticked by a named reviewer who opened the underlying sources — "looked at the draft" is not verification.
• •
  Spot-audit by a second person: pick any three timeline entries and re-trace them to artifacts; all three must resolve verbatim.
• •
  All INFERENCE and OPEN QUESTIONS items were dispositioned explicitly — count in the PR description how many claims were corrected or deleted (a zero count on a complex incident means the gate was skipped, not that the model was perfect).
• •
  Duration and impact arithmetic re-computed by hand or script, not accepted from the draft.

Pitfalls

• Fluency masking gaps. AI drafts read complete even when the artifacts miss a whole workstream (the DB team's huddle nobody transcribed). The OPEN QUESTIONS section and the present-at-incident reviewer are your only defenses.
• Misattributed actions. Transcripts full of "I'll take that" get attributed to the wrong engineer — corrosive in a blameless culture. Verify actor attribution explicitly.
• Plausible action items. Models pad with generic items ("improve monitoring," "add runbook"). Delete any item not traceable to an evidenced failure; postmortem action lists are a budget, not a wishlist.
• One-shot mega-prompts on huge incidents. A 12-hour incident's transcript exceeds usable context; draft per-phase (detection, mitigation, recovery) then merge, verifying each phase.

Restricted-environment variant

• •
  Incident transcripts are among the most sensitive text an org produces (names, customer impact, security detail). Under MSA constraints, run drafting on a self-hosted model (Ollama/vLLM behind LiteLLM) or the client's approved tenant — never a personal hosted account.
• •
  Redact personal names to role labels (ONCALL-1, IC) before processing if the postmortem will circulate beyond the response team; keep the mapping with the incident record.
• •
  Store draft, artifacts, and checklist in the client's systems; the postmortem is their record, and your verification checklist is your professional deliverable attached to it.

Success criteria

• •
  Postmortem produced in roughly half your team's usual writing time (Google's 51% is the benchmark), with the gate adding no more than 30 minutes.
• •
  Checklist on file, second-person spot-audit passed, correction count recorded.
• •
  The published document survives its review meeting without a single factual challenge — challenges about conclusions are healthy; challenges about facts mean the gate failed.
• •
  After three uses, your template + prompt + checklist are in the team repo and someone other than you has run the whole factory.