Turn a Resolved Incident into a Tested Runbook

Tier: Foundation · Workflow area: Reliability & Incident Ops · ~30 minutes

Runbooks written from memory are fiction; runbooks written from shell history are evidence. The workflow here is deliberately narrow: collect the actual artifacts of a resolved incident, have AI reorganize them into a runbook with decision points and verification steps, then prove the runbook by walking it against a rehearsal environment. The validation run is not optional polish — it is the step that separates documentation from decoration.

When to use

• Within a week of any novel incidentthe second occurrence of a failure should never be diagnosed from scratch. Freshness matters: shell history rotates and memory decays.
• Recurring toil with tribal stepsfailover rehearsals, certificate rotations, replica rebuilds that only two people know how to do safely.
• On-call handover gapswhen the roster grows and the new person's first page would otherwise be their first sight of the procedure.

When NOT to use

• For decisions that must stay humanfailover execution, DR invocation and data-loss trade-offs get decision-support documents with named approvers, not follow-the-steps runbooks that invite autopilot.
• From memory aloneif there is no shell history, no timeline and no chat log, write a fresh procedure and test it; do not let AI dramatize recollections into confident steps.
• When the incident is still ambiguousa runbook encoding a wrong root cause propagates the error. RCA first (pb-18), runbook second.

The recipe

1. 1.

   Harvest the raw material while it exists. Three sources, collected verbatim:

   Copy

   # The responder's shell history with timestamps
   HISTTIMEFORMAT='%F %T ' history | tail -200 > incident-2318-shell.txt
   
   # Postgres log slice for the incident window
   awk '$1" "$2 >= "2026-08-02 01:45" && $1" "$2 <= "2026-08-02 04:30"' \
     /var/log/postgresql/postgresql-*.log > incident-2318-dblog.txt
   

   Export the incident channel's chat log too — the "try restarting the wal sender" messages carry the decision sequence the shell history alone loses.

2. 2.

   Prompt for a runbook shape, with invention forbidden. The critical instruction is that gaps become questions, not filler:

   Copy

   Below: shell history, database log slice and chat transcript from a
   resolved incident (replication lag cascade, Postgres 15 streaming
   replica). Draft a runbook titled "Replication lag: diagnose and
   recover" with EXACTLY this structure:
   
   1. SYMPTOMS — what pages/alerts/user reports look like.
   2. VERIFY — read-only commands that confirm this is the same failure,
      each with the expected healthy vs unhealthy output.
   3. DIAGNOSE — ordered checks narrowing the cause, as a decision tree
      (if X shows Y, go to step N).
   4. RECOVER — the fix steps, each with: the exact command, what output
      indicates success, and what to do if it fails.
   5. VERIFY RECOVERY — how to prove the system is healthy again.
   6. ESCALATE — the point at which this runbook stops and a human
      decision (with role named) takes over.
   
   Rules: use ONLY commands that appear in the provided material or are
   standard read-only diagnostics you can name confidently. Where the
   material does not show a step clearly, insert "[GAP: question for
   responder]" instead of inventing. Mark every state-changing command
   with [DANGER].
   
   [paste the three files]
   

3. 3.

   Resolve every GAP with the responder — in one sitting. The gaps are the runbook's honesty markers. A ten-minute review with the engineer who was there converts each [GAP] into either a real step or an explicit "unknown — investigate live." Zero gaps survive to publication.

4. 4.

   Audit the [DANGER] steps for guardrails. Every state-changing command gets three additions, by hand: a precondition check that must pass first, the blast radius in one sentence, and the undo. Example of the finished form:

   Copy

   **Step 7 [DANGER]: restart the WAL sender**
   - Precondition: step 5 confirmed `sender_pid` stuck (no WAL position
     advance for 5+ min) — do not restart a merely slow sender.
   - Blast radius: brief reconnect for all replicas on this primary.
   - Command: `SELECT pg_terminate_backend(<sender_pid>);`
   - Success: new sender pid within 15 s, `pg_stat_replication` advancing.
   - Undo: none needed — sender restarts automatically; if it does not
     within 60 s, go to ESCALATE.
   

5. 5.

   Run the game day — the runbook is unvalidated fiction until someone follows it cold. Reproduce a facsimile of the failure in a rehearsal environment (a lag-injected replica works: SELECT pg_wal_replay_pause(); on the standby). Then a different engineer — not the author, not the original responder — executes the runbook literally, noting every place they had to improvise. Each improvisation is a defect; fix the text, not the person.

6. 6.

   Publish with a validation stamp and an owner. The wiki entry carries: validated by K.S. against lag-injected replica 2026-08-06; next review 2027-02; owner: db-oncall lead. A runbook without a review date is scheduled to rot.

Make it yours

• SQL Serverthe same harvest works from SSMS query history, the ERRORLOG and Agent job history; AlwaysOn incidents should pull sys.dm_hadr_database_replica_states snapshots into the VERIFY section.
• MySQLbuild replication runbooks around SHOW REPLICA STATUS fields, and have the model annotate which fields changed meaning across 5.7/8.0 — then verify those annotations against the docs for your version.
• Managed cloud (RDS/Cloud SQL)shell steps become console/CLI steps; add the support-case escalation path with severity guidance, since some recoveries are only actionable by the provider.

Quality check

• Every command came from evidence or standard diagnosticsgrep the draft for commands absent from the source material and interrogate each one; hallucinated flags in runbooks detonate at 3 a.m.
• Healthy and unhealthy outputs are both showna check without an expected output makes the reader guess under adrenaline.
• The game day used a fresh executorthe author following their own runbook validates nothing; the cold read is the test.
• Escalation names a role and a trigger"escalate if unsure" is not a trigger. "No WAL advance 10 min after step 7 → page DBA lead" is.
• [DANGER] steps have precondition, blast radius, undoall three, or the step is not done.