Mine Tribal Knowledge into Runbooks — and Keep Them Fresh

Tier: Core · Workflow area: Runbooks, docs & knowledge

This playbook extracts it into durable runbooks with an LLM, validates each one by executing it in staging, and then sets up the second half almost everyone skips: a recurring AI-assisted freshness audit that diffs runbook steps against current IaC and tooling, because a stale runbook mid-incident is worse than none.

When to use

• •
  A postmortem action item says "write a runbook for X" and the knowledge exists only in an incident channel.
• •
  Follow-the-sun handoffs keep re-deriving the same recovery steps across timezones.
• •
  Your runbook repo exists but nobody trusts it — commands reference decommissioned hosts and old CLI flags.

Skip AI when a vendor-maintained procedure already exists — link it; don't fork it into a document you now must keep fresh.

Prerequisites

• •
  Raw material: exported Slack incident threads, postmortems, history output from responders, existing wiki fragments.
• •
  A repo-aware assistant (Claude Code or similar) with access to your runbook repo and IaC repo.
• •
  A staging environment where destructive-looking steps can actually be executed.
• •
  A runbook template your team has agreed on (preconditions, steps, verification, rollback, escalation).

The workflow

1. 1.

   Assemble one incident's artifacts per runbook. One directory: thread.txt (Slack export, secrets scrubbed), postmortem.md, commands.txt. Mixing incidents produces chimera runbooks that fit no real failure.

2. 2.

   Draft against your template.

   Copy

   Draft a runbook from the attached incident artifacts using runbook-template.md.
   
   Rules:
   - Every command must appear in, or be directly derivable from, the artifacts.
     Mark anything you inferred with [INFERRED - VERIFY].
   - Include: trigger/symptoms, preconditions, diagnosis steps, remediation
     steps, verification after each remediation step, rollback, escalation.
   - Parameterize environment-specifics as variables ($CLUSTER, $NAMESPACE).
   - List open questions the artifacts do not answer.
   

3. 3.

   Execute-validate in staging. The named runbook owner runs every step in staging (or a game-day sim). Steps that can't run in staging get a human review sign-off recorded inline. Every [INFERRED - VERIFY] marker must be resolved — verified, corrected, or deleted. A runbook nobody has executed is a hypothesis.

4. 4.

   Merge via PR with the owner named. Runbooks live in git next to the services they describe. The PR description links the source incident; the owner and a review date go in front-matter.

5. 5.

   Set up the freshness audit. On a schedule (quarterly, or on major infra PRs), run a repo-aware assistant across runbooks + IaC:

   Copy

   Audit runbooks/ against the current state of infra/ (Terraform), helm/, and
   .github/workflows/.
   
   For each runbook, flag:
   - Commands referencing resources that no longer exist in IaC (instance names,
     ARNs, namespaces, DNS names).
   - CLI invocations inconsistent with tool versions pinned in this repo.
   - Steps contradicting current architecture (e.g. runbook says SSH, IaC shows
     SSM-only; runbook scales an ASG that is now managed by Karpenter).
   Output a table: runbook, line, finding, confidence, suggested fix.
   

6. 6.

   Fix via PRs, never auto-apply. Turn high-confidence findings into small PRs (the assistant drafts them); the runbook owner reviews. Low-confidence findings become issues. Track a staleness metric: findings per audit should trend down.

Tool variants

• Claude Code / repo-aware assistants: best for the freshness audit — they can grep IaC and runbooks in one context and draft the fix PRs.
• incident.io / Rootly / PagerDuty: their AI features draft postmortems and timelines that make excellent runbook source material; export and feed into step 2.
• Plain LLM API: script step 2 for batch-processing a backlog of old postmortems into draft runbooks; still one incident per run.

Verification

• •
  Every published runbook has an execution record (who, when, environment) — check the PR.
• •
  Zero [INFERRED - VERIFY] markers survive into main.
• •
  Freshness audit runs on schedule and its findings are triaged within a sprint.
• •
  Spot-check one audited "clean" runbook manually per cycle to estimate the audit's false-negative rate.

Pitfalls

• Plausible-command hallucination. LLMs produce beautifully formatted commands with wrong flags. Only the execution step catches these reliably — never skip it.
• Runbooks for the wrong failure. Drafting from a single incident encodes that incident's quirks as gospel. The owner should generalize during validation.
• Secrets in Slack exports. Threads contain tokens pasted mid-incident. Scrub before the text touches any AI tool — grep for key, token, Bearer, AKIA.
• Audit findings without owners. An audit report nobody triages is decoration. Route findings to the named runbook owner automatically.

Restricted-environment variant

For client environments under MSA restrictions (typical services-company delivery):

• •
  Client Slack exports and client infra identifiers must not enter public AI tools. Use the client's approved tenant (their Bedrock/Azure OpenAI in their VPC) or a self-hosted model via Ollama/vLLM behind your gateway.
• •
  Redact hostnames, account IDs, and IPs to placeholders before drafting; keep the mapping in the client's own vault.
• •
  Run the freshness audit inside the client's environment — the IaC itself is client confidential.

Success criteria

• •
  A runbook that existed only in heads is now in git, executed once, with an owner.
• •
  On-call engineers outside the original responders successfully use it (measure via handoff feedback).
• •
  The freshness audit has caught at least one real drift item before an incident did.
• •
  Time-to-runbook after a novel incident drops from weeks to days.