Every ops team runs two knowledge bases: the wiki nobody updates and the senior engineer's head. This module installs the pipeline that closes the gap — runbooks mined from real incident exhaust, validated by execution, kept honest by scheduled drift checks, and served back with citations.
ReconcilerLagHigh pages Priya. She knows the shape of this one — the payment reconciler has wedged before — but she doesn't know the fix, because the fix lives in three places and none of them is the wiki.
Last edited fourteen months ago; still names a namespace that got renamed in the spring.
The real procedure — and Rahul is asleep. There's a Slack thread from 2024 where he talked someone through it, but it trails off mid-sentence: "then just bounce the— oh, it recovered."
On the bastion sits the actual command, including the --drain-first flag the wiki version omits and without which the restart drops in-flight settlements.
Priya finds none of this. She pages Rahul.
A groggy answer.
Mitigated. Thirty-eight minutes, of which maybe ninety seconds was the fix and the rest was archaeology.
Every "who knows how the payment reconciler restarts?" at 2 a.m. is the same failure: the real knowledge base failing to be where the on-call engineer is.
Every ops team runs on two knowledge bases: the official one (the wiki nobody updates) and the real one (a senior engineer's head, an old Slack thread, and a shell history on a jump host). The gap between them is measured in incident minutes. This module is about using AI to close that gap industrially: mining tribal knowledge into runbooks that are validated by execution, protecting docs and diagrams against rot with drift checks, and building the platform Q&A assistant that answers with citations and — just as importantly — refuses to answer what it doesn't know.
There's a strategic frame that elevates this beyond documentation hygiene. DORA's AI Capabilities Model names AI-accessible internal data as one of the seven capabilities that determine whether AI adoption translates into organizational performance. The reasoning is direct: 90% of developers now use AI (DORA 2025), but a model that can't see your runbooks, your incident history, and your platform docs can only ever give you generic answers. The teams whose internal knowledge is structured, current, and retrievable get compounding returns from every AI tool they adopt; the teams whose knowledge lives in Rahul's head get autocomplete. Platform teams own this capability, and this module is its build guide.
The verification theme lands differently here than in incident response. The risk isn't an agent restarting the wrong service — it's quieter and slower: a runbook with a stale command that fails during an incident, a confident RAG answer assembled from a deprecated doc, an architecture diagram describing last year's system. Knowledge artifacts fail later, on the reader, which is why every workflow in this module ships with a freshness mechanism, not just a generation step.
DORA's "AI-accessible internal data" capability is the platform team's job: knowledge that is written down, structured, current, and retrievable multiplies every AI tool your org adopts. Generation is the easy half — the engineering is in validation-by-execution and rot detection.
Your first 15 minutes: pick the one runbook your team pastes into Slack most often and ask an AI to turn it into three retrieval-tested Q&A pairs — that's this module's whole method at the smallest possible scale.
The raw material for every runbook you're missing already exists — it's just in the wrong form. An incident channel is a timestamped transcript of what people actually checked, ran, and concluded. A postmortem's timeline is a diagnostic sequence. A senior engineer's shell history on the bastion is a literal record of the real procedure, including the steps the wiki version doesn't mention. LLMs are the first tool that makes converting this exhaust into structured procedure cheap; job postings in our phase-1 research say the quiet part out loud — employers expect engineers to "use LLMs heavily" for exactly this runbook work.
The pipeline:
1. Gather the exhaust. For a chosen failure mode (say, "payment reconciler stuck"), collect: the Slack export of the last two or three incidents where it happened, the related postmortems, relevant alert definitions, and — with permission, filtered — the shell history of whoever fixed it. Filter secrets and customer data before anything reaches a model; in client-account contexts — the services-company delivery work this track keeps returning to — this preprocessing plus a client-approved or self-hosted model is non-negotiable, not optional.
2. Extract the procedure, not a summary. The prompt discipline matters: you don't want "a summary of the incident" — you want the generalized procedure separated from the incident-specific details. Ask for: symptoms as the on-call would first see them, diagnostic steps in dependency order with exact commands, decision points ("if the queue depth is growing → X; if flat → Y"), remediation actions with their blast radius noted, verification steps proving the fix took, and escalation criteria. Ask the model to flag every step where the transcripts disagree with each other or where a step's purpose is unclear — those flags are your interview questions for the humans who were there.
3. Draft into a fixed format. Runbooks are read under stress; format is a safety feature. A structure that has earned its keep:
# Runbook: Payment reconciler stuck
**Service:** payment-reconciler · **Owner:** payments-platform
**Verified:** 2026-07-12 in staging by executing all steps (see freshness check)
## Symptoms
- Alert `ReconcilerLagHigh` firing; `reconciler_queue_depth` rising
- Customer reports of delayed settlement (usually 30+ min after onset)
## Preconditions & access
- kubectl access to `payments` namespace; read access to reconciler dashboard
## Diagnosis
1. Check queue trend: (exact query / dashboard link)
2. Check last deploy: `kubectl rollout history deploy/reconciler -n payments`
3. DECISION: queue growing AND deploy < 2h ago → go to Remediation A
queue growing AND no recent deploy → go to Remediation B
queue flat → this runbook does not apply; see ReconcilerLag runbook
## Remediation A — rollback (blast radius: reconciler only, safe)
`kubectl rollout undo deploy/reconciler -n payments`
## Verification
- `reconciler_queue_depth` falling within 5 min; alert resolves within 15
## Escalation
- Queue still growing after 15 min → page payments-platform secondary
4. Validate by execution — the gate. Here is the module's central discipline: a runbook is validated by running it, not by reading it. The owner (or the lab, in training) executes every step in staging — or, for genuinely unsafe steps, in a game-day exercise — and fixes what the transcript-mining got wrong: the flag that changed, the dashboard that moved, the step that assumed access the on-call won't have. A generated runbook that merely reads plausible is precisely as dangerous as a generated Terraform module that was never planned — same failure class, same gate philosophy as M2. Execution validation is what earns the "Verified" line in the header, and the freshness section below is what keeps that line honest.
A runbook drafted from transcripts but never executed is a plausible-looking trap. Transcript mining reproduces what people typed during chaos — including the dead ends, the typo'd commands that "worked" for unrelated reasons, and the steps that only worked because of state that no longer exists. Execution in staging is the acceptance test; there is no substitute.
5. Mint from every incident. Steady state, this pipeline runs as part of incident closeout: the postmortem workflow from M3 already assembled the artifacts, so "extract or update the runbook" is a ten-minute marginal task while context is hot — the cheapest knowledge capture you will ever do. A useful team rule: no postmortem is closed until its runbook delta (new runbook, update, or explicit "no runbook change needed") is merged.
One more design consideration with a horizon beyond this module: write runbooks as if an agent might one day execute them under approval gates (the bounded-remediation pattern from M3, and M8's agent-operations material). Exact commands rather than prose gestures, explicit decision points, machine-checkable verification steps, and blast-radius annotations per action — everything that makes a runbook safe for a stressed human at 3 a.m. is exactly what makes it consumable by a supervised agent later. Structure is an investment that pays twice.
Operational knowledge has a half-life. Commands deprecate, dashboards move, services rename, and the runbook that was perfect in March lies in November. Stale runbooks are a documented incident-response failure mode — the on-call follows the procedure, the procedure is wrong, and now there are two incidents. Since rot is inevitable, the engineering move is to detect it mechanically rather than hope reviewers notice.
The freshness auditor (UC-51) is a scheduled job — weekly is a fine cadence — in which an AI assistant with repo access walks each runbook and checks its claims against current reality:
--dry-run and --help-parse for CLIs, kubectl ... --dry-run=client server-side where safe, terraform validate for any embedded HCL. Parsing isn't proof the command is right, but it catches the deprecated-flag and renamed-resource class mechanically.The output is a PR, not a report: mechanical fixes proposed as diffs for human review, non-mechanical findings filed as issues against the runbook owner. This is the same drift-gate philosophy the software-engineering track applies to READMEs, pointed at ops knowledge — and the "Verified against staging on DATE" header line it maintains tells the 3 a.m. reader more than any amount of prose polish. An on-call engineer who can see the freshness state of a runbook can calibrate trust in it; one who can't is gambling.
Prioritize the freshness auditor by pager reality: run it strictest against the runbooks linked from your highest-volume alerts. A stale runbook nobody opens is debt; a stale runbook wired to a weekly page is a scheduled incident.
Architecture documentation has the same rot problem with a better solution: for infrastructure, the ground truth is machine-readable. Your Terraform state, K8s manifests, and cloud APIs describe the actual system — so generate the documentation from them instead of from memory (UC-52). Docs remain among the most common AI uses (~61% of developers in our phase-1 research); the ops twist is that infra docs can be regenerated from source at will, which changes what "maintained" means.
The pipeline: point an assistant (or a script it writes) at the IaC repo and state — plus tools like Inframap for graph extraction from Terraform — and generate Mermaid or D2 diagram source plus a narrative: what runs where, what talks to what, where the trust and network boundaries sit, what's stateful. The diagram source is text, lives in the repo, and participates in code review like everything else in this track.
flowchart LR
subgraph vpc-prod
alb[ALB] --> svc[checkout svc]
svc --> db[(RDS postgres)]
svc --> q[[SQS: orders]]
end
q --> worker[reconciler worker]
worker --> db
Two disciplines make this trustworthy rather than merely pretty:
terraform plan sees, and it's often the artifact that makes drift visible to people who don't read plans: architecture review boards, auditors, new joiners.The same from-the-exhaust pattern covers ADRs for infra decisions (UC-53): the debate about multi-AZ vs multi-region, or Karpenter vs cluster-autoscaler, already happened in a thread. Paste it with an ADR template and have the model draft context, options as their advocates argued them, decision, and consequences — including cost figures pulled from Infracost rather than invented. The human edits for honesty and owns the decision; the repo gains a memory. Six months later, these ADRs become retrieval fodder for the assistant in the next section — future-you asking "why are we multi-AZ but not multi-region?" gets a cited answer instead of archaeology.
Now assemble the payoff (UC-44). You have runbooks that are validated and fresh, docs generated from truth, ADRs capturing decisions — a corpus worth retrieving over. The platform Q&A assistant is a RAG service over that corpus, surfaced where engineers already ask (Slack, Backstage), answering "how do I get a staging Postgres?" and "why is my deploy stuck in image pull?" with cited, current answers. This is DORA's AI-accessible-internal-data capability made concrete, and it's the highest-leverage deflection tool a platform team serving hundreds of developers can ship. You built RAG foundations in the Common Core; what follows is the operational hardening that separates a demo from a service teammates trust.
Four disciplines do that hardening:
kubectl procedure delivered with citations to adjacent docs is more dangerous than no bot at all, because the citation styling launders it. Engineer the refusal path explicitly: when retrieval confidence is low or the retrieved chunks don't actually contain the answer, the bot says "I don't have documentation covering this — here's the platform channel / here's how to file a ticket," and logs the question. That log is gold twice over: it's your unanswered-questions feed (each recurring one is a missing doc, i.e., a work item for the mining pipeline above), and it's the honest measure of corpus coverage.The same machinery extends to platform-ticket triage (UC-48): classify and route incoming requests, and auto-answer only above a confidence threshold, with citations, always with a visible path to a human. Below threshold, the bot's job is a good hand-off — classification plus the three most relevant docs attached to the ticket — which is itself a meaningful time save for the platform engineer who picks it up.
A platform assistant that says "I don't know — ask in #platform" preserves trust; one that guesses destroys it. Design, test, and measure the refusal path with the same rigor as the answer path, and treat every logged refusal as a documentation work item.
DORA 2025's core finding is that AI amplifies organizational strengths and weaknesses rather than fixing them — a RAG assistant over a stale, ownerless corpus amplifies staleness at scale. Corpus quality work (mining, validation, freshness) is not preparation for the assistant; it is most of the assistant.
For a platform lead like Divya turning this module into a quarter's roadmap, sequence matters because each stage feeds the next:
| Stage | Build | Gate that makes it real |
|---|---|---|
| 1 | Runbook format + mining pipeline; top 10 pager runbooks first | Each runbook executed in staging; "Verified" header present |
| 2 | Freshness auditor on those runbooks | Auditor's first PR merged; stale-date policy agreed |
| 3 | Diagrams/docs from IaC; ADR habit for infra decisions | Scheduled regeneration diffing in CI; first drift catch |
| 4 | Q&A assistant over stages 1–3 corpus | Eval set (incl. unanswerables) passing; refusal path logging |
| 5 | Deflection measurement + corpus feedback loop | Unanswered-question log reviewed in the platform team's weekly |
Teams reverse this — ship the chatbot first, over the wiki as-is — and get a fluent liar with citations. The corpus work is the moat; the bot is two weeks of glue.
Your team just generated a clean-looking runbook from three incident transcripts. What single step earns it the "Verified" header — and why is reading it, however carefully, not that step?
This module is backed by one hands-on lab: Platform docs RAG bot — build a RAG assistant over a provided IDP documentation set with citation links and refusal-on-unknown behavior, then run it against an eval set of 20 questions including 5 unanswerables, scored on answer accuracy and refusal correctness. The artifact is the working bot plus its eval scores.
Three playbooks carry this module's workflows into copy-adaptable form:
Each ships a client-safe variant (self-hosted or client-tenancy models, secret and customer-data redaction before mining, and per-client corpus isolation) for services-company delivery work.