This module installs the reflex that separates a useful incident AI from a dangerous one: checking that every cited claim exists, says what the summary says it says, and supports the causal story — before you act on it, and out loud in the channel.
payment-svc is throwing 502s. Priya is on call again — but this time the investigation agent beats her to the incident channel.
Tidy: root cause — PR #4312 shrank the connection pool, deployed 23:41; recommend rollback. Three citations hang off it, each a real link.
The night she rolled back a healthy deploy while a Redis eviction storm ran on, she'd have hit rollback on sight.
She runs the three-question check she'd drilled on a game-day.
Does the citation exist? Click — PR #4312 is real.
Does it say what the agent claims? #4312 is real, merged, deployed on time — and it touches checkout-api, not the connection pool in payment-svc at all. The link was genuine; the claim it decorated was wrong.
"PR is real but it's the wrong service — not rolling back yet," she types into the channel, and keeps reading.
The real culprit surfaces: a slow connection leak from a different deploy.
The agent had assembled a fluent story around a true citation, and the only thing between that story and a needless rollback was one engineer clicking through.
That gap — between a citation being real and a citation supporting the claim — is the whole discipline this module installs. Incident response is the hardest test the trust map faces. It is the domain where the 76% resistance is most visceral — nobody wants a language model guessing during a Sev-1 — and simultaneously the domain with the strongest measured evidence for AI, provided it is placed correctly: Google's SRE organization found AI-drafted postmortems scored 10% better than human-written ones in blind review with a projected 51% reduction in writing time, and the incident-tooling market (incident.io Investigations, PagerDuty's SRE agent, Datadog Bits AI SRE) has converged on AI investigation-with-citations as its flagship capability.
Both facts are true because incidents are made of two different kinds of work. There is judgment under uncertainty — deciding severity, choosing mitigations, accepting risk — which stays human. And there is assembly under time pressure — correlating logs with deploys, summarizing timelines, drafting comms for three audiences, retrieving the postmortem from the similar incident eight months ago — which is toil AI demonstrably compresses.
This module teaches the discipline that keeps the two separated at 3 a.m.: citation-checking as a reflex for every AI investigation claim, structured-output triage prompts you control instead of black-box magic, open-source cluster diagnosis with kubectl as the arbiter of truth, claim-by-claim verification for comms and postmortems, incident-history RAG that surfaces institutional memory when it is actually needed, and bounded remediation where an agent's hands are exactly as free as an allowlist and an approval gate allow.
AI's proven incident wins are all assembly — correlation, summarization, drafting, retrieval — under human verification; the moment an AI output steers a decision without its citations being checked, you have silently climbed the trust ladder without deciding to.
The current generation of incident AI shares one architecture: when an alert fires, agents fan out in parallel across your operational surfaces — recent PRs and deploys, logs, metrics, Slack, past incidents — and return ranked hypotheses with citations into the incident channel within minutes. incident.io's Investigations works this way; PagerDuty's SRE agent triages and proposes next steps with human oversight; Datadog's Bits AI SRE does the equivalent inside its own telemetry estate. Vendors attach large automation claims to these products — treat every such number as a vendor claim until your own incidents reproduce it. What is not in dispute is the shape: the deliverable is a cited hypothesis list, and the value you extract from it is bounded by one behavior — whether you check the citations before you act.
Citation-checking is a three-question reflex, applied to each load-bearing claim in the AI's output:
The discipline scales with stakes: for a hypothesis you are about to act on — roll back this deploy, restart that service — all three checks are mandatory, out loud, in the incident channel ("checked: the PR does touch the pool config, and the timing matches — rolling back"). For hypotheses you are merely keeping warm, check one and move on. Narrating the checks matters as much as doing them; it keeps the whole responder group calibrated instead of silently anchoring, and it leaves an audit trail the postmortem will thank you for. This is Level 2 on your M1 map, practiced under adrenaline — which is why M1's game-day advice was to drill it with planted false leads before a real incident grades you on it.
The most dangerous AI output in an incident is a correct-looking hypothesis with real citations that do not actually support it — verify what the citation says, not just that it exists.
Commercial incident AI is a product decision; the technique is portable, and you should own it independently — both because many teams (and most client engagements in a services context) cannot ship incident data to a vendor, and because building it once teaches you exactly what the products are doing. The technique is Common Core material composed for ops: a structured-output prompt over an incident bundle — the artifacts you already have in the first ten minutes: firing alerts, recent deploy log, a metrics excerpt, the incident channel transcript.
A production-shaped triage prompt:
You are assisting incident triage. Using ONLY the four attached
artifacts (alerts.json, deploys.log, metrics.csv,
slack_transcript.txt), return JSON matching this schema:
{
"severity_hypothesis": "SEV1|SEV2|SEV3",
"severity_rationale": "one sentence",
"blast_radius": {"services": [], "customer_facing": true|false,
"evidence": "artifact:line refs"},
"hypotheses": [
{"rank": 1,
"claim": "specific, falsifiable statement",
"evidence": ["alerts.json:3", "deploys.log:17"],
"confidence": "high|medium|low",
"disconfirming_check": "cheapest command/query that would
kill this hypothesis"}
],
"recommended_next_action": "act_now|monitor|defer",
"unknowns": ["what is NOT determinable from these artifacts"]
}
Rules: every claim must cite artifact lines. If evidence is
ambiguous, say so in unknowns rather than raising confidence.
Do not recommend remediation commands.
Every field is a lesson from this track. evidence with line references makes citation-checking mechanical. disconfirming_check per hypothesis imports M2's diagnosis discipline — the model must tell you how to prove it wrong. unknowns is the anti-overconfidence valve: a model forced to enumerate what the artifacts cannot show hallucinates measurably less about what they can. And Do not recommend remediation commands is the trust map enforced in the prompt itself — this is a Level 2 artifact, and the prompt refuses to let it impersonate Level 3. Run this against a synthetic bundle before you ever need it live (lab-06 ships one, with a planted false lead your citation-checking must catch), and wrap it in a script so that during a real incident it is one command, not prompt engineering at 3 a.m. For follow-the-sun teams, the same bundle-plus-prompt pattern generates the handoff summary at shift boundary — one of the highest-value, lowest-risk AI uses in the whole category.
Keep the triage prompt in version control next to your runbooks and treat changes to it like alert-rule changes — reviewed, tested against past incident bundles, and rolled back if signal quality drops.
For Kubernetes incidents, two CNCF-hosted open-source tools give you the investigation pattern with zero vendor dependency — and a free lab path on kind or minikube. They occupy different rungs on the trust ladder, and using both teaches the difference.
K8sGPT is deterministic analyzers plus LLM explanation: it scans for known failure signatures (CrashLoopBackOff, pending pods, misconfigured probes and services, OOM kills) and has the model translate findings into plain English with suggested fixes. Because rule-based analyzers find the facts and the LLM only narrates them, its wrongness surface is narrow — closer to a very good error-message translator than an investigator:
k8sgpt analyze --explain --filter Pod --namespace payments
HolmesGPT is genuinely agentic: given an alert, it decides which data to fetch — pod logs, events, resource states, Prometheus queries — iterates as hypotheses form, and returns a diagnosis grounded in what it actually pulled from your cluster. That is a longer reach and a wider wrongness surface, which is exactly why it prints its toolcalls: the transcript of what it queried is the citation trail, and you review it the same way you review the incident.io hypothesis list.
The discipline that binds both: kubectl remains the source of truth. Before acting on any AI cluster diagnosis, confirm the load-bearing fact directly — kubectl describe pod, kubectl get events --sort-by=.lastTimestamp, kubectl logs --previous. This is not distrust theater; it is the two-tool honesty check, and it costs seconds. Lab-07 operationalizes it: break a kind cluster five ways (crash loop, OOM, bad probe, image pull failure, quota exhaustion), run both tools against each break, and score their diagnoses against the ground truth you planted. Most engineers come out of that lab with a nuanced calibration — excellent on signature failures, weaker on interacting failures — which is precisely the per-failure-class trust table M2 taught for CI, rebuilt for clusters.
K8sGPT narrating deterministic analyzers and HolmesGPT investigating agentically are different trust-map placements of the same domain — the first is a Level 1–2 explainer, the second a Level 2 investigator whose toolcall transcript is its citation trail.
Incident writing is where the evidence for AI is strongest and the verification gate is most often skipped — a bad combination, because incident writing is published: status pages reach customers, exec summaries reach leadership, postmortems become institutional memory. The Google SRE numbers cited above (postmortem drafts rated 10% better in blind review, with a projected 51% reduction in writing time; exec summaries 52% faster at comparable quality) were achieved with humans verifying and finalizing drafts — that gate is part of the result, not overhead on it.
Comms during the incident. Feed the timeline and impact data to a per-audience drafting prompt — status page (plain language, no internals, no blame, no promises you have not made), exec summary (impact, customer exposure, ETA confidence), internal update (technical, candid). Two hard rules govern it:
Postmortems after. The draft assembles itself from artifacts you already have — channel transcript, alert timeline, deploy log, dashboards. What makes the result trustworthy is the reviewer checklist, which forces claim-by-claim verification:
Postmortem verification checklist (reviewer runs before publishing):
[ ] Every timestamp in the timeline matches a source artifact
(alert, deploy log, message) - no narratively smoothed times.
[ ] Every causal statement ("X caused Y") is either verified
against evidence or rewritten as "X preceded Y".
[ ] Every metric/number traces to a query someone can re-run;
the query is linked.
[ ] Quotes attributed to responders appear verbatim in the
transcript.
[ ] Action items name an owner and a check for "done" - not
AI-suggested vagueness ("improve monitoring").
[ ] One deliberate spot-check: pick the draft's single most
important claim and independently re-derive it from raw
artifacts.
The known failure mode this checklist targets: LLMs narrativize. Given a messy timeline, they produce a cleaner story than reality — smoothing timestamps, upgrading correlations to causes, inventing the connective tissue good stories have and real incidents lack. A postmortem that is 5% smoother than the truth is worse than a rough accurate one, because the false coherence gets learned from: it enters the RAG corpus of the next section and misleads the next incident. Lab-08 plants one artifact contradiction in a drafting pipeline and passes only reviewers whose checklist catches it.
The model drafts the story; the artifacts are the truth; the reviewer's job is to make the story lose every argument with the artifacts before anyone else reads it.
"Didn't something like this happen last year?" is one of the highest-value questions in incident response and one of the least reliably answered — the person who remembers is asleep, or gone. This is a retrieval problem, and you learned the machinery in the Common Core; what changes here is the corpus and the quality bar.
The build: embed your postmortem corpus (plus, if governance allows, resolved-incident channel summaries) into a vector store — pgvector is the free, boring, self-hostable default — and expose retrieval at triage time, surfacing the top matches with links: similar incident, root cause then, what mitigated it, how long it took.
Four design decisions matter more than model choice:
Quality is measurable, so measure it: hold out a labeled set of incident pairs your team knows are related, check retrieval hit-rate before trusting the system in anger, and re-run the eval when you change embeddings or chunking (lab-09 does this against a 30-postmortem corpus). One more reason the corpus discipline from the previous section matters: RAG faithfully retrieves whatever you wrote — narrativized postmortems in, confident wrong precedents out.
The final step up the ladder — AI that acts — is where this module gets strict, because it is where M1's Level 3 stops being a diagram. The industry's own sequencing endorses caution: PagerDuty ships its remediation-capable agent with human-oversight execution first and holds fully-autonomous operation to a later, opt-in phase. Build yours the same way. A bounded remediation design has four load-bearing parts:
The allowlist is code, not convention. The agent can execute exactly the actions in a versioned, reviewed action file — nothing else, structurally. Four actions cover a striking share of real remediations: restart a deployment, scale within bounds, roll back to previous version, flip a feature flag.
# remediation-actions.yaml - reviewed like IaC, because it is
actions:
- name: restart-deployment
command: "kubectl rollout restart deployment/{{name}} -n {{ns}}"
scope: {namespaces: [payments, checkout], max_per_hour: 2}
undo: "self-limiting - pods restart"
approval_tier: standard
- name: scale-deployment
command: "kubectl scale deployment/{{name}} -n {{ns}} --replicas={{n}}"
scope: {namespaces: [payments, checkout], replicas: {min: 2, max: 20}}
undo: "scale back to recorded prior value"
approval_tier: standard
- name: rollback-deployment
command: "kubectl rollout undo deployment/{{name}} -n {{ns}}"
scope: {namespaces: [payments, checkout]}
undo: "roll forward - requires new approval"
approval_tier: elevated # touches released code paths
Approval tiers follow blast radius. Standard-tier actions need one responder's explicit yes in the incident channel; elevated-tier actions (rollbacks, anything touching stateful services) need the incident commander or a second responder. The approval message must carry the specifics — action, target, scope, the agent's stated reason — because an approval UX that shows only "approve remediation? y/n" is a rubber stamp factory, and rubber-stamping is how Level 3 silently decays into Level 4. Rate limits (max_per_hour) are the guard against the pathological loop where an agent "fixes" a recurring symptom all night while the cause compounds.
Everything is audited. Every proposal, approval, execution, and result lands in an append-only log with who-approved and artifacts-cited. This is what makes the design defensible to your security team and your client — and it is the dataset from which you will later argue, with evidence, that a specific action has earned a higher trust placement.
The refusal path is tested, not assumed. Red-team your own agent: mid-incident, ask it (through whatever channel it takes requests) to do something off-list — delete a pod stuck terminating, edit a configmap, "just run this one command." The only acceptable behavior is a structured refusal naming the allowlist and, ideally, suggesting the nearest permitted action or paging a human. Lab-10 requires the refusal transcript as a deliverable, because an allowlist you have never seen refuse is a hypothesis, not a control.
Bounded-remediation questions in the assessment hinge on the four parts — enumerable action set, per-action approval at the right tier, append-only audit, demonstrated refusal — and a design missing any one of them is the wrong answer no matter how capable the model.
Your first 15 minutes: the next time an AI investigator hands you a cited hypothesis during on-call, say the three-question citation check out loud in the incident channel before you touch anything — exists? says what's claimed? supports causation? That one narrated habit is this whole module in miniature, and it is the difference between Priya's 03:12 rollback and her 02:41 catch.
Check yourself: An AI investigator gives you a confident root-cause with three log citations, and you click through and confirm all three links resolve to real artifacts. Are you clear to act? What is still unchecked?
unknowns correctly flag missing evidence? Where did confidence exceed what the artifacts supported?This module's companion playbooks cover the citation-checking protocol as an incident-channel checklist, the structured triage prompt with client-safe redaction and follow-the-sun handoff variants, K8sGPT/HolmesGPT setup on kind with the tool-versus-truth worksheet, per-audience comms templates with the commitment scan, the postmortem verification checklist, incident-RAG design notes, and the bounded-remediation reference design. Do this at work this week → pb-15 Run an AI Incident Investigation and Verify Every Citation (Priya's 02:41 catch, as a repeatable protocol), pb-14 Triage an Alert with an AI-Built Incident Bundle (the structured-triage prompt you own), and pb-17 Bounded Auto-Remediation with Approval Gates (the allowlist-plus-approval design). Five labs back the module, and the guided journey deliberately spreads them across two stages rather than one contiguous block: the triage copilot with a planted false lead (lab-06) and the five-failure cluster debugging matrix (lab-07) sit in the Operate stage alongside the bounded remediation agent with approval gate, audit log, and refusal transcript (lab-10), while the postmortem factory with a seeded contradiction (lab-08) and incident-history RAG over a 30-postmortem corpus with retrieval evals (lab-09) anchor the later Institutionalize stage. All run free on kind/minikube, OSS tools, and any LLM API's entry tier.
--explain, filters, and operator mode for the cluster-diagnosis pattern.