Run an AI Incident Investigation and Verify Every Citation

Tier: Core · Hands-on time: ~45 minutes

The number to internalize is different: an investigation is a set of claims with citations, and your job during the incident shifts from gathering evidence to verifying it. This playbook is the verification discipline — it applies identically to a commercial agent's report and to an investigation you run yourself with an LLM over an evidence bundle.

When to use

• •
  A commercial AI investigator posts a root-cause hypothesis during a live incident and you must decide whether to act on it.
• •
  No commercial tool: you are mid-incident with logs, a deploy list, and dashboards, and want a structured multi-hypothesis investigation from an LLM.
• •
  Post-incident, to check whether the AI investigation you followed was actually right (feeds the postmortem, pb-19).

Do not use an unverified AI hypothesis to justify an invasive action (failover, rollback of someone else's deploy, data mutation). Verification first, action second — always.

Prerequisites

• •
  Either a trial/seat of an AI investigation product, or an LLM plus an evidence bundle: incident timeline, relevant log excerpts, deploy/PR list with links, and 2–3 metric screenshots or exported series.
• •
  Access to the systems the citations point at (GitHub, log store, dashboards) — you cannot verify citations you cannot open.
• •
  A practice incident: break something in a staging/kind cluster or replay a past incident's artifacts.

The workflow

1. 1.

   Get the investigation (product or DIY). With a product, open the posted report. DIY, send your evidence bundle with:

   Copy

   Act as an incident investigator. From the evidence above, produce up to 3
   root-cause hypotheses, ranked. For each:
   - HYPOTHESIS: one sentence.
   - SUPPORTING EVIDENCE: bullet list; each bullet quotes the exact log line,
     PR title/number, or metric behavior from the evidence, with its source tag.
   - CONTRADICTING EVIDENCE: anything in the bundle that argues against it.
   - DISCRIMINATING TEST: one check that would confirm or kill this hypothesis.
   Never cite anything not present in the evidence. Mark inferences as INFERENCE.
   

2. 2.

   Verify citations in precision order. Check the top hypothesis's evidence first, most-load-bearing claim first: (a) does the cited artifact exist (open the PR, find the log line verbatim), (b) does it say what the report claims, (c) does the timestamp actually precede the impact? A citation that exists but is misread is the common failure — existence checks alone are not verification.

3. 3.

   Run the discriminating test. Before any remediation, execute the test for the top hypothesis (query the metric, grep the log window, diff the config). This is minutes of work and converts "plausible" into "confirmed" or "dead."

4. 4.

   Feed back contradictions. If the test kills hypothesis #1, reply to the model (or the product's thread) with the contradicting output and ask it to re-rank. Iterating with real signals is faster than restarting.

5. 5.

   Record the verification trail. In the incident channel: Hypothesis 2 confirmed — citations checked (PR #841 verified, log line found 14:02:11, discriminating query attached). This line is what the postmortem's claim-verification gate (pb-19) will look for.

Tool variants

• incident.io Investigations: parallel-search report with cited PRs delivered in Slack; click every cited PR/monitor before acting. Use the thread to challenge weak citations.
• Datadog Bits AI SRE / PagerDuty SRE Agent: same pattern inside their consoles; PagerDuty's agent also pulls past-incident memory — verify that a "similar past incident" is actually similar (same failure mode, not just same service).
• DIY with Claude/ChatGPT + evidence bundle: full control, works in any environment; you own bundle assembly (see pb-14) and get the same ranked-hypotheses structure. Best free-tier lab path.

Verification

• •
  Every citation in the acted-on hypothesis was opened by a human: artifact exists, content matches the claim, timeline is causally possible. No exceptions for "obvious" citations.
• •
  The discriminating test was run and its output saved to the incident channel before remediation started.
• •
  At least one CONTRADICTING EVIDENCE entry or INFERENCE marker exists — a report with only confirming evidence and zero inferences is a red flag for sycophantic pattern-completion, not a clean investigation.
• •
  Cross-check the top hypothesis against the change-correlation from your triage note (pb-14); unexplained disagreement between the two must be resolved, not ignored.

Pitfalls

• Verifying only that links resolve. A real PR cited for the wrong reason passes the click test. Read the diff; confirm it touches the failing path.
• Anchor lock-in. The first AI hypothesis anchors the whole responder channel. Post it as "hypothesis, unverified" — wording changes how aggressively teammates check it.
• The 80% claim as a staffing plan. Vendor automation percentages describe evidence-gathering toil, not judgment. Keep the same number of humans on the bridge; give them verification jobs.
• Evidence bundle bias (DIY). If you only include logs from the service you suspect, the model will confirm your suspicion. Include one adjacent service's evidence deliberately.
• Timezone confusion in timelines. Mixed UTC/IST timestamps in the bundle produce causally impossible narratives that read fine. Normalize to UTC before prompting.

Restricted-environment variant

• •
  Client incidents under MSA: run the DIY variant with a self-hosted model (Ollama/vLLM via LiteLLM); commercial investigators require deep integrations (Slack, GitHub, telemetry) that client security teams must approve first — ask before connecting anything.
• •
  Redact log excerpts with a scripted pass (customer IDs, tokens, emails) before they reach any hosted model; keep the mapping local so citations remain checkable by you.
• •
  If the client runs their own incident.io/Datadog tenancy, use it under their accounts and policies rather than piping their telemetry into yours.
• •
  Store verification trails in the client's incident system — the audit evidence belongs to them.

Success criteria

• •
  On a replayed or synthetic incident with a planted false citation, your precision-order check catches it before the discriminating test stage.
• •
  Time from report-posted to confirmed/killed top hypothesis under 15 minutes.
• •
  Your incident channel shows the three-part trail: hypothesis posted → citations verified by name → discriminating test output attached.
• •
  In a postmortem review, every AI-originated claim that survived into the document can be traced to a human verification event.