At 3 a.m. with a paging alert, the scarce resource is not information — you are drowning in dashboards, logs, and deploy history — it is correlation under stress.
Tier: Core · Workflow area: Incidents & Operations
Correlation under stress is the task AI is built for: hand a model the alert, the error signature, and the last day of changes, and it produces ranked hypotheses with discriminating checks in seconds, immune to the tunnel vision that makes tired humans fixate on their first theory. The iron rule that makes this safe: the AI proposes, you verify, and only a human touches production.
Skip when the cause is obvious (the deploy 4 minutes before the alert — roll it back first, diagnose after) and never let hypothesis generation delay a known-good mitigation. Stabilize, then understand.
Assemble the triage bundle (2 minutes, scripted). Alert text + firing time; the error signature (top 3 distinct stack traces / error messages with counts, not 5,000 raw lines); deploys and config/flag changes in the last 24h across suspect services; a metrics sketch (what stepped, when — even prose: "p99 5x at 14:32, error rate followed at 14:34"); topology one-liner for the involved services.
Ask for ranked hypotheses with discriminating checks.
Production incident, triage mode. Bundle below.
1. Rank the most likely root causes (max 5). For each: the causal chain
from trigger to observed symptoms, which bundle evidence supports it,
which evidence it does NOT explain, and prior likelihood.
2. For each hypothesis: the fastest READ-ONLY check that would confirm or
kill it (specific query, dashboard, or log filter).
3. What evidence is missing from this bundle that would most change your
ranking?
Do not recommend any mutating action. If two hypotheses are near-tied,
say so rather than false-ranking.
Run the checks in likelihood order — yourself. Execute the read-only checks against real systems. Paste results back: Check 1 result: connection pool at 100/100 saturated since 14:31. Check 2: DB CPU normal. Update the ranking. Two or three iterations typically collapse the field to one hypothesis with a verified causal chain. The model's role: keep all hypotheses alive until evidence kills them — the exact discipline stressed humans lose.
Gate every action through a human. When a mitigation emerges (roll back deploy X, raise pool size, disable flag Y), the model may draft the exact command and predict the observable effect — but a human runs it and confirms the prediction ("error rate should drop within 2 min; if not, hypothesis is wrong — stop"). The prediction check matters: a mitigation that "works" for the wrong reason leaves the real cause armed.
Keep the model narrating for the timeline. Ask it to maintain a running incident log from what you paste ("14:41 pool saturation confirmed; 14:44 rollback initiated"). Zero extra cost during the incident; it becomes the spine of the postmortem (pb-39).
(With MCP observability hookups) Let the model run its own read-only Grafana/Datadog queries during steps 2–3 — the loop tightens from paste-and-wait to genuine copilot. Keep the credential scoped read-only; the human-gate on mutations is architecture, not etiquette.
The model ranks by plausibility; production is ruled by facts. Every hypothesis gets a check, every check gets run by you, and no mutation happens on an unverified theory — the discipline of verifying before acting is the entire safety case for AI in the incident loop.
Production logs are the most policy-sensitive data in this track: stack traces embed customer identifiers, tokens, hostnames, query fragments — DPDP Act 2023 applies to personal data in logs. Pre-incident, get a written answer to "what may leave the machine during an incident?" The workable tiers: (1) enterprise tenant inside the org/client cloud (Bedrock, Azure OpenAI) — full bundle allowed; (2) hosted assistant with a scrubbed bundle — error signatures and shapes, hostnames masked, identifiers stripped by the bundle script automatically; (3) fully local — a Qwen-class model via Ollama does competent hypothesis ranking over a clean bundle, and the discriminating-check structure compensates for weaker reasoning. In all tiers the human-gate is unchanged, and scrubbing lives in the bundle script so nobody is redacting by hand at 3 a.m.