Triage an Alert with an AI-Built Incident Bundle

Tier: Core · Hands-on time: ~35 minutes

This playbook turns that assembly into a repeatable AI workflow — feed the model an incident bundle (alert payload + recent deploys + runbook excerpt) and demand a structured triage note in which every statement cites a line in the bundle. PagerDuty's SRE Agent and incident.io both ship this capability commercially; here you build the technique yourself so it works with any LLM and you know exactly how to audit the output.

When to use

• •
  You are on call and an unfamiliar alert fires — especially at 2 a.m. or during a follow-the-sun handoff where the paged engineer did not write the service.
• •
  An alert storm produces 5+ related pages and you need a fast act-now/defer split.
• •
  You are building a triage bot for your team and need the prompt-and-verify pattern before wiring it into PagerDuty or Slack.

Skip AI for alerts you have seen ten times before — your cached diagnosis is faster than any model, and pattern-matching a known alert is exactly what runbooks are for.

Prerequisites

• •
  A current general-purpose AI assistant (chat or CLI); no vendor incident product required.
• •
  The ability to export or paste: the raw alert payload (JSON from Alertmanager/PagerDuty/Datadog), the last 24h of deploys or change events for the service, and the relevant runbook section.
• •
  A synthetic or real (sanitized) alert to practice on — never learn this workflow live on a SEV-1.

The workflow

1. 1.

   Assemble the bundle mechanically, not selectively. Concatenate into one text block, with headers: === ALERT PAYLOAD ===, === RECENT CHANGES (24h) ===, === RUNBOOK EXCERPT ===, === SERVICE METADATA === (owner, tier, dependencies). Do not pre-filter "irrelevant" changes — the deploy you dismiss is often the cause.

2. 2.

   Request a triage note with citation demands. Send the bundle with:

   Copy

   You are assisting an on-call engineer. Using ONLY the bundle above, produce:
   1. SEVERITY HYPOTHESIS: proposed severity + one-line justification.
   2. WHAT FIRED: the alert condition in plain English (metric, threshold, duration).
   3. CHANGE CORRELATION: any change events plausibly related, with timestamps.
   4. ACT-NOW / DEFER: recommendation with the single strongest reason.
   5. FIRST THREE CHECKS: ordered, each a concrete command or dashboard.
   
   Rules: after EVERY factual statement, cite the bundle section and line/field
   it came from in brackets, e.g. [ALERT: annotations.summary] or [CHANGES: line 4].
   If the bundle does not contain evidence for a statement, write UNVERIFIED
   instead of the citation. Do not invent metric names, hosts, or timestamps.
   

3. 3.

   Audit the citations before acting. For each bracketed citation, check the referenced field actually exists in your bundle and says what the note claims. Every UNVERIFIED line is a question for you, not a fact.

4. 4.

   Run the first check yourself. Execute check #1 from the note (a kubectl, promtool, or dashboard look). If reality contradicts the triage note, reply with the contradicting output and ask for a revised hypothesis — the model updates well when fed real signals.

5. 5.

   Log the note into the incident channel with a one-line header: AI triage note — citations audited, checks 1–2 run by <you>. The audit statement is what makes it safe for teammates to build on.

Tool variants

• Plain LLM chat (Claude, ChatGPT, Gemini): paste the bundle; works everywhere, zero integration. Best for learning and for teams without incident tooling budgets.
• PagerDuty SRE Agent / incident.io / Datadog Bits AI SRE: these products auto-assemble the bundle from integrations and post triage suggestions in Slack. Apply the same discipline: treat their output as a cited draft, click through every referenced deploy/monitor before acting.
• Scripted (LLM API + cron/webhook): a small script that builds the bundle from Alertmanager webhook + git log + runbook repo and posts the note to Slack. Keep the citation rules in the system prompt; add a footer "AI-generated — audit citations before acting."

Verification

• •
  Every bracketed citation resolves to a real field or line in the bundle, verbatim — spot-check all of them on your first ten uses, at least three per note thereafter.
• •
  Any metric name, hostname, or timestamp in the note appears character-for-character in the bundle. These are the highest-frequency hallucination sites in ops output.
• •
  The act-now/defer recommendation is consistent with your paging policy — check it against the service tier in the bundle, not against the model's judgment.
• •
  Count UNVERIFIED lines: zero is suspicious (the model may be over-claiming coverage); more than half means your bundle is too thin to triage from.

Pitfalls

• Feeding the alert only. Alert payload without change events produces generic advice ("check CPU"). Change correlation is where AI triage actually earns its keep.
• Letting the model set severity. The model proposes; your paging policy disposes. Vendor agents phrase this as "triage faster without waking a human" — the human still owns the wake-up decision.
• Bundle staleness during long incidents. A note generated at minute 5 is wrong by minute 40. Regenerate with an updated bundle rather than patching the old note conversationally.
• Trusting fluent structure. A perfectly formatted triage note with fabricated citations looks identical to a correct one until you audit. Format quality is not evidence quality.

Restricted-environment variant

Services-company and client-account scenarios (client telemetry under MSA restrictions):

• •
  Redact before pasting: replace client name, account IDs, internal hostnames, and customer identifiers with stable placeholders (CLIENT_A, host-web-01) via a scripted sed/regex pass — manual redaction misses fields inside JSON payloads.
• •
  Prefer a self-hosted model (Ollama/vLLM behind LiteLLM) for raw telemetry; an 8B–70B-class model handles structured triage notes acceptably when the bundle is under a few thousand tokens.
• •
  If the client provides an approved tenant (Bedrock/Azure OpenAI in their VPC), use that and keep the same prompt — the citation discipline is tenancy-independent.
• •
  Check the client's AI policy before wiring any automated bundle-poster to their PagerDuty/Slack.

Success criteria

• •
  From page to audited triage note in under 10 minutes, including your own first check.
• •
  On a synthetic bundle with a planted decoy change, your audit catches whether the model blamed the wrong change.
• •
  A teammate can act on your posted note knowing which statements are bundle-backed and which are UNVERIFIED.
• •
  After two weeks of use you can name at least one case where the citation audit caught a fabricated or misattributed claim — if you cannot, you are probably not auditing.