Lab 06 · Incident Triage Copilot With Citations

The bundle contains a planted false lead that the humans in the transcript (and most models) fall for. Commercial AI-SRE tools (incident.io Investigations, PagerDuty SRE Agent, Datadog Bits) produce exactly this kind of cited hypothesis report; the transferable skill is verifying citations before acting, because a wrong-but-confident triage steers a whole incident channel.

Time & tier

~75 minutes hands-on · Core tier (maps to module M3). $0 — any free-tier chat model.

Setup

Create incident-2417/ with these four artifacts. Incident: checkout errors on 2026-07-24. Read nothing closely yet — arriving cold is part of the exercise.

// alerts.json
[
  {"alert":"PaymentsDBConnectionsHigh","severity":"warning","fired_at":"13:52",
   "channel":"#db-alerts-lowpri","summary":"payments-db connection usage > 95% for 5m"},
  {"alert":"CheckoutHighErrorRate","severity":"critical","fired_at":"14:07",
   "channel":"#oncall","summary":"checkout 5xx rate > 5% for 5m"},
  {"alert":"CheckoutLatencyP95","severity":"critical","fired_at":"14:09",
   "channel":"#oncall","summary":"checkout p95 > 2s for 5m"}
]

# deploys.log
2026-07-20 09:14 UTC deploy checkout-service v181 (weekly release)
2026-07-24 13:28 UTC config-change report-backfill: FULL_REINDEX=true (data-eng, change #4112)
2026-07-24 13:30 UTC cron start report-backfill run #88 (daily)
2026-07-24 14:02 UTC deploy checkout-service v183 (feature: gift-cards)
2026-07-24 14:15 UTC rollback checkout-service v183 -> v181
2026-07-24 14:27 UTC job kill report-backfill run #88 (ops)

# metrics.csv
time_utc,checkout_error_pct,paydb_conns_used,paydb_conns_max,checkout_p95_ms
13:30,0.2,58,100,210
13:35,0.3,71,100,230
13:40,0.4,83,100,290
13:45,0.9,92,100,400
13:50,1.8,97,100,650
13:55,3.5,100,100,1200
14:00,6.2,100,100,2100
14:05,9.8,100,100,3400
14:10,11.5,100,100,3900
14:15,11.9,100,100,4000
14:20,12.1,100,100,4100
14:25,12.0,100,100,4050
14:30,2.1,64,100,600
14:35,0.3,42,100,220

# slack.txt
[14:08] priya (on-call): checkout error page, 5xx spiking
[14:10] dev-raj: we deployed v183 at 14:02, has to be the gift-card change
[14:11] priya: agreed, timing matches. rolling back
[14:15] priya: rollback done
[14:21] priya: still failing?? error rate not moving
[14:23] meera (dba): payments-db pegged at max connections since ~13:55 btw,
        warning fired in #db-alerts-lowpri
[14:26] arjun (data-eng): heads up, we flipped FULL_REINDEX on the daily backfill
        today, run started 13:30, holding way more connections than usual
[14:27] meera: killing that job
[14:31] priya: recovering. errors dropping

Guided steps

1. 1Prompt for structured, cited triage. Paste all four artifacts with: "You are an incident triage copilot. Produce: (1) severity assessment; (2) a timeline of significant events; (3) ranked root-cause hypotheses — for EVERY factual claim cite the artifact and exact line/timestamp in the form [deploys.log 14:02] or [metrics.csv 13:45]; (4) recommended immediate action; (5) what evidence would change your #1 hypothesis. Claims without citations are forbidden."
2. 2Build the citation-verification worksheet. Table in worksheet.md: claim · citation given · citation checks out? (open the artifact and look) · claim actually supported? · verdict (verified / miscited / unsupported / fabricated). Fill a row for every factual claim in the AI report. Distinguish carefully: a citation can be real while the claim over-reads it — "the v183 deploy caused the errors [deploys.log 14:02]" cites a true line but the line only proves a deploy happened.
3. 3Hunt the false lead. The transcript's humans concluded "timing matches" for the v183 deploy. Check that against metrics.csv yourself: when did error rate and DB connections actually start degrading, relative to 14:02? Write one paragraph with three citations proving or refuting deploy causality. If your copilot's #1 hypothesis was the deploy, feed it your paragraph and record whether it revises or rationalizes.
4. 4Interrogate the counterfactual. Ask the copilot: "The rollback completed at 14:15. Using only metrics.csv, did it work?" A trustworthy answer cites 14:20/14:25 error rates staying ~12%. Then: "What single artifact line best explains recovery at 14:30?" (Answer should land on the 14:27 job kill.)
5. 5Write the final triage report yourself (triage-report.md): severity, verified timeline, root cause with citation chain (config change 13:28 → cron start 13:30 → connection climb from 13:40 → saturation 13:55 → checkout starvation), contributing factors (warning alert routed to a low-priority channel; anchoring on the most recent deploy), and the action that actually resolved it. AI may critique your draft; the citations are yours.
6. 6Score your copilot. Count its claims by verdict category. Compute a citation-precision number (verified / total). Note specifically: did it fall for the deploy lead initially, and did it recover when challenged?

Verification

• •
  Worksheet has a row for every factual claim in the AI report, with your four-way verdict per row.
• •
  Your false-lead paragraph cites at least: pre-deploy degradation (metrics 13:40–14:00), rollback non-recovery (metrics 14:15–14:25), and the true trigger (deploys.log 13:28/13:30).
• •
  Final report's causal chain runs config-change → connection exhaustion → checkout failure, not deploy → errors.
• •
  Citation-precision score computed, with fabricated/miscited examples quoted if any occurred.

Completion artifact

incident-2417/ with worksheet.md, triage-report.md, and the raw AI transcript. Self-grading rubric (pass = 8/10):

• False-lead catch (0–3): deploy exonerated with the pre-deploy metrics argument = 3; caught only after hints = 1.
• Citation discipline (0–3): every worksheet row has an artifact-checked verdict = 3.
• Causal chain (0–2): full chain with timestamps, including why rollback failed = 2.
• Copilot scoring (0–1): citation-precision computed honestly = 1.
• Process insight (0–1): report names the alert-routing miss as a contributing factor = 1.

Stuck?

• •
  Copilot refuses to cite: reject the answer and re-prompt "regenerate; uncited claims are forbidden" — do not fill citations in for it.
• •
  Everything looks verified: re-check verdicts for over-reading — "deploy at 14:02 [deploys.log]" is verified; "deploy caused errors [deploys.log]" is unsupported by that line.
• •
  Confused by two things changing (deploy + backfill): that's the design; real incidents stack changes. The discriminator is which change precedes the degradation curve.

Client-safe variant

Real incident bundles are stuffed with client identifiers: hostnames, customer names in error messages, internal channel names. Follow-the-sun teams handing off to India shifts should maintain a redaction script (hostnames→role names, customers→tokens) so triage summaries can use approved AI tenancy; if no tenancy is approved, run a local model — triage summarization is well within local-model capability. The worksheet format is identical and doubles as handoff documentation across timezones.

Answer key (don't read before attempting)

• Root causechange #4112 set FULL_REINDEX=true on the daily report-backfill (13:28); run #88 (13:30) held far more payments-db connections than normal, climbing from 13:35 and saturating the 100-connection pool by 13:55. Checkout shares that database; starved connections drove 5xx and latency.
• The false leadcheckout v183, deployed 14:02 — after error rate had already reached 6.2% (14:00) and the pool had been pegged for 7 minutes. The transcript's "timing matches" is backwards; the rollback's non-effect (errors ~12% through 14:25) is the second exoneration; recovery follows the 14:27 job kill, not any checkout change.
• Contributing factorthe 13:52 PaymentsDBConnectionsHigh warning fired into a low-priority channel and was seen 31 minutes late.