Lab 11 · Incident to Postmortem in Under an Hour

The graded skill is evidence discipline: every claim in your RCA must trace to a specific artifact, and the AI's confident-sounding correlations must be verified before they become "the cause".

Time & tier

Core · ~70 minutes hands-on.

Setup

No code or API spend — any chat assistant works. Create lab11/ and, inside it, four evidence files with the content below. This is your incident bundle; treat it as an on-call handoff.

alerts.txt:

02:14:07Z  PAGE  api-checkout  p99_latency > 2000ms (value 4120ms)
02:14:41Z  PAGE  api-checkout  error_rate > 5% (value 11%)
02:16:02Z  WARN  db-primary    connections_used 191/200
02:18:30Z  PAGE  api-checkout  error_rate > 5% (value 34%)
02:19:55Z  WARN  db-primary    connections_used 200/200
02:22:10Z  INFO  api-search    p99_latency > 2000ms (value 2600ms)
02:41:00Z  INFO  api-checkout  error_rate < 5% (value 0.8%)  [auto-resolved]

deploys.txt:

2026-07-25 21:30Z  api-search    v412  "add related-items query"        rollout: complete
2026-07-25 23:05Z  api-checkout  v889  "swap HTTP client, bump pool"    rollout: complete
2026-07-26 01:50Z  db-primary    -     "routine minor-version patch"    rollout: complete
2026-07-26 02:55Z  api-checkout  v890  "revert v889"                    rollout: complete

app-logs.txt (api-checkout, sampled):

02:13:58Z INFO  checkout ok order=88121 db_ms=12
02:14:05Z WARN  db pool acquire waited 1400ms
02:14:33Z ERROR could not get connection from pool (timeout 3000ms) trace=a1
02:15:10Z ERROR could not get connection from pool (timeout 3000ms) trace=b2
02:17:44Z ERROR could not get connection from pool (timeout 3000ms) trace=c3
02:40:12Z INFO  checkout ok order=88490 db_ms=15

chat-export.txt:

02:16 alice: checkout is on fire, p99 4s, seeing pool timeouts
02:17 raj:   db connections maxed at 200. did something change?
02:19 alice: search also slow now? or is that noise
02:24 raj:   we shipped a db patch at 1:50, could be that
02:27 priya: checkout swapped its http client at 11pm last night, pool config changed?
02:33 raj:   rolling db back would take 40 min, risky
02:38 priya: let's revert checkout v889 first, fastest lever
02:55 priya: v890 out (revert). watching.
03:02 alice: recovering. errors dropping.

Guided steps

1. 1.

   Build the timeline first — no AI. In rca.md, lay out one merged, timestamped timeline across all four files. Doing this by hand first means you can catch the AI when it invents an ordering.

2. 2.

   Ask for ranked hypotheses with disconfirming tests. Paste all four files:

   Copy

   You are helping an on-call engineer. From this evidence only, produce a
   ranked list of root-cause hypotheses. For each: the supporting evidence
   (cite file + timestamp), one piece of evidence that would DISCONFIRM it,
   and a confidence level. Do not assert a single cause. Flag anything that
   looks like correlation without a mechanism.
   

3. 3.

   Verify each hypothesis against the artifacts. For the top hypothesis, physically check every cited timestamp. The AI will likely nominate the 01:50 DB patch (temporally plausible) and the 23:05 client swap. Only one has a mechanism that fits the pool-timeout signature. Write which citations you confirmed and which you rejected.

4. 4.

   Interrogate the tempting red herring. Ask the AI: "Argue that the DB patch is NOT the cause. What evidence is inconsistent with it?" Note that the patch landed 01:50 but the incident began 02:14 — a 24-minute gap with no intermediate signal, versus a config change that shipped hours earlier and needed load to manifest.

5. 5.

   Write the RCA section. State the most-supported cause, the mechanism, and — critically — mark any step that is inference vs. observed. Include a "what we still don't know" line; real postmortems have them.

6. 6.

   Produce the blameless postmortem. Use this skeleton and have the AI draft prose you then edit for accuracy: Summary · Impact (duration, blast radius from the alerts) · Timeline · Root cause · Contributing factors · What went well · What went poorly · Action items (owner + type: prevent/detect/mitigate).

7. 7.

   Three runbook updates. Each must be concrete enough to execute at 3 a.m.: e.g. a "DB pool exhaustion" runbook with the exact metric to check and the fastest-revert decision rule the team actually used. Derive them from the gaps the incident exposed.

Verification

• •
  Your timeline is internally consistent and every RCA claim cites a file + timestamp.
• •
  The RCA correctly identifies the pool-exhaustion mechanism and does not blame the DB patch as primary cause (or, if it does, provides a mechanism that survives step 4).
• •
  Postmortem contains all skeleton sections and at least one honest "went poorly" item and one "still unknown".
• •
  Three runbook updates are concrete (named metric, named command or decision rule), not "improve monitoring".
• •
  No action item names or blames an individual (blameless check).

Completion artifact

rca.md (timeline + hypotheses + verification notes), postmortem.md, runbooks.md. Self-grade (4 points = pass at 3+):

1. 1.
   Every factual claim in the postmortem traces to an artifact; inferences are labeled.
2. 2.
   The red-herring DB patch was explicitly considered and correctly demoted with a stated reason.
3. 3.
   Postmortem is genuinely blameless (systems and decisions, not people).
4. 4.
   Runbook updates are executable, not aspirational.

Stuck?

• AI insists the DB patch caused it: ask for the mechanism by which a minor-version patch exhausts a connection pool 24 minutes later with no intermediate signal. Absent a mechanism, demote it to "contributing/unconfirmed".
• Can't pick a primary cause: the pool-timeout errors (app-logs.txt) plus connections climbing to 200/200 right after a change described as "bump pool" is the load-bearing chain. The client swap changed pool behavior.
• Postmortem reads like blame: replace "Alice deployed a bad change" with "the change passed review and staging because our tests don't exercise pool saturation" — that's the systemic gap.

Client-safe variant

The bundle is synthetic, so run it verbatim anywhere. For a real client incident: the evidence (logs, chat exports) is often the most sensitive data you handle — it can contain customer identifiers, internal hostnames, and security details. Redact before any external LLM (strip order IDs, customer emails, internal IPs, employee names → roles), or run correlation on a client-approved endpoint. Better still, teach the client the timeline-first-then-verify protocol so the sensitive data never leaves their environment; the discipline transfers, the data shouldn't.

Answer key (don't read before attempting)

Primary cause: the 23:05 api-checkout v889 deploy that "swapped HTTP client, bump pool". Despite "bump pool", the new client's pool configuration (or its connection lifecycle) caused connections to be held longer / leaked, so under normal early-morning load connections climbed until they hit the DB primary's 200 ceiling. Mechanism fit: app-logs.txt shows pool-acquire waits then could not get connection from pool timeouts; alerts.txt shows db-primary connections_used climbing 191→200 in step with checkout's error rate; recovery at 02:41 aligns with the 02:55 revert (v890) taking effect. The 01:50 DB patch is the seeded red herring — temporally nearest to the 02:14 onset but with no mechanism linking a routine minor-version patch to pool exhaustion, and a suspicious 24-minute silent gap. The 21:30 api-search v412 and the 02:22 search latency blip are a second red herring (search degraded because it shares the same saturated db-primary, a symptom not a cause). What you genuinely can't know from the bundle: whether it was a leak vs. a raised pool-size-vs-DB-ceiling mismatch vs. slower per-connection queries — v2 postmortem should list this as an open question and an action item to capture pool metrics. Good runbook updates: (1) "DB pool exhaustion" runbook keyed on connections_used approaching the DB max, with fastest-revert-recent-deploy as the first lever; (2) require a pool-saturation load test as a deploy gate for any connection-layer change; (3) add a per-service pool_connections_in_use / pool_max metric and alert, which the incident had to infer from DB-side data.