Lab 08 · Postmortem Factory With a Verification Gate

Google SRE reported AI postmortem drafts scoring 10% better than human-written with 51% less writing time; that only holds when a verification gate stands between draft and publish. Your bundle carries a seeded contradiction between two artifacts. Most models reproduce the human-written summary's wrong claim; your gate must catch it.

Time & tier

~75 minutes hands-on · Core tier (maps to module M3). $0 — any free-tier chat model.

Setup

Reuse the incident-2417/ bundle from Lab 06 (alerts.json, deploys.log, metrics.csv, slack.txt) in a folder factory/bundle/. Add one new artifact — the on-call engineer's hasty timeline note, written mid-incident, containing the seeded contradiction (do not fix it):

# bundle/incident-timeline.md  (on-call notes, written 14:40, unreviewed)
14:07 paged for checkout 5xx
14:10 suspected gift-card deploy v183
14:15 rolled back v183 -- service restored by rollback
14:27 killed report-backfill job as precaution
14:35 all clear

The claim "service restored by rollback" is false: metrics.csv shows error rate still ~12% at 14:20–14:25, recovering only after the 14:27 job kill. Human timeline notes with wrong causality are utterly typical — and models weight fluent human summaries above raw data.

Add the assembler:

# factory/assemble.py
#!/usr/bin/env python3
"""Postmortem factory: assembles incident artifacts into one LLM prompt.
Usage: python3 assemble.py bundle > prompt.txt"""
import os, sys

TEMPLATE = """You are drafting a blameless postmortem. Use ONLY the artifacts below.
Every factual sentence must end with a citation like [metrics.csv 14:20].
Sections: Summary; Impact; Timeline; Root cause; What went well;
What went poorly; Action items (each: owner-role, priority, verification step).
If artifacts conflict, you MUST flag the conflict explicitly rather than pick a side.

"""

bundle = sys.argv[1]
parts = [TEMPLATE]
for name in sorted(os.listdir(bundle)):
    path = os.path.join(bundle, name)
    if os.path.isfile(path):
        parts.append(f"===== ARTIFACT: {name} =====\n{open(path).read()}\n")
sys.stdout.write("\n".join(parts))

And the gate — the artifact that outlives this lab:

# factory/review-gate.md — Postmortem verification checklist (publish-blocking)
For EVERY factual claim in the draft:
  [ ] Citation present and points at a real artifact line
  [ ] Cited line actually supports the claim (not merely related to it)
  [ ] Causal claims ("X fixed Y", "X caused Y") verified against the
      metric/timeline DATA, not against humans' statements about the data
  [ ] Timestamps in the draft match artifact timestamps exactly
Cross-artifact pass:
  [ ] Recovery time per metrics == recovery time claimed in prose
  [ ] Every remediation action is followed in the data by the effect
      attributed to it; actions with no visible effect are labeled as such
  [ ] Conflicts between artifacts listed with which source won and why
Blameless pass:
  [ ] No named individuals; roles only. No "should have"; systems language.
Sign-off: reviewer name, date, count of claims checked / corrected.

Guided steps

1. 1Assemble and generate.python3 assemble.py bundle > prompt.txt; paste prompt.txt into your chat model; save the reply as draft-v1.md. Do not correct the model mid-generation — you are testing the pipeline, not steering it.
2. 2Predict before reviewing. One line in your log: did the draft claim the rollback restored service, flag the conflict, or attribute recovery to the job kill? Models split on this; your prediction calibrates you.
3. 3Run the gate. Work through review-gate.md against draft-v1.md, building verification-log.md: claim · citation · check result · correction. The cross-artifact pass is where the seed lives — "rollback restored service [incident-timeline.md 14:15]" has a valid citation; the citation is to an artifact that is itself wrong. Your gate must compare the claim against metrics.csv 14:20/14:25 and rule the timeline note unreliable.
4. 4Interrogate, then correct. Ask the model: "incident-timeline.md says the rollback restored service; metrics.csv shows 12% errors for 10+ minutes after rollback. Which is right, and how should the postmortem state it?" Record whether it revises cleanly. Then produce postmortem-final.md with corrected causality (recovery follows the 14:27 job kill), the contradiction documented in "What went poorly" (mid-incident notes recorded wrong causality), and an action item to auto-generate timeline notes from the deploy log rather than memory.
5. 5Grade the action items. AI drafts reliably produce vague action items ("improve monitoring"). Rewrite every action item until it has an owner-role, a priority, and a verification step — how you'd prove it was done. Keep before/after.
6. 6Package the factory. Commit assemble.py, review-gate.md, both drafts, the verification log, and a README describing the flow: bundle → assemble → draft → gate → publish. The gate is publish-blocking: no sign-off, no postmortem.

Verification

• •
  python3 assemble.py bundle emits the template plus all five artifacts.
• •
  verification-log.md covers every factual claim in draft-v1 with an explicit check result.
• •
  The seeded contradiction is caught in the cross-artifact pass, with both citations (timeline 14:15 vs metrics 14:20–14:25) quoted.
• •
  postmortem-final.md attributes recovery to the job kill, keeps the rollback as a reasonable-but-ineffective action, and contains zero vague action items.
• •
  Gate sign-off line completed with claims-checked/corrected counts.

Completion artifact

The factory/ directory: pipeline code, gate checklist, draft-v1, verification log, final postmortem, README. Self-grading rubric (pass = 8/10):

• Contradiction catch (0–3): caught via the gate's cross-artifact pass with both citations = 3; caught only after the step-4 hint = 1.
• Gate completeness (0–2): every draft claim logged with a check result = 2.
• Causal correctness (0–2): final postmortem's recovery attribution matches the metrics = 2.
• Action-item quality (0–2): all items have owner-role + priority + verification step = 2.
• Reusability (0–1): README lets a teammate run the factory on a new bundle unaided = 1.

Stuck?

• •
  Draft has no citations: your model ignored the template — regenerate, and if it persists, prepend "Refuse to output any sentence you cannot cite."
• •
  Model flagged the conflict on its own: excellent — your gate still runs; record it as a model win and verify its resolution is the right one.
• •
  Unsure whether "rollback was a reasonable response" survives the correction: yes — blameless means the decision was sound given 14:11 knowledge; the postmortem separates decision quality from outcome.
• •
  Model waffles ("both sources have merit"): data beats notes; make the gate's rule explicit and override.

Client-safe variant

Postmortems are frequently client-visible deliverables, which raises the stakes twice: the bundle contains client telemetry (use approved tenancy or a local model, redact hostnames/customers before assembly — add a redaction step to assemble.py) and a published error becomes a contractual artifact. The verification gate maps directly onto services-company QA norms: the gate sign-off is your reviewer trail for the client, and the claims-checked count is a deliverable quality metric you can report.

Answer key (don't read before attempting)

The seeded contradiction: incident-timeline.md ("14:15 rolled back v183 — service restored by rollback") versus metrics.csv (error rate 11.9→12.1→12.0% at 14:15/14:20/14:25, recovery to 2.1% only at 14:30, after the 14:27 report-backfill kill in deploys.log). Correct resolution: the rollback had no measurable effect; recovery is attributable to the job kill; the timeline note recorded causality by expectation, not observation. A strong final postmortem also carries forward Lab 06's finding — the 13:52 DB warning routed to a low-priority channel — as the detection-gap action item, and flags "mid-incident human notes are an unreliable artifact class" as a process learning.