A failure plus its surrounding evidence — app logs, API traces, recent commits — usually contains the root cause; what it lacks is someone with time to correlate three data sources at 9 a.m. AI does the correlation and drafts ranked hypotheses; the discipline is demanding each one be phrased so it can be verified, not just believed.
Tier: Core · Workflow area: Triage, flakes & defect intelligence
This playbook takes one confirmed failure, feeds the assistant a correlated evidence bundle, and produces a ranked hypothesis list where every entry carries its supporting quotes and a concrete verification step — then works the list top-down until one hypothesis survives testing.
git log for the involved services.Cut the evidence bundle to the window. Gather, per source, only the failure's time slice:
# app logs ±10 min around the failure
aws logs filter-log-events --log-group-name /app/checkout \
--start-time 1754350200000 --end-time 1754351400000 > /tmp/app.log
# recent commits to the suspect services
git -C services/pricing log --oneline --since="3 days ago" > /tmp/commits.txt
# the failing request's trace (OTel/Jaeger export) -> /tmp/trace.json
Small, windowed bundles beat firehoses: the model correlates better and hallucinates less when everything in context is plausibly relevant.
Prompt for hypotheses with the verifiability contract. This prompt's rules are the whole playbook:
FAILURE: checkout E2E "applies loyalty discount" fails since 03:10 UTC —
total is 100.00, expected 90.00. Repro is deterministic on staging.
ATTACHED: app log slice, trace of one failing request, 3 days of commits
for pricing-svc and checkout-api.
Produce up to 4 root-cause hypotheses, ranked by likelihood. EACH must have:
- CLAIM: one sentence naming a specific mechanism (component + change + effect)
- EVIDENCE: verbatim quotes from the attached material supporting it
- AGAINST: any attached evidence that cuts against it (say "none found" honestly)
- VERIFY: one concrete step — a command, query, or experiment — whose outcome
would confirm or kill this hypothesis, and what result means which.
Rules: no hypothesis without quoted evidence. If the evidence is insufficient
for any confident hypothesis, say what additional data to pull instead.
Read the output as a skeptic — check quotes first. Verify each quoted line exists in the real evidence (models paraphrase, and a paraphrase can smuggle in the conclusion). A well-formed entry looks like:
H1 (likely) — pricing-svc commit 4f2c1a9 "cache discount rules per-tenant"
deployed 02:58 serves a stale empty ruleset for tenant=staging-01.
EVIDENCE: trace span pricing.getDiscounts returns [] at 03:12 (trace.json L214);
app log 02:58 "pricing-svc v3.4.0 rollout complete"; commit touches
DiscountRuleCache.java.
AGAINST: none found in the attached window.
VERIFY: curl pricing-svc /internal/cache/discount-rules?tenant=staging-01 —
empty array = confirmed; populated = kill this, promote H2.
Reject entries whose VERIFY step is untestable ("the logic may be wrong somewhere in the cache layer") — send them back for sharpening: "rewrite H3 so a single command can kill it."
Work the list top-down, running each VERIFY step. This is the human half of the correlation: execute the check, record the outcome, and either confirm or strike. Discipline point: run the verify step even when H1 "obviously" fits — the ranked list's value is destroyed the first time a team ships a fix for a hypothesis nobody tested.
Route the confirmed cause with its evidence chain. The bug report (or the fix PR) gets the confirmed hypothesis verbatim: claim, quotes, verification result. That chain is what turns "QA thinks it's the pricing cache" into "the pricing cache is empty for this tenant — here is the curl output" — the difference between a ping-pong ticket and a same-day fix.
When all hypotheses die, escalate the gap, not a guess. The model's honest "insufficient evidence" branch, or a fully-struck list, tells you which data source is missing (no DB slow-query log? no feature-flag audit trail?). Pull it, re-bundle, re-run. Two rounds without a survivor usually means the evidence window is wrong — widen the time slice; intermittent causes start earlier than symptoms.
The deliverable is not an answer — it is a falsifiable list. Every hypothesis carries its own execution: quoted evidence in, one decisive check out. A hypothesis you cannot phrase as "run X; result Y confirms, result Z kills" is not a hypothesis yet, and it does not leave your desk.
Logs and traces are the most PII-dense artifacts in this track — user IDs, emails, tokens in URLs. Scrub bundles with the masking pipeline before any external model sees them, or run the correlation on a client-approved/local model; the verifiability contract survives model downgrades well because quotes and checks carry the rigor, not eloquence. Commit messages are usually safe; commit diffs may embed secrets — prefer --oneline logs plus filenames unless the model truly needs the code.
Keep a small file of your confirmed hypothesis chains — claim, evidence, verify, outcome. Pasting two past chains into future prompts as few-shot examples teaches the model your bar for "verifiable" faster than any rules paragraph, and the file doubles as the strongest possible artifact for the next postmortem.