A freshness alert fires, a volume check fails, a dashboard number looks wrong. The slow parts of what follows — correlating lineage with recent changes, forming hypotheses, writing the consumer note nobody wants to write at 2 a.m. — are exactly where an LLM compresses hours into minutes, provided it proposes and you confirm.
This playbook runs one incident end to end: AI-ranked hypotheses with confirming queries, human-verified root cause, drafted communications, and the post-incident monitor tuning that stops the same alert from crying wolf.
Use this the moment a data-quality alert or consumer report lands and the cause is not already obvious. It assumes you can pull lineage for the affected asset and a list of recent changes (deploys, DAG runs, upstream loads).
Triggers:
Assemble the evidence pack — mechanically, not selectively. Alert details (metric, expected vs observed, when it started), upstream lineage two hops back, last 72 hours of changes (merged PRs, DAG run outcomes, upstream job timings, schema changes), and one query's worth of the symptom. Resist forming a theory while collecting; a pre-formed theory shapes the pack and blinds the model the same way it blinds you.
Ask for ranked hypotheses with disconfirming queries.
Data incident. Evidence below: alert, lineage, recent changes, symptom.
Produce a ranked list of root-cause hypotheses. For each:
- mechanism: the causal chain from trigger to observed symptom
- which evidence supports it and which evidence argues against it
- one cheap SQL query that would CONFIRM or ELIMINATE it (DuckDB/ANSI)
- what else would be broken right now if this hypothesis were true.
Rank by likelihood x cheapness-to-test. Do not commit to one cause;
include one "outside the evidence pack" hypothesis.
[paste evidence pack]
The "what else would be broken" line is the quiet workhorse — it turns each hypothesis into extra checkable predictions.
Run the queries top-down and verify inline. Execute each hypothesis's test before reading the next explanation. Typical shape:
-- H1: upstream orders load ran but loaded partial data
SELECT CAST(loaded_at AS DATE) AS load_date, COUNT(*) AS rows_loaded
FROM raw_orders
WHERE loaded_at >= CURRENT_DATE - INTERVAL 7 DAY
GROUP BY 1 ORDER BY 1;
Paste results back — eliminations matter as much as confirmations, and the model re-ranks usefully when told "H1 eliminated, counts normal." Stop when one hypothesis is confirmed by its query and its side-predictions check out.
Confirm blast radius from lineage, then draft the comms. Two artifacts, two audiences, one prompt each:
Draft a consumer-facing incident note from the confirmed facts below.
Audience: analysts and dashboard owners. Include: what is affected
(tables/dashboards), the impact window, what is NOT affected, current
status, next update time. Under 150 words, no speculation, no jargon,
no root-cause detail beyond one plain sentence.
Facts: [confirmed cause, affected assets from lineage, timeline]
Fact-check every sentence against your confirmed evidence before sending — you own the accuracy, the model owns the typing. After resolution, same pattern for the blameless postmortem: paste the timeline and confirmed cause, require a contributing-factors section and concrete action items, and strip anything that names-and-blames.
Close the loop on the monitor itself. Every incident says something about your alerting: this alert fired late, fired for the wrong reason, or a noisy neighbor had trained everyone to ignore the channel. Feed alert history to the assistant:
Below: 90 days of alert history for this pipeline's monitors (monitor,
fired_at, resolved_as: real/noise/unknown). Classify each monitor:
KEEP (catches real issues), TUNE (propose a threshold or seasonality
adjustment, with reasoning from the history), or RETIRE (noise only).
For TUNE proposals, show what the last 90 days would have looked like
under the new threshold: how many real incidents kept, how many false
alarms removed.
Replay-based proposals ("under the new threshold: 9 of 9 real incidents kept, 31 of 34 false alarms removed") are reviewable; vibes-based threshold changes are not. Apply tuning as a reviewed PR, and note it in the postmortem.
An LLM will produce a fluent, confident root-cause narrative from thin evidence — that fluency is the hazard. Nothing in the incident record, and nothing sent to consumers, rests on an unexecuted query.
Incident evidence packs contain schema names, row counts, and occasionally sample rows — treat them at the same sensitivity as the data. On client work, run triage through Bedrock/Azure OpenAI in the client's tenancy (most client incident processes already require in-boundary tooling), or a local Ollama model for the hypothesis and drafting steps. Strip sample rows down to shapes ("amount column, 4% of rows now NULL, previously 0%") before prompting, keep all confirming queries inside the client warehouse, and route consumer comms through the client's own incident channels — the drafted note is input to their process, never sent by you or by any tool on their behalf.