Stand Up a Pipeline-Failure Triage Copilot for Data On-Call

Tier: Core · Workflow area: Orchestration, ops & cost

This playbook pipes task logs plus DAG context to an LLM that returns a probable cause, similar past incidents, and a suggested next action, wired into your on-call flow. The copilot diagnoses; the human decides.

When to reach for this

• •
  A team with recurring overnight failures where the first 15 minutes of every incident is log spelunking.
• •
  On-call is shared across engineers who don't all know every pipeline — triage quality depends on who's holding the pager.
• •
  You already have decent failure alerting but the alert says only "task X failed", forcing a full context rebuild each time.

Not a fit if your failures are dominated by one known flaky dependency — fix that first; a copilot summarizing the same root cause nightly is expensive alerting.

The workflow

1. 1.

   Assemble the evidence bundle automatically. On task failure (Airflow 3 failure callback or a listener on the API), collect: the last ~200 lines of task logs (tail, not head), the task's operator and rendered params, the DAG-run context (logical date, upstream states, recent code change SHA), and — the differentiator — the last N similar failures from an incident table (same dag/task, error signature via a normalized hash of the exception line).

2. 2.

   Prompt for a structured diagnosis.

   Copy

   A data pipeline task failed. Diagnose it.
   
   Evidence:
   - dag: orders_pipeline, task: load_orders, logical_date: 2026-07-27
   - operator + rendered params: <attached>
   - log tail (200 lines): <attached>
   - upstream task states: all success
   - last deploy: 2026-07-26 21:14 UTC, diff summary: <attached>
   - similar past incidents (same error signature): <3 rows: date, root cause, fix>
   
   Return:
   1. The exact log line(s) you identify as the root error (quote them).
   2. Probable cause, with confidence, reasoning ONLY from the evidence above.
   3. Category: transient-infra | upstream-data | code-regression | config/permission | resource.
   4. Recommended next action: safe-to-retry / needs-fix (what) / escalate (to whom).
   5. If a past incident matches, cite it and say whether the same fix applies.
   If the log tail does not contain enough evidence, say what additional log or
   query you need — do not guess.
   

   Expected output: a five-part diagnosis in which every claim quotes an evidence line. The quoting requirement is what makes the summary checkable at 2 a.m.

3. 3.

   Deliver it where on-call already looks. Post the diagnosis into the alert thread (Slack/PagerDuty note) — not a new dashboard nobody opens. Include deep links: the failed task instance, the quoted log line, the cited past incident. Claude with an MCP connector to the Airflow API is the natural implementation shape (fetch logs and task state as tools); a plain webhook-driven script that calls the API works just as well.

4. 4.

   Keep actions human-gated, with one narrow exception. The copilot recommends; the engineer clicks retry. If — and only if — a failure class has a long, clean history of transient resolution (e.g. a specific warehouse connection reset), you may auto-retry that signature once, with the copilot's note attached. Everything else stays read-only: an LLM with rerun/clear permissions is an incident generator.

5. 5.

   Close the learning loop. After resolution, on-call marks the diagnosis right/wrong and records the actual root cause in the incident table — one Slack reaction plus a short form. That table is both your accuracy metric and the "similar past incidents" corpus that makes next month's diagnoses better. Review misdiagnoses monthly and patch the prompt or the evidence bundle (usually the bundle: missing upstream logs is the top cause of wrong answers).

Verify before you trust it

• •
  Replay 20 historical incidents through the copilot and score root-cause accuracy against the known resolutions before putting it in the alert path.
• •
  Every diagnosis in the pilot quotes log lines that actually exist in the bundle (spot-check for fabricated quotes).
• •
  The category → recommended-action mapping is conservative: no "safe-to-retry" on any code-regression or data-quality signature in the replay set.
• •
  Time-to-first-action measurably drops (compare mean acknowledge-to-action before/after over a few weeks) — otherwise it's decoration.
• •
  Log payloads sent to the model are scrubbed: no credentials, tokens, or row-level data values in the bundle (grep the bundle builder's output in tests).

Pitfalls

• Confident wrong answers become the anchor. A fluent misdiagnosis steers a tired engineer down the wrong path. The quote-your-evidence rule plus the accuracy feedback loop are the countermeasures — never skip them.
• Log noise drowns the signal. Sending 5,000 log lines gets you a summary of the noise. Tail aggressively and normalize known-noisy frameworks first.
• Secrets in logs. Connection strings and tokens leak into task logs constantly. Scrub before the bundle leaves your boundary — this is a hard gate, not a nice-to-have.
• Missing the real evidence. Failures whose cause lives upstream (bad source file, late partition) can't be diagnosed from the failing task's logs alone. Include upstream states and, where cheap, upstream log tails.
• Skipping the feedback table. Without right/wrong marking, accuracy is unmeasurable, the incident corpus never grows, and trust decays on the first bad week.

Tools

• Claude API (with MCP / Airflow REST tools)the diagnosis engine; tool access to logs and task state beats paste-the-logs for completeness.
• Airflow 3 API + failure callbacksthe trigger and evidence source; listener hooks give you failures without touching every DAG.
• PagerDuty / Slack (existing alert path)delivery surface; a diagnosis that isn't in the alert thread doesn't exist at 2 a.m.

Client-safe variant

Logs are the leakiest artifact in a client's stack — they routinely contain hostnames, table names, and stray data values. For client deployments, the bundle builder and scrubber run inside the client's environment, and the model endpoint is whatever they've approved (their cloud provider's hosted models or self-hosted). Start with a shadow-mode pilot: the copilot posts diagnoses to a private channel visible only to the delivery team, accuracy is scored for two weeks, and the client sees the accuracy report before any engineer-facing rollout. That sequencing turns "we bolted an LLM to your alerts" into a measured operational improvement with evidence.