AI for Data Quality & Incident Response

Data quality work has two halves — preventing bad data with tests and contracts, and responding when bad data arrives anyway — and AI now accelerates both. This module teaches the discipline that makes the acceleration safe: pruning generated test suites so they alert on problems instead of on Tuesdays, using LLMs over lineage for root-cause hypotheses you then confirm, and treating LLM-derived data itself as a quality risk with its own guardrails.

The scenario — the 34% drop

Tuesday, the long way. Meena is the data engineer on call for a marketplace's analytics platform — dbt on BigQuery, Airflow, 300 models, Elementary monitors.

The revenue email went out wrong, finance escalated, and Meena spent 96 minutes doing what she'd describe as "grep for data": mechanically intersecting an alert, a lineage graph, and a change log.

Tuesday, the short way. Same alert, 08:12, but it lands first in a triage pipeline the team built in an afternoon.

The LLM didn't decide anything: it read three systems she would have read herself, in seconds instead of 96 minutes, and every claim came with a query she could run. The difference between the two Tuesdays is not intelligence — Meena found the bug both times. It is that assembly became automatic and judgment kept the wheel.

Generated test suites you can trust — profile, propose, prune

Test generation is the easy half; here is what an LLM proposes after profiling a customers table, and why half of it must die:

# AI-proposed dbt tests on stg_customers — BEFORE pruning
columns:
  - name: customer_id
    tests: [not_null, unique]                        # KEEP — structural
  - name: signup_channel
    tests:
      - accepted_values:
          values: ['web', 'ios', 'android', 'referral']   # PRUNE — freezes today's list;
                                                          # breaks the day marketing adds 'whatsapp'
  - name: age
    tests:
      - dbt_utils.accepted_range: {min_value: 18, max_value: 87}  # PRUNE the 87 — that's
                                                                  # today's oldest customer, not a rule
  - name: referral_code
    tests: [not_null]                                # PRUNE — profiled 0.2% nulls are legitimate
  - name: created_at
    tests:
      - dbt_utils.accepted_range: {max_value: "current_timestamp"}  # KEEP — future dates are real bugs

The failure mode has a name: over-fitting to the profile. The LLM read a snapshot and encoded the snapshot as law — current max age, current channel list, current null rate — and every such test is a false alarm with a fuse. The pruning pass is a fixed set of questions applied to every proposed rule:

Run generation where your metadata lives — dbt Copilot proposes tests from project metadata, Great Expectations or Soda suites can be drafted by any assistant fed the profiling output — then prune in review, and record why each kept threshold is believed, because a test whose rationale nobody remembers is a test nobody dares delete. Aim to keep roughly the structural tests (keys, referential integrity, freshness, impossible values) and be ruthless with value-list and range tests, which is where the false-alarm fuses live.

Data contracts — drafting the agreement, generating the enforcement

Contracts formalize the other direction of quality: what downstream consumers may rely on. Here is the shape of a drafted contract clause set for fct_orders:

# Draft contract: fct_orders (LLM-drafted from model SQL + lineage + query logs)
grain: one row per order_id                    # enforced: unique test on order_id
guaranteed_columns:
  order_id: {type: string, nullable: false}
  order_ts: {type: timestamp, nullable: false,
             semantics: "UTC event time, NOT load time"}
  gross_amount: {type: numeric, nullable: false,
                 semantics: "pre-refund, order currency"}
freshness_sla: "complete through T-1 by 07:30 IST"
consumers: [finance_daily_revenue, ops_fulfillment_dash]   # from lineage
change_process: "breaking changes: 14-day notice via #data-contracts"

The drafting workflow starts from what exists — feed the LLM the model's SQL, its schema, its consumers (from lineage or the exposures file), and query logs showing how consumers actually use it — and it produces exactly this strawman, including the semantic clauses humans forget to write down. The draft's job is making the conversation with consumers concrete: finance reads gross_amount's semantics line and either nods or objects, and either outcome is progress. The conversation, as ever, is the human part.

Enforcement is then mechanical and generatable: dbt model contracts pin schema and types at build time, dbt tests cover the semantic promises, and a Soda or Great Expectations check covers freshness SLAs. The one clause worth insisting on in every contract, because it pays during incidents: a declared grain and a uniqueness test proving it — half of all downstream breakage is fan-out, and a contract that guarantees grain converts "the dashboard doubled" from an investigation into a test failure.

RCA with lineage context — hypotheses you can execute

Meena's four-minute triage brief is a small, buildable pipeline: gather context deterministically, let the model rank hypotheses, and require every hypothesis to arrive with its confirming query. The core of it, with structured output so the brief is machine-postable to Slack:

import anthropic
from pydantic import BaseModel

class Hypothesis(BaseModel):
    rank: int
    cause: str                 # one-sentence causal claim
    evidence: list[str]        # verbatim quotes from the provided context only
    confirming_query: str      # SQL the on-call runs to confirm or kill this
    confidence: str            # high / medium / low

class TriageBrief(BaseModel):
    incident_summary: str
    hypotheses: list[Hypothesis]
    blast_radius: list[str]    # affected downstream models/dashboards, from lineage

client = anthropic.Anthropic()

response = client.messages.parse(
    model="claude-opus-4-8",
    max_tokens=4000,
    thinking={"type": "adaptive"},
    messages=[{
        "role": "user",
        "content": f"""A data-quality alert fired. Produce a triage brief.

<alert>{alert_payload}</alert>
<lineage>{upstream_subgraph}</lineage>
<row_count_history>{counts_14d}</row_count_history>
<recent_changes>{merged_pr_diffs_48h}</recent_changes>

Rules: every hypothesis cites only the evidence above, verbatim. Every
hypothesis includes ONE read-only SQL query that would confirm or kill it.
Rank by likelihood. Do not propose fixes — diagnose only.""",
    }],
    output_format=TriageBrief,
)

brief = response.parsed_output
post_to_slack(format_brief(brief))

Three design rules carry the trust:

• Evidence must be verbatim quotes from supplied contextan uncited hypothesis is a guess with confidence formatting, and quoting makes fabrication checkable in seconds.
• Every hypothesis ships with a confirming querythe model ranks, the engineer executes; diagnosis is never accepted, only tested.
• Diagnose-onlythe fix decision stays with the human who owns the blast radius.

Context selection does the real work: alert, lineage subgraph (not the whole graph), metric history, and recent changes cover the overwhelming majority of data incidents, because most data incidents are recent changes meeting old assumptions.

You can buy this pattern instead of building it — Monte Carlo's troubleshooting agent does automated root-cause over its lineage, and Anomalo ships ML-native anomaly explanation — or run it free on the OSS path with Elementary supplying lineage and history to your own script. The pattern is identical in all three: assembly automated, judgment retained.

Monitor tuning — aim the smoke detectors

Alert fatigue is a data problem, and you have the data: months of alert history, each with monitor, timestamp, and what happened next. Export it and hand it to an LLM with a classification job, and the output looks like this:

Each monitor gets one of four labels, and the label carries the proposed change:

• Actionablealerts led to fixes. Keep it.
• Seasonalfires on the pattern the business calls Monday. Give it a seasonality-aware baseline.
• Deadnever fired, or never actioned. Run a "why does this exist" review.
• Miscalibratedfires often, actioned rarely. Widen the threshold or delete it.

The output is a tuning worklist, and the measure of success is precision: fraction of alerts that led to action, before versus after, per monitor. Track it monthly — Monte Carlo and Anomalo expose alert-history views that make the export trivial, and on the OSS path Elementary's artifacts tables hold the same history. A quality program whose alerts are 80% ignorable is not 80% effective; it is 0% effective on the day the real incident lands in the noise, which is the argument for treating monitor precision as a first-class SLO of the quality platform itself.

Synthetic test data — fixtures with teeth

Generated fixtures are the cheapest quality investment in this module: you describe the schema and the traps, the LLM writes the hostile dataset. The prompt shape:

Generate CSV fixtures for orders (25 rows), customers (12), payments (28).
Honor FKs: every order→customer, every payment→order; include 2 orphan
payments for negative tests. Include edge cases: names with Devanagari and
emoji; a customer with two payment currencies; amounts 0, negative, and
99999999.99; timestamps at 2026-03-31T23:59:59+05:30 and UTC midnight
month-end; an order created before its customer's signup_ts; nulls in every
nullable column at least once. Plausible values, deliberately hostile shape.

Wire the output into dbt unit-test fixtures or plain pytest fixtures, and keep it deterministic — fixtures are versioned artifacts, regenerated deliberately, not fresh randomness per CI run, or test failures stop meaning anything. Faker-plus-LLM covers volume when you need thousands of rows; the LLM's real edge over Faker is relational hostility — orphans, FK-honoring edge cases, cross-row contradictions like the order-before-signup — which is exactly what breaks pipelines and what random generators don't produce. And one rule that is compliance, not craft, DPDP-flavored for Indian teams but universal in substance: production PII never becomes test data, however masked — synthetic generation is the pattern because it starts from schema, not from rows.

Data-diff on PRs — review the data, not just the code

Code review shows what changed in the SQL; a data diff shows what changed in the rows, and the LLM's job is translating the second into reviewer language. The workflow: CI builds the changed models into a temp schema, a diff tool (Recce, or data-diff tooling in your dbt CI) compares against production, and an LLM summarizes the delta in semantic terms — "2.3% of rows change owner_id; all changed rows have region = 'APAC'; row count unchanged; two dashboards consume this column" — posted as a PR comment beside the code diff.

The summary turns an unreviewable wall of changed rows into a claim the author must own: if the PR intended to reassign APAC owners, the comment is confirmation; if not, it is the incident that didn't ship. Meena's inner-join bug from the scenario dies in exactly this stage — "row count −34%" on a PR titled "fix duplicate device rows" reads as wrong to every reviewer at a glance. Impact summaries belong on every PR touching marts; they are the cheapest incident prevention this module offers.

Incident comms and postmortems — drafted, then owned

During Meena's incident, two documents needed writing, and both are LLM-draftable from artifacts that already exist. The consumer-facing impact note, generated from the triage brief plus the fix PR: what broke, which dashboards and periods were affected, what was wrong, what to re-pull — drafted in thirty seconds, reviewed by the engineer, posted before the 10:00 email lands wrong twice. And the blameless postmortem, generated from the incident timeline (alert timestamps, Slack thread, PR history): chronology, contributing causes, detection gap, action items — with the LLM explicitly instructed to describe systems and process, never individual fault.

The discipline is the byline rule: the engineer edits and signs, because the LLM drafts fluently and specifically fluently-wrong — a postmortem that misstates the detection gap becomes organizational false memory with a professional tone. The draft removes the blank page, which is the reason most postmortems don't get written; the ownership keeps the document true, which is the reason they're worth writing.

Guardrails for LLM-derived data — the new quality frontier

The final section inverts the module: soon your pipelines will produce LLM-derived columns — classifications, extractions, enrichments — and everything above applies to them with the dial turned up, because LLM output is untrusted input wearing your schema. The guardrail stack, which M8 builds in full:

1. 1Schema validation at the boundarystructured outputs parsed against a declared model (the messages.parse pattern above); malformed output is rejected, never coerced.
2. 2Allowed-value enforcementclassification columns validate against the closed label set; the classify-don't-generate rule means the model selects, and anything outside the list is a rejection, not a new category.
3. 3Confidence thresholds with a human-review queuelow-confidence rows route to review, not to the mart; the threshold is tuned on a hand-labeled sample, not chosen by feel.
4. 4Distribution-drift monitors on the outputlabel proportions tracked like any metric, because silent model or prompt changes shift distributions long before anyone reads individual rows.
5. 5A sampled accuracy audit before any backfillmeasure the misclassification rate on a labeled sample first; backfilling millions of rows and then measuring is the expensive order of operations.

Points 4 and 5 are this module's monitors and pruning discipline pointed at a new source; Great Expectations or Soda on the output tables, plus your anomaly monitors on label distributions, need no new infrastructure. What's genuinely new is epistemic: a failed not_null test means a bug, but a shifted label distribution means either the world changed or the model drifted — and only the sampled audit tells you which.

Check yourself: An assistant profiles a table and proposes forty dbt tests. Why must roughly half of them be pruned, and what is the single question that decides each one?

Where this lands in your week

• This weekrun test generation against your three most-consumed models and do the pruning pass with the four questions; the deliverable is a leaner suite plus a written rationale per kept threshold.
• Within a monthstand up the triage pipeline (an afternoon against Elementary or your observability platform's API; the free path needs nothing but lineage, history, and PR diffs) and let it write the first brief of your next incident. Add the data-diff summary to CI for mart-touching PRs.
• Ongoingalert-history review quarterly with monitor precision as the score, fixtures regenerated when schemas change, and postmortems drafted from timelines while the incident channel is still warm.

The habits compound into one identity shift: the quality platform stops being a pile of checks and becomes an operated system with its own SLOs.

This module anchors LAB-03 — Data-quality copilot: profile a deliberately dirty dataset, have AI propose a Great Expectations/dbt test suite, prune the over-fitted rules, then use an LLM to triage and explain a simulated freshness-and-volume incident end to end. The artifact — the pruned suite plus an incident RCA doc with ranked hypotheses and confirming queries — is Meena's Tuesday, reproduced and documented, and the strongest interview evidence this module produces: anyone can say "I use AI for data quality"; the lab shows you pruning what it over-proposed and confirming what it hypothesized. Do this at work this week → pb-10 Generate a Data Test Suite from Profiling, Then Prune It, pb-11 Triage a Data Incident with AI: Root Cause, Comms, and Monitor Tuning, and pb-12 Add Data-Diff Summaries to Every Transformation PR.