AI-Augmented Drift Monitoring for Classical Models

Tier: Core · Workflow area: Serving & LLMOps

This playbook keeps detection fully deterministic — Evidently computing drift metrics on schedule — and adds an LLM only where language is the bottleneck: drafting the diagnosis narrative and a retrain recommendation from the computed numbers, for a human to verify and send.

When to use

• •
  Deployed classical models with (or needing) scheduled drift checks, where writing up "what changed and does it matter" is the toil.
• •
  Drift reports currently go unread because they are tables of KS-statistics without interpretation.
• •
  A monitoring write-up habit exists but is inconsistent across models and owners.

Detection itself stays statistical — do not ask an LLM to look at two distributions and decide whether they differ; that is what tests are for.

Prerequisites

• •
  A reference dataset (training or a stable window) and access to recent production inputs/predictions for at least one model.
• •
  pip install evidently mlflow — recent Evidently versions use from evidently import Report with presets from evidently.presets (older 0.4-era code imports from evidently.report/evidently.metric_preset; check your installed version).
• •
  Any LLM API (cheap tier is fine — this is a summarization task) or a local model.

The workflow

1. 1.

   Compute drift deterministically. A scheduled job runs an Evidently report — data-drift preset per feature (PSI/KS/Jensen-Shannon as configured), prediction-drift, and data-quality checks (nulls, new categories, range violations). Export the structured JSON, not just the HTML; log both to MLflow as artifacts of a "monitoring" run so history accumulates.

2. 2.

   Build the context package for the narrative. From the JSON, extract: features flagged, their drift scores and directions (means/quantiles before vs after), prediction distribution shift, data-quality failures, plus static context you author once per model — what each feature means in business terms, known seasonal patterns, last retrain date, and the model's performance-vs-drift sensitivity if you have it.

3. 3.

   Prompt for diagnosis, tightly scoped. The instruction that keeps this honest: use only the numbers provided; every claim must reference a specific metric; separate observation ("tenure PSI 0.31, mean up 14%") from hypothesis ("consistent with the March pricing change shifting the acquisition mix — flagged as hypothesis"); end with one of {no action / investigate / retrain recommended} and the specific evidence for it. Ask for a 10-line executive section plus a per-feature appendix.

4. 4.

   Verify like pb-01 taught you. Check every number in the draft against the JSON (they must match exactly — this is the highest-value 5 minutes), sanity-check hypotheses against what you know actually happened, then edit and send under your name. The LLM drafts; you diagnose.

5. 5.

   Wire the loop into alerting. Quiet weeks: report filed to MLflow, no ping. Threshold breaches: narrative posted to the team channel with the HTML report linked. Retrain recommendations: route into your pb-45 flow if you built it — the drift narrative becomes the retrain agent's justification input.

6. 6.

   Calibrate over a month. Track narrative errors caught in review (target: zero number errors after prompt iteration) and whether recommendations matched what you would have decided unaided.

Tool variants

• Evidently + MLflow (shown)fully OSS, self-hosted.
• NannyMLadds performance-estimation-without-labels; same narrative layer applies on its outputs.
• Warehouse-native checks (dbt tests, custom SQL) feeding the same context-package → narrative step, when your monitoring is SQL-first.

Verification

• •
  Every numeric claim in a sample narrative traced to the Evidently JSON — exact match, no rounding drift.
• •
  A synthetic-drift test (inject a shifted feature into "production" data) is detected by Evidently and correctly narrated end-to-end.
• •
  Hypotheses are visibly labeled as hypotheses in the output — checked across three reports.
• •
  A stakeholder who ignored the old reports reads the new one and can say what changed.

Pitfalls

• Letting the LLM see raw data instead of computed metrics. Costlier, slower, and invites invented statistics; the model gets aggregates only.
• Retrain-recommendation inflation. LLMs pattern-match "drift" to "retrain"; the prompt must require evidence linking drift to expected performance impact, and the human must push back — retraining on drifted-but-irrelevant features burns compute and review time.
• Static business context going stale. The feature-meaning notes you wrote in step 2 need an owner and a review cadence, or the narratives slowly diverge from reality.
• Skipping the number check "because it has been right for weeks." The one fabricated statistic will land in the report the CFO reads.

Restricted-environment variant

Evidently and MLflow are local; the only external touchpoint is the LLM call, and the context package contains aggregates, not records — often acceptable even under strict policies. Where it is not, a small local model (Ollama/vLLM) handles this summarization task well; it is among the least demanding LLM tasks in this track.

Success criteria

• •
  Scheduled drift monitoring with machine-readable outputs and accumulated history in MLflow.
• •
  Narratives with zero unverified numbers, clearly separated observations/hypotheses/recommendations.
• •
  Synthetic-drift drill passed end-to-end (detect → narrate → alert).
• •
  Monitoring write-up time per model cut from ~an hour to ~10 minutes of verification, measured.