Validate AI Anomaly Detections Before You Act on Them

Tier: Core · Hands-on time: ~40 minutes

The detection is statistical ("this series deviates from learned patterns"); the explanation is narrative ("likely caused by the 14:20 deploy"). Those are different products with different failure modes, and this playbook is the validation pass that keeps you from paging a team — or rolling back a deploy — over seasonality the model hasn't learned or a narrative the LLM invented. The economics are asymmetric: every false positive you act on taxes an engineer and erodes trust in the whole detector.

When to use

• •
  An anomaly alert fires from Watchdog/Grafana ML/Elastic AIOps and you must decide: act, watch, or dismiss.
• •
  You are tuning anomaly monitors and choosing which series deserve ML detection versus plain thresholds.
• •
  An AI investigator (pb-15) cites "anomalous metric X" as evidence and you need to know if the anomaly is real.

Skip ML anomaly detection entirely for metrics with hard correctness bounds (disk full, cert expiry, queue depth limits) — static thresholds are more accurate, cheaper, and explainable there by construction.

Prerequisites

• •
  An anomaly-detection source: Datadog Watchdog, Grafana ML jobs on a Prometheus stack (free-tier viable), or Elastic AIOps — or, for the lab, Grafana ML pointed at synthetic data where you control ground truth.
• •
  Access to the underlying series and your deploy/change event stream on the same time axis (the validation is mostly overlay work).
• •
  Knowledge of the metric's business rhythm: weekly cycles, batch jobs, marketing spikes, month-end (in India-serving systems: festival-season and salary-day patterns the model may not have enough history to learn).

The workflow

1. 1.

   Separate detection from narrative. Read the anomaly card and split it: what did the detector flag (series, window, deviation direction/magnitude) versus what did the explainer claim (cause, correlated events). Write both down; you validate them independently.

2. 2.

   Validate the detection against context the model may lack. Overlay the flagged window with: (a) the same window last week/month (seasonality check); (b) deploy/config-change events; (c) upstream dependency metrics; (d) known calendar effects (sale events, batch windows, festival traffic). Ask: would an engineer who knows this system call this abnormal? A deviation that recurs every Monday at 02:00 is a cron job, not an anomaly — regardless of model confidence.

3. 3.

   Validate the explanation like a citation. For every causal claim in the AI explanation ("correlates with deploy d41f", "driven by region ap-south-1"): check the referenced event exists, its timestamp actually precedes the deviation, and the correlation holds when you slice the metric by the claimed dimension. An explanation that names a real deploy which happened after the anomaly started is confidently, checkably wrong — this is pb-15's citation discipline applied to metrics.

4. 4.

   Decide with the false-positive economics explicit. Act (page/rollback) only when detection AND at least one causal claim validate. Watch (snooze with a re-fire condition) when detection validates but cause is unknown. Dismiss — and record why — when seasonality or data artifacts explain it. The recorded dismissals are your tuning dataset.

5. 5.

   Tune the detector with your validation log. Weekly or after ~10 events, review the log: which series produce validated anomalies (keep), which produce recurring seasonal false positives (extend training window, add seasonality hints, or demote to threshold alerts), which never fire usefully (remove — every low-value detector dilutes attention). In Datadog, this maps to Watchdog impact settings and monitor scoping; in Grafana ML, to job hyperparameters and holiday calendars.

Tool variants

• Datadog Watchdog (Toto-powered): richest auto-explanations; use its "related events" links as claims to verify, not conclusions to forward. Watchdog Impact/monitor scoping is where dismissal learnings go.
• Grafana ML + Prometheus (lab default): free-tier trainable jobs; you can generate synthetic series with planted anomalies and planted seasonality to practice the full validate/dismiss/tune loop with known ground truth.
• Elastic AIOps: anomaly jobs with influencer analysis — influencers are the same "slice by claimed dimension" check built in; still verify event timing yourself.
• DIY (LLM explains, stats detect): a z-score/ESD detector plus an LLM prompted to explain flagged windows from a bundle of deploy events and neighboring metrics — cheapest path, and it makes the detection/narrative split impossible to forget because you built the seam.

Verification

• •
  For every acted-on anomaly: the causal claim's cited event was opened, timestamp order confirmed (cause precedes effect), and the correlation reproduced by slicing the actual metric — the same open-the-citation standard as pb-15, with timestamps as the load-bearing check.
• •
  Seasonality overlay performed (same-window-last-period) before any dismissal or action; screenshot or query link attached to the decision record.
• •
  On the synthetic lab: the workflow correctly classifies at least 8 of 10 planted events (5 real anomalies, 5 seasonal decoys) — run this before trusting your judgment on production cards.
• •
  Your dismissal log exists, and at least one detector configuration change traces to it.

Pitfalls

• Confidence-score transfer. A 95% detection confidence says nothing about the explanation's accuracy — the narrative is generated separately and inherits none of the statistical rigor. Validate them separately, always.
• Insufficient history for local patterns. Foundation models trained on generic telemetry may not encode YOUR month-end batch or Diwali traffic. New metrics and young accounts produce the most false positives; weight your calendar knowledge higher there.
• Dismissal without recording. Silent snoozes teach the system nothing and hide the false-positive rate that should be driving tuning. The log is the deliverable.
• Acting on anomaly-of-a-proxy. The detector flags request count dropping; the harm hypothesis is revenue. Check the business metric directly before invasive action — proxies deviate for benign reasons (client retries stopping, a scraper leaving).

Restricted-environment variant

• •
  Validation is in-tenant by nature: overlays, slicing, and event checks all run in the client's observability stack with no data egress — the full discipline works under any MSA.
• •
  The DIY variant fits restricted shops best: statistical detection in Prometheus recording rules, explanation via a local model (Ollama) fed a redacted event bundle. Metric names and deploy IDs stay inside the boundary.
• •
  If a client's Datadog/Elastic tenancy includes AI explanations, confirm their AI feature flags and data-processing addenda are approved by the client before enabling — vendor AI features can route telemetry to model endpoints the client hasn't reviewed.

Success criteria

• •
  Synthetic-lab score of 8/10 or better on planted anomalies vs. seasonal decoys before you adjudicate production cards.
• •
  Every acted-on anomaly in the last month has a decision record showing seasonal overlay plus a timestamp-verified causal claim.
• •
  Measured false-positive-action rate (acted, later proven benign) trending down; at least one noisy detector retired or demoted with the dismissal log as justification.
• •
  You can explain to a stakeholder, in plain language, why the detector flagged something AND why you did or didn't believe it — two separate sentences.