Every AI observability answer is a hypothesis about your telemetry, not a fact about your system. This module installs the ninety-second sanity check that turns generated queries, anomaly narratives and alert-tuning proposals into numbers you can defend in front of an exec.
Divya needs a number before the 9 a.m. exec sync: is checkout-api p95 latency actually up, or is the dashboard lying? Priya asks the observability assistant, and in ten seconds it hands back a clean-looking PromQL query — avg(histogram_quantile(0.95, rate(...))) — that renders a reassuring flat line just under target. She almost drops the screenshot into the deck.
Then the ninety-second habit kicks in: she runs the bare selector first. It returns series, good. But the number feels too calm for a morning the on-call channel spent complaining. She reads the query again and sees it — the avg() wrapped around the quantile is averaging p95s across every pod, a pod serving three requests a minute weighted equally with one serving three thousand. Averaging percentiles is statistically meaningless; the "p95" on the slide is a number that moves when latency moves but is nobody's actual p95.
She rewrites it to aggregate the histogram buckets first, and the real p95 lands 40% over target. The exec sync got a true number, and a healthy service was neither paged for a fake problem nor waved through with a real one hidden. Both queries returned a number; both moved when latency moved; only one of them was the p95.
Observability was the first corner of ops that AI genuinely entered: Honeycomb shipped its Query Assistant back in May 2023, and within eighteen months Grafana, Datadog, and Elastic had all shipped natural-language assistants of their own. There's a structural reason it happened here first: query languages are a perfect LLM target — small, formal, well-documented — and a wrong query is (usually) harmless: it returns a chart, not an outage. But "usually harmless" hides the real risk. A plausible-looking wrong query produces a wrong number that you then act on — you page someone, roll back a healthy deploy, or tell an exec the error rate tripled when actually your query stopped matching the metric. The blast radius of observability AI isn't the query; it's the decision made on its output.
So this module applies the track's running theme — verification gates everywhere — to four workflows: natural-language query generation with a sanity-check discipline that catches the wrong-but-plausible query; anomaly explanations that you validate against seasonality and deploy history before acting; alert-noise tuning shipped as measured PRs with before/after precision, never auto-applied; and the unglamorous prerequisite underneath all of it — OpenTelemetry data quality — because every AI ops tool is bounded by the telemetry you feed it.
An AI-generated query is a hypothesis about your telemetry, not an answer about your system. The skill this module teaches is the sanity-check discipline that converts one into the other — and it takes about ninety seconds per query once it's habit.
Every major observability platform now has this feature — Grafana Assistant, Honeycomb Query Assistant, Datadog's Bits assistant — and your coding assistant will happily write PromQL, LogQL, or KQL from a plain-English question too. For an experienced engineer the value isn't that you can't write these queries; it's that the assistant writes in ten seconds the query you'd spend ten minutes assembling from half-remembered function signatures and the metric explorer. The economics are real. So is the failure mode: these languages are full of constructions that are syntactically valid, semantically plausible, and quietly wrong. The discipline is knowing the trap catalogue for each language you use, because generated queries fall into the same traps human learners do — with more confidence.
Ask for "p95 latency for the checkout service over the last hour" and here is a query an assistant will plausibly produce — the one Priya nearly shipped to the exec sync:
# WRONG — quantile computed per-series, then averaged
avg(histogram_quantile(0.95,
rate(http_request_duration_seconds_bucket{service="checkout"}[5m])))
This computes a p95 per time series (per pod, per instance, per label combination) and then averages those percentiles. Averaging percentiles is statistically meaningless — a pod serving 3 requests/min counts the same as one serving 3,000. The correct construction aggregates the histogram buckets first, preserving le, and takes the quantile of the merged distribution:
# RIGHT — merge buckets across series, then take the quantile
histogram_quantile(0.95,
sum by (le) (rate(http_request_duration_seconds_bucket{service="checkout"}[5m])))
Both queries return a number. Both numbers move when latency moves. Only one of them is your p95. Other PromQL traps worth having in your review reflex:
| Trap | What the generated query does | Why it's wrong | Fix |
|---|---|---|---|
Missing rate() on a counter | sum(http_requests_total) | Returns the cumulative total since process start — a huge, meaningless, ever-growing number | sum(rate(http_requests_total[5m])) |
rate() on a gauge | rate(node_memory_available_bytes[5m]) | Rate of a gauge measures its derivative, and counter-reset logic corrupts it on any decrease | avg_over_time(...) or delta(...) for gauges |
| Sum of ratios instead of ratio of sums | avg(rate(errors[5m]) / rate(total[5m])) per pod, averaged | Weighs an idle pod's 100% error rate equally with a busy pod's 0.1% | sum(rate(errors[5m])) / sum(rate(total[5m])) |
| Silently-empty selector | Label value guessed wrong (service="checkout" vs app="checkout-svc") | Empty result renders as a clean flat chart — reads as "no errors" | Verify the selector matches series: run the bare selector first |
| Range/step mismatch | [1m] range with 15s scrape interval on a spiky counter | Too few samples per window; jagged, aliased rates | Range at least 4x the scrape interval |
The silently-empty selector deserves special fear. Models guess label names from convention — service, app, job — and your estate uses whichever one your platform team standardized in 2021. An empty selector doesn't error; it charts as a serene flat line at zero. If the question was "are we seeing errors?", the wrong label name answers "no" with total confidence.
The most dangerous generated query is not the one that errors — it's the one that returns a plausible-looking result from the wrong data. Empty selectors, averaged percentiles, and counters without rate() all produce charts that look like answers.
Ask for "error count per pod for the checkout app over 5 minutes" and the plausible-but-wrong version is:
# WRONG-ish — substring match catches "error_rate=0", "no error", "ErrorHandler"
sum by (pod) (count_over_time({app="checkout"} |= "error" [5m]))
The line filter |= "error" matches the substring anywhere: log lines like error_count=0, ErrorBoundary rendered, or a request path /errors/export all count. If your services emit structured JSON logs — and after the OTel section of this module, they will — parse and filter on the actual severity field:
# RIGHT — parse JSON, filter on the level field
sum by (pod) (count_over_time({app="checkout"} | json | level="error" [5m]))
Second LogQL trap: the stream-selector/parsed-label distinction. {app="checkout"} selects indexed streams cheaply; | json | level="error" filters after parsing, at query cost. Generated queries sometimes put high-cardinality parsed fields into the stream selector (where they don't exist, returning nothing) or scan enormous stream sets with a bare parser (timing out your Loki queriers). Review both halves: is the selector the cheapest one that bounds the data, and is the filter operating on a field that actually exists in your log schema?
Azure-side, the same class of trap wears different syntax. "P95 request duration over the last hour, five-minute resolution":
// WRONG — averaging per-bin percentiles when later summarized further
requests
| where timestamp > ago(1h)
| summarize p95 = percentile(duration, 95) by bin(timestamp, 5m)
| summarize avg(p95) // an average of percentiles again — meaningless
The per-bin percentile chart itself is fine; the trap is any subsequent aggregation of it. If you want the hour's p95, compute it over the hour. And on Application Insights specifically, generated queries routinely forget sampling: when ingestion sampling is on, each stored row represents itemCount original telemetry items, so counts must be sum(itemCount), not count() — a query that forgets this undercounts by your sampling ratio and the chart still looks perfectly sensible.
Regardless of language, run every generated query through the same five checks before its output informs any decision. This is the ninety-second gate:
search/label-values equivalent) and confirm it returns the series or streams you expect — right service, right environment, non-empty. Empty is a finding, not an answer.rate().Keep a "trap file" — every wrong generated query you catch, with the correction and one line on why. It becomes your team's query-review checklist, the few-shot examples you paste into future query prompts, and the best interview-prep artifact this module produces.
The same generation capability extends to dashboards (UC-28): given a service's metadata — its OTel semantic conventions, its RED/USE metrics, its SLOs — an assistant can draft the Grafana dashboard JSON, Jsonnet, or Terraform grafana_dashboard resource. The gate that makes this trustworthy is the one that makes IaC trustworthy in M2: the dashboard is code in a repo, reviewed as a diff, not hand-edited panels in a UI. Every panel's query goes through the five-check discipline once, at review time; after that, drift is visible in version control.
The second AI layer in modern observability platforms is anomaly detection plus explanation — Datadog's Watchdog (powered by its Toto time-series foundation model), Grafana ML's forecasting and outlier detection, Elastic's AIOps features. Understand the architecture, because it determines where verification belongs: the detection is statistical ML over time-series (good at "this metric deviates from its learned pattern"), and the explanation is increasingly an LLM narrative over the detection plus correlated context ("checkout latency is elevated, coinciding with deploy X and elevated Redis connection errors").
These are different trust objects. The detection has false positives with known causes; the explanation has hallucinations with unknown causes. Validate them separately.
Validating the detection. Before treating an anomaly as real, check the three innocent explanations that generate most false positives:
Validating the explanation. LLM anomaly narratives are correlation engines with confident prose. "Latency increased due to the 14:02 deploy of checkout-service" means the model found two events near each other in time. Treat it like an AI incident hypothesis in M3: extract the causal claim and run the cheapest disconfirming check — did latency rise on pods running the old version too? In regions the deploy hadn't reached? A verified narrative is a head start; an unverified one forwarded to an incident channel is how AI loses its production privileges.
There is also an economics discipline here. Every anomaly alert routed to a human costs attention; a detector tuned to catch everything buries the on-call in explanations of nothing. Track anomaly-alert precision the same way the next section tracks alert precision — what fraction led to action? — and tune sensitivity accordingly.
Anomaly detection tells you a metric deviated; it cannot tell you whether the deviation matters. Seasonality, change events, and telemetry artifacts explain most detections — check all three before the narrative reaches a human decision.
Alert fatigue is the oldest problem in ops, and it's where the 76%-resistance signal from the 2025 Stack Overflow survey (developers who won't use AI for deployment/monitoring) should shape your design most directly: alert-rule changes touch what wakes humans up, so AI proposes, humans merge, and nothing auto-applies. What AI changes is the economics of the analysis — mining ninety days of firing history for patterns used to be a quarter-long project proposed at every retro; it's now an afternoon.
The loop, end to end:
for: duration from 2m to 10m for a rule whose sub-10-minute firings were never actionable); grouping and inhibition (Alertmanager group_by and inhibit_rules so that a node failure doesn't page once per affected pod, and a parent alert suppresses its symptoms); and deletion or demotion (rules whose precision rounds to zero get deleted, or demoted from paging to a ticket/dashboard tier). Deletion needs the strongest human review: a rule that never fired actionably might still be the tripwire for a rare disaster, and firing history can't see events that haven't happened. This judgment doesn't get delegated.for: duration; 0 of those were actioned"). Your alert rules are in git — Prometheus rule files, Terraform-managed monitors — precisely so this review can happen as code review.Scenario questions on AI alert tuning score the workflow, not the tooling: mine firing history, compute precision, propose minimal changes as human-reviewed PRs, measure before/after including missed detections. Any answer where AI applies threshold changes directly to production monitoring is a wrong answer in this track.
Adjacent and easier (UC-30): generating parsing rules — grok patterns, Vector VRL transforms, ingest pipelines — from sample log lines. This is the rare observability task with a complete mechanical gate: parsing either produces the right fields from the sample corpus or it doesn't. Paste 20–30 representative log lines (including the ugly ones — multiline stack traces, embedded JSON, the legacy service that logs differently), have the assistant draft the rules plus unit tests (Vector's vector test, or your pipeline's equivalent), and iterate until tests pass. The tests live in the repo with the config; future log-format drift breaks a test instead of silently producing _grokparsefailure fields at 2 a.m. When a task offers a deterministic gate this cheap, take it.
Here is the dependency that makes or breaks everything above, and every AI-SRE product you will ever evaluate: anomaly-detection quality is bounded by telemetry quality. Practitioner guidance around AI SRE tooling is blunt about OTel compliance being a prerequisite — an AI investigator can only correlate what's instrumented, and only across services if they describe the world consistently. If checkout emits http.response.status_code and payments emits a hand-rolled status, no model bridges that gap; it just fails to find the correlation and you conclude the tool is dumb. This reframes instrumentation work (UC-31): not a vendor chore, but the data engineering that determines whether the AI layer you're paying for can function. Three practices:
http.request.method, db.system.name, k8s.pod.name, resource attributes like service.name and deployment.environment.name. Assistants are genuinely good at generating semconv-following instrumentation, because the conventions are well-documented public text — ask for "spans for this handler following current OTel HTTP semantic conventions" and review against the spec. The failure mode to catch is invented attribute names that look conventional (http.status instead of http.response.status_code) — the same plausible-but-wrong pattern as generated queries, and grep-able in review.user.id, session.id, and ID-bearing request paths as metric attributes, and each unbounded attribute multiplies your time-series count and your bill. Review with the cardinality question explicit: bounded attributes (method, status code, region) can be metric labels; unbounded values (user ID, full URL) belong on spans and logs only. The Collector's transform processors are the enforcement point; M6 picks up the cost side.When an AI observability tool underperforms, audit the telemetry before blaming the model — inconsistent service names, missing deploy markers, broken trace propagation, and non-semconv attributes are the usual culprits, and they're your platform's job to fix, not the vendor's.
Closing the loop (UC-32): SLOs make everything above principled — alert tuning gets a target (page on error-budget burn, not raw symptoms), anomaly triage gets a materiality test (does this threaten the SLO?), and dashboards get a spine. The classic barrier is that writing good SLOs is a blank-page problem requiring both service context and SLO literacy. AI collapses the blank page: give an assistant the service's description, RED metrics, dependencies, and user journeys, and ask for candidate SLIs, threshold options with trade-off notes, and a draft error-budget policy (what happens at 50% budget consumed, at exhaustion — who slows releases, who gets paged).
The draft is a workshop input, not a decision. Thresholds encode business judgment — is 99.5% checkout availability acceptable? — and error-budget policies allocate organizational consequences; both belong to the team and its stakeholders. The draft makes the workshop concrete (arguing about a written 99.5% beats arguing about vibes), and tooling like Sloth or OpenSLO turns the agreed result into recording rules and burn-rate alerts — as code, in the repo, reviewed like everything else in this module. Multi-window burn-rate alerts generated from an SLO spec are, satisfyingly, the principled replacement for many of the noisy threshold alerts your firing-history analysis flagged for deletion.
Your first 15 minutes: the next time an assistant writes you a query, run the bare selector before its number informs any decision or reaches a slide — empty is a finding, not an answer. That one reflex is the front of the ninety-second gate, and it is what stood between Priya and a wrong number in the exec deck.
Check yourself: An assistant hands you a PromQL query that runs without error and returns a clean, plausible-looking chart. Before that number informs any decision, what is the first check you run — and why is "it returned a chart" the most dangerous kind of pass?
rate(), substring log matching, wrong label names. Deliverable: the wrong query, the corrected query, and which check caught it.This module is backed by one hands-on lab (lab-11): NL-to-query + alert tuning — use an assistant to generate PromQL answering ten questions against a demo Prometheus, catching the seeded traps with the sanity-check discipline, then analyze provided alert-firing history and ship three noise-reduction changes as PRs with measured before/after precision. The artifact is a query notebook plus an alert-tuning PR with metrics. Do this at work this week → pb-22 Natural-Language to PromQL/LogQL with Sanity Checks (Priya's ninety-second gate, as a template), pb-23 Validate AI Anomaly Detections Before You Act on Them (seasonality / change event / telemetry artifact), and pb-25 Alert-Noise Reduction as Measured, Reviewed PRs (mine history, compute precision, ship diffs). The accompanying playbooks give you copy-adaptable templates for the five-step query sanity check, the wrong-query trap catalogue per language, the anomaly-validation triage (seasonality / change event / telemetry artifact), the alert-precision analysis prompt, and the OTel instrumentation review checklist — each with a client-safe variant for services-company work where telemetry can't leave the client boundary (self-hosted model or client-tenancy AI, and redaction of hostnames/customer identifiers before any external assistant sees a query result).
rate()