Lab 11 · NL-to-query + alert tuning

Then you'll analyse 28 days of alert history, propose three noise-reduction changes as reviewed PRs, and prove the precision/recall improvement by replaying the history against old versus new rules.

This is the observability lab for Module 4. The running theme: AI drafts the query and the rule change, but the objective replay harness — not the model — decides whether the change is an improvement.

Time & tier

Core · ~70 minutes hands-on. Runs at ₹0: Prometheus in Docker, a CSV replay harness, and Claude only for query drafting.

Setup

Self-contained. Docker, Python 3.11+, and an Anthropic API key for the NL-to-query half.

mkdir alert-tuning && cd alert-tuning
python -m venv .venv && source .venv/bin/activate
pip install anthropic requests
docker run -d --name prom -p 9090:9090 prom/prometheus
export ANTHROPIC_API_KEY="sk-ant-..."

Generate 28 days of alert-firing history with a ground-truth actionable label (whether the fired alert corresponded to a real problem an engineer had to act on). Save as gen_alert_history.py:

import csv, random
random.seed(7); rows = []
def add(name, sev, ns, count, p_act, noise, real):
    for _ in range(count):
        actionable = random.random() < p_act
        peak_rng, dur_rng = (real if actionable else noise)
        rows.append({"alertname": name, "severity": sev, "namespace": ns,
            "peak_value": round(random.uniform(*peak_rng), 1),
            "duration_min": random.randint(*dur_rng), "actionable": int(actionable)})
add("HighCPU",         "warning",  "prod", 120, 0.08, ((80,95),(2,18)),  ((90,99),(15,60)))
add("ProbeFlap",       "critical", "prod",  60, 0.05, ((1,1),(1,4)),     ((1,1),(5,20)))
add("PodCrashLooping", "critical", "dev",   45, 0.00, ((3,12),(5,30)),   ((3,12),(5,30)))
add("ErrorBudgetBurn", "critical", "prod",  12, 0.90, ((2,5),(5,15)),    ((14,40),(10,60)))
add("DiskPressure",    "warning",  "prod",   8, 0.75, ((80,84),(10,25)), ((85,99),(30,120)))
add("PodCrashLooping", "critical", "prod",  10, 0.80, ((3,5),(4,9)),     ((5,20),(10,45)))
random.shuffle(rows)
with open("alert_history.csv","w",newline="") as f:
    w = csv.DictWriter(f, fieldnames=rows[0].keys()); w.writeheader(); w.writerows(rows)
print(f"wrote {len(rows)} events ({sum(r['actionable'] for r in rows)} actionable)")

Run python gen_alert_history.py — you'll get 255 events, ~45 actionable. The noise is deliberate: HighCPU fires on benign short spikes, ProbeFlap blips for a minute and self-heals, and a dev-namespace PodCrashLooping rule pages prod on-call for non-prod churn.

Guided steps

Part A — NL to PromQL

1. 1.

   Draft queries with an explicit sanity-check habit. Create nl2promql.py:

   Copy

   import sys, anthropic
   client = anthropic.Anthropic()
   SYSTEM = ("You write PromQL. Return ONLY the query on the first line, then a second "
             "line starting 'CHECK:' describing what a correct result should look like "
             "(units, expected range) so the operator can sanity-check it.")
   print(client.messages.create(model="claude-opus-4-8", max_tokens=300, system=SYSTEM,
       messages=[{"role":"user","content":sys.argv[1]}]).content[0].text)
   

2. 2.

   Run the ten questions. For each, read the CHECK: line and ask "does this query actually measure that?" before trusting it. Examples to run:

   • •
     "95th percentile request latency for the checkout job over 5 minutes"
   • •
     "per-second rate of 5xx responses by service"
   • •
     "pods in CrashLoopBackOff right now"
   • •
     "memory working set as a fraction of the limit, per pod"
   • •
     "predicted hours until the disk fills, based on the last 6 hours"

   The last one is the trap: linear-predict_linear extrapolation is easy to write and easy to get directionally wrong (sign errors, wrong window). The CHECK: line is what catches it.

3. 3.

   Record trust calibration. In a table, note for each query whether it was correct as written, correct after one nudge, or subtly wrong — this is the "where AI is trustworthy for queries" evidence.

Part B — Alert tuning as a measured PR

4. 1.

   Build the replay harness. This is the objective scorer. Create retune.py:

   Copy

   import csv, json, sys
   RULES = json.load(open(sys.argv[1]))
   def fires(ev):
       r = RULES.get(ev["alertname"], {})
       if r.get("drop_namespace") == ev["namespace"]: return False
       if float(ev["peak_value"]) < r.get("threshold", 0): return False
       if int(ev["duration_min"]) < r.get("for_min", 0): return False
       return True
   events = list(csv.DictReader(open("alert_history.csv")))
   fired = [e for e in events if fires(e)]
   tp = sum(int(e["actionable"]) for e in fired)
   all_act = sum(int(e["actionable"]) for e in events)
   prec = tp/len(fired) if fired else 0
   rec = tp/all_act if all_act else 0
   print(f"fired={len(fired)}/{len(events)}  precision={prec:.2f}  recall={rec:.2f}")
   

5. 2.

   Score the current (noisy) rules. Save rules_before.json:

   Copy

   {"HighCPU": {"threshold": 80}, "ProbeFlap": {}, "PodCrashLooping": {},
    "ErrorBudgetBurn": {}, "DiskPressure": {}}
   

   python retune.py rules_before.json → roughly precision=0.18 recall=1.00. Everything fires; 82% of pages are noise.

6. 3.

   Have AI propose noise-reduction changes, then encode them. Ask the assistant: "Given these alert families and that many HighCPU/ProbeFlap fires are short benign spikes and one PodCrashLooping rule fires in the dev namespace, propose three noise-reduction changes." Expect: raise the HighCPU threshold and add a for duration; add a for to ProbeFlap so single-minute blips self-heal; inhibit PodCrashLooping in dev. Encode as rules_after.json:

   Copy

   {"HighCPU": {"threshold": 92, "for_min": 15},
    "ProbeFlap": {"for_min": 5},
    "PodCrashLooping": {"drop_namespace": "dev"},
    "ErrorBudgetBurn": {"threshold": 10},
    "DiskPressure": {"threshold": 85, "for_min": 30}}
   

7. 4.

   Prove the improvement. python retune.py rules_after.json → roughly precision=0.86 recall=0.98. You cut fired alerts from 255 to ~51 while keeping nearly all actionable ones. That delta is the PR's justification — a reviewer can see the numbers, not just trust the AI.

Verification

• •
  nl2promql.py returns ten queries, each with a CHECK: line; your calibration table flags at least one query that was wrong or needed a nudge.
• •
  retune.py rules_before.json shows low precision (~0.18) with recall 1.0.
• •
  retune.py rules_after.json shows precision materially higher (≥ 0.80) with recall still ≥ 0.90.
• •
  Your alert-tuning PR description contains both before and after numbers.

Completion artifact

A query notebook or queries.md with the ten NL questions, their PromQL, and per-query trust-calibration notes; plus rules_before.json, rules_after.json, and a TUNING-PR.md PR description quoting the measured before/after precision and recall. Self-grade (pass at 4/5):

1. 1.
   Ten queries drafted, each with a sanity-check note; at least one flagged as wrong/needing a nudge.
2. 2.
   Replay harness scores rules against labelled history.
3. 3.
   Three distinct noise-reduction changes (threshold, for duration, namespace inhibition) encoded.
4. 4.
   After-precision ≥ 0.80 and after-recall ≥ 0.90, both quoted in the PR.
5. 5.
   The change is framed as a reviewed PR, not an auto-apply.

Stuck?

• AI query looks plausible but returns nothing in Prometheus: the demo Prometheus scrapes only itself, so app metrics are absent — that's fine, you're grading the query shape and the CHECK: reasoning, not live data.
• After-recall dropped below 0.9: your HighCPUfor_min is too aggressive; 15 minutes keeps the long real spikes while dropping short benign ones. Sweep 10/15 and re-score.
• Precision barely moved: confirm the drop_namespace: dev inhibition is actually applied — the dev PodCrashLooping events are pure noise and dropping them is the single biggest precision win.

Client-safe variant

Client alert histories are sensitive operational data. The replay harness runs entirely on a CSV you can keep inside the client VPC — no query text or metric values ever leave. The NL-to-query half is the exposure point: route it through the client's approved endpoint, or use a self-hosted model, and redact client hostnames/service names from the questions. The deliverable clients value most is exactly this artifact — a rule change with a measured noise reduction they can put in a change ticket.

Answer key

Part A's trap is the disk-fill prediction query. predict_linear(node_filesystem_free_bytes[6h], 4*3600) predicts free bytes four hours out — but the question asked for hours until full, which requires solving for when the prediction crosses zero, not evaluating it at a fixed horizon. An assistant frequently returns the fixed-horizon form, which is a different question. The CHECK: line ("result should be a number of hours, positive, decreasing as the disk fills") is what exposes the mismatch; without it the wrong query looks authoritative. Part B's trap is chasing precision to 1.0 by over-tightening — push HighCPU for_min to 30 and you'll drop a couple of genuine sustained-CPU incidents, trading a recall miss for marginal precision. The right stopping point keeps recall ≥ 0.9; report both numbers so the reviewer sees the trade-off you chose.