Alert-Noise Reduction as Measured, Reviewed PRs

Tier: Core · Hands-on time: ~75 minutes

The industry's AI-SRE guides converge on alert correlation as the entry-point use case, and they converge on the delivery mechanism too: given how engineers distrust unverified AI changes, every tuning change ships as a reviewed PR with before/after precision-and-recall numbers attached — never auto-applied. This playbook is that measured-PR recipe.

When to use

• •
  On-call reports drowning: high page counts, low action rates, ritual acknowledge-and-ignore behavior.
• •
  After an alert-storm incident where one failure paged fourteen times across layers.
• •
  Quarterly alert hygiene, turned from an unfunded wish into a repeatable 90-minute exercise.

Do not run this during or immediately after a major incident — tuning decisions made in trauma either overcorrect into silence or enshrine panic thresholds.

Prerequisites

• •
  30–90 days of alert firing history with dispositions: Alertmanager notification logs, PagerDuty/Opsgenie analytics export (CSV/API), ideally joined with "did a human act?" signals (ack-then-resolve time, incident created or not).
• •
  Alert rules in git (Prometheus rules, Alertmanager routing/inhibition config, or vendor monitors exported to code). If they are not in git, that is step zero — this playbook's safety model IS the PR.
• •
  An LLM with a large context or a willingness to chunk; a spreadsheet-capable teammate to spot-check stats.

The workflow

1. 1.

   Export and label the history. Produce one row per firing: rule name, timestamps, duration, labels, disposition (actioned / self-resolved before ack / acked-no-action / storm-member). Disposition is the ground truth for everything downstream; spend the time to get it, scripting joins between Alertmanager and your incident tool.

2. 2.

   Mine for noise patterns with claims tied to data.

   Copy

   Analyze this alert firing history (CSV). Identify:
   1. NOISY RULES: fired often, rarely actioned — rank by pages-per-action.
   2. FLAPPERS: short-duration self-resolvers (candidates for longer `for:`).
   3. STORM CLUSTERS: rules that co-fire within 5 minutes (candidates for
      grouping or inhibition, with the implied parent-child hypothesis).
   4. DEAD RULES: never fired (candidates for deletion or broken-rule check).
   For every finding, cite the supporting rows (rule name + firing IDs/dates
   + the computed stat). No recommendation without its numbers.
   

   Spot-check the arithmetic: recompute pages-per-action for two rules by hand or pivot table. Models miscount; the citations make miscounting catchable.

3. 3.

   Generate the change set as diffs, one concern per PR. For each accepted finding, have the model draft the concrete edit: raised threshold with rationale, extended for: duration, Alertmanager group_by/inhibition rule with the parent-child claim stated, deletions listed separately. Small PRs — a threshold change and an inhibition rule need different reviewers' attention.

4. 4.

   Attach the measurement: replay before/after. This is the load-bearing step. Replay the proposed rules against the same history window and compute per-PR: pages before → pages after; recall check — of the firings that led to real incidents, how many still fire? (must be 100% unless explicitly accepted otherwise, in writing, in the PR); precision estimate — of remaining pages, what fraction were actioned? For Prometheus, replay = re-evaluating rule expressions with new thresholds/durations over historical data (promtool + recorded series, or a script against the HTTP API). The PR description template: problem, change, pages/week before → after, incidents-still-caught 100%, dispositions of what was cut.

5. 5.

   Review, merge, and verify live. The reviewer's job: check the replay honestly covers incident recall (open the cited incident firings and confirm the after-rules catch them), sanity-check the inhibition hypothesis (does the parent alert REALLY imply the children are redundant, including during partial failures?), then merge. Tag rules with tuned-by: <PR> and re-measure after two weeks of live traffic; revert is one git revert away — which is the whole reason this ships as PRs.

Tool variants

• LLM + Alertmanager/PagerDuty exports (lab default): everything above with CSVs and prompts; zero new tooling, works against any stack.
• Keep (OSS AIOps): open-source alert aggregation/correlation with AI enrichment — good middle path; still export its proposed groupings into reviewed config rather than clicking apply.
• Datadog Bits / vendor correlation features: native storm-clustering suggestions; translate accepted suggestions into monitor-as-code changes in git so the PR discipline and revert path survive the vendor UI.
• PagerDuty Event Intelligence / Grafana IRM grouping: platform-level grouping that overlaps with Alertmanager inhibition — pick ONE layer to own dedup per alert family, or you'll debug double-suppression at 3 a.m.

Verification

• •
  Arithmetic spot-check done: at least two of the model's ranked findings recomputed independently and matching; every finding's cited firing rows sampled to confirm they exist and have the claimed dispositions — an analysis whose citations don't resolve is discarded whole, not partially trusted.
• •
  Replay evidence present in every PR: before/after page counts AND the incident-recall line, with the incident-linked firings named so the reviewer can open them.
• •
  Reviewer explicitly confirmed the inhibition parent-child claims against at least one historical partial-failure case (the scenario where inhibition rules silently eat real pages).
• •
  Two-week post-merge live measurement recorded and compared to the replay prediction; material divergence reopens the PR discussion.

Pitfalls

• Optimizing acked-no-action away as "noise." Some acked-no-action pages are early warnings a senior engineer metabolized silently. Interview the on-callers about the top cuts before merging — dispositions lie at the margins.
• Survivorship in the window. A 30-day window with no major incident makes every incident-catching rule look like noise. Use 90 days or deliberately include the last big incident's window in the replay.
• Batch mega-PRs. One PR changing 25 rules gets rubber-stamped, and rubber-stamping is the failure mode this whole design exists to prevent. One concern per PR, even if it takes a week to land the series.
• Tuning thresholds that should be SLO burn rates. If a rule keeps needing threshold nudges, the fix is often converting it to an SLO-based multi-window burn-rate alert (pb-24), not another nudge — recognize the smell.

Restricted-environment variant

• •
  Alert histories carry hostnames, customer-tenant labels, and architecture. Under client MSAs: redact label values to stable placeholders in the export script, or run the analysis on a local model (the mining task is aggregation-heavy and works well chunked through an 8B–70B local model; do the heavy counting in pandas first and let the model reason over the aggregates — this also reduces miscounting).
• •
  Replays run entirely in-tenant against the client's Prometheus/vendor APIs — the measurement half never leaves the boundary.
• •
  Changes land through the client's change-management: your PR + replay evidence maps directly onto their CAB template, which is precisely why the measured-PR format wins in services contexts.

Success criteria

• •
  A landed series of small PRs with replay evidence, achieving a meaningful page reduction (teams typically find 25–50% of pages are cuttable) with incident recall demonstrably intact on both replay and the two-week live check.
• •
  Every merged change revertible by one git operation, and tagged to its analysis.
• •
  On-call satisfaction check: the next rotation's engineers report fewer useless wakes — ask them, don't infer it.
• •
  The exercise is scheduled to recur (quarterly), with the export/replay scripts committed so the next run starts at step 2.