Cluster the Morning Alert Flood

Tier: Core · Workflow area: Monitoring & change · ~30 minutes

This playbook builds a daily morning-sweep routine: export overnight alerts from the RMM and monitoring stack, cluster them into probable-cause groups with a standing prompt, verify one representative per cluster, and act per-cluster instead of per-alert. It also draws the hard line — AI groups and summarizes; it never decides what pages a human. (UC-37)

When to use

• Overnight accumulations50+ unpaged alerts each morning across RMM, infrastructure monitoring and SaaS status noise; the sweep is where an hour disappears daily.
• Alert-storm morningsa network blip at 3 AM generated 400 downstream alerts; clustering finds the four real events inside them.
• Handover between shifts or from an NOCa clustered summary is a better baton than a raw queue.

When NOT to use

• Live paging decisionswhat wakes a human is a threshold and routing rule set in the monitoring platform, deliberately, in advance. Do not put a model in that path.
• Active incident triageduring an incident you work the incident process; clustering is for the backlog, not the fire.
• As a reason to keep noisy monitorsif the same cluster appears every morning, the fix is tuning the monitor, not summarizing it faster. Clustering that hides bad hygiene is a step backward.

The recipe

1. 1.

   Export the overnight window with the fields that cluster well. From the RMM and monitoring tools, pull alerts from the last 16 hours: timestamp, source system, device/host, alert type, severity, message text, current state (active/auto-resolved). CSV is fine. Include auto-resolved alerts — flapping shows up only when you can see the resolves.

2. 2.

   Run the standing cluster prompt.

   Copy

   Below are overnight alerts from our monitoring systems (CSV).
   Cluster them into probable-cause groups:
   - Group alerts likely sharing ONE underlying cause (same subnet,
     same host family, same time window, cascade patterns like
     host-down -> service-down -> check-failed)
   - For each cluster: name, alert count, time span, probable cause
     hypothesis, confidence (high/med/low), and ONE representative
     alert I should verify first
   - Separate sections: FLAPPING (alerted and auto-resolved 3+ times),
     SINGLETONS (no cluster, listed individually — do not force-fit),
     and STALE (active >7 days — hygiene candidates)
   - Order clusters by severity of probable cause, not by count
   Do not mark anything resolved; do not recommend paging or priority
   changes. Group and hypothesize only.
   
   [paste CSV]
   

3. 3.

   Read the shape before the detail. A typical useful output:

   Copy

   CLUSTER 1 — Branch-05 connectivity (HIGH confidence) — 118 alerts
   02:41–03:07. All device-unreachable alerts share subnet 10.5.x;
   pattern matches WAN drop with recovery at 03:07. Representative:
   ALR-88231 (edge router BR05-RTR-01 unreachable 02:41).
   Note: 3 devices did NOT auto-recover — listed separately.
   
   CLUSTER 2 — Disk space, finance file server (MED) — 6 alerts ...
   SINGLETONS — 4 alerts, including ALR-88410: backup job failed on
   SQL-02 (only occurrence, no related alerts)
   

   The "did NOT auto-recover" remainder and the singletons are usually where the real morning work lives — a good clustering makes the exceptions visible, not just the mass.

4. 4.

   Verify one representative per cluster in the console. Open the representative alert in the monitoring tool and confirm the hypothesis holds (the WAN drop is in the router logs; the disk really is at 92%). One verification validates or busts the whole cluster at a fraction of per-alert cost. A busted hypothesis means you triage that cluster manually — and note what the prompt got wrong.

5. 5.

   Act per-cluster, close per-cluster. One ticket per cluster (not 118), referencing the alert IDs; bulk-acknowledge the cluster's alerts against it. Singletons get individual thirty-second looks — the lone backup failure matters precisely because nothing else explains it.

6. 6.

   Feed the flapping and stale lists into tuning. Weekly, take the accumulated FLAPPING and STALE sections and fix the top two: adjust a threshold, add a dependency so downstream checks suppress during upstream outages, or delete a monitor nobody acts on. Track queue depth over a month — the flood should shrink because monitors improved, with clustering as the bridge, not the destination.

7. 7.

   Keep paging rules out of the loop — visibly. Write it in the runbook: clustering output never changes severity, never closes alerts unverified, never modifies notification rules. Threshold and paging changes go through normal change control with a human name attached.

Make it yours

• RMM-centric shop (NinjaOne, Datto, Atera)export via the alerts API on a schedule; some platforms ship native AI alert grouping — pilot it against your prompt's output for two weeks and keep whichever finds the exceptions (non-recovered devices, singletons) more reliably.
• Enterprise monitoring (PRTG, Zabbix, LogicMonitor, Datadog)use built-in dependency and correlation features as the first layer (they prevent cascades at the source), and this prompt as the cross-tool layer where alerts from separate systems meet.
• Small shop, email-alert realitypipe the alert mailbox to CSV (Power Automate or a script) and run the same prompt; the sweep habit matters more than the sophistication of the pipeline.

Quality check

• One representative verified per cluster, every morning. The day you skip verification is the day a "WAN blip" cluster hides a dying switch.
• Singletons are never auto-dismissed. Force-fit clustering that absorbs the lone backup failure into "misc" is the failure mode to watch in the prompt's output.
• The queue trend is measured. Morning alert count, week over week — clustering should buy time that gets reinvested in tuning, and the number proves whether it did.
• Paging config is diffed monthly against change records, so nobody — human or workflow — has quietly moved a threshold outside change control.