A genuine incident — the service hard down, or error rate at 40% — went unanswered for hours even though alerts fired correctly. The on-call engineer saw the notification and swiped it away, because it looked exactly like the last two hundred notifications, all of which were noise. Postmortem line: "the alert worked; nobody believed it."
A team that added monitoring with enthusiasm and no thresholding discipline: a page for every 500, every latency blip over an arbitrary number, every drift score a hair above baseline, every failed health check including deploy-time restarts. Within weeks the on-call phone buzzed dozens of times a day. Muting, filtering to a side channel, and reflex-dismissal followed — so the one page that mattered drowned with the rest. Linked modules: 4.8 (Monitoring — tiered alerts), 4.2 (latency percentiles), 4.7 (CloudWatch).
Alerts spend human attention, and attention is a small, non-refillable daily budget. Every false or unactionable page trains the responder that pages are safe to ignore — classic conditioning, and it cannot be reversed by a memo asking people to "take alerts seriously". The design errors are recognisable: single events promoted to pages (one 500 in 1,800 requests is 0.06%, not an incident); thresholds set at normal operating values with no headroom (paging at p95 > 20 ms when normal p95 is 19 ms); and every signal mapped to the same maximum severity, so the channel carries no information about urgency. Confirm it with two numbers from the alert history: pages per week, and the fraction of pages that led to any action. Dozens per week with under 10% actionable is the disease in figures.
Rebuild the policy as tiers that match response cost to failure cost. Page — wake a human now — only for "the service is not serving": down, or error rate over a threshold with real headroom (ours: 2% sustained for five minutes, against a normal 0.2%). Ticket — handle within a day or two — for model signals: a feature PSI at 0.2 or more, rolling recall under the gate floor for two consecutive weeks. Digest — read on Friday — for everything else: p95 creep, single 500s, watch-level drift. Delete every alert that has no written action attached, then declare pager bankruptcy: announce the reset, so the next page arrives with credibility restored.
Every new alert must answer three questions in review before it ships: what action does the responder take, how fast must that action happen, and what is the expected false-fire rate? No action, no alert — it becomes a dashboard line instead. Set page thresholds from measured normals plus generous headroom, never from round numbers. Track pages-per-week and actionable-fraction as team health metrics, and prune monthly. The goal is a pager that fires a few times a year and is trusted absolutely — a quiet pager is not a broken pager.
Q: "You join a team whose on-call phone gets thirty alerts a day and people have started ignoring them. What do you do?" Strong answer: name the failure mode — alert fatigue — and its mechanism: unactionable pages train dismissal, so signal dies with the noise. Give the audit (pages per week versus fraction actionable), then the tier redesign: page only for service-down with headroom above measured normals, tickets for drift and quality-gate breaches, digest for the rest, and delete anything without a written action. The differentiator is saying "a page must be rare enough to be believed" — it shows you design for the human, not only the metric.