Alerts That Don't Cry Wolf: Taming Monitoring Noise

There's a specific, quiet way monitoring fails. Not with a missed outage — with a muted channel. The alerts got so frequent and so rarely actionable that the team started ignoring them, and the one that actually mattered scrolled by with the rest. That's alert fatigue, and it's more dangerous than having no monitoring at all, because it comes with a false sense of safety.

The real cost of noise#

Every alert asks for a slice of human attention, and attention is the scarcest resource on any team. A good alert spends that attention well: it fires, someone acts, the problem gets smaller. A bad alert spends it for nothing — and worse, it trains people to disregard the channel. After the tenth "down" that self-resolved in twenty seconds, the eleventh gets a shrug. The cost isn't the interruption; it's the erosion of trust in the entire system.

Why monitors cry wolf#

The usual culprits, in rough order of how often they're the problem:

• Transient blips. A single dropped packet or a 200ms DNS hiccup gets treated as a full outage.
• Re-notifying. The same ongoing outage pages every check interval instead of once.
• No severity. A disk at 71% pages with the same urgency as a checkout that's returning 500s.
• Expected downtime. Deploys and maintenance trip alerts everyone already knew about.
• Flapping. A service on the edge of healthy toggles up/down/up/down, firing on each swing.

The good news: five targeted fixes handle nearly all of it.

Fix 1: alert on transitions, not on states#

The single most important rule. A monitor checks constantly, but you only want to hear about the edges: the moment it went down, and the moment it recovered. A service that's been down for an hour should have paged you once, an hour ago — not 120 times since.

Pair this with a recovery alert so an incident has a clear close, and with flap-damping so a service teetering on the edge doesn't fire on every oscillation. Good monitoring has a small state machine behind each monitor doing exactly this.

Fix 2: confirm before you page#

Most false positives are transient. The fix is to demand a little more evidence before declaring an outage:

• Consecutive failures. Require 2–3 failed checks in a row. A one-off blip never reaches the threshold.
• Multi-region agreement. Check from several locations and only declare "down" when a majority agree. One failing region is almost always a network path problem, not your service.

The trade-off is a few extra seconds of detection latency — a small price for alerts you can trust. This is the same idea introduced in what uptime monitoring is, applied deliberately.

Fix 3: severity and routing#

Not every problem is an emergency, and pretending otherwise is how you get people to snooze the emergencies. Assign a severity and route accordingly:

A simple three-tier model

1. Page. User-facing and urgent — the site is down, checkout is failing, a cert expires today. Wakes a human.
2. Notify. Needs attention soon but not now — disk at 85%, response time degrading, a non-critical service down. Goes to a team channel.
3. Log. Worth recording, not worth interrupting anyone — a single retryable failure, an informational threshold. Dashboard or digest only.

Then route by who can act and which service: database alerts to the data folks, front-end to the app team. An alert that reaches someone who can't fix it is noise for them and a delay for the person who can.

Fix 4: suppress expected noise#

A huge share of "false" alerts aren't false at all — they're expected. You just knew about the downtime already. Two tools handle this without the cardinal sin of disabling a monitor and forgetting to turn it back on:

• Maintenance windows. Schedule a window and down/recovery alerts for the affected services are suppressed for its duration — while the monitor keeps recording, and anything outside the window still alerts.
• Snooze. Muting one monitor's alerts for an hour while you do ad-hoc work, with an automatic un-mute so you can't leave it off by accident.

Fix 5: make each alert actionable#

An alert that fires is only half the job. A good one also tells the recipient how to start:

• What broke, specifically — the monitor name, the error class (timeout vs 500 vs cert expiring), the region.
• Since when, and whether it's still ongoing.
• Where to look — a link straight to the monitor's detail and recent checks.

The difference between "prod-api is DOWN" and "prod-api: connection timeout from 3/4 regions, started 2m ago, cert still valid" is the difference between an alert that starts a five-minute investigation and one that skips it.

A 10-minute alert audit#

Run this on your current setup:

1. Scroll your last two weeks of alerts. For each, ask: did anyone do something? If mostly no, you have a noise problem.
2. Find your loudest monitor. Add confirmation (consecutive failures or multi-region) and a recovery alert.
3. Tag every alert source as page / notify / log, and re-route anything mis-tiered.
4. Set up a maintenance-window habit so deploys stop generating pages.
5. Rewrite your noisiest alert template to include error class, duration, and a direct link.

Do that and the channel becomes trustworthy again — which is the entire point. Ensju bakes most of this in: transition-only alerts, multi-region down-rules, per-monitor snooze, and maintenance-window suppression are defaults, not add-ons. If you're routing alerts to a team, pair this with a clear public status page so users self-serve instead of adding to the noise.

Frequently asked questions#

What is alert fatigue?

Alert fatigue is what happens when a monitoring system sends so many low-value alerts that people stop reading them. The danger isn't the noise itself — it's that a real, urgent alert gets ignored or muted along with everything else.

How do I stop getting false-positive downtime alerts?

Require confirmation before alerting: two or three consecutive failed checks, or agreement across multiple regions, instead of paging on a single failure. That filters out transient network blips, which are the most common cause of false positives.

Should every alert wake someone up?

No. Split alerts by severity. Urgent, user-facing failures should page a person. Warnings and informational alerts should go to a channel people check during working hours. If everything pages, nothing is truly urgent.

How do I avoid alerts during planned maintenance?

Use maintenance windows to suppress down and recovery alerts for the affected services during the planned time, and a per-monitor snooze for ad-hoc work. Both stop the expected noise without disabling the monitor, so real problems outside the window still alert.