Fixing alert fatigue: what actually works

Most posts about alert fatigue tell you to “align alerts with user impact” and stop there. That’s true but useless on its own — nobody’s stuck because they don’t know the goal, they’re stuck because the path from “200 pages a week” to “5 pages a week, all real” is not obvious. Here’s what that path actually looks like, with the specific mechanics.

Start by counting, not by guessing

Before touching a single threshold, pull your PagerDuty (or Opsgenie, or whatever) incident log for the last 60 days and dump it into a spreadsheet. Three columns: alert name, whether it paged during business hours or off-hours, and whether the on-call engineer did anything besides acknowledge it. You’re looking for “ack, no action” — not “resolved,” because plenty of alerts auto-resolve without anyone doing anything, which is its own signal.

On a team I worked with, this exercise showed that 6 alert types produced 71% of all pages, and 4 of those 6 had zero recorded actions in 60 days. That’s not a rounding error, that’s almost three-quarters of your on-call burden coming from six lines of YAML. You don’t need a strategy to fix that — you need to open the alertmanager config and delete or reroute six rules.

Do this before you read any more of this article. Seriously — the numbers you get back will tell you which of the tips below actually apply to your situation.

Separate “page” from “ticket” from “dashboard,” and mean it

Most teams have exactly one severity that matters in practice: pages. Everything else gets treated the same whether it’s tagged P2 or P4. If your tooling only has one notification channel, you don’t have a severity system, you have a filter that doesn’t filter.

Concretely, in Alertmanager this means three separate receivers, not three labels on one receiver:

route:
  routes:
    - match:
        severity: page
      receiver: pagerduty-oncall
      group_wait: 10s
    - match:
        severity: ticket
      receiver: jira-backlog
      group_wait: 5m
    - match:
        severity: info
      receiver: grafana-only

The page route should have almost nothing in it. If you’re not sure whether something belongs there, it doesn’t — default to ticket and promote it later once it’s proven itself. Promoting an alert to paging status after it’s caused two real incidents is easy. Demoting a noisy page after your team has stopped trusting it takes months, because trust doesn’t rebuild on the same schedule it breaks on.

Runbook-or-delete is a real rule, not a suggestion

This is the one people nod along to and then don’t do, because it means admitting some alerts you’re proud of don’t deserve to exist.

The test: open the alert right now and try to write two sentences — what a human checks first, and what they do based on what they find. If you can’t write those two sentences in under a minute, the alert doesn’t know what it wants from the on-call engineer, and neither will they at 3am.

I’ve seen teams solve this by literally requiring a runbook_url label on every alerting rule and having CI reject PRs that add a paging alert without one. That sounds bureaucratic until you realize it takes about ten minutes to set up and it’s caught real garbage before it shipped, multiple times.

The threshold problem isn’t the number, it’s the shape

A flat threshold assumes your traffic looks the same at 3pm and 3am. It almost never does. Before reaching for anomaly detection or ML-based alerting (usually overkill, and hard to debug when it’s wrong), try comparing current values against a trailing window of the same metric instead of a fixed number:

# instead of: error_rate > 0.05
error_rate > (avg_over_time(error_rate[1h] offset 1d) * 4)
  and rate(requests[5m]) > 50

That second line — the volume floor — is the part people skip and then wonder why the alert fires on three errors during a slow Tuesday morning. Small denominators make ratios lie to you. If you only get 12 requests in a window, one error is an 8% error rate and looks like a crisis. Don’t alert on ratios below a minimum sample size, full stop.

Flapping isn’t a bug in your alert, it’s a bug in your for: clause

If a condition trips right at the boundary, you’ll get alert-resolve-alert-resolve until it settles. The fix most people reach for first is widening the threshold, which usually just moves the flapping to a different boundary. The actual fix is almost always the for: duration:

- alert: HighLatency
  expr: histogram_quantile(0.95, rate(request_duration_bucket[5m])) > 0.5
  for: 10m

for: 10m means the condition has to stay true continuously for ten minutes before it fires, not just be true at one evaluation. This alone kills most flapping without touching the threshold at all. If you’re still flapping after adding a reasonable for: window, the metric itself is too noisy at that resolution and you should be alerting on a rate or an average, not a raw instant value.

One incident, one page — correlation is worth doing, but do it last

When a database dies, everything downstream fails within the same 30 seconds: API timeouts, queue backups, dependent service alerts. Twelve pages for one root cause trains people to stop opening the second, third, and fourth one.

Alertmanager’s group_by handles a good chunk of this for free if you group on something like cluster or service_dependency instead of the default of grouping by alert name:

route:
  group_by: ['cluster', 'root_service']
  group_wait: 30s
  group_interval: 5m

But don’t reach for correlation tooling before you’ve fixed the underlying thresholds — grouping ten bad alerts into one notification just means you get a well-organized noisy page instead of ten disorganized ones. Fix the noise first, group second.

The part nobody wants to hear: some of your alerts are protecting a dashboard, not a person

Every team has alerts that exist because someone got burned once, two years ago, by something extremely specific, and added an alert so it could never happen again — even though the conditions that caused it haven’t existed since. Those alerts don’t get removed because removing them feels like inviting the incident back.

They’re usually the alerts with zero actions in your 60-day log. If it hasn’t fired with a real response in two months, it’s not protecting you, it’s just sitting there waiting to erode trust in everything else. Delete it. If the fear is strong enough, turn it into a weekly dashboard check instead of a page — you keep the visibility, you lose the 3am interruption for nothing.

What to actually do this week

If you only do one thing from this list, pull the 60-day log and find your six worst alerts. That’s a two-hour task, it requires no new tooling, and it will very likely account for most of your pain. Everything else here — severity routing, runbook requirements, trailing-window thresholds — is worth doing, but it compounds over months. The audit pays off by Friday.