Ask any senior engineer why they left their last job and on-call will come up within the first two minutes. Not "we had too many incidents" — that's a different problem. It's almost always the same thing: "I was getting paged at 2am for alerts that resolved themselves before I even opened my laptop."
This is the on-call tax. It's not the cost of keeping systems reliable. It's the cost of a broken alerting system — one that pages humans for things that either don't need humans or that automation should have already resolved. And it's quietly costing most engineering organizations more than they realize.
The Math Nobody Does
Most engineering managers know their on-call rotation is noisy. Few have actually run the numbers on what that noise costs.
📟 The real cost of alert noise — a typical mid-size eng team
For a team with 6 engineers rotating on-call, that's over $130,000 per year in engineering productivity lost to alerts that shouldn't have paged anyone.
But the productivity number understates the actual cost. It doesn't account for the harder-to-quantify effects: lower cognitive capacity the day after a noisy night, lower morale across the team, and the compounding effect of engineers who start disengaging from on-call alerts altogether — because they've learned that most of them don't matter.
The Habituation Problem
High alert volume doesn't just waste time. It degrades your team's ability to respond to the alerts that do matter.
When engineers are paged repeatedly for things that auto-resolve or turn out to be noise, two things happen. First, response times slow — there's an unconscious hesitation before taking the alert seriously. Second, alert fatigue causes genuine incidents to be initially classified as "probably another false positive," adding critical minutes to MTTD when the real incident finally arrives.
The paradox: The more noise in your alerting system, the slower your team responds to real incidents. High alert volume actively degrades your MTTR.
This is the part that shows up in your SLO data eventually — not as a clear cause-and-effect, but as a general drift in incident response quality that's hard to attribute cleanly.
What "Fixing Alerts" Actually Means
Most teams try to fix alert noise by adding more alerts or tuning individual thresholds. This makes the problem worse. Alert noise isn't a threshold problem — it's a signal architecture problem.
The real fixes are structural
Alert correlation. Multiple alerts firing for the same underlying condition should generate one page, not seven. If your database goes down and seventeen downstream services start alerting, your on-call engineer should see one grouped incident, not seventeen independent pages.
Auto-remediation for known patterns. If you can write a runbook that says "when X happens, do Y," you can automate that runbook. Roughly 20–30% of incident types in most organizations repeat often enough that they should never require human intervention — a restart, a cache flush, a traffic shift, a rollback trigger.
Severity-based routing. Not all alerts should page someone. P1s page on-call immediately. P2s create tickets. P3s go into a queue for review during business hours. Most alert systems are configured to page on everything, because configuring nuanced routing takes time nobody schedules.
Regular alert audits. Alerts that haven't fired in 90 days should be reviewed. Alerts that fire and auto-resolve without human action should be candidates for auto-remediation or removal. This is a monthly 2-hour session — the kind of operational hygiene that almost never gets prioritized until the problem is severe.
The Retention Calculus
Replacing a senior engineer costs between 1.5× and 2× their annual salary when you account for recruiting, onboarding, and ramp time. For a $180K engineer, that's $270K–$360K.
The comparison: A comprehensive alert remediation project — correlation setup, auto-remediation for the top 5 incident patterns, severity routing, and a runbook automation framework — typically takes 3–6 weeks of engineering time or can be implemented by an outside team in 2. The math against even a single attrition event is stark.
This isn't a hypothetical. Engineering managers consistently cite on-call health as one of the top three factors engineers mention in exit interviews. And unlike compensation, which is constrained by budget and market rates, on-call health is entirely within your control.
What Good On-Call Looks Like
Teams with healthy on-call systems share a few characteristics. These aren't aspirational — they're achievable in 6–12 weeks for most engineering organizations:
- Fewer than 2 non-actionable pages per on-call shift. If engineers are getting paged more than twice per shift for things that don't require their action, the signal architecture needs work.
- Auto-remediation covers at least 25% of incident types. Look at your last 90 days of incidents. The ones that followed a predictable pattern and required the same manual steps every time are candidates for automation.
- MTTD under 15 minutes for P1s. This requires AI-assisted anomaly detection, not just threshold alerts — customers shouldn't be your detection mechanism for critical failures.
- Monthly on-call retrospective. Not a postmortem for a specific incident — a review of the entire on-call rotation's health, including page volume, noise ratio, and engineer feedback.
The organizations that get here don't do it all at once. They pick the highest-impact change (usually alert correlation and noise reduction) and make one focused improvement per month. The on-call experience improves visibly within 6 weeks — fast enough to retain the engineers who were most at risk of leaving.