The Problem With Most On-Call Setups

The average on-call rotation fails for one of two reasons: too many alerts that mean nothing, or no structure for what to do when something real happens.

Both lead to the same outcome: a team that dreads being paged, stops taking alerts seriously, or burns out and leaves.

Good on-call isn't about suffering through more incidents. It's about building a system where the right person gets the right alert with the right context to act fast.

Start With Rotation Schedules

Weekly rotations are the most common default. They work for small teams but create problems at scale: one person carries a week of risk, and the context switch after a long weekend on-call is brutal.

Recommended approach for teams of 4+:

Primary on-call: One engineer per shift (typically 7-day or 12-hour rotations)
Secondary on-call: A backup who escalates if primary doesn't respond within 5 minutes
Incident commander: A rotating role for major incidents only, separate from day-to-day paging

Keep rotations predictable. Engineers should know their on-call weeks at least a month in advance. Surprises create resentment.

Two-Tier Escalation

A single-tier on-call setup (one person, no backup) fails whenever that person is unavailable: on a call, driving, or simply asleep through their phone. A two-tier setup fixes this:

Alert fires
  → Primary on-call notified immediately
  → No acknowledgment after 5 minutes
  → Secondary on-call paged
  → No acknowledgment after another 5 minutes
  → Team lead or manager paged

The 5-minute delay is intentional. Primary on-call needs time to acknowledge without the secondary being flooded unnecessarily. Adjust based on your SLA requirements.

Write Runbooks Before You Need Them

The worst time to figure out what to do is at 2 AM with a production incident in progress.

For every alert you configure, write a runbook that answers:

What does this alert mean?
What's the likely cause?
What's the first thing to check?
How do you resolve it?
Who else should you notify?

Runbooks don't need to be perfect. A rough guide that gets you to resolution in 20 minutes beats no guide that takes 90 minutes.

Post-Incident Reviews Keep the Rotation Healthy

Every significant incident, anything that woke someone up, caused customer impact, or took longer than 30 minutes to resolve, should have a post-incident review (PIR).

The PIR isn't about blame. It's about three things:

What happened? A clear timeline of events.
Why did it happen? The contributing factors, not just the proximate cause.
What changes? Concrete action items: updated runbooks, new alerts, architectural changes, or better test coverage.

A team that runs PIRs consistently builds better on-call over time. A team that skips them repeats the same incidents.

Connect Your Monitoring to Your Alerting

On-call only works if the alerts reaching your engineers are accurate and actionable. This means your monitoring setup and your paging system need to be tightly integrated.

The best setups have:

Deduplication: One incident, one page: not ten alerts for the same root cause
Context in the alert: Not just "database is slow" but the region, the affected service, and a link to the relevant dashboard
Severity tiers: Not every alert needs to wake someone up. Route low-severity issues to a Slack channel for review during business hours

Noise is the enemy of good on-call. Every false positive trains your team to ignore alerts.

Measure the Health of Your On-Call Rotation

Track these metrics to know if your on-call is working:

Pages per week: Baseline, then track trends
False positive rate: How many pages led to no action?
Time to acknowledge: Are people responding within your SLA?
MTTR (Mean Time to Resolve): Is it improving over time?
On-call satisfaction score: A simple weekly survey question goes a long way

A healthy on-call rotation gets quieter over time as teams fix root causes and improve alerting thresholds.

How to Set Up On-Call Rotations That Don't Burn Out Your Team