Google never open-sourced Outalator, its internal on-call tool, but on-call is hard for other reasons. Google SRE and Datadog focus on two things: keeping toil under 50%, and keeping schedules sustainable. The tool's job is to collapse alerts into incidents so people can collaborate, pass work to the next shift, and review.
Most teams can start with three steps: set up metrics, make the rotation sustainable, then pick a tool. The first two need no extra budget.
Tooling is a small part of on-call
Kubernetes grew out of Borg, Google's internal container orchestrator. Prometheus grew out of Borgmon, its internal monitoring system. So it's natural to ask whether Outalator has an open-source counterpart.
That question misses the point. Site Reliability Engineering spends nearly three chapters on on-call, and tooling takes up only a small part of them. On-call quality depends on how culture, mechanisms, and tools work together. Tools are the easiest layer to copy, and the easiest to overrate.
Google SRE: Keep toil under 50%
Google SRE uses software engineering to solve operations problems and to reduce toil. In practice, they set a public, measurable goal: no SRE spends more than 50% of their time on toil. The rest goes to engineering projects that remove future toil or add capabilities to services.
The goal is 50%, not "as little as possible," because a vague goal can't be measured. With a measurable cap, overruns become visible, someone owns them, and the team adds people or cuts load.
Google SRE lists three costs of toil left unchecked:
- Career stagnation. When engineering time gets squeezed out, growth slows or stops. Unpleasant work is worth doing when it's unavoidable and high-impact, but repetitive work doesn't build a career.
- Morale and retention. Everyone has a limit for toil, and past it come burnout and resentment. When toil overflows, a team's strongest engineers are often the first to look elsewhere.
- An eroded role. SRE is an engineering organization. Once a team sinks into toil, development teams hand over ops work they should keep, and other teams start to expect SRE to take it. That trend is hard to reverse.
The biggest source of toil is interrupt-driven work, followed by on-call. The first is mostly non-urgent, service-related work. The second is urgent incident response. Google's rule of thumb is that an SRE team needs at least six to eight people to keep on-call toil low. Without enough people, the rotation can't spread the load, and it turns into the same people on duty all the time.
Outalator: Handle alerts as incidents
Outalator is Google's internal on-call tool, where SREs manage the full alert lifecycle on one platform. Its features are modest:
- Alert grouping: related alerts collapse into one incident, and SREs work at the incident level instead of alert by alert. This step cuts a large amount of duplicate notification and duplicate work. It's the foundation of noise reduction.
- Labels: incidents get labels that describe their context, so you can filter, report, and review by label.
- Analytics: it analyzes alert volume trends, response efficiency, and resolution efficiency by team, person, service, and data center. Teams use this to find weak spots in on-call.
- Handoff and reports: select a set of incidents and email their titles, labels, and key notes to the next on-call engineer. A report mode supports the weekly production reviews many teams run.
Everything Outalator does is plain work: collapse alerts into incidents and let people collaborate and review. A good on-call tool doesn't deliver reliability by itself. It gives people the capacity to deliver it.
Datadog: Build sustainability into the rotation rules
Google SRE offers a conceptual framework. Datadog's approach reads more like a scheduling playbook, with rules like these:
- A minimum team size. Ideally, six to eight people share a rotation. If you can't staff that, three to four is the minimum for a sustainable schedule. This is close to Google's guideline.
- A maximum shift length. Shifts usually run 8–12 hours, and scheduling accounts for workload, vacations, and swaps.
- On-call means only on-call work. The engineer on duty handles alerts, checks, alert rules, and SOPs, and doesn't build new features. This rule recognizes on-call as a job of its own.
- Moderate frequency. Too often leads to burnout. Too rarely leaves little reason to improve the process.
Beyond scheduling, Datadog invests in two areas that other teams often overlook:
- Support for the person on duty. Give people training and materials before they go on call. Datadog uses a primary and a backup. If the primary doesn't respond, the system notifies the backup. If the backup can't handle it, the page escalates to the direct manager. This backup chain keeps a missed page at 3 AM from turning into an incident.
- Managers in the rotation. Direct managers take shifts, both to set an example and to feel the pressure firsthand. That experience gives them a reason to improve the process and tools. Managers who stay away from the front line lose touch over time, and the on-call process gets worse.
Both point to the same lesson: Design on-call as real work
Both approaches lead to the same conclusion. How well you run on-call has little to do with the tool you buy. What counts is whether you treat it as work to design, measure, and keep investing in.
| Layer | Practice |
|---|---|
| Culture | Put people first and think long term; recognize that on-call wears people down and deserves compensation and protection |
| Mechanisms | Teams of six to eight, sustainable schedules, clear backup and escalation chains, managers on call, and a 50% cap on toil |
| Tools | Group alerts into incidents to support collaboration, handoff, metrics, and reviews |
Of the three, tools are the easiest to put in place. Culture and mechanisms are the hard part.
Three first steps
Copying Google's model wholesale isn't realistic for most teams. The line between on-call and overtime blurs, and "on duty with no rotation" becomes normal. Short-term goals drive planning, and technical debt piles up. In many companies, the ratio of SREs to developers is close to 1:100, so a six-person SRE team is hard to staff. These are real structural barriers.
You can still start with three low-cost steps that pay off quickly.
First, set up metrics
An ops lead often sees "too many alerts and an exhausted team" but not the actual workload of alert handling. Without that, you can't plan headcount or see what to fix. A basic on-call dashboard includes at least five metrics:
| Metric | Meaning | Goal |
|---|---|---|
| Noise-reduction ratio | Compression ratio after grouping | Higher is better |
| Response ratio | Share of alerts that get acknowledged; shows whether alerts are useful | Higher is better |
| Total alert volume | Alerts per time window; SLO-based alerting can cut it sharply | Lower is better |
| MTTA (mean time to acknowledge) | Reflects handling efficiency and team pressure | Keep it in a healthy range |
| MTTR (mean time to resolve) | Reflects observability, infrastructure, and understanding of the business | Shorter is better |
Once you can see these five metrics reliably, decisions like "add people or fix alerts" and "which alert to delete" have a basis.
Second, make the rotation sustainable
If you can't field six people, keep a minimum rotation of three or four. Cap shifts at 8–12 hours, and define a primary, a backup, and escalation to a manager. None of this needs extra budget, only rules that you write down and follow.
Third, pick an existing tool
Outalator was never open-sourced, but similar products exist. They ingest alerts from many monitoring systems on one platform and connect to IM and a mobile app. Common features include grouping and noise reduction, acknowledgment, escalation, dispatch, and scheduling with swaps.
Flashduty On-call is built by the team behind the open-source monitoring tool Nightingale. It connects to mainstream monitoring systems and IM tools. If you're comparing options, see Flashduty vs. PagerDuty.
A tool is only the start. Collapse alerts into incidents, put the metrics on the table, and keep the rotation sustainable. With those three in place, your on-call can keep improving.
