SLO-based alerting: moving from thresholds to burn rate alerts
Threshold-based alerts have a fundamental problem: a 5% error rate at 2 AM with zero traffic looks identical to a 5% error rate during peak load. Both fire the same alert. One is nothing, one is a disaster. SLO-based burn rate alerting fixes this by measuring how fast you're depleting your error budget — which makes the severity of a problem proportional to its actual business impact.
What SLOs and error budgets mean in practice
An SLO (Service Level Objective) is a commitment about reliability. A common one: "99.9% of requests succeed over a rolling 30-day window." That 0.1% tolerance is your error budget — the total amount of failure you're allowed before you've violated the SLO.
For a 30-day window with a 99.9% SLO:
- Total requests in 30 days at 1,000 req/min: 43,200,000
- Allowed failures: 43,200
The key insight: a 1% error rate sustained for just one hour burns through ~139% of your entire monthly budget. That's not a warning alert — that's a service emergency.
The error budget replenishes at a fixed rate (1 month = full budget). The burn rate measures how fast you're consuming it relative to that replenishment. A burn rate of 1.0 means you'll exhaust the budget in exactly 30 days — that's acceptable. A burn rate of 14.4 means you'll exhaust it in 50 hours. That's worth a page.
Calculating burn rate
The formula is straightforward:
def burn_rate(error_rate: float, slo_target: float) -> float: """ Returns how many times faster than the budget you're consuming errors. burn_rate=1.0 → exhausts budget in exactly 30 days (acceptable) burn_rate=14.4 → exhausts budget in ~50 hours (page immediately) burn_rate=36 → exhausts budget in ~20 hours (page now) """ error_budget = 1.0 - slo_target return error_rate / error_budget # Reference points for a 99.9% SLO print(burn_rate(0.001, 0.999)) # 1.0 — exactly on pace print(burn_rate(0.006, 0.999)) # 6.0 — budget gone in 5 days → ticket print(burn_rate(0.0144, 0.999)) # 14.4 — budget gone in 50h → page print(burn_rate(0.05, 0.999)) # 50.0 — budget gone in ~14h → page now This is the number your alerting rules should fire on. Not the raw error rate. Not request latency. The burn rate.
The two-window approach
A single burn rate window produces false positives. A short spike during a deploy bumps your 5-minute error rate to 10%, fires a critical alert, and resolves itself before anyone responds. Alert fatigue follows.
The solution from Google's SRE workbook: require both a long window and a short window to exceed a threshold before firing. The long window confirms the problem is real (not a blip). The short window confirms it's happening right now (not hours ago).
from dataclasses import dataclass @dataclass class AlertRule: name: str long_window_minutes: int short_window_minutes: int burn_rate_threshold: float severity: str ALERT_RULES = [ AlertRule("critical", 60, 5, 14.4, "page"), AlertRule("high", 360, 30, 6.0, "page"), AlertRule("medium", 1440, 120, 3.0, "ticket"), AlertRule("low", 4320, 360, 1.0, "ticket"), ] def should_alert( long_window_error_rate: float, short_window_error_rate: float, slo_target: float, rule: AlertRule, ) -> bool: long_burn = burn_rate(long_window_error_rate, slo_target) short_burn = burn_rate(short_window_error_rate, slo_target) return long_burn >= rule.burn_rate_threshold and short_burn >= rule.burn_rate_threshold # 1-hour error rate: 1.5%, 5-minute error rate: 2.2% for rule in ALERT_RULES: if should_alert(0.015, 0.022, 0.999, rule): print(f"ALERT [{rule.severity.upper()}]: {rule.name}") # Output: ALERT [PAGE]: critical Both conditions must hold. If only the short window fires, it's a transient spike. If only the long window fires, the incident already recovered — log it, don't page anyone.
A burn rate evaluator in Go
Here's a minimal implementation that evaluates alert rules against pre-fetched error rates (e.g., from a Prometheus recording rule):
package sloalert import "fmt" type SLO struct { Target float64 // e.g. 0.999 for 99.9% } func (s SLO) BurnRate(errorRate float64) float64 { budget := 1.0 - s.Target if budget == 0 { return 0 } return errorRate / budget } type AlertRule struct { Name string BurnRateThreshold float64 Severity string } type WindowRates struct { Long float64 Short float64 } func Evaluate(slo SLO, rates WindowRates, rules []AlertRule) []string { longBurn := slo.BurnRate(rates.Long) shortBurn := slo.BurnRate(rates.Short) var fired []string for _, rule := range rules { if longBurn >= rule.BurnRateThreshold && shortBurn >= rule.BurnRateThreshold { fired = append(fired, fmt.Sprintf( "[%s] %s — long=%.1fx short=%.1fx threshold=%.1fx", rule.Severity, rule.Name, longBurn, shortBurn, rule.BurnRateThreshold, )) } } return fired } Usage in a real system: call a Prometheus query API (or any metrics store) to get error rates for each window, pass them into Evaluate, then route the results to PagerDuty, Slack, or whatever you use. The evaluation logic stays pure and testable.
Reference thresholds for 99.9% SLO
| Burn rate | Budget exhausted in | Action |
|---|---|---|
| ≤ 1× | ≥ 30 days | Nothing |
| 3× | 10 days | Ticket (business hours) |
| 6× | 5 days | Ticket (urgent) |
| 14.4× | ~50 hours | Page |
| 36× | ~20 hours | Page immediately |
Adjust these if your SLO target changes. For 99.95%, your error budget is 5× smaller, so the same absolute error rate translates to a 5× higher burn rate — the thresholds hold, the inputs change.
The takeaway
Threshold-based alerting answers "is the error rate high right now?" Burn rate alerting answers "are we on track to violate our reliability commitment within hours?" The second question is the one worth waking someone up for.
The two-window check is the practical minimum to implement before going live. Start with the four standard rules (critical/high/medium/low), tune burn rate thresholds to match your SLO window, and revisit the numbers quarterly as your traffic patterns change. For a broader set of observability and reliability practices, the articles section covers related topics.
I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.