SLO burn rate alerts: an interactive error budget simulator
An SLO such as “99.9% of requests succeed over 30 days” comes with an error budget: the 0.1% of requests allowed to fail, the same as 43 minutes of total outage a month. Burn rate is how fast that budget is being spent, where 1x would use it up in exactly 30 days. Alerting on burn rate pages people when the budget is in real danger, instead of whenever an error rate crosses a line.
The playground runs three days of a service’s error rate, with an outage, a slow leak and a brief blip, past three alerting rules. Switch incidents on and off and see who gets paged, when, and for what.
A 99.9% availability SLO allows 0.1% of requests to fail: over 30 days, the same as 43.2 minutes of total outage. These three days hold a 12-minute outage, 60% of requests failing; a slow leak, about 0.8% failing for 20 hours; a 2-minute blip, 5% failing, and use 41% of the month's budget. The leak hovers around the 1% threshold, so that rule pages and clears over and over: 42 pages in all. The 0.2% threshold catches it at once. Burn-rate alerts page 4.5 h in, once the leak has burned enough budget to matter. Both thresholds page for the blip, which used under 0.3% of the budget; the burn-rate rules stay quiet, because an hour's burn rate barely moves. All three page for the outage within 2 minutes.
Why fixed thresholds page badly
A 12-minute outage with 60% of requests failing uses about 17% of the month’s budget, and every rule pages for it within two minutes. The other incidents separate them. A leak of about 0.8% errors for 20 hours uses about 22% of the budget, more than the outage. A 1% threshold sits inside its noise, so it pages and clears 40 times. A 0.2% threshold catches it at once, but it also pages for a 2-minute blip that used a quarter of one percent of the budget, and on a real service with bursty errors it would page all week. There is no threshold that is both quiet and complete, because the error rate alone says nothing about how much budget is at stake.
Multi-window, multi-burn-rate alerts
The rules from Google’s SRE workbook page on two conditions. A fast burn, 14.4 times the sustainable rate over the last hour, spends 2% of the month’s budget in that hour. A slow burn, 6 times over the last six hours, spends 5% in that time. Each also checks a short window, 5 or 30 minutes, so the alert clears soon after the problem stops instead of an hour later. In the playground the burn-rate rules page for the outage after 2 minutes, for the leak after about 4.5 hours, once it has spent enough budget to be worth waking someone, and never for the blip.
- Page on budget, ticket on trends. A third rule, about 1x over three days, opens a ticket rather than a page: the budget will run out this month if nothing changes.
- Measure what users feel. Count failed or too-slow requests at the load balancer or client, not CPU or queue depth, which are causes, not symptoms.
- Low traffic needs care. With few requests, a handful of failures is a big percentage; use longer windows or synthetic traffic.
- Spend the budget. When budget is left at the end of the month, ship faster; when it is gone, prioritise reliability. That trade, not the alerts, is the point of an SLO.
Burn-rate alerts in Prometheus
# Prometheus: page on a fast or a slow burn of a 99.9% SLO's error budget
groups:
- name: checkout-slo
rules:
- alert: ErrorBudgetFastBurn # 2% of the month's budget in an hour
expr: |
( sum(rate(http_requests_total{code=~"5.."}[1h])) / sum(rate(http_requests_total[1h])) ) > (14.4 * 0.001)
and
( sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) ) > (14.4 * 0.001)
labels: { severity: page }
- alert: ErrorBudgetSlowBurn # 5% of the month's budget in six hours
expr: |
( sum(rate(http_requests_total{code=~"5.."}[6h])) / sum(rate(http_requests_total[6h])) ) > (6 * 0.001)
and
( sum(rate(http_requests_total{code=~"5.."}[30m])) / sum(rate(http_requests_total[30m])) ) > (6 * 0.001)
labels: { severity: page }In a system design interview
When asked how you would monitor the system, name SLOs and how they page: “Checkout’s SLO is 99.9% of requests succeeding in under 1 second over 30 days, measured at the load balancer. On-call is paged on burn rate: 14.4x over an hour or 6x over six hours, each confirmed by a short window. Slower burns open a ticket. Dashboards show budget remaining, and when it runs out, risky launches wait.” Expect the follow-ups: latency SLOs, SLOs for batch jobs, and alerts for dependencies you do not own.
Read the guide to SLOs and error budgets or observability and operations, see canary deployments for catching bad releases before they burn budget, or watch a strong candidate design a metrics and monitoring system.
Questions people ask
What is an error budget?
The amount of failure an SLO allows. A 99.9% availability SLO over 30 days permits 0.1% of requests to fail, equivalent to about 43 minutes of complete outage a month. Teams spend the budget on releases and experiments, and slow down when it runs out.
What is burn rate?
How fast the error budget is being consumed relative to the SLO: the current error rate divided by the allowed error rate. A burn rate of 1 uses exactly the whole budget over the SLO period; a burn rate of 14.4 sustained for an hour uses 2% of a 30-day budget.
Why alert on burn rate instead of error rate thresholds?
A fixed threshold cannot tell a brief, harmless spike from a sustained problem, so low thresholds page too often and high ones miss slow leaks or flap. Burn-rate alerts page when a meaningful share of the budget is being spent, which matches how much users are actually affected.
What are multi-window, multi-burn-rate alerts?
The alerting scheme from Google’s SRE workbook: page if the burn rate exceeds 14.4 over 1 hour and 5 minutes, or 6 over 6 hours and 30 minutes, and open a ticket for slower burns over days. The long window measures significance; the short one makes the alert reset quickly once the problem stops.
How do you choose an SLO?
From what users need and what the system can achieve: measure current reliability, pick a target users would not notice slightly missing, and express it as good events over valid events in a window, such as 99.9% of checkout requests succeeding in under a second over 30 days.
What should an SLO measure?
Symptoms users experience, such as failed or slow requests measured at the load balancer or the client, rather than causes like CPU usage or queue depth. Causes belong on dashboards for diagnosis; symptoms belong in SLOs and pages.
What happens when the error budget runs out?
By agreement between product and engineering, risky changes slow or stop and reliability work takes priority until the budget recovers. When budget remains, teams can release faster. This trade-off, made explicit, is the main purpose of SLOs.