SLOs Are a Contract, Not a Dashboard Metric
Service Level Objectives sound straightforward. Pick a metric, set a target, monitor it. In practice, getting SLOs right requires making uncomfortable decisions about what you will and won't invest engineering time in. The SLO isn't just a number — it's a statement about how much reliability your service commits to providing, and by extension, how much unreliability you're explicitly accepting.
I've rolled out SLO frameworks at two organizations. The first attempt failed because we set targets nobody believed in. The second succeeded because we tied error budgets to concrete operational decisions. Here's the difference.
Choosing the Right SLI
A Service Level Indicator is the metric that underpins your SLO. The most common mistake is choosing an SLI that's easy to measure rather than one that reflects user experience. CPU utilization is easy to measure. It tells you nothing about whether users can complete their transactions.
For request-driven services, availability and latency are the two SLIs that matter most. Availability means "what percentage of requests succeed?" Latency means "how fast are the successful ones?"
Defining "success" is where things get interesting. A 500 error is clearly a failure. A 400 error is the client's fault — should it count? A 200 response that took 30 seconds — is that a success or a failure? These edge cases matter because they shift your SLO by percentage points.
Our team's definitions:
# Prometheus recording rules for SLI calculation
groups:
- name: sli_availability
rules:
- record: sli:request_success:ratio_rate5m
expr: |
sum(rate(http_requests_total{code=~"2..|3.."}[5m]))
/
sum(rate(http_requests_total{code!~"4[0-24-9].|431"}[5m]))
# Excludes 400, 404, 429 from denominator (client errors)
# Includes 403, 431 as server-side auth failuresWe exclude 400 and 404 from both numerator and denominator — they're client errors that don't reflect service health. We include 403 because our auth service failing causes 403s, which is a service problem. We include 429 (rate limiting) in the denominator but not the numerator, because rate limiting is the service choosing to reject requests — that's degraded availability from the user's perspective even though it's intentional.
Error Budget Calculation
If your SLO is 99.9% availability over a 30-day rolling window, your error budget is 0.1% — roughly 43 minutes of downtime or equivalent failed requests per month. That's not a lot. But it's deliberately not a lot. The error budget exists to create tension between shipping speed and reliability.
# Error budget remaining (Prometheus)
- record: error_budget:remaining:ratio
expr: |
1 - (
(1 - sli:request_success:ratio_rate30d)
/
(1 - 0.999) # 99.9% SLO target
)When the error budget is healthy (above 50%), teams ship features aggressively. When it's low (below 25%), feature work pauses and reliability improvements take priority. When it's exhausted (0% or negative), all deployments stop except reliability fixes.
This sounds draconian. It is. But it's also the mechanism that makes SLOs operational rather than aspirational. Without error budget policies, an SLO is just a number on a dashboard that nobody acts on. With them, it's a decision framework that directly affects sprint planning.
Multi-Window Burn Rate Alerts
Traditional threshold alerts ("alert when error rate exceeds 1%") are noisy and poorly calibrated. A 1% error spike lasting 30 seconds is barely visible in the SLO. A sustained 0.3% error rate over 6 hours consumes a significant chunk of budget. Threshold alerts catch the first but miss the second.
Burn rate alerts solve this. A burn rate of 1x means you're consuming the error budget at exactly the expected rate — you'll exhaust it at the end of the SLO window. A burn rate of 14.4x means you'll exhaust a 30-day budget in roughly 2 days if the current error rate continues.
# Multi-window burn rate alerting
groups:
- name: slo_burn_rate
rules:
# Fast burn: 14.4x over 1h, confirmed over 5m
- alert: SLOHighBurnRate
expr: |
(
(1 - sli:request_success:ratio_rate1h) / (1 - 0.999)
) > 14.4
and
(
(1 - sli:request_success:ratio_rate5m) / (1 - 0.999)
) > 14.4
for: 2m
labels:
severity: critical
annotations:
summary: "High error burn rate - budget exhaustion in ~2 days"
# Slow burn: 3x over 6h, confirmed over 30m
- alert: SLOSlowBurnRate
expr: |
(
(1 - sli:request_success:ratio_rate6h) / (1 - 0.999)
) > 3
and
(
(1 - sli:request_success:ratio_rate30m) / (1 - 0.999)
) > 3
for: 5m
labels:
severity: warning
annotations:
summary: "Elevated error burn rate - budget exhaustion in ~10 days"The dual-window approach (long window + short window) prevents false alarms from brief spikes. The 1-hour window detects sustained issues, and the 5-minute confirmation window ensures the problem is still happening now — not a resolved spike from 45 minutes ago that's still dragging the 1-hour average.
Practical Error Budget Policies
Writing the policy is the easy part. Getting engineering leadership and product management to agree to it is the hard part. Here's the policy template we use, after several rounds of negotiation:
Budget above 75%: Standard development velocity. Feature work proceeds normally. Reliability work happens during planned maintenance windows.
Budget between 25-75%: Heightened awareness. Deploy monitors for the primary error contributors. Schedule reliability improvements for the next sprint. No change to feature velocity unless the trend is clearly declining.
Budget below 25%: Reliability sprint. 50% of engineering capacity redirected to reliability improvements. New features still ship but require additional review for potential reliability impact.
Budget exhausted: Feature freeze. Only reliability fixes and critical security patches deploy to production. Postmortem for each incident that consumed more than 10% of the budget. Resume normal development when budget recovers above 25%.
The specific thresholds aren't magic numbers. They're starting points that you calibrate based on your team's rhythm. The important thing is that they exist, they're written down, and engineering leadership has signed off on them. A policy that only the SRE team knows about isn't a policy — it's a suggestion.
Latency SLOs
Availability SLOs measure whether requests succeed. Latency SLOs measure whether they succeed fast enough. The typical formulation is "99% of requests complete in under 300ms." This is a percentile target — the p99 latency must be below 300ms.
Don't use average latency for SLOs. Averages hide the tail. A service with 50ms average latency might have a p99 of 2 seconds — meaning 1% of your users wait 40x longer than the average. The users in that tail aren't comforted by the fast average.
We track p50, p95, and p99 latency, with the SLO target on p99. The p50 tells us about typical user experience. The gap between p50 and p99 tells us about consistency. A service where p50 is 30ms and p99 is 280ms has a very different character than one where p50 is 250ms and p99 is 290ms — even though they have similar p99 values. The first is fast but occasionally slow. The second is consistently mediocre.
Rolling Out SLOs to an Organization
The technical setup — SLI definitions, Prometheus rules, Grafana dashboards — is the straightforward part. The organizational rollout is where SLO adoption succeeds or fails.
Start with one service. Pick the one that's most critical and most understood by the team. Define its SLIs, set conservative targets (99.5% instead of 99.9% — you can tighten later), and run the error budget calculation in shadow mode for a month. Shadow mode means you track the metrics and generate reports, but you don't enforce the error budget policies yet. This builds confidence in the measurements before you attach consequences.
During the shadow period, you'll discover that your SLI definitions need adjustment. Maybe your availability calculation counts health check traffic, which inflates the success rate. Maybe your latency percentiles include warmup requests after deployments, which deflates performance. These measurement quirks are normal — every team finds them — and it's much better to discover them during shadow mode than after you've declared a feature freeze based on bad data.
SLO Documentation Template
Each service's SLO gets a one-page document that answers five questions:
What are we measuring? (The SLI definition, including what counts as success, what's excluded, and where the data comes from.)
What's the target? (The SLO percentage and the window — 30-day rolling, calendar month, etc.)
What's the budget? (The calculated error budget in concrete terms — minutes of downtime, number of failed requests, etc.)
What happens at each threshold? (The error budget policy — what changes when budget drops below 75%, 25%, 0%.)
Who's responsible? (The team that owns the SLI measurement, the team that responds to budget alerts, and the escalation path when the budget is exhausted.)
This document lives in the service's repository, not in a wiki that nobody reads. When the on-call engineer gets a burn rate alert at 3 AM, they shouldn't need to search Confluence for the error budget policy. It should be in the same repo as the runbook and the alerting rules.
Common SLO Anti-Patterns
Setting targets at 100%. Nothing is 100% reliable. An SLO of 100% means your error budget is zero, which means any failure — no matter how brief — exhausts the budget and triggers a feature freeze. This isn't ambitious; it's meaningless, because you'll either never enforce it (making it useless) or always enforce it (making feature work impossible).
Too many SLOs per service. I've seen teams define separate SLOs for availability, p50 latency, p95 latency, p99 latency, throughput, and error rate. That's six error budgets to track, six sets of alerts, and six potential feature freezes. Start with two: availability and p99 latency. Add more only if you have evidence that the existing SLOs don't capture a failure mode that affects users.
SLOs without error budget policies. An SLO without consequences is a dashboard metric. It tells you how the service performed, but it doesn't drive behavior. The entire value proposition of SLOs over traditional monitoring is that they create a feedback loop between reliability and development velocity. Without the error budget policy, you've built an expensive monitoring system with no action mechanism.