Site Reliability Engineering: SLOs, Error Budgets and Incidents
Run services by the numbers: SLIs, SLOs and error budgets, burn-rate alerts, toil and on-call, incident response and postmortems, capacity, safe releases and chaos experiments.
Intermediate · Cloud & DevOps · about 18 hours · 5 modules · certificate on completion
Taught by Lachlaneè Mitchèll, Rodrigoò Silvà, AI Learning Companions.
About this course
Reliability isn't an accident or a heroic effort; it's a set of numbers and practices a team agrees and keeps. This course teaches site reliability engineering through one checkout service and its month of traffic. You'll choose SLIs and set SLOs from real measurements, turn them into error budgets, write the policy for when the budget runs out, and replay the month through alerting rules to see why multiwindow, multi-burn-rate alerts page quickly for real problems and stay quiet otherwise. You'll measure a team's toil and on-call load, and build automation that's safe by default, idempotent and tested.
Then you'll handle what goes wrong: run an incident with clear roles, rebuild its timeline in UTC from logs, alerts and chat, measure what it cost in budget, and write a blameless postmortem. The last module plans capacity with queueing maths and headroom, judges canary releases with a statistical test, and runs a chaos experiment on two local services with a hypothesis and abort conditions. Every lab was run, and its checks quote the real output. Observability with Prometheus, Grafana and OpenTelemetry implements the alerts in PromQL, and Incident Response Fundamentals covers the extra steps that security incidents need.
What you'll learn
- Define SLIs from user journeys and set SLOs with targets, windows and a documented rationale
- Calculate error budgets and burn rates, write an error budget policy and design multiwindow burn-rate alerts
- Measure toil and on-call load, and automate operational tasks so that they're idempotent, safe and tested
- Run an incident with clear roles and communication, build a UTC timeline and write a blameless postmortem
- Plan capacity with headroom, gate canary releases statistically and run chaos experiments with abort conditions
Skills you'll gain
Service level objectives, Error budgets, Burn-rate alerting, Toil reduction, On-call design, Incident command, Blameless postmortems, Capacity planning, Progressive delivery, Chaos engineering
Tools
Python, pandas, pytest, FastAPI, uvicorn, httpx
Before you start
Python and the command line at the level of Python Scripting and Linux for DevOps; pandas basics help. Observability with Prometheus, Grafana and OpenTelemetry is a good companion for the monitoring side, and CI/CD and Automated Testing for releases. You need Python 3.10 or later with pandas, numpy, pytest, fastapi, uvicorn and httpx.
Modules
SLIs and SLOs: defining reliability
Reliability as users experience it: why 100% is the wrong target, how to choose service level indicators, and how to set objectives worth keeping.
- Reliability as a feature, and why 100% is wrong (12 min)
- Choosing service level indicators (13 min)
- Setting service level objectives (12 min)
- Hands-on lab: Measure SLIs and choose SLOs from a month of traffic
Error budgets and alerting on burn rate
Turn an SLO into an error budget, agree what happens when it runs out, and page on how fast it's burning, with the maths behind multiwindow, multi-burn-rate alerts.
- Error budget maths (12 min)
- Error budget policies (11 min)
- Alerting on burn rate (14 min)
- Hands-on lab: Spend an error budget and replay a month of alerts
Toil, automation and on-call
Measure the operational work that grows with the service and crowds out engineering, automate it safely, and design an on-call rotation that people can sustain.
- Toil: what it is and how to measure it (11 min)
- Automating operations safely (13 min)
- Sustainable on-call (12 min)
- Hands-on lab: Measure toil and on-call load, then automate safely
Incident response and blameless postmortems
Run an incident with clear roles and steady communication, keep a timeline as you go, mitigate before you diagnose, and learn from it with a blameless postmortem.
- Incident roles and severity (12 min)
- Mitigation, communication and the timeline (13 min)
- Blameless postmortems (13 min)
- Hands-on lab: Build an incident timeline and write the postmortem
Capacity, progressive delivery and chaos engineering
Plan capacity with headroom and queueing in mind, release changes gradually with automated canary analysis, and test resilience with careful chaos experiments.
- Capacity planning (13 min)
- Release engineering and progressive delivery (14 min)
- Chaos engineering basics (12 min)
- Hands-on lab: Gate canaries and run a chaos experiment
Project: Define, defend and test the reliability of a service
Take a service you run, or the course's checkout, shop and recommender services, and make its reliability explicit and defended. Define two or three SLIs for its main user journeys with exact definitions of good and valid events, set SLOs with a window and a rationale, and work out the error budgets. Write an error budget policy the product owner could sign. Design multiwindow, multi-burn-rate alerts for each SLO and replay at least four weeks of real or generated data through them to show their detection times and false alarms. Measure toil for at least two weeks (or use the course's work log), automate one source with a safe, tested script, and set out the on-call rotation and paging policy. Finally, run a chaos experiment with a stated hypothesis and abort conditions, fix what it reveals, and write a blameless postmortem for the failure it found or for a game day you ran.
Part of
The products this course names are trademarks of their owners, who haven't made or endorsed it.