Observability with Prometheus, Grafana and OpenTelemetry

Instrument services with metrics, logs and traces, query them with PromQL, alert on SLO burn rates you can test, and investigate problems with Grafana dashboards and distributed traces.

Intermediate · Cloud & DevOps · about 18 hours · 5 modules · certificate on completion

Taught by Katarína Novákovà, Chukwuemekà Obiechè, AI Learning Companions.

About this course

When a service slows down at two in the morning, you need answers fast: what's broken, for whom, since when and why. Observability gives you them, from three signals working together: metrics that show trends and trigger alerts, traces that follow a request through every service, and logs that explain each event. This course builds all three into a pair of Python services, orders and payments, running beside a complete open-source stack on your laptop: Prometheus, Alertmanager, Grafana, the OpenTelemetry Collector and Jaeger, all in Docker Compose.

You'll instrument FastAPI code with prometheus_client and OpenTelemetry, write structured logs that carry trace IDs, and query metrics with PromQL, practising against unit tests with known answers. You'll turn queries into recording rules and alerts, including multiwindow burn-rate alerts for an SLO, prove them with promtool before they reach production, and route them with Alertmanager. Finally you'll provision Grafana dashboards as code, derive metrics from spans, follow exemplars from a latency graph to the slow request's trace, and track down faults injected into the services. Every lab has expected output you can check, and the capstone makes a new service observable end to end. Choosing SLOs and running incidents are left to the Site Reliability Engineering course; here SLOs are for alerting.

What you'll learn

  • Instrument a Python service with Prometheus metrics, structured JSON logs and OpenTelemetry traces
  • Query metrics with PromQL: rates, aggregations, error ratios, percentiles and joins
  • Write recording and alerting rules, including SLO burn-rate alerts, and unit-test them with promtool
  • Route, group and inhibit notifications with Alertmanager
  • Build Grafana dashboards as code, and follow a problem from an alert to a trace and its logs

Skills you'll gain

Observability, Prometheus, PromQL, Grafana dashboards, OpenTelemetry, Distributed tracing, Alerting, SLO-based alerting, Structured logging

Tools

Prometheus, Grafana, OpenTelemetry Collector, Jaeger, Alertmanager, Docker Compose, Python 3

Before you start

Basic Python (Python Programming with PyCharm covers it) and the command line (Linux Command Line Essentials). Some Docker Compose helps; Containerisation and Orchestration with Docker and Kubernetes covers it. You need Docker with Compose v2, Python 3.10 or later with pip, and 8 GB of memory. Everything runs locally.

Modules

  1. Observability Foundations: Metrics, Logs and Traces

    What observability adds to monitoring, what metrics, logs and traces each tell you, how Prometheus collects metrics, and how OpenTelemetry and the course's local stack fit together.

    • Monitoring, observability and the three signals (12 min)
    • How Prometheus collects and stores metrics (13 min)
    • OpenTelemetry and the local stack (12 min)
    • Hands-on lab: Start the observability stack

  2. Instrumenting a Python Service

    Add the three signals to a FastAPI service: Prometheus metrics with prometheus_client, structured JSON logs that carry trace IDs, and OpenTelemetry traces that follow requests from one service to the next.

    • Metrics with prometheus_client (13 min)
    • Structured logs that correlate (11 min)
    • Tracing with OpenTelemetry in Python (13 min)
    • Hands-on lab: Instrument the orders service

  3. Querying with PromQL

    Select and filter series, turn counters into rates, aggregate across instances, and compute error ratios, percentiles and joins with PromQL, practising against unit tests with known answers.

    • Selectors, instant vectors and range vectors (12 min)
    • Rates, counters and aggregation (13 min)
    • Histograms, ratios and joins (13 min)
    • Hands-on lab: Practise PromQL against known answers

  4. Recording Rules, Alerts and SLO-Based Alerting

    Precompute queries with recording rules, write alerting rules that page on symptoms, route and quieten notifications with Alertmanager, and alert on SLO error-budget burn rates, all tested before deployment.

    • Recording rules (12 min)
    • Alerting rules and Alertmanager (13 min)
    • SLO-based alerts with burn rates (13 min)
    • Hands-on lab: Burn-rate alerts you can test

  5. Dashboards and Tracing in Practice

    Build Grafana dashboards as code, use traces and span-derived metrics from the Collector, choose a sampling strategy, and connect metrics, traces and logs to find the cause of a problem.

    • Building Grafana dashboards (13 min)
    • Traces in practice: the Collector, sampling and Jaeger (12 min)
    • Connecting the signals (11 min)
    • Hands-on lab: Dashboards, exemplars and a fault to find

Project: Make a new service observable end to end

The shop is adding an inventory service: orders will call it to reserve stock before charging the payment. Write the service in FastAPI and add it to the stack, then make it as observable as orders and payments: RED metrics with low-cardinality labels and a business metric, structured logs with trace IDs, and OpenTelemetry spans that continue the orders trace, with attributes and events where they help. Define an availability SLO and a latency SLO for placing an order, and implement them as recording rules and multiwindow burn-rate alerts with promtool unit tests; route pages and tickets with Alertmanager. Provision a Grafana dashboard covering all three services, with exemplars linking to traces. Finally, inject a fault into inventory, let the alerts find it, and write up the investigation as a short runbook entry.

Part of

The products this course names are trademarks of their owners, who haven't made or endorsed it.

Start the course free