LLMOps: Evaluating, Tracing and Monitoring LLM Applications
Run LLM features like production software: golden sets and judges, regression gates with replayed recordings, OpenTelemetry traces, cost and guardrail metrics, and A/B tests.
Advanced · AI Engineering · about 17 hours · 5 modules · certificate on completion
Taught by Dariusz Wojciechowskì, Amèlia Whitè, AI Learning Companions.
About this course
Shipping an LLM feature is the start of the work, not the end: prompts change, models are upgraded, costs creep, guardrails misfire and users find questions nobody tested. This advanced course teaches the operational discipline that keeps LLM applications dependable, using one customer-support assistant throughout. You'll build a golden set as a versioned asset, score it with programmatic checks before reaching for an LLM judge, run judges as cached, versioned components, and compare versions pairwise without being fooled by position bias.
You'll then gate releases the way software teams do, with case-by-case regression reports, McNemar's test, safety blockers and recorded answers that make the suite deterministic; trace every request with OpenTelemetry using the GenAI semantic conventions; turn usage logs into cost per successful task, latency tails and guardrail precision; and run A/B tests with the right unit, size and checks. Every lab runs offline with recorded or simulated data, and the code that calls Claude reads the model from the CLAUDE_MODEL environment variable, so it works with whichever current model you choose.
What you'll learn
- Build versioned golden sets and score them with programmatic checks, cached rubric judges and pairwise comparisons
- Gate prompt and model changes with case-by-case regression, McNemar's test, blockers and replayed recordings
- Trace LLM requests with OpenTelemetry spans and GenAI conventions, and analyse latency, retries and tokens
- Report cost per feature and per successful task, latency tails and guardrail precision from usage logs
- Design and analyse A/B tests of LLM changes with the right unit, size and sample ratio checks
Skills you'll gain
LLMOps, LLM evaluation, LLM-as-a-judge, Regression testing, OpenTelemetry tracing, Cost tracking, Guardrail metrics, A/B testing, Python
Tools
Claude API, Anthropic Python SDK, OpenTelemetry, pytest, pandas, SciPy, statsmodels
Before you start
Engineering with Foundation Models (evaluation and judge validation) or equivalent experience building LLM features, Python with pandas, and basic statistics (Statistics for Data Science and AI). You need Python 3.10 or later; the labs install anthropic, opentelemetry-sdk, SciPy, statsmodels and pytest. An Anthropic API key is optional: every lab runs offline.
Modules
Evaluation datasets and scoring
Build a golden set as a versioned asset, score it with programmatic checks first, and run rubric judges and pairwise comparisons inside a harness that caches, versions and reports uncertainty.
- Golden sets as versioned assets (12 min)
- Programmatic and reference-based checks (12 min)
- Rubric judges and pairwise comparison in a harness (13 min)
- Hands-on lab: Build an evaluation harness for a support assistant
Regression testing prompts and models
Version prompts and model choices like code, compare runs case by case with a paired test and release blockers, and make the suite deterministic by recording and replaying model answers.
- Prompts and models as versioned dependencies (11 min)
- Case-by-case regression and the release gate (12 min)
- Record and replay (11 min)
- Hands-on lab: Gate a prompt change with replayed regression tests
Tracing LLM applications
Instrument every step of an LLM request with OpenTelemetry spans, name model calls with the GenAI semantic conventions, and read traces to find latency, retries, token use and evaluation cases.
- Traces and spans for LLM pipelines (12 min)
- GenAI semantic conventions and what to capture (12 min)
- Reading traces: latency, failures and evaluation data (11 min)
- Hands-on lab: Instrument an assistant with OpenTelemetry and analyse its traces
Cost, latency and guardrail metrics
Turn usage fields into cost per feature, tenant and successful task, measure time to first token and tail latency, and judge guardrails by their precision and misses rather than their block rate.
- Cost per call, per feature and per successful task (12 min)
- Latency: time to first token, speed and tails (11 min)
- Guardrail and refusal metrics (12 min)
- Hands-on lab: Cost, latency and guardrail reports from usage logs
Online feedback and A/B tests
Collect feedback that can be linked and trusted, run A/B tests of prompt and model changes with the right unit, size and checks, and keep evaluating after launch.
- Feedback you can learn from (11 min)
- A/B testing LLM changes (13 min)
- From experiment to continuous evaluation (11 min)
- Hands-on lab: Analyse an A/B test of two prompt versions
Project: Run LLMOps for a customer-support assistant
Quillfern Homeware's support assistant is about to get a new prompt and possibly a new model, and the team wants to release changes like this safely and often. Build the LLMOps around it. Extend the golden set to at least 40 cases in at least five slices, with programmatic checks for every case and a rubric judge where checks can't reach; wire it into a regression gate with case-by-case diffs, McNemar's test, safety blockers and replayed recordings that run in pytest; instrument the assistant with OpenTelemetry spans using the GenAI conventions and analyse a batch of traces; report cost per feature and per successful task, latency percentiles and guardrail precision; and plan the A/B test for the change, with the unit, metrics, sample size and analysis, run on simulated data. Everything must run offline with recorded answers; live Claude calls are optional and read the model from CLAUDE_MODEL.
Part of
- Generative AI Engineer Professional Certificate
- LLMOps: Large Language Models in Production Specialization
The products this course names are trademarks of their owners, who haven't made or endorsed it.