Monitoring Models in Production: Drift, Data Quality and Alerts

Catch broken pipelines, measure drift with statistics you build yourself, track performance when labels arrive late, and design alerts and retraining policies people trust.

Advanced · AI Engineering · about 17 hours · 5 modules · certificate on completion

Taught by Rodrigoò Silvà, Tatiànna Morrisòn, AI Learning Companions.

About this course

A model can keep answering requests long after its answers stop being good. This advanced course teaches you to notice: what to record when a model makes predictions, how to catch the pipeline bugs that cause most urgent incidents, and how to tell real changes in the world from noise. You'll build the core of a monitoring system yourself with NumPy and SciPy rather than a library whose API changes, so you know exactly what each number means: population stability index, Kolmogorov-Smirnov and chi-squared tests, Jensen-Shannon distance and a domain classifier.

The labs follow one fraud model through 20 simulated weeks with a unit bug, an outage, a new payment channel, rising amounts and a new fraud pattern. You'll see which signals catch which incident, why p-values mislead at production volumes, how to report performance honestly when labels arrive weeks late, how to estimate it in the meantime and where that estimate fails, and how to design alerts from backtests rather than rules of thumb. The course ends with retraining policies compared over a simulated year and a capstone that proves a monitoring system with incident drills.

What you'll learn

  • Design prediction logs and label tables, and catch pipeline incidents with data quality checks
  • Implement PSI, KS, chi-squared and Jensen-Shannon drift measures and know what each one misses
  • Measure performance on mature labels, estimate it before labels arrive, and detect concept drift
  • Learn alert thresholds from history, backtest alert rules and build a model-health dashboard
  • Choose and justify a retraining policy, accounting for label delay and feedback loops

Skills you'll gain

Model monitoring, Data drift, Concept drift, Data quality, Population stability index, Statistical testing, Performance estimation, Alert design, Retraining strategy

Tools

Python, NumPy, SciPy, pandas, scikit-learn, Matplotlib

Before you start

Statistics for Data Science and AI (distributions and hypothesis tests) and Machine Learning Foundations with Python, or equivalent; Deploying and Serving Machine Learning Models helps for context. You need Python 3.10 or later with NumPy, SciPy, pandas, scikit-learn and Matplotlib; no monitoring product is required.

Modules

  1. What to monitor, and the data to do it with

    Record the prediction log and the labels that monitoring depends on, tell the kinds of drift apart, and catch broken pipelines with cheap data quality checks before any statistics run.

    • The prediction log: monitoring starts with records (12 min)
    • Kinds of drift, and why labels arrive late (12 min)
    • Data quality checks that catch broken pipelines (11 min)
    • Hands-on lab: Build the prediction log and catch pipeline incidents

  2. Measuring drift with statistics

    Implement PSI, the Kolmogorov-Smirnov and chi-squared tests and Jensen-Shannon distance yourself, learn what each one sees and misses, and why p-values mislead at production volumes.

    • The population stability index, done properly (13 min)
    • KS and chi-squared tests, and p-values at scale (12 min)
    • Jensen-Shannon distance, and choosing a measure (12 min)
    • Hands-on lab: Build a drift report from scratch

  3. Performance when labels arrive late

    Measure performance only on windows whose labels have matured, estimate it from calibrated scores in the meantime, and detect concept drift in the stream of errors once labels arrive.

    • Measuring performance on mature labels (12 min)
    • Estimating performance before labels arrive (12 min)
    • Detecting concept drift in the error stream (11 min)
    • Hands-on lab: Estimate performance without labels, then check it

  4. Alerts and dashboards people trust

    Learn thresholds from history, backtest alert rules before they page anyone, build a model-health dashboard, and investigate an alert with a runbook.

    • Designing alerts: thresholds, persistence and severity (13 min)
    • Dashboards for model health (11 min)
    • Investigating an alert (11 min)
    • Hands-on lab: Backtest alert rules and build a health dashboard

  5. Retraining triggers and closing the loop

    Choose when to retrain, retrain safely on the right window, and keep feedback loops and the monitoring system itself under watch.

    • When to retrain: schedules and triggers (12 min)
    • Retraining safely (12 min)
    • Feedback loops and monitoring the monitor (11 min)
    • Hands-on lab: Compare retraining policies over a simulated year

Project: A monitoring system for a payment-fraud model

The fraud team's model scores every instant payment, and the last incident went unnoticed for three weeks. Build the monitoring system that would have caught it. Starting from the course's simulator (run it with a new seed, or adapt it), design the prediction log and label table; write data quality checks; compute drift with at least three measures, with thresholds learnt from history and backtested; report performance on mature labels with an estimate for immature weeks; detect degradation in the error stream; and draw a dashboard. Then prove it works with incident drills: inject at least four incidents of different kinds (such as a unit bug, missing data, a new category, gradual covariate drift and concept drift) into a fresh simulation, and report which signals caught each one, and how quickly. Finish with a retraining policy, supported by a simulation, and a runbook for the on-call engineer.

Part of

The products this course names are trademarks of their owners, who haven't made or endorsed it.

Start the course free