Deploying and Serving Machine Learning Models
Serve models fast and safely: validated APIs, ONNX, gRPC and micro-batching, lean containers, canary and shadow releases, and load tests that tell the truth.
Advanced · AI Engineering · about 17 hours · 5 modules · certificate on completion
Taught by Mouhamadouù Bâ, Wei-Chéng Liú, AI Learning Companions.
About this course
A model is only useful once it answers requests reliably, quickly and safely, and most of the hard problems in getting there aren't about the model. This advanced course takes one transaction-risk model all the way to production shape on your own machine: a batch job that survives crashes, a FastAPI server that validates every feature and won't take traffic until the model is verified and warm, an ONNX export proven to make the same decisions, a gRPC service with deadlines, micro-batching measured rather than assumed, and a lean container that knows its CPU budget.
The last module is about releasing without fear: a sticky canary routed by a stable hash, a shadow that can never affect a response, comparisons on the same live requests, and an open-loop load test that avoids coordinated omission and turns into a capacity plan. Every lab was run end to end and its checks quote the real output. It builds on Advanced FastAPI and Production Deployment and Containerisation and Orchestration with Docker and Kubernetes for the general web and container skills, and leads on to Monitoring Models in Production: Drift, Data Quality and Alerts.
What you'll learn
- Build batch scoring jobs that resume after failure and record the model behind every prediction
- Serve models with FastAPI using strict validation, verified loading, warm-up and readiness checks
- Export scikit-learn and PyTorch models to ONNX and prove the exported model makes the same decisions
- Serve over gRPC, add micro-batching where measurements justify it, and package lean serving images
- Release models through sticky canaries and shadows, and size services from open-loop load tests
Skills you'll gain
Model serving, Batch inference, FastAPI, ONNX, gRPC, Dynamic batching, Canary releases, Shadow deployment, Load testing, Capacity planning
Tools
FastAPI, pydantic, onnxruntime, skl2onnx, gRPC, Docker, httpx, pytest
Before you start
MLOps Foundations: Experiments, Models and Reproducibility with MLflow (or experience packaging models), and Introduction to FastAPI and Modern Web Services or equivalent FastAPI basics. Docker basics help for module 4. You need Python 3.10 or later and Docker; the labs install scikit-learn, FastAPI, onnxruntime, skl2onnx and grpcio with pip.
Modules
Serving patterns and the prediction contract
Set latency and throughput objectives, build batch jobs that survive failure, and define the contract a prediction service keeps with its clients, over REST or gRPC.
- Latency, throughput and serving objectives (12 min)
- Batch inference jobs that survive failure (12 min)
- REST, gRPC and the prediction contract (12 min)
- Hands-on lab: Build a resumable batch scoring job
A production model server with FastAPI
Load and warm up a model once, validate every request against the features it was trained on, report liveness and readiness separately, and handle concurrency the way inference needs.
- Loading, warm-up and readiness (12 min)
- Validating prediction requests with pydantic (12 min)
- Concurrency in a model server (12 min)
- Hands-on lab: Serve the risk model with validation, warm-up and readiness
Portable, fast models with ONNX
Export scikit-learn and PyTorch models to ONNX, run them with onnxruntime, prove the exported model makes the same decisions, and measure what the change buys you.
- ONNX and onnxruntime (10 min)
- Exporting scikit-learn and PyTorch models (13 min)
- Parity, precision and speed (12 min)
- Hands-on lab: Export the risk model to ONNX, prove parity and benchmark it
gRPC, micro-batching and containers
Serve the ONNX model over gRPC with deadlines and status codes, batch concurrent requests into single model calls, and package a lean serving image that knows its CPU budget.
- gRPC model services (12 min)
- Dynamic batching (12 min)
- Packaging a model server in a container (12 min)
- Hands-on lab: Serve over gRPC, batch requests and build a container
Safe releases, load testing and capacity
Release a candidate model through a sticky canary and a shadow, compare it on live requests, load-test the service without fooling yourself, and turn the results into a capacity plan.
- Canary and shadow releases in practice (13 min)
- Load testing a model service (13 min)
- Capacity, scaling and cost (11 min)
- Hands-on lab: Release a candidate with canary and shadow, then load-test it
Project: Ship a transaction-risk service ready for production traffic
A payments team wants the transaction-risk model scoring live card payments, with a nightly batch re-score of the day's transactions for the fraud analysts. Build the serving system for it from the course's model and data (or a model of your own with at least five features). Export the model to ONNX and prove parity; serve it over HTTP with strict validation, verified loading, warm-up, separate liveness and readiness, a bounded batch endpoint and the model version on every answer; add either a gRPC endpoint or micro-batching, justified by measurement; package it as a lean, non-root container image with a health check and a set thread budget; write the resumable nightly batch job; release a second model version through a sticky canary with a shadow comparison and a documented rollback; and load-test the service open-loop to find its knee and size it for a peak of 300 requests per second with headroom.
Part of
The products this course names are trademarks of their owners, who haven't made or endorsed it.