Foundations of AI System Design

Scope a machine learning system and design its data, features, models, deployment and monitoring, with a design document that catches data leakage before it ships.

Intermediate · AI Engineering · about 21 hours · 5 modules · certificate on completion

Taught by Hiroshì Yamadà, Wei-Chéng Liú, AI Learning Companions.

About this course

Most machine learning projects that fail in production don't fail because of the algorithm. They fail because the problem was framed loosely, the training data contained information the model would never see in real use, nobody decided how predictions would reach the people who act on them, or nobody noticed when the world changed. This course teaches the design work around the model: scoping a project with its stakeholders, requirements and success metrics; collecting, storing and versioning data; engineering features that exist at prediction time; selecting, tuning and evaluating models honestly; and deploying and monitoring them.

You'll work through one system end to end: a same-day delivery company that wants to warn customers before an order arrives late. You'll cost the trivial strategies and a rule baseline, build a versioned Parquet snapshot with DuckDB, catch three kinds of data leakage, compare models with time-aware cross-validation and set the warning threshold by cost, then package the model, batch-score a new week, run a shadow candidate and check for drift. Each lab adds sections to a design document, and the capstone asks you to design and prototype a new system, from ingestion to monitoring, on a fresh dataset with leaks of its own to find.

What you'll learn

  • Turn a business problem into an ML problem statement with stakeholders, requirements, success metrics and a feasibility check
  • Design data collection, storage and versioning for an ML system, and handle missing data without leakage
  • Detect and fix target, preprocessing, temporal and group leakage in ML pipelines
  • Engineer, encode, scale and select features inside reproducible pipelines
  • Select, tune and evaluate models with validation that matches production and thresholds set by cost
  • Choose deployment and release strategies, and plan monitoring and retraining in a design document

Skills you'll gain

ML system design, Problem framing, Data management, Data versioning, Data leakage detection, Feature engineering, Model evaluation, Deployment strategy, Model monitoring, Design documents

Tools

Python 3, pandas, scikit-learn, DuckDB, Parquet

Before you start

Python and pandas basics, and supervised learning essentials: splitting data, fitting and scoring a scikit-learn model (Machine Learning Foundations with Python covers them; Data Analysis with Python and pandas helps). Install Python 3.10 or newer, then: pip install pandas scikit-learn duckdb joblib. The labs generate their own synthetic data.

Modules

  1. Project Scoping and Requirements

    Turn a vague request into a machine learning problem worth solving: the decision it supports, its stakeholders and requirements, how success is measured and whether it's feasible, written into the first sections of a design document.

    • From a business goal to an ML problem statement (12 min)
    • Stakeholders, requirements and constraints (12 min)
    • Success metrics and feasibility (13 min)
    • Hands-on lab: Frame the problem and set the baselines

  2. Data Engineering and Management

    Plan where training data comes from and how it is labelled, stored, versioned and cleaned, so any model can be traced to the exact data it learnt from.

    • Collecting data and labels (12 min)
    • Storing data for ML: databases, warehouses, lakes and Parquet (13 min)
    • Versioning data and handling missing values (14 min)
    • Hands-on lab: Build a versioned training snapshot with DuckDB

  3. Advanced Feature Engineering

    Build features that carry signal and exist at prediction time: extraction from raw records, leakage detection, categorical encodings, scaling and selection, all fitted inside pipelines.

    • Extracting features from raw data (12 min)
    • Data leakage: how it happens and how to catch it (14 min)
    • Encoding categories and scaling numbers (12 min)
    • Selecting features (12 min)
    • Hands-on lab: Catch data leakage and engineer a better feature

  4. Model Development and Evaluation

    Choose candidate models, validate them the way they'll be used, tune them without fooling yourself, and judge them with metrics and thresholds that reflect what errors cost.

    • Choosing a model family (12 min)
    • Cross-validation that matches production (12 min)
    • Tuning hyperparameters (12 min)
    • Metrics and thresholds that match the decision (13 min)
    • Hands-on lab: Select, tune and evaluate with time-aware validation

  5. Deployment and Monitoring

    Choose how predictions reach the people and systems that use them, release new models safely, and plan the monitoring and retraining that keep a model useful.

    • Deployment strategies: batch, online and edge (12 min)
    • Serving a model and releasing it safely (12 min)
    • Monitoring and retraining (13 min)
    • Hands-on lab: Package, batch-score, shadow and monitor

Project: Design and prototype an SLA-breach prediction system

A software company's support team misses its resolution deadlines (SLAs) on about a third of tickets and wants to know, the moment a ticket is created, which ones are at risk, so a team lead can reassign or escalate them. make_tickets.py generates ten weeks of tickets with their outcomes (tickets.csv) and a new week without outcomes (new_tickets.csv); tickets_data_dictionary.csv says when each column becomes known. Design the system in a design document (all eleven sections of the template) and build a working prototype: ingest the CSV into a versioned Parquet snapshot with a manifest, audit the columns for leakage (several are known only after the ticket is answered or resolved), engineer features, compare at least three candidates with time-aware cross-validation, choose a threshold from costs you justify, test once on the last two weeks, batch-score the new week with a versioned artefact, and compute drift between the training data and the new week. Then recommend batch or online serving for this system, with a release and monitoring plan.

Part of

The products this course names are trademarks of their owners, who haven't made or endorsed it.

Start the course free