Enterprise Data and ML Platforms
Architect data lakes, lakehouses, streaming pipelines, hybrid cloud deployments and governed MLOps for an enterprise, and build working miniatures of each on your laptop.
Advanced · AI Engineering · about 26 hours · 5 modules · certificate on completion
Taught by Ismaìlè Rahimovì, Cataliná Ionescù, AI Learning Companions.
About this course
Enterprises rarely struggle to train a model; they struggle to give it trustworthy data at the right speed, run it in the places their data and rules allow, and prove later what it did and why. This course is about the platforms that make that possible. You'll learn why organisations modernise from scattered databases and warehouses, how a zoned data lake is organised and fed, how lakehouse table formats such as Delta Lake and Apache Iceberg add ACID transactions and time travel to files in object storage, how stream processors handle late and duplicated events with event-time windows and watermarks, how hybrid architectures keep data and models consistent across on-premises sites and clouds under residency and privacy rules, and how enterprise MLOps automates retraining while keeping humans accountable.
The labs follow a hydropower company's turbine data, and each builds a working miniature of a platform component on your laptop with Python, DuckDB and SQLite: a raw, clean and curated lake with incremental ingestion and late corrections; a small table format with a transaction log, optimistic concurrency, time travel, data skipping, compaction and vacuum, compared with a real Delta Lake log; a stream processor with watermarks and a real-time feature checked against batch SQL; cross-environment synchronisation with reconciliation, pseudonymisation and policy checks; and a governed retraining gate with a registry and audit log. The capstone is a cloud-agnostic lakehouse and MLOps architecture, documented with a diagram and decision records and backed by a runnable walking skeleton.
What you'll learn
- Architect a scalable, zoned data lake with incremental, idempotent ingestion
- Design lakehouse tables with ACID transactions, time travel and query optimisation, and explain how Delta Lake and Apache Iceberg work
- Design streaming pipelines with event-time windows, watermarks, state and real-time features for ML
- Construct hybrid cloud ML systems with reliable synchronisation, portable deployment, security and compliance
- Establish enterprise MLOps with maturity assessments, continuous training and governed model promotion
Skills you'll gain
Data lake design, Lakehouse architecture, Delta Lake and Iceberg, Stream processing, Event-time windowing, Real-time features, Hybrid cloud architecture, Data governance, MLOps maturity, Model governance
Tools
Python 3, DuckDB, SQLite, pandas, scikit-learn, Parquet, Delta Lake
Before you start
Python, SQL and the ML workflow from training to deployment; Foundations of AI System Design and Data Engineering Foundations: Batch and Streaming Pipelines are good preparation. Install Python 3.10 or newer, then: pip install pandas numpy scikit-learn duckdb pyarrow. The optional Delta Lake steps add pip install deltalake. Everything runs locally on sample data: no cloud account is needed.
Modules
Modernisation Strategy and Data Lakes
Why organisations modernise their data platforms, how a zoned data lake is organised, how to design its storage layer, and how data gets in: full, incremental, change data capture and streaming ingestion.
- Why organisations modernise their data platforms (12 min)
- Data lake architecture and zones (12 min)
- Designing the storage layer (12 min)
- Ingestion strategies (13 min)
- Hands-on lab: Build a zoned data lake with incremental ingestion
Lakehouse Architecture
Combine the lake's open storage with a warehouse's guarantees: lakehouse concepts, how Delta Lake and Apache Iceberg work, ACID transactions through a transaction log, and query optimisation.
- Lakehouse concepts (12 min)
- Delta Lake and Apache Iceberg (13 min)
- ACID transactions on a data lake (12 min)
- Query optimisation on lakehouse tables (12 min)
- Hands-on lab: Build a table format in miniature
Streaming Data Patterns
Process events as they happen: stream processing engines and delivery guarantees, event-driven architectures, event-time windows with watermarks and state, and real-time features for ML.
- Stream processing engines (12 min)
- Event-driven architectures (12 min)
- Windowing, watermarks and state (13 min)
- Real-time feature engineering (12 min)
- Hands-on lab: Event-time windows, watermarks and a real-time feature
Hybrid Cloud Architectures
Design ML systems that span on-premises sites and clouds: why organisations go hybrid, keeping data in step between environments, deploying the same model everywhere, and securing it all.
- Why hybrid and multi-cloud (12 min)
- Synchronising data between environments (12 min)
- Deploying models across clouds (12 min)
- Security and compliance across environments (13 min)
- Hands-on lab: Synchronise, reconcile and govern across environments
Enterprise MLOps
Run ML at enterprise scale: maturity models, automated CI, CD and continuous training pipelines, model governance with registries, model cards and approvals, and safe promotion of retrained models.
- MLOps maturity models (12 min)
- Automated pipelines: CI, CD and continuous training (12 min)
- Model governance (13 min)
- Continuous training and safe promotion (12 min)
- Hands-on lab: Govern continuous training with a promotion gate
Project: Design a cloud-agnostic lakehouse with streaming and MLOps
A hydropower company runs plants in two countries. Turbine sensors stream readings every few seconds, the maintenance system keeps work orders in an on-premises database, and operators' personal data must stay in its country of origin. The company wants a lakehouse that serves dashboards, data science and a fault-prediction model, on-premises and in a cloud region, without being tied to one vendor. Design it: an architecture diagram (start from the Mermaid diagram in architecture_template.md), the eight sections of the architecture document, and at least five architecture decision records (adr_template.md) for the choices that matter most, such as the table format, the stream processor, where training runs, how personal data is protected and the promotion gate. Then build a walking skeleton that runs locally, reusing and extending your labs: bronze, silver and gold tables with incremental ingestion, a streaming feature written to an online store (SQLite is enough) and to the offline store, a point-in-time training set, training with a governed promotion into a registry, and a reconciliation report between two environments.
Part of
The products this course names are trademarks of their owners, who haven't made or endorsed it.