Data Quality, Testing and Data Contracts
Measure data quality, test data in pipelines with pandera, agree data contracts between producers and consumers, and catch and handle bad data before it does harm.
Intermediate · Data Engineering · about 16 hours · 5 modules · certificate on completion
Taught by Tatiànna Morrisòn, Ximènà Castillò, AI Learning Companions.
About this course
Bad data rarely announces itself. A temperature typed in Fahrenheit, a clinic code that doesn't exist, a nightly load that quietly stopped: each one passes through a pipeline and ends up in a dashboard, a report or a model that someone trusts. This course teaches the practices that catch these problems early and handle them well when they slip through. You'll measure quality along the six common dimensions, write data tests as code with pandera and unit-test the rules themselves, and agree data contracts that say what producers promise and what consumers can rely on, with a checker that spots breaking changes before they ship.
The second half moves to production: freshness and volume monitors, anomaly checks that don't cry wolf every Sunday, lineage that tells you who an incident affects, and the write-audit-publish pattern and circuit breakers that keep bad batches away from the people who rely on them. Every lab works on the same clinic network's data, with problems planted at known places, so you can see exactly what each check catches and what it misses. The capstone has you build the whole layer for a pipeline and run a game day against it.
What you'll learn
- Measure data against the six common dimensions of quality and report the results as a scorecard
- Write pandera schemas and unit tests that validate data in a pipeline and quarantine the rows that fail
- Write a data contract and classify schema changes as safe or breaking for the consumers who rely on it
- Monitor freshness, volume and distributions with checks that respect seasonality, and use lineage to find impact and causes
- Stop bad data with write-audit-publish, and run a data incident from detection to a blameless review
Skills you'll gain
Data quality dimensions, Data profiling, Data validation with pandera, Data contracts, Schema evolution, Data observability, Anomaly detection, Data lineage, Data incident response
Tools
Python, pandas, pandera, pytest, PyYAML, DuckDB
Before you start
Python with pandas, as in Data Analysis with Python and pandas, and basic SQL from SQL for Data Analysis with DataGrip. Data Engineering Foundations: Batch and Streaming Pipelines helps. Install Python 3.10 or later and pip install "pandera[pandas]" pytest pyyaml duckdb. Monitoring Models in Production: Drift, Data Quality and Alerts and Site Reliability Engineering: SLOs, Error Budgets and Incidents make good next steps.
Modules
What data quality means, and how to measure it
The dimensions of data quality, how to turn them into rules you can measure, and where in a pipeline quality breaks, with a scorecard built for a clinic network's visit extract.
- The dimensions of data quality (12 min)
- Turning dimensions into measurable rules (12 min)
- Where quality breaks in a pipeline (11 min)
- Hands-on lab: Score a clinic extract on six dimensions
Testing data with pandera
Data validation as code: pandera schemas and checks, lazy validation and failure reports, quarantine, unit tests for the rules themselves, and how Great Expectations, Soda and dbt tests compare.
- Schemas and checks in pandera (13 min)
- Validating in a pipeline: failures, quarantine and tests (12 min)
- Other tools: Great Expectations, Soda and dbt tests (10 min)
- Hands-on lab: Validate and quarantine visits with pandera
Schema evolution and data contracts
How schemas change without breaking consumers, what a data contract between producers and consumers contains, and how to enforce contracts in CI and at run time, with a checker that classifies changes as safe or breaking.
- Schema evolution: safe and breaking changes (12 min)
- What a data contract contains (12 min)
- Enforcing contracts in CI and at run time (12 min)
- Hands-on lab: Write a data contract and catch a breaking change
Monitoring: freshness, volume, anomalies and lineage
Watching data in production: freshness and volume checks, anomaly detection that respects seasonality and resists outliers, and lineage that shows what a problem affects and where it came from.
- Freshness and volume checks (11 min)
- Anomaly detection without the false alarms (13 min)
- Lineage: impact and root cause (11 min)
- Hands-on lab: Monitor loads and trace the impact
Handling data incidents
When bad data gets through: severity, containment, communication, backfills and blameless reviews, and the patterns, write-audit-publish and circuit breakers, that keep bad data away from the people who rely on it.
- The data incident lifecycle (12 min)
- Prevention patterns and blameless reviews (12 min)
- Hands-on lab: Stop a bad batch with write-audit-publish
Project: Build a quality and contract layer for a pipeline
Take a pipeline, either the clinic visits from this course extended with a second table (such as laboratory results) or a dataset of your own with at least two tables, and give it the protection a team would rely on. Write a data contract for each published dataset, enforce it with pandera in CI and at run time, monitor freshness, volume and at least one distribution metric with checks that respect seasonality, record its lineage, and publish every batch through write-audit-publish with a circuit breaker. Then run a game day: inject at least three different problems, such as a breaking schema change, a partial load and a silent unit change, show which checks catch each one (and admit which don't), and write a blameless review of one of them.
Part of
The products this course names are trademarks of their owners, who haven't made or endorsed it.