Orchestrating Data Pipelines with Apache Airflow

Write, schedule, test and operate Airflow 2 pipelines in Python, with TaskFlow, data intervals, backfills, sensors and retries, and know what changes in Airflow 3.

Intermediate · Data Engineering · about 15 hours · 5 modules · certificate on completion

Taught by Rodrigoò Silvà, Ngo Thi Huừngé, AI Learning Companions.

About this course

Apache Airflow schedules and watches over the batch pipelines that many data teams depend on, as Python code. This course teaches Airflow 2 (2.11.2, the newest 2.x release) by building the pipelines of an online shop: daily order exports to check, totals to load into a warehouse, a flaky payments API to survive and a week of history to backfill. You'll write DAGs with classic operators and the TaskFlow API, pass small values between tasks with XCom, use templates and the run's context, and learn exactly which data interval each run covers, which is the key to correct backfills.

Then you'll make pipelines dependable: idempotent tasks, retries and timeouts, sensors that wait without holding a worker, fail-fast quality checks, trigger rules and failure alerts, with credentials in connections and settings in variables. The last module covers testing DAGs with pytest, running Airflow in production, scheduling DAGs from data with datasets, and what changes in Airflow 3, so your code is ready to move. Every lab runs on a laptop, with airflow standalone or the official Docker Compose set-up, and was tested with Airflow 2.11.2. Afterwards, Analytics Engineering with dbt and DuckDB and Big Data Processing with Apache Spark cover the tools Airflow most often orchestrates.

What you'll learn

  • Explain Airflow's architecture, and run it locally with airflow standalone or the official Docker Compose quick start
  • Write DAGs with classic operators and the TaskFlow API, passing small values between tasks with XCom
  • Schedule DAGs and reason about logical dates, data intervals, catchup and backfills
  • Make pipelines reliable with idempotent tasks, retries, sensors, fail-fast checks, trigger rules and alerts
  • Keep credentials in connections and settings in variables, and test DAGs with pytest
  • Identify what to change when moving DAGs from Airflow 2 to Airflow 3

Skills you'll gain

Apache Airflow, Workflow orchestration, DAG design, TaskFlow API, Scheduling and backfills, Idempotent tasks, Sensors and retries, Connections and secrets, Testing DAGs with pytest

Tools

Apache Airflow 2, Python 3, Docker Compose, SQLite, pytest

Before you start

Data Engineering Foundations: Batch and Streaming Pipelines, or the same ideas: batch loads, idempotency and backfills. Confident Python (functions, modules, virtual environments) and basic SQL. Linux, macOS or WSL 2 with Python 3.12 (Airflow 2.11.2 supports 3.9 to 3.12), or Docker with Compose. The labs use Airflow 2.11.2 and SQLite, which comes with Python.

Modules

  1. Airflow's building blocks

    What Airflow does and how its parts fit together, how DAGs, tasks and operators describe a workflow, and how to run Airflow on a laptop.

    • What Airflow does, and how it's built (12 min)
    • DAGs, tasks and operators (12 min)
    • Run Airflow on your computer (12 min)
    • Hands-on lab: Run Airflow and your first DAG

  2. Writing DAGs with the TaskFlow API

    Write tasks as Python functions, pass small values between them with XCom, and use Jinja templates and the run's context.

    • TaskFlow: tasks as Python functions (12 min)
    • XCom: passing small values between tasks (12 min)
    • Templates and the run's context (12 min)
    • Hands-on lab: Build an ETL DAG with TaskFlow

  3. Scheduling, data intervals and backfills

    How schedules, logical dates and data intervals work, what catchup and backfills do, and how to write tasks that give the same result however often they run.

    • Schedules, logical dates and data intervals (14 min)
    • Catchup and backfills (12 min)
    • Idempotent, deterministic tasks (12 min)
    • Hands-on lab: Backfill a week of orders safely

  4. Reliable pipelines

    Retries, timeouts, fail-fast checks and alerts; sensors that wait without wasting workers; and connections and variables that keep settings and credentials out of DAG code.

    • Retries, timeouts and failure handling (12 min)
    • Sensors and deferrable operators (12 min)
    • Connections, variables and secrets (12 min)
    • Hands-on lab: Wait, retry and fail fast

  5. Testing, operating and upgrading

    Test DAGs with pytest, run Airflow in production, schedule DAGs from data with datasets, and see what changes in Airflow 3.

    • Testing DAGs (12 min)
    • Running Airflow in production (12 min)
    • Data-aware scheduling, and Airflow 3 (14 min)
    • Hands-on lab: Test your DAGs with pytest

Project: Orchestrate an online shop's daily pipelines

An online shop exports its orders once a day, and the finance and marketing teams need trustworthy daily figures by category, plus a report as soon as each day is loaded. Using the data from make_orders.py, build an Airflow 2 project that waits for each day's export, checks it (fail fast on files that are too small or have missing or negative amounts), loads the day's orders and paid revenue into a warehouse idempotently, keyed by data interval, and marks a dataset as updated; a second DAG, scheduled on that dataset, builds the report. Use retries for transient failures, a failure callback for alerts, a connection for the warehouse and a variable for thresholds, keep business logic in a tested module, backfill the whole week, and write notes on what you would change to run the project on Airflow 3.

Part of

The products this course names are trademarks of their owners, who haven't made or endorsed it.

Start the course free