Big Data Processing with Apache Spark

Process large datasets with PySpark DataFrames and Spark SQL, and make jobs fast: partitions, shuffles, joins, windows, caching, Parquet, the Spark UI and tuning.

Intermediate · Data Engineering · about 14 hours · 5 modules · certificate on completion

Taught by Camilà Torresè, Davaadorjì Gantulàg, AI Learning Companions.

About this course

Apache Spark processes data that is too big or too slow for one machine, by splitting it into partitions and working on them in parallel. This course teaches Spark through PySpark, with a month of synthetic sky-survey observations that Spark generates on your laptop: you'll see how lazy transformations become a query plan, jobs, stages and tasks, read files with explicit schemas and handle rows Spark can't parse, and write the same query with DataFrames and Spark SQL.

Then you'll learn what makes Spark jobs fast or slow: partitions and the shuffles between them, broadcast and sort-merge joins, window functions, caching, and partitioned Parquet that's quick to read and safe to rewrite one partition at a time. The last module shows how to read the Spark UI, find and fix skew, use adaptive query execution and run applications with spark-submit. Every lab prints results you can check, and the capstone turns raw survey files into a clean, partitioned dataset with a tuning report. Afterwards, Orchestrating Data Pipelines with Apache Airflow shows how to schedule such jobs, and Real-Time Data Streaming with Apache Kafka covers streams.

What you'll learn

  • Explain how Spark runs a job: driver, executors, lazy evaluation, the query plan, stages and tasks
  • Read, clean and aggregate data with PySpark DataFrames and Spark SQL, with explicit schemas
  • Control partitions and shuffles, and choose between broadcast and sort-merge joins
  • Use window functions and caching, and write partitioned Parquet that can be rewritten safely
  • Find bottlenecks and skew in the Spark UI, tune jobs, and run them with spark-submit

Skills you'll gain

Apache Spark, PySpark DataFrames, Spark SQL, Partitioning and shuffles, Join strategies, Window functions, Parquet, Performance tuning, Spark UI

Tools

PySpark, Spark SQL, Spark UI, spark-submit, Python 3, Java

Before you start

Data Engineering Foundations: Batch and Streaming Pipelines (Parquet, partitioning, idempotent loads) and SQL as in SQL for Data Analysis with DataGrip: joins, GROUP BY and window functions. Comfortable Python. Install Python 3.10 or later and Java 17 or 21, then pip install pyspark in a virtual environment; the labs were tested with PySpark 4.2.0 and need no cluster or download.

Modules

  1. Spark's model: distributed and lazy

    When a job needs Spark, how a Spark application runs, and how lazy DataFrames become plans, jobs, stages and tasks.

    • When you need Spark, and how it runs (12 min)
    • DataFrames, transformations and actions (12 min)
    • Jobs, stages, tasks and the query plan (12 min)
    • Hands-on lab: Run Spark locally and read its plans

  2. DataFrames and Spark SQL

    Read files with explicit schemas and deal with bad rows, transform and aggregate with built-in functions, and write the same query in Spark SQL.

    • Reading data with schemas (12 min)
    • Transforming columns and aggregating (12 min)
    • Spark SQL, views and ANSI mode (12 min)
    • Hands-on lab: Clean and summarise survey observations

  3. Partitions, shuffles and joins

    How data is split into partitions, what a shuffle costs and how to avoid one, and how Spark chooses between broadcast and sort-merge joins.

    • Partitions and parallelism (12 min)
    • Shuffles and what they cost (12 min)
    • Join strategies and broadcast joins (12 min)
    • Hands-on lab: Watch shuffles and choose a join

  4. Windows, caching and Parquet

    Rank and compare rows with window functions, reuse results with caching, and write partitioned Parquet that's quick to read and safe to rewrite.

    • Window functions (12 min)
    • Caching and persistence (10 min)
    • Writing Parquet: partitions, files and overwrites (12 min)
    • Hands-on lab: Rank, cache and write a partitioned dataset

  5. The Spark UI and tuning

    Read what a running application is doing in the Spark UI, find and fix skew, let adaptive query execution help, and run applications with spark-submit.

    • Reading the Spark UI (12 min)
    • Skew, adaptive query execution and tuning (14 min)
    • Running Spark applications (12 min)
    • Hands-on lab: Find and fix skew

Project: Process a month of sky-survey observations

An observatory's survey sends raw observation files every night, and its scientists need a clean, query-ready dataset and per-object brightness statistics. Generate the raw data with make_survey.py (gzipped CSV, two million rows by default, with unreadable magnitudes, bands in capitals, rows delivered twice and one very common object), then write a PySpark application, run with spark-submit, that reads it with an explicit schema, sets bad rows aside with a reason, removes duplicates, joins the objects with a broadcast join, computes per-object and per-band statistics with aggregations and window functions (number of observations, mean and range of magnitude, and the biggest change between consecutive observations), and writes partitioned Parquet that a rerun replaces cleanly. Finish with a short tuning report based on the Spark UI.

Part of

The products this course names are trademarks of their owners, who haven't made or endorsed it.

Start the course free