Real-Time Data Streaming with Apache Kafka
Run Kafka 4 in KRaft mode, write reliable Python producers and consumers, use transactions and schemas, and build windowed stream processing that copes with late data.
Advanced · Data Engineering · about 18 hours · 5 modules · certificate on completion
Taught by Chaiyapòn Srisomboòn, Rodrigoò Silvà, AI Learning Companions.
About this course
Apache Kafka is where many organisations keep their events: orders, payments, sensor readings and clicks, stored as durable logs that many applications read independently. This course teaches how Kafka works and how to build on it correctly. You'll run Kafka 4, which needs no ZooKeeper, in KRaft mode on your laptop, see how keys choose partitions and what that means for ordering, and write Python producers and consumers with confluent-kafka that report every delivery and commit offsets on purpose. Then you'll make pipelines reliable with replication, idempotent producers and transactions, and find out exactly where exactly-once stops.
The second half covers what keeps streaming systems running for years: schemas that evolve without breaking consumers, with the registry wire format and Avro's compatibility rules, and stream processing with tables, event-time windows, grace periods, joins and fault-tolerant state. Every module has a lab against a real broker, with starter code whose output is the same on every run: you'll crash consumers on purpose to see at-most-once and at-least-once, abort a transaction and read around it, and build a windowed processor that drops late events. The capstone is a real-time delivery monitor that survives crashes and schema changes.
What you'll learn
- Run Kafka 4 in KRaft mode and design topics, keys and partitions for the ordering and parallelism you need
- Write Python producers and consumer groups that deliver at least once, with idempotent producers and deliberate offset commits
- Build exactly-once consume-transform-produce pipelines with transactions, and explain what they don't cover
- Evolve Avro schemas safely using backward, forward and full compatibility and the registry wire format
- Process streams with tables, event-time windows, grace periods and joins, and keep their state fault-tolerant
Skills you'll gain
Apache Kafka, Event streaming design, Partitioning and ordering, Consumer groups and offsets, Delivery semantics, Kafka transactions, Schema evolution, Stream processing
Tools
Apache Kafka, Docker, Python, confluent-kafka, fastavro
Before you start
Confident Python and the command line (Linux Command Line Essentials), and the batch and streaming basics of Data Engineering Foundations: Batch and Streaming Pipelines. Docker basics from Containerisation and Orchestration with Docker and Kubernetes help. Install Python 3.10 or later and Docker, or Java 17 or later for the Kafka download. Enterprise Data and ML Platforms covers streaming architecture across a whole organisation.
Modules
Kafka's model: logs, topics, partitions and offsets
What an event log is and how Kafka stores one: brokers, topics, partitions, offsets, retention and compaction, how keys decide partitions and ordering, and how to run Kafka 4 in KRaft mode on a laptop.
- Event streaming and the log (12 min)
- Partitions, keys and ordering (12 min)
- Running Kafka in KRaft mode (12 min)
- Hands-on lab: Run a broker and watch keys find their partitions
Producers and consumers in Python
How a producer batches, acknowledges and reports on what it sends, how consumer groups share partitions and rebalance, and how the moment you commit an offset decides between at-most-once and at-least-once delivery.
- How a producer sends: batching, acks and delivery reports (12 min)
- Consumer groups, assignment and rebalancing (13 min)
- Offsets, commits and delivery semantics (12 min)
- Hands-on lab: Produce and consume delivery events with confluent-kafka
Reliability: replication, idempotence and transactions
How Kafka keeps data when brokers fail, how idempotent producers stop retries from duplicating records, and how transactions give exactly-once consume-transform-produce, along with what they can't promise.
- Replication, in-sync replicas and durability (12 min)
- Idempotent producers (11 min)
- Transactions and exactly-once processing (13 min)
- Hands-on lab: Validate payments exactly once
Schemas and evolution on the wire
Why streams need schemas, how a schema registry and its wire format work, and which schema changes are backward, forward or fully compatible, checked by actually reading old data with new schemas and new data with old ones.
- Serialisation formats and the schema registry (12 min)
- Compatibility rules and safe changes (12 min)
- Hands-on lab: Evolve an Avro schema safely
Stream processing: tables, windows, joins and state
Processing streams on Kafka: the duality of streams and tables, event time, windows and grace periods, stream-table and stream-stream joins with their co-partitioning rule, fault-tolerant state, and the limits of exactly-once.
- Streams and tables (12 min)
- Time and windows in Kafka (12 min)
- Joins, state and the limits of exactly-once (13 min)
- Hands-on lab: Windowed revenue with a stream-table join
Project: Build a real-time delivery monitor
A food-delivery platform wants a live view of its city: orders per zone per minute, late deliveries, and riders who go quiet. Build the streaming pipeline on Kafka 4 in KRaft mode with Python (or Kafka Streams if you know Java). A simulator produces order and rider-location events with event timestamps, some out of order and some duplicated, keyed so each order's events stay in order. A validator checks every event against an Avro schema in the registry wire format and splits valid from rejected events in transactions, exactly once. A windowed processor joins orders with a compacted zone table and emits final per-zone counts for one-minute windows with a grace period, plus an alert when a delivery takes longer than its promised time. Show that the pipeline survives a consumer crash without double counting, that a compatible schema change ships without downtime, and that consumer lag stays under control.
Part of
The products this course names are trademarks of their owners, who haven't made or endorsed it.