Introduction
Course welcome
()
1. Kafka Background Concepts
Why do I need a message broker?
()
Understanding Kafka components
()
Running Kafka on the LHM virtual machine
()
Understanding basic Kafka usage
()
2. Viewing Kafka Operations
Topics
()
Exploring the KafkaEsque UI features
()
Replaying basic examples with KafkaEsque
()
3. Streaming NOAA Weather Data with Kafka Python
Topics
()
Understanding basic Kafka Python producers and consumers
()
Constructing a Python producer using keys and multiple partitions
()
Constructing a Python consumer using keys, offsets, and time windows
()
4. Moving Kafka Topic Data to External Storage
Using a PySpark consumer to stream data to HDFS
()
Using a Python consumer to write data to MariaDB (MySQL)
()
Using a Python consumer to write data to HBASE
()
5. Edge Image Streaming with Kafka Python
Topics
()
Detecting edge 3D-printing problems
()
Understanding image capture, streaming, and real-time CNN analysis using Kafka producers and consumers
()
Emulating an edge 3D-printing streaming example on the LHM virtual machine
()
6. Data Pipelines and Kafka Connect
Topics
()
Understanding Kafka Connect design and use cases
()
How to manage Kafka connectors
()
Kafka Connect examples using files, HDFS, and MariaDB (MySQL)
()
7. Installation Considerations
Topics
()
Installing Zookeeper and Kafka
()
Understanding hardware choices
()
Creating a Kafka cluster
()
Configuring partitions
()
8. Basic Administration Topics
Topics
()
Understanding partition allocation and management
()
Understanding file management and compaction
()
Learning Kafka topic operations
()
Using Kafka console consumer and producer tools
()
Conclusion
Summary and next steps
()