Introduction
An overview of Apache Airflow
()
1. Introducing Apache Airflow
Apache Airflow concepts
()
Airflow architecture
()
Windows users: install WSL2
()
Windows and macOS users: installing Airflow
()
Running Airflow standalone
()
2. Running Workflows on Airflow
Navigating and exploring the Airflow UI
()
Running an example DAG
()
Creating your first DAG
()
Running your first DAG
()
Creating a DAG using the "with" keyword
()
3. Defining Tasks and Dependencies
Specifying task dependencies
()
Using the bitshift operator for dependency specification
()
Using bash scripts in the DAG
()
Clearing task state to rerun tasks
()
4. Running Python-Based Tasks
Introducing the Python operator
()
Python operators and dependencies
()
Passing parameters to Python callables
()
Injecting task context
()
Introducing XCom
()
Accessing values in XCom
()
Accessing values in XCom using custom keys
()
Implementing a data transformation DAG pipeline part-I
()
Implementing a data transformation DAG pipeline part-II
()
5. Executing SQL Queries in Tasks
Setting up the SQLite database and connection
()
Introducing the SQLExecuteQueryOperator
()
Implementing a SQL based DAG pipeline part-I
()
Implementing a SQL based DAG pipeline part-II
()
6. Conditional Branching and TaskGroups
Introducing branching
()
Conditional branching with variables
()
Task groups and edge labels
()
7. Catch Up, Backfill, CRON Expressions, and Trigger Configurations
Catch up with all previously scheduled runs
()
Backfill scheduled runs
()
Configuring logical dates
()
Data interval and run configuration
()
Conclusion
Summary and next steps
()
Ex_Files_Learning_Apache_Airflow.zip
(98 KB)