A DAG is a graph, not a script. The order in which you write the tasks in the file does not determine the order they run — the dependencies you declare between them do. Airflow runs a task as soon as all of its upstream tasks have succeeded, and runs independent tasks in parallel.
Declaring order#
The >> and << operators set upstream/downstream relationships:
load_staging >> build_metrics # build_metrics runs after load_stagingbuild_metrics << load_staging # the same thing, written the other wayChain several:
extract >> transform >> publishFan out to a list — every task in the list runs after load_staging, in
parallel:
load_staging >> [build_current_load, build_dangerous_load, build_vehicle_structure]Fan in — a task that waits for all of a list:
[build_current_load, build_dangerous_load, build_vehicle_structure] >> notify_doneCombine both in one line:
load_staging >> [build_a, build_b, build_c] >> notify_doneFor longer sequences, chain and cross_downstream from
airflow.models.baseoperator are clearer than a wall of >>:
from airflow.models.baseoperator import chain
chain(extract, transform, [mart_a, mart_b], publish)A bare list does nothing
Writing [task_a, task_b, task_c] on its own line — with no >> — is a common
mistake. It creates a Python list and discards it. The tasks still run (they are
attached to the DAG), but with no ordering between them and everything else.
You need an actual >> to declare a dependency.
A worked structure#
A typical pipeline: load raw into staging, then build several independent marts from staging, then run a rollup once all marts exist.
with DAG(dag_id="orders_pipeline", ...) as dag: load_staging = PythonOperator( task_id="load_raw_orders_to_staging", python_callable=load_raw_orders_to_staging, )
build_customer_metrics = PythonOperator( task_id="build_customer_metrics", python_callable=build_customer_metrics, ) build_daily_revenue = PythonOperator( task_id="build_daily_revenue", python_callable=build_daily_revenue, ) build_segment_revenue = PythonOperator( task_id="build_segment_revenue", python_callable=build_segment_revenue, )
rollup_30d = PythonOperator( task_id="rollup_last_30_days", python_callable=rollup_last_30_days, )
load_staging >> [ build_customer_metrics, build_daily_revenue, build_segment_revenue, ] >> rollup_30dThe graph:
build_customer_metrics / \load_staging -----> build_daily_revenue ----> rollup_last_30_days \ / build_segment_revenueThe three mart tasks have no dependency on each other, so Airflow runs them in
parallel (up to the executor's capacity). rollup_last_30_days waits for all
three.
Trigger rules#
By default a task runs only when all its upstream tasks succeed
(trigger_rule="all_success"). Other rules change that:
| Rule | The task runs when... |
|---|---|
all_success (default) | every upstream succeeded |
all_done | every upstream finished, success or fail |
one_success | at least one upstream succeeded |
one_failed | at least one upstream failed |
none_failed | no upstream failed (success or skipped is fine) |
all_done is useful for a cleanup or notification task that must run whether the
pipeline passed or failed:
[build_a, build_b] >> cleanup # cleanup has trigger_rule="all_done"Branching#
BranchPythonOperator runs a function that returns the task_id (or list of
ids) to follow; the other branches are skipped:
from airflow.operators.python import BranchPythonOperator
def choose_path(**context): if is_month_end(context["data_interval_end"]): return "run_monthly_rollup" return "skip_monthly_rollup"
branch = BranchPythonOperator(task_id="choose_path", python_callable=choose_path)branch >> [run_monthly_rollup, skip_monthly_rollup]A task downstream of a branch needs trigger_rule="none_failed_min_one_success"
so a skipped branch does not skip it too.
Keeping a DAG readable#
- One DAG per pipeline. If two workflows have no data dependency, they are two DAGs, not two halves of one.
- Group related tasks.
TaskGroupcollapses a cluster of tasks into one box in the UI without changing the graph. - Name tasks after what they produce, not
task_1,task_2— the graph is documentation. - Avoid cross-DAG dependencies where you can. If DAG B must wait for DAG A,
an
ExternalTaskSensoror a Dataset does it, but it couples two schedules; often the right fix is to make them one DAG.
Common mistakes#
A bare list with no >>#
[a, b, c] alone declares nothing. The tasks run unordered. Add a real
dependency.
Order implied by file position#
Tasks run based on dependencies, not the order lines appear. If b must follow
a, write a >> b.
A dependency that is not a real data dependency#
Chaining b after a "to be safe" when b does not use a's output just
makes the pipeline slower and hides the true structure.
One enormous DAG#
Fifty tasks spanning three unrelated pipelines is hard to read and to operate. Split by pipeline.
Forgetting trigger rules after a branch or a failure-tolerant step#
A task downstream of a skipped branch, or of an all_done cleanup, needs a
matching trigger_rule or it will be skipped or blocked.
Quick reference#
| Syntax | Meaning |
|---|---|
a >> b | b runs after a |
a >> [b, c] | b and c run after a, in parallel |
[b, c] >> d | d runs after both b and c |
a >> [b, c] >> d | fan out then fan in |
chain(a, b, [c, d], e) | readable long sequences |
trigger_rule="all_done" | run regardless of upstream success |
BranchPythonOperator | pick which downstream tasks to run |
TaskGroup | visually group tasks without changing the graph |