Orchestration and Airflow Fundamentals

A data platform is a set of jobs that have to run in the right order, on a schedule, with retries when something fails and a record of what happened. Orchestration is the layer that manages that, and Apache Airflow is the most common tool for it.

This article covers what orchestration is for and how Airflow is put together. The rest of the section builds and schedules real DAGs.

The problem: cron is not enough#

The simplest scheduler is cron: run this command at this time. It works until the jobs start depending on each other and failing.

• Dependencies. "Load raw, then build staging from it, then build three marts from staging." Cron has no idea that the second job must wait for the first, or that the marts must wait for staging. You end up encoding order in sleep calls and hoping.
• Failure handling. A cron job that fails is just gone. There is no retry, no alert, no record — you find out when a dashboard is empty.
• Backfill. "The pipeline was broken for three days; rerun it for each of those days." Cron cannot express "run this for a range of past dates".
• Visibility. Which runs succeeded? Which task in the pipeline failed, and why? How long does each step take? Cron gives you none of this.

An orchestrator makes the workflow itself a first-class object: a graph of tasks with dependencies, a schedule, a retry policy, and a full history you can inspect.

What Airflow is#

Airflow is four parts working together:

DAG files (Python)          you write these; they define the workflows      |scheduler                   parses the DAGs, decides what should run now,      |                      creates runs and queues task instancesexecutor                    runs the queued task instances      |                      (as local subprocesses, on Celery workers, on k8s)metadata database           the source of truth: every run, task, state, log ref      |web UI                      view runs, inspect task logs, trigger, retry, pause

The Docker stack runs exactly this: Postgres as the metadata database, a scheduler service, and a webserver service, with LocalExecutor so tasks run as subprocesses of the scheduler.

Core objects#

A DAG file describes the shape; a DAG run and its task instances are what actually happen, and they live in the metadata database.

The execution model#

scheduler loop:  1. parse every .py file in the dags/ folder into DAG objects  2. for each DAG, is a new data interval due? -> create a DAG run  3. for each DAG run, which task instances have all upstream deps met?     -> mark them queued  4. executor picks up queued task instances and runs them  5. each task instance writes its state and logs back to the metadata DB  6. downstream tasks become eligible as their upstreams reach success

Two consequences to internalise:

• The scheduler re-parses your DAG files constantly. Anything at the top level of a DAG file — not inside a task function — runs on every parse. Heavy work or a slow import at module level slows the whole scheduler. Keep the top level to defining the DAG.
• A task instance can run more than once. Retries, manual re-runs, and backfills all re-execute a task. Tasks must be idempotent: running one twice must leave the system in the same state as running it once.

The architecture in a bit more detail#

The scheduler decides what should run; the executor decides where it runs. Which executor you configure changes the deployment, not the DAG code.

The metadata database is the coordination point for all of them: the scheduler writes "this task instance should run", the executor (or a worker) picks it up, runs it, and writes the result back. Nothing is held only in memory, which is why Airflow survives a scheduler restart mid-run.

The webserver is stateless — it just renders what is in the metadata database. You can restart it any time without affecting running tasks.

A note on versions#

Airflow has changed names over its major versions, and older code and answers online still use the old ones:

This section uses the current names and notes the old ones where they still turn up in the wild.

Airflow orchestrates; it does not transform#

The single most important design rule: the DAG file wires tasks together; it does not contain the business logic.

dags/orders_pipeline.py       defines the DAG and its tasks (thin)transform/raw_to_staging.py   the actual pandas / SQL work (a plain function)

Each PythonOperator points at a python_callable — an ordinary function that lives in its own module, imports nothing from Airflow, and can be tested and run on its own. The orchestrator decides when and in what order; the function decides what. Mixing them makes both harder to change and nearly impossible to test.

When to use Airflow#

Airflow fits when:

• jobs run on a schedule and depend on each other
• you need retries, alerting, and a history of every run
• you need to rerun the pipeline for past dates (backfill)
• several people need to see what the pipeline is doing

It is more than you need when:

• there is one script, run by hand, occasionally
• the work is a single job with no dependencies and no schedule — a cron entry or a systemd timer is simpler
• you need sub-second latency — Airflow schedules on the order of seconds to minutes; for continuous processing use a stream consumer (see Kafka in a Data Pipeline)

Common mistakes#

Business logic in the DAG file#

Transformation code at the top level of a DAG file runs on every scheduler parse and cannot be tested in isolation. Put it in a function in another module and point the operator at it.

Assuming a task runs once#

Retries and backfills re-run tasks. Design every task to be safe to run twice.

Heavy work at module level#

Imports and computation outside a task function execute on every parse. Keep the module level to DAG definition.

Treating Airflow as a real-time engine#

The scheduler works in seconds-to-minutes cycles. For per-event processing, use a stream consumer, not a DAG on a one-minute schedule.

Quick reference#

See also#