Scheduling and Backfill

Airflow's scheduling model confuses almost everyone at first because a run does not mean "right now". A run is tied to a data interval — a window of time the run is responsible for — and it executes after that window has closed. Getting this right is the difference between a pipeline that backfills cleanly and one that produces wrong results.

The schedule#

schedule (older DAGs: schedule_interval) controls how often a run is created. Three forms:

schedule="@hourly"                       # presetschedule="0 6 * * *"                     # cron: every day at 06:00schedule=timedelta(hours=6)              # every 6 hours from start_dateschedule=None                            # never; trigger manually only

Presets: @once, @hourly, @daily, @weekly, @monthly, @yearly.

Prefer a preset or a plain cron expression. A one-minute schedule ("* * * * *") is a warning sign — if you need per-event latency, use a stream consumer, not Airflow.

Data intervals and when a run executes#

start_date and schedule together carve time into intervals. A run is created for each interval, and it starts after the interval ends.

schedule = @hourly, start_date = 2026-03-01 00:00
interval  [00:00, 01:00)   run created ~01:00, processes 00:00-01:00 datainterval  [01:00, 02:00)   run created ~02:00, processes 01:00-02:00 datainterval  [02:00, 03:00)   run created ~03:00, ...

The delay is intentional: the hour's data is only complete once the hour is over. Inside a task, the window is available as data_interval_start and data_interval_end (see Operators and Tasks).

logical_date is the run's label — historically called execution_date, which misled people into thinking it was "when the task runs". It is the start of the interval, not the wall-clock run time.

catchup#

When you enable a DAG (or un-pause it after downtime), Airflow looks at every interval between start_date and now that has no run yet.

• catchup=True — create a run for every one of those missed intervals. With start_date set months ago, enabling the DAG triggers a flood of runs.
• catchup=False — create only the most recent missed interval and go forward from there. The historical intervals are simply skipped.

start_date months ago, enabled today
catchup=True    -> hundreds of runs queued at oncecatchup=False   -> one run for the latest interval, then normal schedule

Set catchup=False on almost every DAG. When you genuinely need history processed, backfill it explicitly, with a range you control.

Backfill#

Backfilling means running a DAG for a range of past intervals on purpose:

airflow dags backfill orders_pipeline \  --start-date 2026-02-01 --end-date 2026-02-28

This creates and runs one DAG run per interval in the range. Reasons to backfill:

• a new DAG needs history loaded
• a bug corrupted output for a period, now fixed, and those days must be redone
• retention on a source is about to expire and you want the data captured

Backfill is only safe if every task is idempotent — running an interval again must overwrite, not duplicate. On the SQL side that means delete-partition then insert, or a ReplacingMergeTree keyed on the business key (see Building Staging and Data Marts). A backfill over a non-idempotent DAG multiplies your data.

Interval-aware tasks#

A task must process the interval it was given, not "the latest data". Compare:

# WRONG: depends on wall-clock time; a backfill reprocesses today, not the target daydef load(**context):    end = datetime.now()    start = end - timedelta(hours=1)    load_window(start, end)
# RIGHT: processes exactly this run's interval, on time or during a backfilldef load(**context):    load_window(        start=context["data_interval_start"],        end=context["data_interval_end"],    )

With the right version, airflow dags backfill ... --start-date 2026-02-01 runs the 2026-02-01 00:00 interval task against 2026-02-01 00:00–01:00 data, exactly as the original scheduled run would have.

Controlling concurrency#

For a backfill of a DAG that writes to one table, set max_active_runs=1 so the intervals are processed in order and never collide.

Common mistakes#

Expecting a run at the interval's start#

A run for 00:00–01:00 executes near 01:00, not 00:00. The data has to be complete first.

catchup=True with an old start_date#

Enabling the DAG queues every missed interval. Use catchup=False and backfill deliberately.

datetime.now() in a task#

Breaks on-time reruns and backfills. Use data_interval_start / data_interval_end.

Backfilling a non-idempotent DAG#

Each re-run appends again. Make tasks overwrite their interval before running any backfill.

A one-minute schedule#

If you need that latency, Airflow is the wrong tool. Use a stream consumer.

Quick reference#

See also#