Running and Operating Airflow

Writing a DAG is half the job; the other half is watching it run, finding out why a task failed, and rerunning the right thing. This article covers the web UI, how the scheduler picks up your files, retry and alert behaviour, and the operational problems that come up in practice.

The scheduler, webserver, and metadata database are set up in Building the Data Stack; the UI is at http://localhost:8080.

The web UI#

DAGs list#

The landing page: every DAG the scheduler has parsed. For each one:

• a pause toggle — a paused DAG is not scheduled (existing runs continue)
• the schedule, last run, and next run
• small run-history dots, green for success, red for failed
• quick links into the grid and graph views

If a DAG you just wrote is missing here, it did not parse — see import errors below.

Grid view#

The main operational screen for one DAG. Columns are DAG runs (newest on the right), rows are tasks, and each cell is a task instance coloured by state:

Click a cell to open that task instance: its logs, its duration, its try number, and the actions Clear (reset it and re-run, cascading to downstream) and Mark Success / Failed.

Graph view#

The DAG's dependency structure for a selected run, with each task coloured by state. This is where you see which task in a chain failed and what is blocked behind it.

Code and other tabs#

• Code — the parsed source of the DAG file, so you can confirm what Airflow is actually running.
• Task Duration / Landing Times — trends, for spotting a task that is getting slower.
• Audit Log — who paused, triggered, or cleared what.

Reading task logs#

Every task instance writes a log. Open it from the grid cell, or on disk under the mounted logs/ directory:

logs/dag_id=orders_pipeline/run_id=scheduled__2026-03-01T00:00:00+00:00/task_id=load_staging/attempt=1.log

The log contains Airflow's own lines (rendering the command, setting up the context) and then everything the task printed or logged. print() and the logging module both land here. When a task fails, the traceback is at the bottom.

Each retry is a separate attempt=N.log, so you can compare a failed attempt with a later successful one.

Rerunning things#

Clear is the normal recovery action: it resets the task instances to no_status, and the scheduler re-queues them. Because your tasks are idempotent, re-running is safe.

How the scheduler sees your files#

The scheduler parses every .py in the dags/ folder on a loop (every 30 seconds by default). Each parse:

1. imports the file as a module — so imports and top-level code run
2. collects any DAG objects it creates
3. updates the scheduler's view of DAGs, schedules, and structure

Implications:

• A new or edited DAG appears within a parse cycle, not instantly.
• Top-level code runs on every parse. A slow import or a network call at module level slows every cycle for every DAG. Keep the module level to defining the DAG.
• A syntax error or a failing import means the file does not parse and the DAG does not appear; the error shows on the DAGs page and in the scheduler log.

Retries and alerting#

Retries come from default_args:

default_args = {    "retries": 2,    "retry_delay": timedelta(minutes=5),    "retry_exponential_backoff": True,   # 5m, 10m, 20m instead of 5m, 5m}

A task that raises is marked up_for_retry, waits retry_delay, and runs again. After retries failed attempts it is failed, and downstream tasks (with the default all_success rule) do not run.

Alerting options:

• email_on_failure in default_args — needs SMTP configured.
• on_failure_callback — a function called with the context when a task fails; the usual hook for Slack, PagerDuty, or a custom sink.
• SLA (sla=timedelta(...) on a task) — fires a miss callback if the task is not done within the SLA of the interval start.

For a data pipeline, an on_failure_callback that posts the DAG id, task id, run, and a link to the log is the practical minimum.

Operational failures you will hit#

DAG does not appear#

The file did not parse. Check the DAGs page for a red import-error banner, or:

docker compose exec airflow-scheduler python /opt/airflow/dags/orders_pipeline.pydocker compose logs airflow-scheduler | grep -i error

A bare python file.py that runs without error means the file at least imports.

Task stuck in queued or scheduled#

The scheduler created the task instance but the executor has not run it. Usual causes: the scheduler is down, the executor is at capacity (max_active_tasks, a pool limit), or depends_on_past is waiting on a previous run that never succeeded.

Edited DAG not taking effect#

You are inside a parse interval, or the edit introduced an import error so the old version is still cached. Check the Code tab to see what Airflow parsed.

Runs pile up after un-pausing#

catchup=True with an old start_date. Pause, set catchup=False, and clear the unwanted runs — or backfill only the range you want.

Wrong times / off-by-one intervals#

Airflow works in UTC by default. start_date should be timezone-aware, and a run for [T, T+1) executes after T+1 — the "missing" first run is the interval that has not closed yet.

Permission errors on dags/ or logs/#

The container writes as a specific uid. In the Docker stack the Airflow services run as user: "50000:0" so files in the mounted directories get the right owner; running as the wrong user causes Permission denied on those paths.

Common mistakes#

Using Mark Success to get past a failure#

It tells downstream tasks the data is ready when it is not. Fix the task and Clear it instead.

Ignoring import errors on the DAGs page#

A DAG with an import error is simply not running. The banner is easy to miss; check it after every DAG change.

No failure callback#

A pipeline that fails silently is discovered by a stakeholder. Wire an on_failure_callback or email_on_failure.

Debugging by waiting for the schedule#

Use airflow tasks test <dag> <task> <date> to run a single task immediately, with logs to the console and no metadata writes.

Heavy work at module level slowing the scheduler#

Every parse pays for it. Move it into a task.

Quick reference#

See also#