Writing a Dockerfile

Most services in a stack run a stock image unchanged — postgres:13, clickhouse/clickhouse-server:latest. Sometimes an image is almost right but missing something your code needs. A Dockerfile is a recipe for building a new image on top of an existing one.

This article uses one concrete case: an Airflow image that needs an extra system library and a set of Python packages so the project's DAGs can run.

When you need a custom image#

Reach for a Dockerfile when the stock image lacks something that must be present before the container starts:

• a system package a Python library links against (here, libgomp1, which LightGBM needs)
• Python packages your code imports (pandas, clickhouse-connect, and so on)
• a tool the entrypoint calls
• a baked-in configuration file

Things that change per environment — connection strings, credentials, which command to run — do not belong in the image. They go in environment variables and Compose configuration, so the same image works everywhere.

The anatomy of a Dockerfile#

A Dockerfile is a list of instructions. Each one runs in order and produces a new layer — a diff on top of the previous file system state.

FROM apache/airflow:2.8.1-python3.10
# switch to root to install system librariesUSER root
# libgomp1 is required by LightGBMRUN apt-get update && \    apt-get install -y --no-install-recommends libgomp1 && \    apt-get clean && \    rm -rf /var/lib/apt/lists/*
# back to the airflow user before installing Python packagesUSER airflow
# install compatible package versionsRUN pip install --no-cache-dir \    "numpy>=2.0.0,<2.1.0" \    "pandas>=2.2.3" \    "pyarrow>=17.0.0" \    "scikit-learn>=1.5.0,<1.6.0" \    "lightgbm>=4.0.0" \    "clickhouse-connect" \    "joblib"

Read it top to bottom:

FROM apache/airflow:2.8.1-python3.10#

Every Dockerfile starts from a base image. This one is the official Airflow image, pinned to version 2.8.1 on Python 3.10. Pinning matters: FROM apache/airflow:latest would silently change the Airflow version whenever the image is rebuilt.

USER root / USER airflow#

The Airflow image runs as a non-root user called airflow for safety. Installing system packages with apt-get needs root, so the Dockerfile switches to root, installs, then switches back. Anything after the final USER airflow — including the running container — is that unprivileged user.

Leaving the image as root would be a security smell and can cause file-owner problems with mounted directories.

RUN apt-get ... && apt-get clean && rm -rf /var/lib/apt/lists/*#

RUN executes a shell command while building. This one installs libgomp1 (the OpenMP runtime LightGBM links against), then deletes the apt package lists in the same instruction.

Doing the cleanup in the same RUN matters because of layers: each RUN is a permanent layer. If the cleanup were a separate instruction, the downloaded package lists would still exist in the earlier layer and bloat the image. Chaining with && keeps the whole install-and-clean in one layer.

--no-install-recommends skips optional extra packages; apt-get clean removes the downloaded .deb archives.

RUN pip install --no-cache-dir ...#

Installs the Python packages the DAGs import, with version constraints so the set resolves the same way every build. --no-cache-dir tells pip not to keep its download cache in the image.

The versions are constrained ("numpy>=2.0.0,<2.1.0") because these libraries have compatibility windows with each other and with the base image's own dependencies. Pinning ranges avoids a build that works today and breaks next week when a new release lands.

Layers and build caching#

Docker caches each layer. On a rebuild it reuses a cached layer as long as the instruction and everything before it are unchanged; the first change invalidates that layer and every layer after it.

The practical rule: order instructions from least to most frequently changed. System packages change rarely, so they go near the top. If you later add COPY . /app to bring in source code, put it after the pip install, because source changes on every commit and you do not want to reinstall packages each time.

FROM ...                  rarely changes      cached across most buildsRUN apt-get install ...   rarely changes      cached across most buildsRUN pip install ...       changes sometimes   cached until dependencies changeCOPY . /app               changes often       rebuilt often, but cheap

.dockerignore#

When a build copies files in, Docker first sends the build directory to the daemon as the "build context". A .dockerignore file keeps large or irrelevant paths out of it:

.git__pycache__venvlogs*.pyc

This speeds up builds and avoids accidentally baking secrets or local junk into an image.

Building and using the image#

Build from a directory containing the Dockerfile:

docker build -t my-airflow:local .

-t names and tags the result. The . is the build context.

In Compose you rarely call docker build directly. A service says build: . and Compose builds the image the first time and on docker compose build:

services:  airflow-scheduler:    build: .          # build from the Dockerfile in this directory    # image: my-airflow:local   # or reference a pre-built image

Common mistakes#

Not pinning the base image#

FROM some/image:latest makes builds non-reproducible. Pin a version tag.

Cleaning up in a separate RUN#

RUN apt-get install ... then RUN rm -rf /var/lib/apt/lists/* leaves the removed files in an earlier layer. Chain them with && in one RUN.

Copying source before installing dependencies#

If COPY . /app comes before pip install, every source change busts the dependency layer and reinstalls everything. Install dependencies first.

Putting configuration in the image#

Baking a database URL or credentials into the image ties it to one environment and leaks secrets. Pass those at run time through environment variables.

Running as root#

Unless the base image requires it, drop back to a non-root user before the end of the Dockerfile.

Quick reference#

See also#