Docker Compose Basics

Running one container is a docker run command. Running eight, wired together, is unmanageable that way. Docker Compose describes a set of containers in a single docker-compose.yaml file and manages them as one unit: one command brings them all up, one command tears them all down.

This article covers the parts of the file format. The next article uses them to build a real stack.

The shape of a Compose file#

services:  clickhouse:    image: clickhouse/clickhouse-server:latest    ports:      - "8123:8123"    environment:      - CLICKHOUSE_USER=clickhouse    volumes:      - clickhouse_data:/var/lib/clickhouse
volumes:  clickhouse_data:

Two top-level keys matter most:

• services — one entry per container. The key (clickhouse) is the service name.
• volumes — named storage that services can mount and that survives docker compose down.

Compose manages everything under these as a project, named after the directory by default.

image vs. build#

A service gets its image one of two ways:

services:  postgres:    image: postgres:13        # pull a published image
  airflow-scheduler:    build: .                  # build from the Dockerfile in this directory

Use image for stock services. Use build when you need a custom image (see Writing a Dockerfile). A service can have both: build to build it and image to name the result.

ports — reaching a container from your machine#

ports:  - "8123:8123"       # host 8123  ->  container 8123  - "8085:8080"       # host 8085  ->  container 8080

The format is "HOST:CONTAINER". The left number is the port on your machine; the right is the port the process listens on inside the container. They do not have to match — the Kafka UI listens on 8080 inside its container but is published on 8085 so it does not clash with Airflow.

A service only needs published ports if something outside the Compose network talks to it. Services talk to each other without any ports entry at all (next section).

The Compose network#

Compose puts every service on one private network and registers each service name as a hostname on it. Inside that network, clickhouse resolves to the ClickHouse container, kafka to the Kafka container, and so on.

your machine   |  reaches published ports at localhost:<host-port>   vcompose network  "project_default"   postgres     other services reach it as   postgres:5432   clickhouse   other services reach it as   clickhouse:8123   kafka        other services reach it as   kafka:29092

This is the single most important idea for wiring a stack:

• From your machine: use localhost:<published-port>.
• From another container: use <service-name>:<container-port>.

Airflow connects to Postgres with the host postgres, not localhost. Kafka UI connects to Kafka with kafka:29092. Using localhost from inside a container points at that container itself and fails.

environment — configuration per environment#

environment:  - AIRFLOW__CORE__EXECUTOR=LocalExecutor  - AIRFLOW__CORE__LOAD_EXAMPLES=False

or the map form:

environment:  KAFKA_NODE_ID: 1  KAFKA_PROCESS_ROLES: 'broker,controller'

Environment variables are how a stock image is configured without rebuilding it. Both forms are equivalent; pick one per file for consistency. Values that are secret in a real deployment (POSTGRES_PASSWORD) are fine as plain text in a local development file, but should come from a secret store in production.

volumes — storage that survives#

A container's own file system is deleted when the container is removed. Anything that must persist goes in a volume.

Named volume — Docker manages the storage; good for a database's data directory:

services:  postgres:    volumes:      - postgres_data:/var/lib/postgresql/data
volumes:  postgres_data:      # declared here, at the top level

Bind mount — a path on your machine is mounted into the container; good for source code and config you edit:

services:  airflow-scheduler:    volumes:      - ./dags:/opt/airflow/dags      - ./logs:/opt/airflow/logs

The format is SOURCE:TARGET. A named volume's source is a name declared under the top-level volumes:; a bind mount's source is a path starting with ./ or /.

docker compose down keeps named volumes. docker compose down -v deletes them — that is how you reset a database to empty.

depends_on — start order#

depends_on:  postgres:    condition: service_healthy

Plain depends_on: [postgres] only waits for the Postgres container to start, not for Postgres itself to be ready to accept connections. The condition: service_healthy form waits for the dependency's healthcheck to pass first, which is what you usually want.

healthcheck — "is it actually ready?"#

healthcheck:  test: ["CMD-SHELL", "pg_isready -U airflow"]  interval: 5s  timeout: 5s  retries: 5  start_period: 30s

Compose runs test on a schedule and marks the service healthy or unhealthy. interval is how often, retries how many failures before unhealthy, timeout how long one check may take, and start_period a grace window at startup during which failures do not count (useful for a service like Kafka that takes time to initialise its storage).

Healthchecks pair with depends_on: condition: service_healthy to sequence a stack correctly.

restart — what happens when a container exits#

restart: always            # restart whenever it stops, including on rebootrestart: unless-stopped    # same, but not if you stopped it manuallyrestart: "no"              # default: do not restart

Long-running services use always or unless-stopped. A one-shot init container that is meant to run once and exit uses the default.

Running a Compose project#

docker compose up -d           # start everything in the backgrounddocker compose ps              # status of each servicedocker compose logs -f kafka   # follow one service's logsdocker compose exec kafka bash # a shell inside a running servicedocker compose down            # stop and remove containers (keep volumes)docker compose down -v         # also delete named volumesdocker compose up -d --build   # rebuild images first, then start

up creates the network, pulls or builds images, and starts services in dependency order. -d (detached) runs them in the background.

Common mistakes#

Using localhost between containers#

Inside a container, localhost is that container. Use the other service's name: postgres, clickhouse, kafka.

Publishing a port just to let services talk#

Services communicate over the Compose network without any ports entry. Publish only what you open in a browser or connect to from your host.

depends_on without a health condition#

Plain depends_on waits for the container, not the service. Add a healthcheck to the dependency and condition: service_healthy.

Expecting data after down -v#

-v deletes named volumes. Run docker compose down without -v to keep a database's data.

Editing a running container instead of the file#

Changes made with exec are lost on recreate. Change the Compose file or the Dockerfile and re-run up.

Quick reference#

See also#