Building the Data Stack

This article builds a complete local data platform: a metadata database, an Airflow orchestrator, ClickHouse for analytics, Kafka and Kafka UI for event streaming, MinIO for object storage, and Metabase for dashboards. Every service is added one at a time, with the reasoning for each setting.

If the Compose keys below are unfamiliar, read Docker Compose Basics first.

What we are building#

airflow-webserver / scheduler  -->  postgres   (Airflow metadata)        |        |  runs DAGs that move data        vproducers  -->  kafka  -->  consumer  -->  clickhouse  -->  metabase                  |                            ^                kafka-ui                       |                                             minio   (files / artifacts)

Each service has one job:

We start from an empty file:

services:

1. Postgres — Airflow's metadata store#

Airflow keeps its own state — DAG runs, task history, connections — in a relational database. Postgres is the standard choice.

  postgres:    image: postgres:13    environment:      - POSTGRES_USER=airflow      - POSTGRES_PASSWORD=airflow      - POSTGRES_DB=airflow    volumes:      - postgres_data:/var/lib/postgresql/data    healthcheck:      test: ["CMD-SHELL", "pg_isready -U airflow"]      interval: 5s      timeout: 5s      retries: 5

• image: postgres:13 — pinned version, so every environment runs the same Postgres.
• environment — the official Postgres image reads these on first start to create the user, password, and database. In local development plain-text credentials are acceptable; in production they come from a secret store.
• volumes: postgres_data:/var/lib/postgresql/data — Postgres writes its data here. Mounting a named volume means the Airflow history survives docker compose down.
• healthcheck — pg_isready returns success only once Postgres accepts connections. The Airflow services depend on this being healthy, not just on the container existing.

No ports entry: nothing outside the Compose network connects to Postgres directly. Airflow reaches it as postgres:5432.

2. A custom Airflow image#

The Airflow services need Python packages the project imports and a system library for LightGBM. That is a job for a Dockerfile in the project directory (covered in Writing a Dockerfile):

FROM apache/airflow:2.8.1-python3.10USER rootRUN apt-get update && \    apt-get install -y --no-install-recommends libgomp1 && \    apt-get clean && rm -rf /var/lib/apt/lists/*USER airflowRUN pip install --no-cache-dir \    "numpy>=2.0.0,<2.1.0" "pandas>=2.2.3" "pyarrow>=17.0.0" \    "scikit-learn>=1.5.0,<1.6.0" "lightgbm>=4.0.0" \    "clickhouse-connect" "joblib"

Every Airflow service below uses build: . so Compose builds this image once and shares it.

3. Airflow init — a one-shot setup container#

Before Airflow can run, its metadata schema has to be created and an admin user added. This is a container that runs once and exits.

  airflow-init:    build: .    user: "50000:0"  # run as the airflow user    depends_on:      postgres:        condition: service_healthy    environment:      - AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres/airflow    command: >      bash -c "mkdir -p /opt/airflow/dags /opt/airflow/logs &&      airflow db init &&      airflow users create --username admin --password admin --firstname Airflow --lastname Admin --role Admin --email admin@example.com"

• depends_on: postgres: condition: service_healthy — the init only runs once Postgres is actually accepting connections.
• AIRFLOW__CORE__SQL_ALCHEMY_CONN — Airflow's setting for its metadata database, in SECTION__KEY environment-variable form. The host is postgres, the Compose service name.
• user: "50000:0" — the numeric uid/gid of the airflow user, so files it creates in mounted directories have the right owner.
• command — creates the log/DAG directories, initialises the database schema, and creates the admin / admin login. It has no restart, so once it finishes it stays exited.

4. Airflow webserver and scheduler#

Airflow is split into two long-running processes. The webserver serves the UI and API; the scheduler reads the DAGs, decides when each task should run, and launches it. They run as two separate services from the same custom image, and share almost all of their configuration. What Airflow is and how DAGs work is covered in the Airflow section; this is just how to run it.

The shared base#

Both services start with the same four keys:

  airflow-webserver:    build: .    restart: always    user: "50000:0"    depends_on:      postgres:        condition: service_healthy

• build: . — use the custom image from the Dockerfile in this directory (section 2), not a stock image. Compose builds it once and both Airflow services share the result.
• restart: always — Airflow is meant to run continuously; if the process dies or the machine reboots, bring it back.
• user: "50000:0" — run as uid 50000, gid 0, which is the airflow user inside the official image. This matters because of the bind mounts below: files the container writes into ./dags and ./logs on your machine get this owner, and running as the wrong user causes permission errors on those directories.
• depends_on: postgres: condition: service_healthy — do not start until Postgres is actually accepting connections. Plain depends_on would only wait for the container to exist; Airflow's first action is a database query, so it needs the real "ready" signal from Postgres's healthcheck.

Mounting DAGs and logs#

    volumes:      - ./dags:/opt/airflow/dags      - ./logs:/opt/airflow/logs

These are bind mounts (a host path on the left), not named volumes. That is deliberate:

• ./dags holds the workflow .py files. Because it is mounted live from your project directory, editing a DAG on your machine is picked up by the scheduler within seconds — no image rebuild, no restart. Named-volume storage would hide the files inside Docker and defeat that.
• ./logs is mounted so task logs land in your project directory where you can open them directly, and so the webserver and scheduler share the same log files.

Environment#

    environment:      - AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres/airflow      - AIRFLOW__CORE__EXECUTOR=LocalExecutor      - AIRFLOW__CORE__LOAD_EXAMPLES=False

Airflow reads any setting from an environment variable named AIRFLOW__<SECTION>__<KEY> (double underscores). These three:

• AIRFLOW__CORE__SQL_ALCHEMY_CONN — where Airflow keeps its metadata. The host in the URL is postgres — the Compose service name from section 1, which resolves on the Compose network — with the airflow / airflow credentials that service was created with.
• AIRFLOW__CORE__EXECUTOR=LocalExecutor — run each task as a local subprocess of the scheduler. That is enough for one machine; a real cluster would use the Celery or Kubernetes executor and more services.
• AIRFLOW__CORE__LOAD_EXAMPLES=False — do not seed the database with Airflow's bundled example DAGs, so the UI shows only your own.

The webserver: a port and a healthcheck#

    ports:      - "8080:8080"    command: webserver    healthcheck:      test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]      interval: 10s      timeout: 10s      retries: 5

• command: webserver — the image can run several Airflow subcommands; this service runs the web server.
• ports: "8080:8080" — publish the UI so you can open http://localhost:8080 from your machine. This is the only Airflow service that needs a published port.
• healthcheck — Airflow exposes a /health endpoint; curl --fail makes a non-2xx response a failed check. Marking the webserver healthy is useful for scripting and for anything that waits on it.

The scheduler: everything above, minus the UI#

  airflow-scheduler:    build: .    restart: always    user: "50000:0"    depends_on:      postgres:        condition: service_healthy    volumes:      - ./dags:/opt/airflow/dags      - ./logs:/opt/airflow/logs    environment:      - AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres/airflow      - AIRFLOW__CORE__EXECUTOR=LocalExecutor      - AIRFLOW__CORE__LOAD_EXAMPLES=False    command: scheduler

Identical base, mounts, and environment — only two differences:

• command: scheduler instead of webserver.
• No ports and no healthcheck. The scheduler has no HTTP interface to publish or probe. It must run for anything to be scheduled, but nothing in this file waits on it, so a healthcheck would add noise without value.

5. ClickHouse — the analytical database#

Where pipelines land data for analysis.

  clickhouse:    image: clickhouse/clickhouse-server:latest    container_name: clickhouse    restart: always    ports:      - "8123:8123"  # HTTP interface      - "9000:9000"  # native TCP client    environment:      - CLICKHOUSE_USER=clickhouse      - CLICKHOUSE_PASSWORD=clickhouse      - CLICKHOUSE_DB=default    volumes:      - clickhouse_data:/var/lib/clickhouse    healthcheck:      test: ["CMD", "wget", "--spider", "-q", "http://localhost:8123/ping"]      interval: 10s      timeout: 5s      retries: 3

• Two ports. 8123 is the HTTP interface that clickhouse-connect uses from Python; 9000 is the native protocol for the clickhouse-client CLI and some tools. Publish both so either works from the host.
• container_name: clickhouse — a fixed name instead of the generated project_clickhouse_1. Convenient for docker exec, and it is the hostname other containers use.
• environment — the image creates this user and database on first start.
• volumes: clickhouse_data — the table data. Losing this volume means losing every table.
• healthcheck — GET /ping returns Ok. once the server is up.

6. Kafka — the event streaming broker#

Kafka has the longest configuration in the file because a broker needs to be told how to run without ZooKeeper (KRaft mode) and exactly which network addresses to bind and to advertise. Taken in pieces it is not complicated. Running Kafka Locally covers the underlying model in more depth; this section explains why each line is here.

Image and basics#

  kafka:    image: confluentinc/cp-kafka:7.6.0    container_name: kafka    restart: always    ports:      - "9092:9092"

• image: confluentinc/cp-kafka:7.6.0 — Confluent's Kafka image, pinned. This version supports KRaft, so no separate ZooKeeper service is needed.
• container_name: kafka — a fixed name. Other containers reach the broker at the hostname kafka, and several settings below refer to it by that name, so pinning it keeps everything consistent.
• ports: "9092:9092" — publish one port to your host. As the listener configuration below shows, 9092 is the external listener; the internal one that other containers use is never published.

KRaft: broker and controller in one process#

    environment:      KAFKA_NODE_ID: 1      CLUSTER_ID: 'MkU3OEVBNTcwNTg0NDM0M2'      KAFKA_PROCESS_ROLES: 'broker,controller'      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka:29093'      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'

Older Kafka stored cluster metadata (which topics and partitions exist, who leads each partition) in a separate ZooKeeper service. KRaft moves that into Kafka itself:

• KAFKA_PROCESS_ROLES: 'broker,controller' — this single process plays both roles: it serves data as a broker and runs the metadata quorum as a controller. In production those roles are usually separate machines; for local use, one process is fine.
• KAFKA_NODE_ID: 1 — the id of this node in the cluster. With one node it is just 1.
• KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka:29093' — the controller quorum is a single voter, node 1, reachable on the Compose network at kafka:29093.
• KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER' — which of the listeners defined below the controller quorum talks on.
• CLUSTER_ID — a fixed identifier stamped into the stored metadata on first boot. Any stable base64 string works. Do not change it after the first start: the broker will refuse to start against a kafka_data volume that was formatted with a different id. To start fresh, wipe the volume with docker compose down -v.

Listeners: bind vs. advertise#

      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT'      KAFKA_INTER_BROKER_LISTENER_NAME: 'INTERNAL'      KAFKA_LISTENERS: 'INTERNAL://0.0.0.0:29092,EXTERNAL://0.0.0.0:9092,CONTROLLER://0.0.0.0:29093'      KAFKA_ADVERTISED_LISTENERS: 'INTERNAL://kafka:29092,EXTERNAL://127.0.0.1:9092'

This block is where most "I can't connect to Kafka" problems come from. There are two different address lists:

• KAFKA_LISTENERS — the addresses the broker binds inside the container. 0.0.0.0 means "all interfaces". Three named listeners: INTERNAL on 29092 for other containers, EXTERNAL on 9092 for your host, CONTROLLER on 29093 for the KRaft quorum.
• KAFKA_ADVERTISED_LISTENERS — the addresses the broker hands back to a client after its first connection, telling it where to send all further traffic. A client inside the Compose network is told kafka:29092; a client on your machine is told 127.0.0.1:9092. If a client bootstraps successfully but is then handed an address it cannot reach, it "connects, then hangs" — the classic failure.
• KAFKA_LISTENER_SECURITY_PROTOCOL_MAP — the security protocol for each listener. Everything here is PLAINTEXT (no TLS, no auth), which is fine for a local development broker and nothing else.
• KAFKA_INTER_BROKER_LISTENER_NAME: 'INTERNAL' — which listener brokers use to talk to each other. Irrelevant with one broker, but required.

The rule that falls out of this: connect from your machine with 127.0.0.1:9092, and from another container with kafka:29092.

Single-broker settings#

      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1      KAFKA_LOG_DIRS: '/var/lib/kafka/data'

• The three replication / in-sync-replica values default to 3, which assumes a multi-broker cluster. With one broker there is nowhere to put extra copies, so they must be set to 1 or the internal topics fail to create and the broker will not start.
• KAFKA_LOG_DIRS — where Kafka writes its log segments inside the container. It points at the mounted volume below.

Storage and healthcheck#

    volumes:      - kafka_data:/var/lib/kafka/data    healthcheck:      # lightweight port check so the broker is not stressed on startup      test: ["CMD-SHELL", "nc -z localhost 29092 || exit 1"]      interval: 5s      timeout: 5s      retries: 3      start_period: 30s  # give Kafka 30 seconds to create its files first

• volumes: kafka_data:/var/lib/kafka/data — a named volume for the log data and the KRaft metadata, so topics and messages survive docker compose down.
• healthcheck — nc -z just checks that something is listening on the internal port. It deliberately does not run a real Kafka API call, which would add load while the broker is still starting.
• start_period: 30s — on first boot Kafka formats its storage directory, which takes a while. Failures during this grace window do not count against retries, so the service is not marked unhealthy just for being slow to start.

7. Kafka UI — a web view of the broker#

  kafka-ui:    image: provectuslabs/kafka-ui:latest    container_name: kafka-ui    restart: always    ports:      - "8085:8080"    depends_on:      kafka:        condition: service_healthy    environment:      - KAFKA_CLUSTERS_0_NAME=local-cluster      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:29092      - DYNAMIC_CONFIG_ENABLED=true

• ports: "8085:8080" — Kafka UI listens on 8080 inside its container; published on 8085 so it does not collide with the Airflow webserver.
• depends_on: kafka: condition: service_healthy — it has nothing to show until the broker is up.
• KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:29092 — it connects over the Compose network using the service name and the INTERNAL listener port, not localhost:9092.
• No volume: Kafka UI is stateless, it just reads the broker.

8. MinIO — S3-compatible object storage#

MinIO is a storage server that speaks the Amazon S3 API. It is where a pipeline keeps files that do not belong in a table — raw exports, image frames, model artifacts — and because it talks S3, the same boto3 or s3fs code that works against AWS works against it, just pointed at a different endpoint.

Image and the explicit command#

  minio:    image: minio/minio:latest    container_name: minio    restart: always    command: server /data --address ":9001" --console-address ":9002"

• command: server /data --address ":9001" --console-address ":9002" — unlike the other images, MinIO's entrypoint does not start a useful server on its own; you have to tell it what to do. server /data means "run the object server and store objects under /data". --address ":9001" puts the S3 API on port 9001, and --console-address ":9002" puts the web console (the browser UI) on 9002. Splitting them onto two ports lets you publish or firewall them independently.
• container_name: minio — a fixed name, so code in other containers can use the hostname minio.

Ports: API and console#

    ports:      - "9001:9001"  # S3 API      - "9002:9002"  # web console

Two separate things are published:

• 9001 is the S3 endpoint. Client code targets http://minio:9001 from another container, or http://localhost:9001 from your host.
• 9002 is the browser console for creating buckets and inspecting objects by hand. You open http://localhost:9002.

Application code never touches 9002, and a human never needs 9001 directly — keeping them apart makes that separation explicit.

Credentials#

    environment:      - MINIO_ROOT_USER=minioadmin      - MINIO_ROOT_PASSWORD=minioadmin

These create the root account on first start. In S3 client code they are the access key and secret key:

import boto3
s3 = boto3.client(    "s3",    endpoint_url="http://minio:9001",    aws_access_key_id="minioadmin",    aws_secret_access_key="minioadmin",)

minioadmin / minioadmin is MinIO's well-known default and is fine for a local box. A real deployment sets a strong secret and pulls it from a secret store, exactly like the Postgres and ClickHouse passwords.

Storage and healthcheck#

    volumes:      - minio_data:/data    healthcheck:      test: ["CMD", "curl", "-f", "http://localhost:9001/minio/health/live"]      interval: 10s      timeout: 5s      retries: 3

• volumes: minio_data:/data — the /data directory from the command is backed by a named volume, so stored objects survive docker compose down.
• healthcheck — MinIO exposes /minio/health/live on the API port; curl -f turns a non-2xx response into a failed check.

9. Metabase — dashboards#

A BI tool that connects to ClickHouse and builds charts and dashboards over the marts.

  metabase:    image: metabase/metabase:latest    container_name: metabase    ports:      - "3000:3000"    volumes:      - ./metabase-data:/metabase.db    restart: unless-stopped

• ports: "3000:3000" — the Metabase UI.
• volumes: ./metabase-data:/metabase.db — a bind mount holding Metabase's own small embedded database (its saved questions and settings). Keeping it in the project directory means the dashboards survive and can be inspected.
• restart: unless-stopped — comes back after a reboot, but stays down if you stop it deliberately.

10. The top-level volumes block#

Every named volume a service mounts has to be declared once at the top level:

volumes:  postgres_data:  clickhouse_data:  kafka_data:  minio_data:

Bind mounts (./dags, ./logs, ./metabase-data) are not listed here; only Docker-managed named volumes are.

Bringing it up#

docker compose up -d --build     # build the Airflow image, then start all servicesdocker compose ps                # watch services become healthydocker compose logs -f airflow-scheduler

Order is handled by depends_on + healthchecks: Postgres first, then airflow-init runs and exits, then the Airflow services; Kafka comes up on its own and Kafka UI waits for it.

Then open:

• Airflow — http://localhost:8080 (admin / admin)
• ClickHouse HTTP — http://localhost:8123
• Kafka UI — http://localhost:8085
• MinIO console — http://localhost:9002
• Metabase — http://localhost:3000

To reset everything, including all data:

docker compose down -v

Common mistakes#

Forgetting to declare a named volume#

A service mounts kafka_data:/var/lib/kafka/data but kafka_data is not under the top-level volumes:. Compose errors out. Declare every named volume once.

One depends_on chain, no healthchecks#

airflow-webserver starts before Postgres accepts connections and crashes on its first query. Use condition: service_healthy on the dependency.

Reusing a host port#

Airflow's webserver and Kafka UI both listen on 8080 inside their containers. They must be published on different host ports (8080 and 8085).

Changing Kafka's CLUSTER_ID after first run#

The stored metadata is keyed to it. Either keep it fixed or wipe the kafka_data volume with docker compose down -v.

Editing DAGs and expecting a rebuild#

DAG files are bind-mounted, so edits apply live. Only a change to the Dockerfile needs docker compose up -d --build.

Quick reference#

See also#