Why Containers

A data project rarely runs on one program. Even a small one needs a database, a message broker, an orchestrator, maybe object storage and a dashboard. Getting all of those installed, configured, and talking to each other on one machine is tedious, and it breaks the moment someone else tries to reproduce it.

Containers solve that. This article explains what a container actually is and why it has become the default way to run this kind of stack. The rest of the section builds a real multi-service setup with Docker.

The problem containers solve#

Installing software directly on your machine has three recurring problems.

Version drift. ClickHouse 24.3 on your laptop, 23.8 on a colleague's, something else in production. Each version has slightly different behaviour, and a bug that appears on one machine cannot be reproduced on another.

Conflicting dependencies. Two tools that both need Python, but different versions. A system library that one service upgrades and another depends on the old copy of. On a shared machine these collisions are constant.

Setup that lives in someone's head. "Install these five packages, set these environment variables, create this directory with these permissions, then run this command." Written down it is a page of instructions that goes stale; not written down it is lost when that person leaves.

A container packages a piece of software together with everything it needs to run — its libraries, its configuration, its file system — into one unit that behaves the same on any machine with a container runtime.

Image and container#

Two words that are easy to mix up.

An image is the read-only package: a snapshot of a file system plus metadata that says which command to run. Images are built once and shared (through a registry such as Docker Hub). postgres:13, clickhouse/clickhouse-server:latest, and confluentinc/cp-kafka:7.6.0 are images.

A container is a running instance of an image. Starting a container gives the image's file system a thin writable layer on top, runs its command, and gives it an isolated view of processes and the network. You can start many containers from the same image; each is independent.

image  (built once, immutable)      postgres:13  |  docker runcontainer  (a running process)      one Postgres server, isolated  |  docker run  (again)container  (another one)            a second, independent Postgres server

The mental model: an image is like a class, a container is like an instance.

Container vs. virtual machine#

Both isolate software, but at different levels.

A virtual machine emulates hardware and runs a full guest operating system, including its own kernel. It is heavy: gigabytes of disk, tens of seconds to boot, meaningful memory overhead per VM.

A container shares the host's kernel and isolates only the parts above it — the file system, process list, network, and resource limits. It is light: an image is tens to hundreds of megabytes, a container starts in under a second, and the overhead compared to running the process directly is small.

For running a stack of data services on one machine, containers are the right level: light enough to run eight of them at once, isolated enough that their dependencies never collide.

Why data infrastructure runs in containers#

Four properties matter for the kind of stack this section builds.

Reproducibility. The image pins the exact version of every service. Someone who checks out the project and runs one command gets the same ClickHouse, the same Kafka, the same Airflow as everyone else.

One-command startup. With the services described in a file, bringing the whole environment up is a single command, and tearing it down is another. There is no multi-page setup document to keep current.

Isolation. Each service has its own file system and dependencies. Kafka's Java runtime, ClickHouse's libraries, and Airflow's Python packages never interfere with each other or with anything already on your machine.

Parity with production. The same images that run on your laptop can run on a server. The gap between "works on my machine" and "works in production" shrinks to configuration, not software versions.

The pieces of Docker#

• The Docker Engine (daemon) is a background service that builds images and runs containers.
• The docker CLI is what you type; it sends commands to the daemon.
• A registry (Docker Hub by default) stores images so they can be pulled by name and tag.
• Docker Compose describes a set of containers, their configuration, and how they connect, in one YAML file, and manages them together.

A first container#

Pull and run an image directly:

docker run --rm -p 8123:8123 clickhouse/clickhouse-server:latest

• docker run starts a container from the image, pulling it first if needed.
• --rm deletes the container when it stops, so nothing accumulates.
• -p 8123:8123 publishes the container's port 8123 on the host, so a client on your machine can reach it at localhost:8123.

Useful commands while a container runs:

docker ps                       # running containersdocker logs <name-or-id>        # its outputdocker exec -it <name> bash     # a shell inside itdocker stop <name>              # stop it

Running services one docker run at a time, each with its own flags, does not scale past two or three. That is what Compose is for, and it is where this section is headed.

Common mistakes#

Treating a container like a server you log into and configure#

A container should be fully described by its image and its run configuration. Changes made by hand inside a running container are lost when it is recreated. Put the change in the image or the Compose file instead.

Expecting data to persist by default#

A container's writable layer is deleted with the container. Anything that must survive — a database's files — has to be stored in a volume, covered in the Compose articles.

Confusing the image tag latest with "a fixed version"#

latest is just the default tag; it moves as the image is republished. Pin a real version (postgres:13) when reproducibility matters.

Publishing every port#

-p exposes a container to your host network. Only publish the ports you actually need to reach from outside the container's own network.

Quick reference#

See also#