Running Kafka Locally

To develop against Kafka you need one broker running and a way to look inside it. This article covers the KRaft (no ZooKeeper) and listener configuration that trips people up, and the CLI and UI tools for creating and inspecting topics and consumer groups.

The broker itself runs as the kafka and kafka-ui services in the local stack — see Building the Data Stack, which has the Compose configuration and how to start it (docker compose up -d, wait for kafka to report healthy). The sections below explain the settings that service uses.

KRaft mode, no ZooKeeper#

Modern Kafka runs its own metadata quorum instead of a separate ZooKeeper service. The kafka service settings that enable it:

• KAFKA_PROCESS_ROLES: 'broker,controller' — this one process is both the data broker and the metadata controller.
• CLUSTER_ID — a fixed identifier for the cluster's stored metadata. Any stable base64 string works locally. Do not change it after the first start or the broker refuses the existing kafka_data volume.
• KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka:29093' — the quorum is just node 1, reachable inside the network at kafka:29093.
• KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER' — which listener the controller quorum uses.
• The *_REPLICATION_FACTOR and *_MIN_ISR values are 1 because there is one broker; it cannot hold a second copy of anything.

Listeners: why clients connect (or do not)#

A listener is a name://host:port the broker binds to. The kafka service defines three:

The mechanism that matters: a Kafka client's first connection is to a bootstrap address, and the broker replies with the list of advertised addresses to use for all further traffic. So even if you bootstrap correctly, you fail on the second step if the advertised address is wrong for where you are.

process on your machine   ->  bootstrap 127.0.0.1:9092   broker replies: "use EXTERNAL -> 127.0.0.1:9092"     works
process in another container  ->  bootstrap kafka:29092   broker replies: "use INTERNAL -> kafka:29092"        works
process in a container using 127.0.0.1:9092   connects, then broker says "use 127.0.0.1:9092"   -> that is the container itself, not the broker      hangs / times out

KAFKA_INTER_BROKER_LISTENER_NAME: 'INTERNAL' tells brokers to talk to each other over the internal listener. ports: "9092:9092" publishes only the external listener to your host.

The rule to remember:

• From your machine: 127.0.0.1:9092
• From another container: kafka:29092

Kafka UI#

Open http://localhost:8085. Kafka UI connects to the broker over the Compose network at kafka:29092 (KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS), so it never needs a published Kafka port.

What it is for:

• Topics — list, create (name, partitions, replication factor, retention), view per-partition offsets and size, browse and search messages, and see the message key, value, headers, and timestamp.
• Consumers — every consumer group, which partitions each member owns, the committed offset per partition, and the lag. This is the fastest way to check whether a consumer is keeping up.
• Brokers — the broker list and their configuration.
• With DYNAMIC_CONFIG_ENABLED=true, you can also edit topic configuration from the UI.

The kafka-topics CLI#

The broker image ships the standard CLI tools. Run them inside the container, pointing at the internal listener (localhost:29092 from the broker's own point of view):

# create a topicdocker compose exec kafka kafka-topics --bootstrap-server localhost:29092 \  --create --topic order-events --partitions 3 --replication-factor 1
# list topicsdocker compose exec kafka kafka-topics --bootstrap-server localhost:29092 --list
# describe one: partitions, leader, replicas, ISRdocker compose exec kafka kafka-topics --bootstrap-server localhost:29092 \  --describe --topic order-events
# add partitions (you cannot remove them)docker compose exec kafka kafka-topics --bootstrap-server localhost:29092 \  --alter --topic order-events --partitions 6

Producers can auto-create a topic on first write if the broker allows it, but creating it explicitly lets you set the partition count deliberately.

Changing topic configuration#

Retention and other per-topic settings are changed with kafka-configs:

# set retention to 3 daysdocker compose exec kafka kafka-configs --bootstrap-server localhost:29092 \  --alter --entity-type topics --entity-name order-events \  --add-config retention.ms=259200000
# switch a topic to compactiondocker compose exec kafka kafka-configs --bootstrap-server localhost:29092 \  --alter --entity-type topics --entity-name order-events \  --add-config cleanup.policy=compact
# show a topic's non-default configdocker compose exec kafka kafka-configs --bootstrap-server localhost:29092 \  --describe --entity-type topics --entity-name order-events

Inspecting consumer groups#

# list groupsdocker compose exec kafka kafka-consumer-groups --bootstrap-server localhost:29092 --list
# per-partition committed offset, log-end offset, and LAG for a groupdocker compose exec kafka kafka-consumer-groups --bootstrap-server localhost:29092 \  --describe --group warehouse-sink
# move a group's offsets (only when no members are active)docker compose exec kafka kafka-consumer-groups --bootstrap-server localhost:29092 \  --group warehouse-sink --topic order-events \  --reset-offsets --to-earliest --execute

Resetting offsets to --to-earliest makes a group replay a topic from the beginning — useful for a backfill or after fixing a bug in a consumer.

Console producer and consumer#

Quick manual testing without writing code:

# type lines; each becomes a message valuedocker compose exec -it kafka kafka-console-producer \  --bootstrap-server localhost:29092 --topic order-events
# print messages as they arrive (Ctrl+C to stop)docker compose exec kafka kafka-console-consumer \  --bootstrap-server localhost:29092 --topic order-events --from-beginning

Common mistakes#

Connecting to kafka:29092 from the host#

That hostname only resolves inside the Compose network. From your machine use 127.0.0.1:9092.

Connecting to 127.0.0.1:9092 from another container#

Inside a container, 127.0.0.1 is that container. Use kafka:29092.

Changing CLUSTER_ID after the first start#

The stored metadata is tied to it; the broker will not start against the old volume. Wipe it with docker compose down -v to start fresh.

Running the CLI with the wrong bootstrap#

Inside the container, the CLI reaches the broker at localhost:29092 (the internal listener), not 9092.

Expecting replication or fault tolerance from one broker#

A single broker cannot keep two copies of a partition. Replication factor stays 1 locally, and a lost kafka_data volume is lost data.

Quick reference#

See also#