Kafka Fundamentals

Apache Kafka is a distributed, append-only log for event streams. Systems append events to it; other systems read those events at their own pace. The two sides never call each other directly and do not have to be online at the same time.

That one sentence hides a lot. This article unpacks it: the problem Kafka was built for, what "an append-only log" actually means as a data structure, and how the pieces — brokers, topics, partitions, producers, consumers — relate. The rest of the section goes deeper into each piece and connects Kafka to the rest of a data platform.

The problem: point-to-point integration does not scale#

Imagine a system where an order is placed and several things must happen: inventory is updated, a confirmation email is sent, an analytics table is refreshed, a fraud check runs, a warehouse is notified.

The obvious design is direct calls: the order service calls each of the other five. That works for a while, then it does not:

• Coupling. The order service now knows about five other services — their addresses, their APIs, their failure modes. Adding a sixth consumer means changing and redeploying the order service.
• Backpressure. If the fraud check is slow, the order service waits, or has to build its own queue, retry logic, and timeout handling for each callee.
• No history. If the analytics job was down for an hour, those events are gone. There is no way to replay them.
• Fan-out multiplies failure. Five calls, each of which can fail independently, on the hot path of placing an order.

Kafka inverts the relationship. The order service appends one event — "order 101 placed" — to a topic and moves on. Each downstream system reads that topic independently, at its own speed, and keeps its own place. Adding a consumer requires no change to the producer. A consumer that was down catches up from where it left off. The producer is decoupled from how many consumers exist and how fast they are.

Without Kafka                          With Kafka
order --> inventory                    order --> "order-events" topic      --> email                                       |      --> analytics                        inventory / analytics / fraud      --> fraud check                      each read the topic on its own      --> warehouse  producer knows all consumers        producer knows only the topic

The core idea: an append-only log#

Strip Kafka down and what remains is a log: a file you can only append to, where every record gets the next sequential number.

offset:  0     1     2     3     4     5   ...       [ e0 ][ e1 ][ e2 ][ e3 ][ e4 ][ e5 ] <- new records appended here

This structure has properties that make it a good backbone for event data:

• Appends are cheap and ordered. Writing is a sequential disk write, and records are automatically in the order they arrived.
• Reads are independent. A reader holds a position (an offset) and moves forward. Two readers at different positions do not affect each other.
• Records are not consumed on read. Reading record 3 does not remove it. Kafka keeps records for a configured retention period (by time or size), so a new reader can start at offset 0 and replay everything.
• The position is the only state a reader needs. To resume after a restart, a reader just needs to remember its offset.

A queue deletes a message once it is delivered. A Kafka topic keeps the message and lets each reader track its own offset. That difference is why Kafka supports replay, multiple independent consumers, and reprocessing.

Brokers and the cluster#

A broker is one Kafka server. A cluster is one or more brokers working together. In production a cluster has several brokers so that data is replicated and the system survives a machine failure; for local development one broker is enough (that is what the Docker stack runs).

Brokers store the log data on disk, serve reads and writes, and — in modern Kafka — run a built-in metadata quorum (KRaft) that tracks which topics and partitions exist and which broker is responsible for each. Older Kafka needed a separate ZooKeeper service for this; KRaft folds it into the brokers.

Topics, partitions, offsets#

A topic is a named stream, such as order-events. It is the unit you produce to and subscribe to.

A topic is split into partitions. Each partition is a separate append-only log. A topic with three partitions is three independent logs that together hold the topic's data.

Topic "order-events"
  partition 0:  [0][1][2][3][4]  partition 1:  [0][1][2]  partition 2:  [0][1][2][3]

Partitions exist for two reasons:

• Parallelism. Different partitions can be read by different consumers at the same time, so throughput scales with partition count.
• Ordering. Kafka guarantees order within a partition, not across the whole topic. Records that must stay in order relative to each other have to go to the same partition.

Which partition a record lands in is decided by its key: Kafka hashes the key and maps it to a partition, so all records with the same key (say, the same customer_id) go to the same partition and keep their relative order. With no key, records are spread across partitions and only per-partition order holds.

An offset is a record's position within its partition: partition 0 offset 4, partition 2 offset 1. An offset is only meaningful together with its partition.

Topics, Partitions, and Offsets covers partition count, keys, retention, and replication in detail.

Producers#

A producer is a client that appends records to a topic. It:

1. serializes the record's value (and optional key) to bytes
2. picks a partition (by key hash, or round-robin without a key)
3. batches records in memory and sends them to the broker
4. receives an acknowledgement per batch

Producers are asynchronous by design: produce() buffers, and delivery is confirmed later through a callback or an explicit flush. Producing Messages with Python covers the client.

Consumers and consumer groups#

A consumer reads records from a topic. It always reads as part of a consumer group, identified by a group.id.

Kafka divides a topic's partitions among the members of a group, so each partition is read by exactly one consumer in that group at a time:

Topic with 3 partitions, group "analytics" with 2 consumers
  partition 0  -->  consumer A  partition 1  -->  consumer A  partition 2  -->  consumer B

• Adding consumers to a group (up to the partition count) spreads the load.
• If a consumer leaves or joins, Kafka rebalances: it reassigns partitions among the remaining members.
• Different groups are independent. A group analytics and a group fraud both reading order-events each receive every record and each track their own progress.

Progress is stored as a committed offset per partition per group, kept by Kafka itself in an internal topic. On restart, a consumer resumes from its committed offset. If a group has no committed offset yet, auto.offset.reset decides where it starts: earliest (beginning of the partition) or latest (only new records).

Consuming Messages with Python covers the poll loop, offset management, and rebalancing.

The full picture#

producers          -->  append to a topic                          |                        topic = N partitions, each an ordered log                          |                        retained for a configured time / size                          |consumer groups    -->  each group reads every record and tracks its own                        offsets; within a group, partitions split across members

Delivery guarantees#

Kafka's default is at-least-once delivery: a consumer may see a record more than once — for example, if it processes a record, then crashes before committing the offset, and on restart re-reads from the last committed position.

The practical response is to make downstream processing idempotent: writing the same record twice produces the same result as writing it once. Usually that means de-duplicating on a business key (an order_id) or using a storage engine that collapses duplicates.

The other modes:

• At-most-once — commit the offset before processing. A crash then skips records. Rarely what you want for data.
• Exactly-once — Kafka supports it for processing that stays within Kafka (transactions, idempotent producers). Across an external system like a database it is hard; at-least-once plus an idempotent sink is the common, simpler target.

Kafka in a Data Pipeline works through what at-least-once means when the consumer writes to ClickHouse.

When Kafka is the right tool#

Kafka fits when:

• several independent consumers need the same stream of events
• producers and consumers should be decoupled and run at different rates or times
• you need to replay history, or add a consumer that reprocesses old data
• throughput is high and needs to be spread across partitions and machines
• events must be buffered so a burst does not overwhelm a slow downstream system

It is heavier than you need when:

• one producer hands work to one worker — a simple task queue is enough
• the data is a bounded file processed once — a batch job is simpler
• you need request/response — Kafka is one-way, fire-and-forget
• the volume is tiny and there is exactly one consumer that is always available

Common mistakes#

Expecting global ordering across a topic#

Order is guaranteed only within a partition. For related records, use the same key so they land in one partition.

Treating a topic like a queue that drains#

Records stay until retention expires. "Consumed" means "this group's committed offset moved past it", not "removed from the topic".

One partition for a high-throughput topic#

A single partition can be read by only one consumer per group, so it caps parallelism. Choose the partition count for the throughput you need.

Assuming exactly-once by default#

The default is at-least-once. Design for duplicates unless you have specifically built for more.

Putting large payloads in Kafka#

Kafka is tuned for many small records, not multi-megabyte blobs. Store large objects elsewhere (object storage) and put a reference in the event.

Quick reference#

See also#