Topics, Partitions, and Offsets

Everything you configure and reason about in Kafka comes back to three things: the topic you produce to, the partitions it is split into, and the offsets that mark position within a partition. Understanding how these behave prevents most of the surprises people hit later — lost ordering, uneven load, a consumer that cannot keep up, a topic that fills the disk.

What a topic is#

A topic is a named, durable stream of records. It is the level at which you grant access, set retention, and choose a partition count. Producers write to a topic by name; consumers subscribe to it by name.

A record in a topic has:

Kafka does not care what the value bytes mean. Structure and schema are a convention between the producer and the consumers — see the message-contract discussion in Kafka in a Data Pipeline.

Partitions: the unit of parallelism and ordering#

A topic is physically one or more partitions, each an independent append-only log stored on a broker.

Topic "order-events", 3 partitions
  p0:  [0][1][2][3][4][5]  p1:  [0][1][2]  p2:  [0][1][2][3]

Two guarantees follow directly from this layout:

• Order is per partition. Kafka guarantees that records in one partition are read in the order they were written. It guarantees nothing about the order of records in different partitions. There is no global order across a topic.
• Parallelism is capped by partition count. Within a consumer group, each partition is read by exactly one consumer. A topic with 3 partitions can be processed by at most 3 consumers in a group working in parallel; a fourth sits idle.

Choosing a partition count#

The partition count is the main scaling knob. Guidelines:

• Start from target throughput. If one consumer instance can process ~2,000 records/second and you expect peaks of ~10,000/second, you need at least 5 partitions so 5 consumers can share the load.
• You can increase partitions later, but not decrease them. Adding partitions also changes how keys map to partitions (below), so do it deliberately.
• Do not over-partition. Each partition has overhead on the broker (open files, memory, replication traffic). Thousands of partitions on a small cluster is a common mistake. Tens per topic is normal; low hundreds is a lot.
• More partitions = more open ordering scopes. If you need strict ordering for an entity, that entity's records must all be in one partition, so ordering requirements do not benefit from more partitions.

Keys and the partitioner#

When a producer sends a record, Kafka decides its partition:

• With a key: partition = hash(key) % partition_count. Every record with the same key goes to the same partition, so records for one customer_id (or order_id, or device id) are ordered relative to each other.
• Without a key: the producer spreads records across partitions (a sticky round-robin in recent clients), maximising even load. Only per-partition order holds, which for keyless records means effectively no useful order.

key = "customer-42"  ->  hash  ->  always partition 1key = "customer-99"  ->  hash  ->  always partition 0no key               ->  spread across p0, p1, p2

Choosing a key is a design decision:

• Use a key when consumers must see an entity's events in order (state changes, a running total, anything where "before" and "after" matter).
• Pick a key with enough distinct values that load stays even. Keying every record by a constant, or by a low-cardinality field like country, funnels most traffic into one partition — a hot partition.
• Changing the partition count later re-maps keys, so a key that was in partition 1 may move to partition 3. Existing data does not move; only new records use the new mapping. Plan the partition count up front when ordering matters.

Offsets#

An offset is a monotonically increasing integer identifying a record's position within a partition. Partition 0 offset 5 and partition 2 offset 5 are unrelated records.

Kafka tracks three kinds of position:

• Log-start offset — the earliest offset still retained (records before it have expired).
• Log-end offset — the offset the next produced record will get; the current head of the partition.
• Committed offset — per consumer group, per partition: the position a group has acknowledged reading up to.

Consumer lag is log-end offset − committed offset: how many records a group is behind. Lag that keeps growing means consumers cannot keep up with producers, and is the primary health metric for a streaming consumer.

p0:  [0][1][2][3][4][5][6][7][8]      ^log-start        ^committed        ^log-end                        |<--- lag = 3 --->|

Committed offsets are stored by Kafka in an internal compacted topic (__consumer_offsets), so a restarted consumer resumes from the right place without any external state.

Retention: how long records live#

Records are not deleted when read. A topic keeps them according to its retention policy:

Retention is enforced per partition, and Kafka deletes whole segments (the files a partition's log is chunked into), not individual records, so the log-start offset jumps forward in steps.

Consequences for a data pipeline:

• A consumer that is down longer than retention loses data. If retention is 7 days and a consumer is offline for 8, the first day's records are gone before it comes back. Monitor lag against retention.
• Retention sets the replay window. You can reprocess only as far back as retention keeps. Longer retention costs disk.

Log compaction#

With cleanup.policy=compact, Kafka keeps at least the most recent value for each key and removes older values for the same key over time. The topic becomes a changelog / snapshot rather than a full history.

Use compaction for topics that represent current state — "the latest profile for each user", "the current price for each product" — where a new consumer should be able to rebuild the full current state by reading the topic from the start without replaying every historical change. Use plain delete retention for event streams where every event matters.

Replication (brief)#

On a multi-broker cluster, each partition has a replication factor: the number of brokers that keep a copy. One replica is the leader (handles reads and writes); the others are followers that copy from it. The set of replicas that are fully caught up is the in-sync replica set (ISR).

• replication.factor = 3 and min.insync.replicas = 2 is a common production setting: survive one broker failure, and refuse writes if fewer than two replicas are in sync (with producer acks=all).
• On a single-broker local setup, replication factor is 1 — there is nowhere to put a second copy, and a broker failure means data loss. That is fine for development and is why the Docker stack sets every replication factor to 1.

Common mistakes#

Keying by a low-cardinality field#

key = country sends most traffic to one partition. Choose a key with many distinct values, or use no key if order does not matter.

Setting the partition count too low and needing order later#

You cannot reduce partitions, and increasing them re-maps keys. Size the partition count for peak throughput at creation time.

Ignoring lag until it is a crisis#

Growing lag means consumers are falling behind. Alert on it, and compare it to retention so you know how much runway you have before data is lost.

Expecting a global order across partitions#

There is none. If total ordering matters, you need a single partition (and give up parallelism) or you order downstream by an event timestamp.

Using delete retention for a state topic#

If a topic represents "current value per key" and consumers must rebuild state from it, use cleanup.policy=compact, not age-based deletion.

Quick reference#

See also#