Kafka in a Data Pipeline

The previous articles covered Kafka in isolation: the log, topics and partitions, producing and consuming. This one places Kafka inside a full data pipeline and works through what "reliable" means across the whole path, from the system that emits an event to the table an analyst queries.

Where Kafka fits#

In the data path, Kafka is an ingestion buffer: it sits between the systems that produce events and the storage that consumes them.

source systems              Kafka                  warehouse                       "order-events" topic
app / service   -->    [ append events ]    -->   consumer  -->  raw.ordersdevice / sensor  -->                          (batch + insert)      |                                                                    v                                                            staging / marts (SQL)

Putting a durable log in the middle buys four things:

• Decoupling of rates. Producers emit events whenever they happen; the consumer reads and loads at whatever rate the warehouse accepts. A burst of events is absorbed by the log instead of overwhelming ClickHouse.
• Decoupling of availability. The consumer (or the warehouse) can be down for maintenance. Events accumulate in the topic and are processed on catch-up, as long as the outage is shorter than retention.
• Multiple consumers. The same order-events topic can feed the warehouse loader, a real-time alerting job, and a metrics exporter — three independent consumer groups, each at its own position.
• Replay. A bug in the loader that corrupted a day of data can be fixed and the day reprocessed by resetting the consumer group's offsets, as long as the records are still within retention.

Streaming vs. micro-batch consumption#

A consumer can process records one at a time or in groups. For a warehouse sink, micro-batch is almost always right:

The consumer article builds the micro-batch loop: accumulate N records (or wait T seconds), write them in one client.insert(), then commit the offset. Choose the batch size from the warehouse's preferences (thousands of rows per insert for ClickHouse) and the latency you can tolerate.

End-to-end reliability#

Reliability is a property of the whole chain, not any one link. Walk it from left to right.

At the producer#

• enable.idempotence=True so a retried send does not append a duplicate to a partition.
• Include a business key in every event — order_id — that uniquely identifies it. Everything downstream depends on this for de-duplication.
• Key the record by the entity whose events must stay ordered (key = customer_id), so state changes arrive in sequence.

In the topic#

• Retention long enough to cover the worst realistic consumer outage plus a margin — often 7 days. This is also the maximum replay window.
• Partition count sized for peak throughput, because you cannot reduce it and increasing it re-maps keys.

At the consumer#

• enable.auto.commit=False; commit after the batch is durably written. This gives at-least-once: on a crash between write and commit, the batch is re-read and re-written.
• Route unparseable records to a dead-letter destination so one bad message cannot block a partition.

At the sink#

Because the consumer is at-least-once, the sink must be idempotent — writing the same batch twice must equal writing it once. Two ways in ClickHouse:

• ReplacingMergeTree keyed on the business key. Duplicate order_ids collapse to the latest version during merges; queries use FINAL or re-aggregate to see the collapsed result.
• Replace-partition-then-insert for time-partitioned marts: drop the day's partition and rebuild it, so re-running the load rewrites the same partition.

Both are covered in Building Staging and Data Marts.

producer (idempotent)  ->  topic (retained)  ->  consumer (commit after write)                                                        |                                                   at-least-once                                                        v                                              sink de-dups on order_id                                                        |                                                exactly-once *effect*

You do not get exactly-once delivery, but you get an exactly-once result, which is what matters for a warehouse.

The message contract#

Kafka stores bytes; the meaning is a contract between the producer and every consumer. For JSON, treat it as a real interface:

• Document the shape. Field names, types, which are required, what the business key is.
• Only make additive changes without coordination. A new optional field is safe. Removing or renaming a field, or changing a type, breaks consumers and needs a rollout plan (support both shapes for a transition period).
• Version explicitly if the shape must change incompatibly — a schema_version field, or a new topic (order-events.v2) that consumers migrate to.
• Teams that need this enforced use Avro or Protobuf with a Schema Registry, which rejects an incompatible producer at publish time. That is beyond this section; the minimum without it is discipline and documentation.

Backfill and replay#

Because the topic retains records, you can reprocess.

• Fix-and-replay: a bug in the loader wrote wrong values for a period. Deploy the fix, then reset the consumer group's offsets to the start of that period (kafka-consumer-groups --reset-offsets --to-datetime ...), and let it reprocess. The idempotent sink makes the rewrite safe.
• New consumer, full history: a new job needs all historical events. Give it a fresh group.id with auto.offset.reset=earliest; it reads the topic from the log-start offset forward.
• Limit: you can only go back as far as retention. For history beyond that, the source of truth is the warehouse's raw layer, not Kafka.

Monitoring#

The signals that tell you the pipeline is healthy:

When to add stream processing#

The consumer here does light work: parse, batch, insert. When you need per-record transformation, joins between streams, windowed aggregation, or stateful enrichment before the data lands, that is the job of a stream processing framework — Kafka Streams, ksqlDB, or Apache Flink. Those are a larger topic and out of scope for this section. The dividing line: if the logic fits in "decode, validate, batch, write", a plain consumer is enough; if it needs state and windows over the stream itself, reach for a stream processor.

Common mistakes#

An at-least-once consumer with a non-idempotent sink#

Replays after a crash create duplicate rows. Make the sink de-duplicate on a business key.

No business key in the event#

Without a stable unique id, the sink cannot de-duplicate and replay is unsafe. Add one at the producer.

Retention shorter than the outage you need to survive#

If the consumer can be down for a day, retention of a few hours guarantees data loss. Size retention against realistic downtime.

Changing the JSON shape without a plan#

Removing or renaming a field breaks live consumers. Additive changes only, or a versioned rollout.

Treating lag as a dashboard curiosity#

Growing lag is the early warning that the pipeline is failing. Alert on it, and compare it to retention.

Quick reference#

See also#