Every format covered so far in this module has been column-oriented, built for scanning millions of rows at once. Apache Avro is a serialization format that solves a different problem: one small, complete record at a time — exactly the shape of a single event on a Kafka topic.
Row-oriented, on purpose#
Row-Oriented vs. Column-Oriented Storage covered why a row-oriented layout fits whole-record reads and writes. A single Kafka message is exactly that: one complete record, produced and consumed as a unit, never scanned alongside a billion others for one column's average. Avro is row-oriented for the same reason a transactional database usually is — the read pattern it serves never touches "one column across many records," only "one record, all its fields, right now."
The schema travels with the mechanism, not just the data#
Every Avro record is paired with a JSON schema describing its fields and types. In a Kafka pipeline, that schema usually lives in a Schema Registry rather than being repeated in every message — the producer registers a schema once, and every message just references it by ID, keeping the per-message overhead small while still making the schema available to every consumer.
Backward, forward, and full compatibility, precisely#
Confluent's Schema Registry documentation defines the compatibility rules Avro's schema resolution is built to support:
- Backward compatible — a consumer using the new schema can read data written with the old one. Safe to deploy consumers before producers finish upgrading.
- Forward compatible — a consumer using the old schema can read data written with the new one. Safe to deploy producers before consumers finish upgrading.
- Full compatible — both directions hold. Producers and consumers can upgrade in any order, independently.
Which schema changes are actually safe#
Adding a field WITH a default value -> safe (backward compatible)Removing a field that HAD a default value -> safe (backward compatible)Renaming a field -> NOT safe — looks like a remove + an unrelated addChanging a field's type incompatibly -> NOT safe (e.g. string -> int)A rename is the mistake that looks harmless
Renaming customer_id to client_id feels like a trivial change and is
one of the most common ways teams accidentally break compatibility — to a
schema compatibility checker, this looks exactly like deleting one field
and adding an unrelated new one, which fails backward compatibility the
moment an old consumer tries to read a record written with the new name
and finds customer_id simply missing. The safe version of a rename is
adding the new field alongside the old one, migrating consumers, and only
removing the old field once nothing reads it anymore.
Where this fits the rest of this site#
Producing Messages with Python and Consuming Messages with Python cover the Kafka side of this pairing without needing Avro specifically — adding Avro and a Schema Registry on top of that same producer/consumer pattern is what gives a Kafka pipeline the compatibility guarantees this article describes, instead of relying on every producer and consumer staying in perfect, manually coordinated sync about the message shape.
Common mistakes#
Renaming instead of add-then-deprecate#
Covered above — the single most common way a seemingly small schema change breaks every consumer still running the previous schema version.
Adding a required field with no default#
A new field with no default value is not backward compatible — an old consumer reading new data has no value to fall back to, and a new consumer reading old data has nothing to populate the field with either. Defaults are what make additions and removals safe in the first place.
Treating plain JSON payloads as if they had the same guarantees#
A JSON message with no registered schema has no compatibility checking at all — nothing stops a producer from silently changing a field's type or name, and nothing tells a consumer that happened until it breaks in production. Avro's schema registry enforces these rules at write time; unstructured JSON enforces nothing.
Further reading#
- Apache Avro documentation — the full specification, including schema resolution rules.
- Apache Avro — Wikipedia — background and history.
- Schema Evolution and Compatibility — Confluent documentation — the full set of compatibility types, including the transitive variants.
- Schema Registry overview — Confluent documentation — how the registry enforces these rules in a real Kafka deployment.