Ingesting Messy Files covers what can go wrong reading a CSV. This module is about a different question: once data is at rest — written to disk, to object storage, to a data lake — what format should it actually be stored in? The answer changes how much space it takes, how fast it can be queried, and whether a schema change breaks every consumer at once.
Text formats: readable, self-describing, expensive to parse#
CSV and JSON are text formats — every value is stored as
human-readable characters. A number like 1204 takes four bytes to store
its digits, whether the actual value needs one byte or eight. Nothing
about the file's layout helps a reader skip data it does not need; parsing
a text file means reading it byte by byte and re-discovering its structure
every single time.
Binary formats: compact, fast, opaque without the right tool#
A binary format stores values in the machine's own representation — an integer as a fixed number of bytes, not as printed digits — and typically adds a compact, structured layout on top. Apache Parquet and Apache Avro are both binary formats; neither can be usefully opened in a plain text editor, and both are dramatically smaller and faster to read than the equivalent CSV.
The trade is legibility for performance
A CSV can be inspected with cat, diffed in version control, and
understood by anyone without special tooling. A binary format gives that
up in exchange for size and speed — the right choice depends on whether a
file is meant for a human to glance at or a pipeline to process at scale.
Row-oriented vs. column-oriented: the layout question#
Independent of text-vs-binary, every tabular format also makes a second choice: store each row's values together, or store each column's values together. This single decision is responsible for most of the practical difference between the formats covered in this module, and gets a full article of its own next: Row-Oriented vs. Column-Oriented Storage.
Two questions that decide the format#
- Is this file read mostly whole, or mostly by a few columns at a time? A service reading every field of one record at a time (an application's own storage, an event log consumed end to end) fits a row-oriented format. An analytical query touching three columns out of fifty across a billion rows fits a column-oriented one.
- Does the schema change over time, and does every consumer upgrade at the same moment? A format with strong, explicit schema evolution rules — Avro and Schema Evolution covers this in depth — matters far more when producers and consumers deploy independently than when one team controls both ends.
Common mistakes#
Choosing a format because it is popular, not because it fits the read pattern#
Parquet's popularity in analytics does not make it the right choice for a system that mostly reads and writes whole records one at a time — that is exactly the row-oriented use case Avro or a plain row store fits better.
Treating a binary format's compactness as free#
Binary formats are smaller and faster to read, not free to produce — writing them well (choosing row group sizes, compression codecs, sort order) takes real design decisions, covered across the rest of this module.
Assuming a text format is inherently safer or simpler#
A CSV's readability does not protect against the encoding and schema-drift problems Ingesting Messy Files covers — a binary format with an explicit schema often catches exactly those problems earlier, at write time, instead of leaving them to be discovered on read.
Further reading#
- Apache Parquet and Apache Avro — the two formats this module covers in depth.
- Apache Parquet — Wikipedia and Apache Avro — Wikipedia — background and history for each.
- Apache ORC — a third columnar format, originally built for the Hadoop ecosystem, following the same column-oriented principles as Parquet.