Compression and Encoding Trade-offs

Row-Oriented vs. Column-Oriented Storage mentioned that columnar data compresses better without explaining why. It comes down to two distinct layers working together — encoding, which uses what a column's values actually look like, and compression, a generic byte-level technique applied on top of whatever encoding produced.

Encoding: using what the column already knows about itself#

A status column with four repeating values does not need to store the full string on every row. Dictionary encoding — the same idea behind Pandas' category dtype, covered in Memory-Efficient Data Representations — stores each distinct value once and replaces every occurrence with a small integer code:

Raw column:      pending, shipped, pending, pending, deliveredDictionary:       0 = pending, 1 = shipped, 2 = deliveredEncoded column:   0, 1, 0, 0, 2

Run-length encoding goes further when a column is also sorted or naturally clustered: instead of one code per row, store each run of repeated values once, with a count:

Sorted column:    0, 0, 0, 1, 1, 2, 2, 2, 2Run-length form:  (0, count=3), (1, count=2), (2, count=4)

Delta encoding helps a third common case — a mostly-increasing numeric column, like a timestamp or an ID — by storing each value as the difference from the previous one rather than the full number, which is usually a much smaller value to represent.

Compression: a second, separate layer#

After encoding, a general-purpose compression codec — Snappy, gzip, or Zstandard — compresses the already-encoded bytes further, using standard pattern-matching techniques that know nothing about columns, rows, or schemas at all:

The trade-off: smaller files cost CPU to read back#

A more aggressively compressed file is smaller — less disk space, less network transfer, less I/O time. It also costs more CPU to decompress on every read. For a file read once and rarely touched again, a higher compression ratio is close to free. For a file scanned repeatedly by many queries, decompression cost is paid every single time, and a faster codec with a slightly worse ratio can outperform a slower one with a better ratio in total query time.

Common mistakes#

Picking the highest compression ratio by default#

The codec with the smallest output is not automatically the fastest to query — decompression happens on every read, and a table scanned frequently often performs better with a faster codec even at a larger file size.

Storing an unsorted column that would have benefited from sorting#

Run-length encoding is dramatically more effective on a sorted or naturally-clustered column than a shuffled one. Choosing a useful sort order before writing — even just the column most queries filter on — can be a larger win than the choice of compression codec itself.

Treating encoding and compression as the same setting#

A file format's "compression" setting usually controls the codec layer only; encoding choices (dictionary, run-length, delta) are often automatic or configured separately. Conflating the two makes it easy to tune the wrong knob when trying to improve either file size or query speed.

Further reading#

• Data compression — the general field this article's codec layer belongs to.
• Dictionary coder, Run-length encoding, and Delta encoding — the three encoding techniques compared above.
• Zstd — the modern general-purpose codec increasingly used as a default across the ecosystem.
• Memory-Efficient Data Representations — the same dictionary-encoding idea, applied to an in-memory Pandas column instead of a file on disk.

See also#