Parquet In Depth

The row group and column chunk structure behind every Parquet file, why the footer is read before any actual data, and the small-files problem that catches most teams new to writing it.

IntermediateJune 5, 2026

Row-Oriented vs. Column-Oriented Storage covered the layout idea in the abstract. Apache Parquet is that idea, standardized into an actual file format — this article covers its structure concretely enough to explain why it behaves the way it does. Full documentation covers every detail this article does not.

Row groups, column chunks, and pages#

A Parquet file is not one flat column-oriented blob — it is internally subdivided, and that subdivision is what makes selective reads possible:

Text

Parquet file └── Row group 1  (e.g. rows 0 - 999,999)      ├── Column chunk: order_id    (pages of order_id values)      ├── Column chunk: customer_id (pages of customer_id values)      └── Column chunk: amount      (pages of amount values) └── Row group 2  (rows 1,000,000 - 1,999,999)      ├── Column chunk: order_id      ├── Column chunk: customer_id      └── Column chunk: amount └── Footer: schema + per-column-chunk statistics (min, max, null count)

A row group is a horizontal slice of the table — a batch of rows, stored together. Within each row group, data is laid out by column chunk, split further into pages, the smallest unit actually read and decompressed. A query touching only amount reads the amount column chunk from every row group and nothing else — the column-oriented benefit from the previous article, made concrete.

Every Parquet reader reads the footer before touching a single data page: the schema, and for every column chunk, statistics including its minimum and maximum value. This is what Predicate and Projection Pushdown builds on — a reader can decide whole row groups are irrelevant to a query before reading any of their actual data, using only the footer's statistics.

Row group size is a real trade-off#

Text

Too small:  many row groups -> more per-group overhead,            metadata bloat, less benefit from columnar compression            (fewer repeated values per chunk to compress against)
Too large:  fewer, bigger row groups -> coarser pushdown granularity,            more memory needed to hold one group during a scan

There is no universally correct row group size — it depends on typical query selectivity and available memory. Most tools default to something in the range of 128MB–1GB per row group, tuned from there against real query patterns rather than guessed in advance.

Reading Parquet from Python#

Python

import pandas as pd
df = pd.read_parquet("orders.parquet", columns=["order_id", "amount"])

Passing columns explicitly is not just a convenience — it lets the underlying reader skip every other column chunk entirely, the same projection pushdown the next article covers in depth, available here with one keyword argument.

Parquet files are immutable#

A Parquet file, once written, is not updated in place — there is no operation that changes one row's value inside an existing file. A "update" in a Parquet-backed system means writing a new file (or new row groups) and updating which files the table considers current; this is the same idea Schema as Code applies to rebuilding a schema from scratch rather than hand-editing it, one level down at the file layer.

Common mistakes#

Writing many small Parquet files instead of fewer larger ones#

A pipeline that writes one small Parquet file per micro-batch accumulates thousands of tiny files — each with its own footer, its own per-file overhead, and none big enough for row groups to pay for themselves. This "small files problem" is one of the most common practical issues in Parquet-backed data lakes; periodic compaction (rewriting many small files into fewer larger ones) is the standard fix.

Never checking actual row group sizes in production#

Default settings do not automatically suit every table's shape and query pattern. Checking a file's real row group sizes and adjusting the writer's target size against observed query performance is worth doing once a table is queried heavily, rather than trusting the default indefinitely.

Treating Parquet as something rows get updated inside of#

Code that expects an UPDATE-like operation on a Parquet file is applying the wrong mental model — every update is really "write new data, then repoint what's considered current," whether that repointing is manual or handled by a table format layered on top of Parquet.