Partitioning Files on Disk

Encoding a column's value into a directory path so a query engine can skip entire files without opening any of them, the Hive-style layout most engines expect, and why partitioning by the wrong column recreates the small-files problem.

IntermediateJune 8, 2026

Predicate and Projection Pushdown covered skipping row groups inside a file using its footer statistics. This article covers the layer above that: skipping entire files, using nothing but their path — before a single footer is even read.

Encoding a column into the directory structure#

The de facto standard, sometimes called Hive-style partitioning, encodes partition column values directly into the folder path:

Text

orders/  year=2026/    month=05/      part-0001.parquet      part-0002.parquet    month=06/      part-0001.parquet

year and month are not stored as columns inside these files at all — they are implied entirely by the path. Spark's own documentation on Parquet partition discovery and Apache Hive's partitioning model both describe the same convention, which is why most engines that read Parquet also understand this layout without any special configuration.

Partition pruning: skipping files without opening them#

A query filtering WHERE year = 2026 AND month = 6 never needs to list, open, or read the footer of anything under month=05/ — the engine prunes those paths using the directory structure alone:

Text

Query: WHERE year = 2026 AND month = 6
year=2026/month=05/*.parquet   -> pruned, path doesn't matchyear=2026/month=06/*.parquet   -> read (predicate pushdown applies                                    within these files, per the                                    previous article)

This composes directly with predicate pushdown: partition pruning eliminates whole files first, and predicate pushdown then eliminates row groups within whatever files remain — two layers of the same "skip what cannot match" idea, operating at different granularities.

This is not the same thing as database table partitioning#

Partitioning and Sharding covers partitioning a database table — rows assigned to a partition by a key, often for distributing load across nodes. File-on-disk partitioning serves a narrower, read-optimization purpose: helping a query engine skip files it does not need to open, with no implication about which machine anything lives on. The same hot-partition and cardinality reasoning from that article still applies here, just applied to directory counts instead of nodes.

Choosing a partition column#

A good partition column is one that is both frequently filtered on and has moderate cardinality — enough distinct values to meaningfully narrow a scan, not so many that partitioning produces one file per directory:

Over-partitioning recreates the small-files problem

Partitioning by a high-cardinality column — customer_id, a UUID — creates one directory per distinct value, each holding a handful of tiny files. Parquet In Depth already covers why many small files are expensive; partitioning on the wrong column is one of the most common ways teams back into that exact problem without realizing partitioning was the cause.

A date-based column — year/month, or just a single date partition — is the most common real-world choice precisely because so many analytical queries already filter on a time range, and dates naturally bound cardinality to something reasonable (365 partitions a year, not millions).

Beyond manually managed directories#

Everything above is the plain, manually managed version of file partitioning — the layout Apache Hive popularized and most engines still understand natively. Table formats like Apache Iceberg build on top of this idea, tracking partition metadata explicitly instead of inferring it from directory paths, which avoids several of this article's common mistakes automatically — at the cost of adopting a format-specific table layer instead of plain files in a data lake.

Common mistakes#

Partitioning by a column nothing actually filters on#

Partition pruning only helps when a query's WHERE clause matches the partition column. Partitioning by a column that is never filtered adds directory overhead with none of the pruning benefit.

Partitioning too finely#

Beyond the small-files problem, deeply nested partition paths (year=/month=/day=/hour=/) can make directory listing itself slow — the engine has to enumerate a large tree before it can even start pruning.

Expecting partition columns to also appear as regular columns automatically#

Some tools require explicitly registering partition columns before they appear in query results; a raw file scan that ignores the directory structure will not see year or month as columns at all unless the engine's partition discovery is actually engaged.