File Formats Quick Reference A compact lookup for row vs. column layout, Parquet structure, encoding and compression, pushdown, disk partitioning, and Avro compatibility rules.
Beginner June 10, 2026
Before you use this reference
Choosing a layout#
Parquet structure#
File -> Row group (batch of rows) -> Column chunk (one column's values in this row group) -> Page (smallest decompressed unit) -> Footer: schema + per-column-chunk min/max/null-count stats import pandas as pd df = pd . read_parquet ( "orders.parquet" , columns = [ "order_id" , "amount" ] ) Encoding vs. compression#
Snappy: fastest, lowest ratio Zstd: fast, tunable, strong ratio Gzip: slower, higher ratio Pushdown#
Projection pushdown: read only referenced columns Predicate pushdown: skip row groups whose min/max can't match (breaks if the filter wraps the column in a function, e.g. WHERE UPPER(status) = 'X') Partition pruning: skip whole files/directories by path, before opening any of them Hive-style disk partitioning#
Layout popularized by Apache Hive ;
Apache Iceberg and similar
table formats track this metadata explicitly instead of inferring it from
paths.
orders/year=2026/month=06/part-0001.parquet Avro schema compatibility#
Backward: new schema reads old data Full: both directions hold — Forward: old schema reads new data producers/consumers upgrade in any order Further reading#
See also#