Reading and Writing Data at Scale

Why schema inference costs an extra pass over the data, how Spark's Parquet reader gets pushdown and partition pruning for free, and the retry that silently duplicates a table because a write used the wrong mode.

Intermediate

Everything earlier in this module covered how Spark computes. This article covers the two ends every job actually touches: getting data in, and getting a result back out, at a scale where doing either one carelessly has a real, measurable cost.

Schema inference costs an extra pass#

Python
df = spark.read.csv("orders.csv", header=True, inferSchema=True)

inferSchema=True is convenient and expensive: Spark reads the entire file once just to guess each column's type before reading it again to actually load the data. For a large or frequently-run job, declaring the schema explicitly avoids that first pass entirely — and avoids the same type-guessing surprises Ingesting Messy Files covers for pandas.read_csv, where an inferred type can silently turn out wrong:

Python
from pyspark.sql.types import StructType, StructField, IntegerType, StringType, DoubleType
schema = StructType([    StructField("order_id", IntegerType()),    StructField("customer_id", StringType()),    StructField("amount", DoubleType()),])df = spark.read.csv("orders.csv", header=True, schema=schema)

Reading Parquet gets pushdown for free#

Reading an Apache Parquet source through the DataFrame API automatically applies the same optimizations Predicate and Projection Pushdown covers at the file-format level — Spark's Parquet reader reads each file's footer statistics and skips row groups that cannot match a filter, and reads only the column chunks a query actually selects:

Python
df = spark.read.parquet("orders/")result = df.filter(df.order_date >= "2026-06-01").select("customer_id", "amount")

Because Catalyst sees the whole plan before executing it, the filter and select here reach the Parquet reader itself, not just the DataFrame after it is fully materialized — exactly the file-level skipping Parquet In Depth describes, now triggered automatically by DataFrame operations instead of a hand-written query.

Reading a partitioned directory#

A source laid out with Hive-style partitioning — orders/year=2026/month=06/ — is discovered automatically, with year and month appearing as real DataFrame columns even though they only exist in the directory path:

Python
df = spark.read.parquet("orders/")recent = df.filter((df.year == 2026) & (df.month == 6))

Spark prunes directories that cannot match year == 2026 AND month == 6 before opening a single file under them — the exact partition pruning mechanism covered in Partitioning Files on Disk, happening transparently underneath this filter.

Writing: mode matters more than it looks#

Python
df.write.mode("overwrite").partitionBy("year", "month").parquet("output/")

partitionBy writes Hive-style partitioned output, so future reads get the same pruning benefit. The mode argument decides what happens if the destination already has data:

Text
"overwrite"       replace existing data entirely"append"          add new files alongside existing ones"errorIfExists"   fail if the destination already has data (the default)

append is not automatically idempotent

A job using mode("append") that gets retried after a partial failure loses idempotence — or simply run twice by accident — writes its output a second time, duplicating every row. Data Ingestion Fundamentals covers this exact problem for ingestion pipelines; the fix here is the same idea, applied to a Spark write: either overwrite the specific partition being rewritten, or deduplicate downstream, rather than trusting that a job will only ever run exactly once successfully.

Common mistakes#

Using inferSchema=True in a production pipeline#

Convenient for exploration, costly and occasionally wrong when run repeatedly at scale. A declared schema is both faster and more predictable once a pipeline is no longer a one-off script.

Writing without partitionBy for data that will be filtered later#

Output written as one flat, unpartitioned set of files loses the directory-level pruning Partitioning Files on Disk describes for every downstream reader — a cost paid on every future query against data that could have been partitioned once, at write time.

Retrying a failed append write without deduplication#

The most common source of silently duplicated data in a Spark pipeline. overwrite on the specific partition being rewritten, or an explicit dedup step downstream, closes this gap; assuming a retry is harmless does not.

Further reading#

See also#

Partitioning in Spark
Controlling partition count before writing, to avoid the small-files problem.
PySpark Performance Pitfalls
More ways a read or write step becomes the actual bottleneck.