Partitioning in Spark

Distributed Processing Fundamentals introduced partitions as the unit of parallel work — one slice of data per executor. This article covers how many partitions a job actually ends up with, and why that number is worth checking rather than trusting to a default.

Where the partition count comes from#

Reading a file creates an initial partition count based on the source's own layout — roughly one partition per block for a file in object storage, or a number derived from the file's row groups for a Parquet source — the data parallelism Apache Spark is built around. After any operation that reshuffles data — a groupBy, a join — the resulting partition count is controlled by a single setting:

spark.conf.set("spark.sql.shuffle.partitions", 200)  # the actual default

200 is Spark's default shuffle partition count, chosen as a reasonable middle ground decades ago and left unchanged since — reasonable for a mid-sized cluster on a mid-sized dataset, and wrong in both directions often enough that checking it explicitly is worth the habit.

Too few partitions: an idle cluster#

With fewer partitions than available executor cores, some executors sit idle for the entire stage — ten partitions on a hundred-core cluster means ninety cores do nothing while the other ten process one partition each. The fix is more partitions, not more cores.

Too many partitions: overhead eats the actual work#

Each partition, however small, carries fixed scheduling overhead — launching a task, tracking its completion. With enough tiny partitions, that overhead dominates the total job time even though very little actual computation is happening. This is the exact same shape as Parquet In Depth's small-files problem and Partitioning and Sharding's over-partitioning caution — the identical trade-off, appearing a third time in a third layer of the stack.

repartition() vs. coalesce(): not interchangeable#

Both change the partition count; they do it in different ways with different costs:

df.repartition(50)   # full shuffle: redistributes data evenly across                      # exactly 50 new partitions — can increase or                      # decrease the count
df.coalesce(50)       # merges existing adjacent partitions without a                       # full shuffle — cheaper, but can only decrease                       # the count, and can leave partitions uneven

coalesce avoids the shuffle's network cost by combining existing partitions in place rather than redistributing every row — the right choice when reducing partition count after a stage that already produced reasonably balanced data. repartition pays for a full shuffle but guarantees an even split, and is the only option when increasing partition count or when the current partitions are badly imbalanced to begin with.

Partitioning on write#

Writing output without specifying a partition scheme inherits whatever partition count the DataFrame currently has — often the default 200, regardless of how much data is actually being written. A small result written with 200 partitions produces exactly the small-files problem Parquet In Depth warns about, just generated from the Spark side instead of an ingestion pipeline:

df.coalesce(4).write.parquet("output/")

Coalescing to a sensible number immediately before writing is a common, cheap fix for output that would otherwise scatter into far more files than the data actually warrants.

Common mistakes#

Leaving the default 200 shuffle partitions unexamined#

Right for some jobs, badly wrong for a tiny dataset (pure overhead) or a massive one (each partition far too large). Checking and setting this explicitly, based on the actual data size and cluster shape, beats trusting the default by default.

Using repartition() where coalesce() would do#

Reducing partition count with repartition() pays for a full shuffle the job did not need — coalesce() accomplishes the same reduction more cheaply whenever an even data distribution is not specifically required.

Writing output without controlling partition count#

Inheriting whatever partition count a DataFrame happens to have at write time, rather than setting it deliberately, is one of the most common ways a Spark job produces the small-files problem downstream consumers then have to deal with.

Further reading#

• Performance Tuning — Spark SQL documentation — spark.sql.shuffle.partitions and Adaptive Query Execution's automatic partition coalescing.
• PySpark DataFrame API Reference — the full repartition and coalesce method signatures.
• MapReduce — the earlier framework whose fixed, less flexible partitioning Spark's tunable approach improved on.
• Partitioning and Sharding — the same too-few/too-many trade-off, for stored data instead of a running job.

See also#