The rest of this module explains how Spark distributes work. These three patterns are common enough, specific enough, and expensive enough to be worth checking by name before a longer investigation into why a job is slow — or, in the first case, why it crashed outright.
collect() pulling an entire distributed dataset into one process#
.collect() gathers every row from every partition, across every
executor, into the driver's single process — the one machine
coordinating the job, not a distributed store itself:
# Fine for a small resulttop_customers = df.groupBy("customer_id").sum("amount").orderBy("sum(amount)", ascending=False).limit(10).collect()
# Almost certainly wrong: pulls the entire distributed dataset# into one machine's memoryeverything = df.collect()A DataFrame with a billion rows, spread across a hundred executors'
memory, does not fit on the driver — .collect() on it fails with an
out-of-memory error, or
degrades the driver badly enough to stall the whole job. .show(20) for
a bounded preview, .take(n) for a bounded
sample, or writing the full result to storage with .write are the usual
alternatives; .collect() belongs only on a result already known to be
small.
Python UDFs: opaque to Catalyst, and a real serialization cost#
DataFrames and the Catalyst Optimizer already covers a Python UDF blocking Catalyst's optimization. The cost goes further than lost optimization: Spark itself runs on the JVM, and PySpark bridges to it with Py4J, so a Python UDF forces every row to be serialized out of the JVM, sent across that bridge to a separate Python process, computed there, and serialized back — real, per-row cross-process overhead that a built-in Spark SQL function never pays:
# Opaque UDF: serialization cost on every row, invisible to Catalystfrom pyspark.sql.functions import udfto_upper = udf(lambda s: s.upper())df.withColumn("status_upper", to_upper(df.status))
# Built-in function: runs inside the JVM, fully visible to Catalystfrom pyspark.sql.functions import upperdf.withColumn("status_upper", upper(df.status))Checking whether a built-in Spark SQL function already covers the transformation — string, date, and math functions cover an enormous share of real cases — before writing a UDF avoids both costs at once.
Data skew: one partition, one slow task, one idle cluster#
A shuffle groups data by key across partitions, breaking the load balancing even partition sizes normally provide — and if a handful of keys account for a disproportionate share of the rows (a single customer with a million orders, in a dataset otherwise averaging a few dozen per customer), the partition holding that key becomes dramatically larger than the rest:
Partition A: 50,000 rows (key = customer 42, an outlier)Partition B: 1,200 rowsPartition C: 980 rowsPartition D: 1,050 rows
Job finishes when the SLOWEST partition finishes —A alone can dominate the entire stage's runtime while B, C, and Dsit idle, already done.This is the same hot-partition
phenomenon covered for stored data, showing up a third time — first in
partitioning theory, then in file-on-disk layout, now in a running Spark
shuffle. Adaptive Query Execution's
skew-join handling — spark.sql.adaptive.skewJoin.enabled — detects and
splits an oversized partition automatically in modern Spark versions;
recognizing the symptom in the Spark UI (one task running long after every
other task in its stage completed) is what confirms skew is the actual
cause before reaching for that or any other fix.
Common mistakes#
Calling .collect() out of habit on an unbounded result#
The single most common way a Spark job crashes rather than merely runs
slowly. Bound the result first — .limit(), an aggregation, a filter — or
write it to storage instead of pulling it into the driver at all.
Writing a UDF for something a built-in function already does#
Checking the Spark SQL functions reference before writing custom Python logic avoids both the optimization loss and the serialization cost in one step, for a large share of common transformations.
Diagnosing a slow job without checking the Spark UI first#
Every pitfall in this article has a visible signature in Spark's own job
UI — one task taking far longer than its peers (skew), a stage with far
more tasks than expected (over-partitioning), a driver running out of
memory (collect()). Systematic Query Profiling's
measure-before-guessing discipline applies here directly: the UI usually
shows which of these it actually is before any code needs to change.
Further reading#
- Performance Tuning — Spark SQL documentation — Adaptive Query Execution, including automatic skew-join handling.
- PySpark SQL Functions Reference — the built-in function library worth checking before writing a UDF.
- Partitioning and Sharding — the same hot-key phenomenon, in its original, storage-layer form.