Distributed Processing Quick Reference

A compact lookup for lazy evaluation, the DataFrame vs RDD choice, partition sizing, narrow vs wide transformations, broadcast joins, and the three most common PySpark performance pitfalls.

Beginner

Before you use this reference

This page is a lookup, not a tutorial. The explanations live in Distributed Processing Fundamentals, Lazy Evaluation and the DAG, DataFrames and the Catalyst Optimizer, Partitioning in Spark, Transformations, Actions, and Shuffles, Reading and Writing Data at Scale, and PySpark Performance Pitfalls.

Transformations vs. actions#

Text
Transformation (lazy, builds the plan): filter, select, groupBy, joinAction (triggers execution):            count, collect, show, write

Narrow vs. wide transformations#

TypeExamplesData movement
Narrowfilter, select, withColumnNone — each partition computed independently
WidegroupBy, join, distinct, repartitionShuffle — data moves across the network

DataFrame vs. RDD#

Text
RDD:       arbitrary Python functions, opaque to the optimizerDataFrame: named columns + structured ops -> Catalyst can optimize

Prefer the DataFrame API; drop to RDDs only when nothing else expresses the operation.

Reading and writing#

Python
# Explicit schema — skips the extra inference passdf = spark.read.csv("orders.csv", header=True, schema=schema)
# Partitioned write, with an explicit modedf.write.mode("overwrite").partitionBy("year", "month").parquet("output/")
ModeBehavior
overwriteReplace existing data
appendAdd alongside existing data — not automatically idempotent
errorIfExistsFail if the destination already has data (default)

Partition sizing#

Text
Default spark.sql.shuffle.partitions = 200 — check it, don't trust it.Target: ~2-4 partitions per CPU core, ~128MB-1GB of data per partition.
repartition(n):  full shuffle, even split, can increase or decrease countcoalesce(n):     merges adjacent partitions, cheaper, decrease only

Avoiding a shuffle: broadcast join#

Python
from pyspark.sql.functions import broadcastresult = big_df.join(broadcast(small_df), "key")

Three pitfalls to check first#

Text
.collect() on an unbounded result:  can OOM the driverPython UDF instead of a built-in:   opaque to Catalyst + serialization costData skew:                          one oversized partition dominates a stage

Further reading#

See also#

Distributed Processing Fundamentals
Why this module's tools are worth reaching for in the first place.
PySpark Performance Pitfalls
The full explanation behind the three pitfalls above.