Transformations, Actions, and Shuffles

Why a filter costs almost nothing and a groupBy can dominate a job's entire runtime — the narrow/wide distinction between transformations, and what a shuffle actually moves across the network.

IntermediateJune 23, 2026

Lazy Evaluation and the DAG split Spark operations into transformations (build the plan) and actions (run it). Not every transformation costs the same, and the difference between the cheap ones and the expensive ones is the single most useful thing to understand before trying to speed up a slow Spark job.

Narrow transformations: no data has to move#

A narrow transformation computes each output partition from exactly one input partition — every executor works entirely on its own local data, with nothing to send anywhere — the ideal case for data parallelism:

Text

Partition A  --filter-->  Partition A'Partition B  --filter-->  Partition B'Partition C  --filter-->  Partition C'

filter, select, and withColumn are narrow: filtering partition A's rows only ever needs partition A's own data. These run essentially in parallel with no coordination cost between executors at all.

Wide transformations: data has to be regrouped#

A wide transformation computes an output partition from data spread across multiple input partitions — often every partition in the job:

Text

Partition A (has some key="x" rows, some key="y" rows)Partition B (has some key="x" rows, some key="y" rows)Partition C (has some key="x" rows, some key="y" rows)                    |            groupBy(key) needs every "x" row together,            regardless of which partition it started in                    |Partition A' (all key="x" rows)   Partition B' (all key="y" rows)

groupBy, join, distinct, and repartition are wide: computing a correct group-by sum requires every row sharing a key to end up on the same executor, even though they started out scattered across every partition arbitrarily. Making that happen is a shuffle: every executor serializes the relevant rows, sends them across the network to whichever executor now owns that key, and the receiving executor deserializes and combines them.

A shuffle is usually where a Spark job's time actually goes

Network transfer, serialization, and often disk spilling for data too large to shuffle entirely in memory make a shuffle dramatically more expensive than any narrow transformation. A job that looks slow rarely has its filters or column selections to blame — checking which stage in the Spark UI corresponds to a shuffle is usually the fastest way to find the real bottleneck.

Reducing shuffle cost: filter and select before the shuffle#

Every column and row that survives to a shuffle has to be serialized and moved — narrowing the data before a wide transformation directly shrinks what the shuffle has to move:

Python

# Filters and selects columns before the join reduces what# actually needs to move across the network during the shufflesmall = orders_df.filter(orders_df.status == "shipped").select("customer_id", "amount")result = small.join(customers_df, "customer_id")

DataFrames and the Catalyst Optimizer covers Catalyst doing exactly this reordering automatically when it can prove it is safe — writing the filter first regardless is still worth the habit, since it keeps the plan easy to read and does not depend on the optimizer catching every case.

Avoiding the shuffle entirely: broadcast joins#

When one side of a join is small enough to fit comfortably in memory on every executor — a dimension table with a few thousand rows, joined against a fact table with a billion — Spark can send a full copy of the small table to every executor instead of shuffling the enormous one:

Python

from pyspark.sql.functions import broadcast
result = orders_df.join(broadcast(customers_df), "customer_id")

This avoids the shuffle for the large table entirely — each executor already has everything it needs to complete its share of the join locally. Join Algorithms covers this same small-side/large-side asymmetry for a traditional database's hash join; a Spark broadcast join is the distributed-cluster version of the identical idea.

Common mistakes#

Chaining several wide transformations without checking if they can combine#

Each unnecessary shuffle repeats the full network and serialization cost. Checking whether two groupBy stages could be expressed as one, or whether an intermediate repartition is actually needed, is worth doing before accepting a job's shuffle count as fixed.

Joining a small table without broadcasting it#

Spark can sometimes detect this automatically based on size estimates, but an explicit broadcast() hint removes the guesswork — especially useful when statistics are stale or Spark's own size estimate is wrong, the same estimation-can-be-wrong caution Cost-Based Optimization covers for a traditional query planner.

Assuming every transformation has the same cost#

A chain of ten filter calls costs close to nothing extra; one groupBy in the middle of that chain can dominate the entire job's runtime. Narrow-vs-wide is the first question worth asking about any slow stage, before looking at anything else.