Lazy Evaluation and the DAG

Why a chain of Spark transformations returns instantly with no output, what actually triggers computation, and how deferring execution lets Spark optimize the whole plan before running any of it.

BeginnerJune 20, 2026

Every Pandas operation covered elsewhere on this site runs the moment it is called — df[df.amount > 0] filters immediately, right there, on that line. Spark's DataFrame API looks similar and behaves completely differently: most calls do not run anything at all.

Transformations build a plan; they do not run one#

Python

filtered = orders_df.filter(orders_df.amount > 0)selected = filtered.select("customer_id", "amount")

Both lines return instantly, regardless of whether orders_df has a thousand rows or a trillion. Nothing has been filtered, nothing has been selected — Spark has only recorded that these steps should eventually happen, building an internal plan rather than touching any actual data. This is lazy evaluation: delay the work until the result is actually needed.

What actually triggers computation#

An action — .count(), .collect(), .show(), writing to a file — is what makes Spark run anything at all:

Python

total = selected.count()   # only now does Spark actually read,                            # filter, and select the data

Everything recorded up to this point — the filter, the column selection — runs now, together, as one coordinated job across every executor. The distinction between building a plan (filter, select, groupBy, called transformations) and running one (count, collect, write, called actions) is the single most important mental model for reading Spark code correctly, covered further in Transformations, Actions, and Shuffles.

Why defer execution at all#

Waiting to execute anything until an action is called lets Spark see the entire chain of transformations before running a single one of them — and optimize across all of it at once, the same way Predicate and Projection Pushdown lets a query engine skip data it will never need. A filter placed after a join in the code can be moved before the join internally, if Spark can prove that produces the same result while touching far less data — an optimization only possible because Spark saw the whole plan first, rather than executing the join the moment it was written.

The DAG, and why it also enables fault tolerance#

Spark represents the full chain of transformations as a directed acyclic graph — each transformation is a node, pointing back to the transformation (or raw data) it depends on:

Text

raw orders  ->  filter(amount > 0)  ->  select(customer_id, amount)  ->  count()   (source)        (transformation)         (transformation)          (action)

This DAG is not just an optimization tool — it is also how Spark recovers from a failed executor. If a machine holding one partition dies mid-job, Spark does not need a checkpoint of that partition's data; it can recompute exactly that partition by replaying the DAG from the original source, because the DAG fully describes how to reproduce any part of the result from scratch. This is the practical payoff of the RDD paper's original design, referenced in Distributed Processing Fundamentals.

Common mistakes#

Assuming a transformation ran because the line finished executing#

orders_df.filter(...) returning instantly does not mean the filter happened — it means Spark noted that it should happen eventually. Nothing about the source data has been touched yet.

Being surprised an action takes far longer than the lines before it#

The very first .count() or .show() in a chain of transformations often takes noticeably longer than expected, because it is the moment every preceding transformation actually runs — not a slow action, but the sum of everything the "instant" lines before it had deferred.

Calling multiple actions on the same chain without caching#

Each action re-triggers the full DAG from the source, unless the intermediate result is explicitly cached. Calling .count() and then .show() on the same uncached DataFrame recomputes the entire chain twice — a real, avoidable cost once the underlying computation is nontrivial.