DataFrames and the Catalyst Optimizer

Lazy Evaluation and the DAG established that Spark sees the whole chain of transformations before running any of it. This article covers what Spark actually does with that visibility — and why it can only do it for one of Spark's two APIs.

RDDs: the original, low-level API#

Spark's original abstraction, the RDD (Resilient Distributed Dataset), is a distributed collection with almost no structure attached — no schema, no column names, just partitions of arbitrary Python objects transformed by arbitrary functions, the same data parallelism idea MapReduce popularized before Spark, minus MapReduce's requirement to write intermediate results to disk between every stage:

rdd = sc.textFile("orders.csv")parsed = rdd.map(lambda line: line.split(","))amounts = parsed.map(lambda fields: float(fields[2]))

Every map here takes a Python function — lambda line: line.split(",") — that Spark cannot look inside. It is a black box: Spark knows to call it on every row, and nothing about what the function actually does.

DataFrames: structured, and therefore optimizable#

The DataFrame API expresses the same idea in terms Spark's planner can actually understand — named columns, declared types, and a fixed vocabulary of operations (filter, select, groupBy) instead of arbitrary functions:

df = spark.read.csv("orders.csv", header=True, inferSchema=True)amounts = df.select("amount").filter(df.amount > 0)

filter and select are not opaque Python functions — they are structured operations Spark's query planner can inspect, reorder, and optimize, because it fully understands what each one means.

The Catalyst optimizer: the same pipeline as any query planner#

Spark SQL's Catalyst optimizer processes a DataFrame plan through stages that map directly onto Query Performance Fundamentals' parse-plan-optimize-execute model, covered there for a traditional database:

Parse       -> build an initial logical plan from the DataFrame callsAnalyze     -> resolve column names and types against the actual schemaOptimize    -> logical optimizations: push filters below joins, combine               adjacent operations, eliminate unused columnsPhysical plan -> choose join algorithms and execution strategy               (the same hash/sort-merge/broadcast choice covered in               Join Algorithms)

A filter written after a join in the code can be pushed below the join during the optimize stage — evaluated first, against less data, exactly like Predicate and Projection Pushdown describes for a file scan. This only works because Catalyst understands that a filter call means "keep rows matching this condition" — the same call expressed as an opaque rdd.filter(lambda row: ...) blocks this reordering, because Catalyst cannot prove what an arbitrary Python function does without running it.

Common mistakes#

Dropping into the RDD API out of old habit or a Spark tutorial's example#

Most structured data problems are expressible entirely in the DataFrame API today, even on older Apache Spark tutorials still written against RDDs. Reaching for rdd.map() when df.select() would do the same job gives up Catalyst's optimization for no benefit.

Assuming every DataFrame operation is automatically optimized#

A DataFrame call wrapping a Python UDF is still opaque at exactly that point in the plan — the DataFrame API's optimizability is a property of its structured operations, not a guarantee that applies unconditionally to every line of code that happens to use it.

Not knowing which API a piece of legacy code actually uses#

RDD-based code and DataFrame-based code can look superficially similar at a glance. Confirming which API a slow job actually uses is worth doing before assuming Catalyst is even in a position to help.

Further reading#

• Performance Tuning — Spark SQL documentation — the official reference for Catalyst's optimizations and configuration.
• PySpark DataFrame API Reference — the full structured operation vocabulary Catalyst can reason about.
• MapReduce — the earlier programming model Spark's in-memory, DAG-based execution improved on.
• Query Performance Fundamentals — the same four-stage optimization pipeline, for a traditional database.
• Join Algorithms — the physical-plan join choice Catalyst makes during the physical planning stage.

See also#