Distributed Processing Fundamentals

What actually runs out first when a Pandas job stops working — memory or a single CPU core — and why the fix is the same partitioning idea covered elsewhere on this site, just applied to computation instead of storage.

BeginnerJune 19, 2026

Every Pandas article on this site assumes the data fits on one machine, in one process's memory. This module is about what happens once that assumption breaks, and Apache Spark specifically — the tool most of the data engineering world reaches for when it does.

What actually runs out first#

"Pandas doesn't scale" usually means one of two distinct problems, and the fix is different for each:

Memory. A DataFrame that does not fit in the machine's RAM at all — pd.read_csv on a file bigger than available memory fails outright, or succeeds and leaves no room for anything else.
A single CPU core. Pandas operations run on one core; a machine with sixty-four cores computing a Pandas aggregation uses exactly one of them, leaving the other sixty-three idle the entire time.

Buying a bigger machine addresses both, temporarily, and has a ceiling — the largest single machine available is still finite. Distributed processing takes the other approach: split the data and the computation across many machines, so no single one has to hold it all or compute it alone.

The same partitioning idea, applied to computation#

Partitioning and Sharding covers splitting stored data across nodes so no single one holds all of it. Distributed processing applies the identical idea to computation: split a DataFrame into partitions, send each partition to a different machine (called an executor in Spark), and have each one compute on its own slice in parallel:

Text

                    Driver (coordinates the job)                         |        ------------------------------------        |            |             |   Executor 1    Executor 2    Executor 3   (partition A) (partition B) (partition C)

A groupBy().sum() that would run on one core in Pandas runs on every executor simultaneously in Spark, each computing a partial result over its own partition — the same shape as the row-group-parallel reads Parquet In Depth covers on the storage side, now applied to the actual computation.

Spark specifically: what it adds beyond "just split the data"#

Splitting data across machines — data parallelism — is the easy part; making that split reliable and fast is where Apache Spark earns its place. Earlier frameworks like MapReduce solved the same splitting problem by writing intermediate results to disk between every stage — reliable, but slow for anything iterative. Spark originated from a research paper — Resilient Distributed Datasets — that solved a specific problem earlier distributed frameworks handled poorly: recovering automatically from a machine failing partway through a job, without restarting the entire computation from scratch, while keeping data in memory between stages instead of round-tripping through disk. The rest of this module covers the ideas built on top of that foundation: lazy evaluation, the DataFrame API, partitioning tuning, and the operations — shuffles specifically — that are expensive enough to design around deliberately.

When this module's tools are the right reach#

Not every large dataset needs Spark. A file that fits in memory with downcast dtypes and chunked reading does not benefit from distributing across machines — that overhead is pure cost with nothing to show for it below a certain data size. Spark earns its complexity once a single machine's memory or a single core's throughput is the actual, measured bottleneck — not before.

Common mistakes#

Reaching for Spark before confirming a single machine can't do it#

Distributing a job that would fit comfortably on one machine adds coordination overhead — network communication between executors, job scheduling — for no benefit. Memory-Efficient Data Representations covers extending how far a single machine can go before that trade-off is worth making.

Assuming more executors always means more speed#

Coordination and data movement between executors have a real cost. Partitioning in Spark covers why an under- or over-partitioned job can be slower with more executors, not faster.

Treating "distributed" as a synonym for "fast"#

Distribution trades single-machine simplicity for the ability to scale past what one machine could ever hold — it is not automatically faster for a job small enough to run comfortably on one.