Memory-Efficient Data Representations

A DataFrame that will not fit in memory is not always genuinely too big — often it is stored in a shape that costs far more memory than the data actually requires. Before reaching for a bigger machine or a distributed tool, it is worth checking whether the representation itself is the problem.

Measure before optimizing#

Pandas reports memory usage lazily by default — the real number, including the actual content of object columns, needs deep=True:

df.memory_usage(deep=True).sort_values(ascending=False)

Without deep=True, an object column (Pandas' catch-all for strings) reports only the size of the pointers to its Python string objects, not the strings themselves — often understating that column's real footprint by an order of magnitude.

What a dtype actually costs#

Every value in a numeric NumPy array occupies a fixed number of bytes determined entirely by its dtype — int64 costs 8 bytes per value whether the value is 3 or 3 billion. Most columns never need the full range a 64-bit type provides:

df["quantity"] = pd.to_numeric(df["quantity"], downcast="integer")df["price"] = pd.to_numeric(df["price"], downcast="float")

downcast="integer" picks the smallest integer type — int8, int16, int32, or the original int64 — that still fits every value actually present in the column. A quantity column that never exceeds 100 fits in int8 (range -128 to 127); one running up to a few thousand fits int16 instead. Either is a 4x to 8x reduction over the default int64.

Category: dictionary-encoding repeated strings#

A status column with a million rows but only four distinct values ("pending", "shipped", "delivered", "cancelled") stores the same handful of Python string objects a million times over as object dtype. Pandas' category dtype fixes this with the same idea a database's dictionary-encoded column uses: store each unique value once, and store a small integer code per row pointing at it:

df["status"] = df["status"].astype("category")

This is a general categorical variable encoding technique, not a Pandas-specific trick — the same low-cardinality-column-as-codes-plus-lookup-table idea appears in ClickHouse's LowCardinality(String), mentioned in the SQL Quick Reference, and in most columnar storage formats. Wherever a column has a small, repeating set of distinct values relative to its row count, this representation applies.

Chunked reading for files too large for memory at all#

Downcasting and category reduce memory for data already loaded. A file that will not fit in memory at all needs to be processed in pieces instead:

totals = {}for chunk in pd.read_csv("large_export.csv", chunksize=100_000):    for customer_id, total in chunk.groupby("customer_id")["amount"].sum().items():        totals[customer_id] = totals.get(customer_id, 0) + total

chunksize turns read_csv into an iterator of smaller DataFrames instead of one call that loads the whole file — each chunk is processed and discarded before the next one is read, so peak memory usage depends on the chunk size, not the file size. The trade-off is that any aggregation spanning the whole file — like the running totals dict above — has to be built up incrementally across chunks, rather than computed in one groupby() over the full DataFrame.

Common mistakes#

Downcasting without checking whether the range is guaranteed#

An int8 column is a bet that no future value will exceed 127. When that assumption comes from "today's data happens to fit," not from a real constraint on the domain, an integer overflow or a silent wraparound is one new row away.

Using category on a high-cardinality column#

category saves memory when a column repeats a small set of values many times. Applied to a column that is nearly all unique values — a UUID column, for instance — it adds the overhead of a lookup table with almost no repetition to amortize it against, costing more than plain object.

Recomputing a global aggregation naively across chunks#

chunk.groupby(...).sum() for each chunk gives per-chunk sums, not the overall total — combining them correctly, as in the running totals dict above, requires summing the per-chunk results together, not just concatenating them.

Further reading#

• pandas.to_numeric — the downcast parameter's exact behavior.
• Categorical variable — the general statistical concept behind the category dtype.
• Integer overflow — what happens when a downcast type's range is exceeded later.
• pandas.read_csv — the chunksize and dtype parameters referenced above.

See also#