Partitioning and Sharding

A single machine has a ceiling: a fixed amount of disk, memory, and network throughput. Partitioning — also called sharding when each partition lives on its own server — raises that ceiling by splitting data across many machines, each responsible for a subset.

The partition key decides everything#

Every row needs a rule that says which partition it belongs to. That rule — the partition key — determines whether load actually spreads evenly or piles onto one node while the others sit idle:

Partition by order_date, one partition per day:
  2026-06-01 -> Partition A  2026-06-02 -> Partition B  2026-06-03 -> Partition C   <- today: every new write lands here
Every write for "today" hits the same partition. The other twopartitions serve reads only — one machine does all the write work.

This is a hot partition: a key choice that is reasonable for one purpose (range queries by date) and disastrous for another (load balancing across writes), on the same data, at the same time.

Range vs. hash partitioning#

Range partitioning keeps keys in order across partitions — everything from A to M on one node, N to Z on another. This makes range scans efficient (fetch three contiguous partitions instead of scattering the read across all of them) and is exactly the pattern that produces hot partitions when the key increases monotonically — a timestamp, an auto-incrementing ID — because every new row's key is always the largest one, always landing on the same node.

Hash partitioning applies a hash function to the key first, then assigns the result to a partition. This spreads writes evenly regardless of whether the underlying key is monotonic, at the cost of range scans: rows that were adjacent by key are now scattered across every partition, so a range query has to touch all of them.

Consistent hashing: adding a node without reshuffling everything#

A naive hash partitioning scheme — partition = hash(key) % N — has a costly property: changing N, by adding or removing one node, changes the result of the modulo for almost every key, forcing nearly all data to move. Consistent hashing avoids this by hashing both keys and nodes onto the same ring and assigning each key to the next node clockwise from it:

Hash ring (simplified, 4 positions shown):
        Node A       /       \  Node D         Node B       \       /        Node C
key "order-42" hashes near Node B -> stored on Node B.Adding Node E between A and B only moves the keys thatwere already between A and B — everyone else is unaffected.

Only the keys between the new node and its counterclockwise neighbor move — roughly 1/N of the data for N nodes, instead of nearly all of it. This is the mechanism behind resizing a cluster without a full data migration every time.

Where this is already in use elsewhere on this site#

ClickHouse MergeTree and Table Design covers PARTITION BY — ClickHouse's own range-style partitioning, chosen coarse specifically to avoid the hot-partition and per-partition-overhead problems this article describes. Topics, Partitions, and Offsets covers a hash-partitioning decision from the other direction: Kafka assigns a message to a partition by hashing its key, for the same load-spreading reason covered above. Neither article names the underlying theory explicitly — this is where it comes from.

Common mistakes#

Partitioning by a monotonically increasing key for write-heavy data#

A timestamp or auto-incrementing ID as the partition key concentrates every new write onto whichever partition currently holds "the latest" — the textbook hot-partition case. Hash the key, or partition on something less correlated with recency, when write throughput matters more than range scans.

Choosing too few partitions#

Too few partitions caps how far the system can actually scale out — with four partitions, a cluster can never usefully run more than four writers in parallel for that data, no matter how many machines are added.

Choosing far more partitions than the cluster can use#

Each partition carries its own overhead — metadata, open file handles, and in the case of ClickHouse's PARTITION BY, a real per-partition storage cost. Partitioning far more finely than the actual node count and query pattern require adds this overhead without a corresponding benefit.

Further reading#

• Partition (database) and Shard (database architecture) — the general vocabulary this article builds on.
• Consistent hashing — the full algorithm, including virtual nodes for more even distribution than the simplified ring shown above.
• ClickHouse MergeTree and Table Design — PARTITION BY as a concrete, practical instance of range partitioning.

See also#