Every practical system covered elsewhere on this site — a Kafka cluster, a sharded ClickHouse deployment, a distributed job scheduler — is, underneath its specific API, a set of independent machines coordinating to behave like one system. This module is about the handful of unavoidable trade-offs that coordination forces, regardless of which specific tool implements them.
What makes a system "distributed"#
A distributed system is a collection of independent computers that communicate only by passing messages over a network, and that present themselves to a caller as a single coherent system. The independence is the important part: each machine has its own memory, its own clock, and can fail — or simply run slowly — without the others knowing why, or even knowing that it happened at all.
A network call is not a slower function call#
A function call on one machine has one failure mode: it works, or it raises an exception you catch immediately. A network call has a third outcome a function call cannot have at all — it can succeed on the far side and the response can still never arrive, because the network dropped it, the client's socket timed out first, or the machine that would have sent it crashed a moment later:
Client Server |--- write("orders", row) --->| | | (row written successfully) | X <----- ack -----| (ack lost in the network) | | (client sees a timeout — but the write already happened)From the client's side, a timeout looks identical whether the server never received the request, received it and is still working, or received it, finished, and the response was what got lost. All three require different handling, and the network gives no way to tell them apart. Idempotence — covered throughout Data Ingestion & APIs for exactly this reason — is one of the main tools for making that ambiguity safe to live with: design the write so that retrying it, just in case, causes no harm even if it already happened.
The assumptions a single machine lets you make for free#
L. Peter Deutsch and colleagues at Sun Microsystems catalogued the false assumptions programmers new to distributed systems repeatedly make, now known as the fallacies of distributed computing. The first four explain most of what the rest of this module is about:
- The network is reliable. It drops packets, silently, under normal operation — not just during an outage.
- Latency is zero. A call to another machine costs real, often variable, time — sometimes far more than the work being requested.
- Bandwidth is infinite. Moving data between machines has a real cost that moving data between two variables in the same process does not.
- The network is secure. Every message between machines crosses a boundary a single-process system never has to defend.
None of these are exotic edge cases reserved for planet-scale systems — they apply the moment a pipeline calls an API on another machine, writes to a database on another machine, or waits on a message queue. They are the reason the same operation that is trivial in a single Python function becomes a design decision once it crosses a network.
What the rest of this module covers#
Every later article here is really one question, seen from a different angle: given that machines fail independently and the network is not reliable, what does a distributed system have to give up, and what can it still guarantee? The CAP Theorem names the trade-off directly; the articles after it — partitioning, replication, consensus, failure detection, and clocks — are the concrete mechanisms every real distributed system, including the ones used as running examples elsewhere on this site, builds out of that same trade-off.
Common mistakes#
Treating a network call like a function call that happens to be slower#
The failure modes are not the same, only the latency is. Code that assumes "no response" means "nothing happened" will eventually be wrong in a way that duplicates or loses data.
Assuming a single machine's clock and view of the world is shared#
Two machines' clocks drift, and neither one has a complete, instantaneous view of what the other is doing. Clocks and Ordering in Distributed Systems covers what to use instead of wall-clock time when order actually matters.
Designing for the common case and never testing the failure case#
A distributed system's hardest bugs live in the cases that are individually rare but not, over enough machines and enough time, actually rare at all — a dropped packet, a slow disk, a machine that restarts mid-write. The rest of this module exists because these cases need to be designed for explicitly, not discovered in production.
Further reading#
- Fallacies of distributed computing — the full list of eight, with the history behind them.
- Idempotence — the property that makes retry-after-uncertain-failure safe.
- CAP theorem — the trade-off this whole module builds on, covered in depth next.