Consensus and Leader Election

Replication and Consistency Models assumed a leader already exists. This article covers how a group of nodes, none of which trusts the others' view of the world any more than its own, actually agrees on who that leader is — and keeps agreeing after failures.

Why a heuristic is not enough#

"Whichever node responds to a heartbeat first becomes leader" sounds reasonable and fails in exactly the scenario that matters most: a network partition splits the cluster into two groups, and each group, unable to see the other, independently elects its own leader. Both now accept writes, believing themselves to be the sole leader — a split-brain scenario, and the reason leader election needs a protocol with an actual correctness proof behind it, not an intuitive-sounding rule.

Majority quorum: the mechanism that prevents split-brain#

The fix is structural: a new leader must be confirmed by a majority of all nodes, not just the ones it can currently reach:

5-node cluster, partitioned into a group of 3 and a group of 2:
  Group of 3 (majority: 3 out of 5) -> CAN elect a leader  Group of 2 (minority: 2 out of 5) -> CANNOT elect a leader

Because a majority requires more than half of all nodes, only one side of any partition can ever contain one — the two groups can never both reach majority at the same time. The minority side simply cannot proceed, which is the point: unavailability on the minority side is the cost of guaranteeing there is never more than one leader.

Raft: leader election in practice#

Raft, designed specifically to be easier to reason about than its predecessor Paxos, elects a leader through randomized timeouts:

Every node starts as a follower with a randomized election timeout.
If a follower hears nothing from a leader before its timeout fires,it becomes a candidate and requests votes from every other node.
A candidate that receives votes from a majority becomes the new leaderand starts sending heartbeats to reset everyone else's timeout.

The randomization matters: if every node's timeout fired at the same moment, they would all become candidates simultaneously and split the vote repeatedly. Random timeouts make it likely that one node times out first, requests votes before anyone else becomes a candidate, and wins outright.

Once elected, the leader is the only node that accepts new writes; it replicates them to followers and considers a write committed once a majority — not all, just a majority — of nodes have stored it. This is the same majority-quorum idea from leader election, applied continuously to every write, not just to the initial vote.

Where this is already running in systems covered elsewhere on this site#

Modern Kafka clusters elect a controller — the node responsible for partition leadership and cluster metadata — using KRaft, Kafka's own Raft-based consensus layer, replacing the older external ZooKeeper-based coordination. The mechanism is exactly the one described above: a majority of controller nodes must agree before cluster metadata changes are considered committed.

Common mistakes#

Treating leader election as instant and free#

A cluster has no leader for the duration of an election — typically a handful of network round trips, but not zero. A system that assumes writes can always proceed needs to account for this brief unavailability window explicitly, not discover it during an incident.

Rolling a custom leader-election scheme instead of using a proven one#

Ad-hoc heuristics — "whoever has the lowest node ID," "whoever responds fastest" — tend to work in testing and fail exactly during the partition scenarios they were never actually verified against. Raft and Paxos have formal correctness proofs behind them precisely because this class of bug is subtle enough that intuition reliably gets it wrong.

Confusing "elected a leader" with "cluster is healthy"#

A freshly elected leader in a system using asynchronous replication may be missing the most recent writes the previous leader had not yet replicated before failing. Consensus guarantees agreement on committed values, not that no acknowledged write can ever be lost — that guarantee depends on the majority-commit rule actually being followed for every write, not only for elections.

Further reading#

• Raft Consensus Algorithm and the paper, In Search of an Understandable Consensus Algorithm — the full protocol, deliberately written to be approachable.
• Paxos (computer science) — the earlier, more general consensus family Raft was designed to simplify.
• Leader election and Split-brain (computing) — the problem this article's majority-quorum mechanism solves.

See also#