Failure Detection and Timeouts

Why a distributed system can never directly observe that a node is down — only the absence of a response, which a dead node and a slow one produce identically — and the timeout trade-off that follows from that.

IntermediateMay 31, 2026

Consensus and Leader Election assumed nodes can tell when a leader has failed and trigger a new election. This article covers the genuinely hard problem underneath that assumption: a distributed system has no direct way to observe that a remote node is down — only the absence of a response, and a dead node produces exactly the same absence as a node that is simply slow.

The failure detector is fundamentally a guess#

Every failure-detection mechanism reduces to the same shape: send a periodic heartbeat, and declare a node failed if no heartbeat — or response to one — arrives within some timeout. The timeout is a genuine trade-off with no value that is simply "correct":

Text

Short timeout:  detects real failures fast                but flags a merely slow node as dead too                (a long garbage-collection pause, a disk stall,                 a brief network hiccup)
Long timeout:   fewer false positives                but a real failure goes undetected for longer,                leaving the system degraded the whole time

A node that is falsely declared dead because of a garbage-collection pause or a disk stall is not actually gone — it can resume exactly where it left off moments later, potentially still believing itself to be the leader, exactly the split-brain risk Consensus and Leader Election covers. Failure detection and consensus are not two separate problems — consensus's majority-quorum rule is precisely what limits the damage a falsely-declared-dead node can do once it resumes.

Fencing: making sure a "dead" node that comes back cannot cause harm#

Detecting a failure and acting on it safely are different problems. STONITH — "shoot the other node in the head," the deliberately blunt name for this class of technique — forcibly isolates a node believed to have failed, typically by power-cycling it or revoking its access to shared resources, specifically so it cannot resume acting as leader after the fact:

Text

1. Node A stops responding to heartbeats.2. The cluster elects Node B as the new leader.3. Before Node B starts accepting writes, Node A is fenced —   forcibly prevented from writing, even if it "wakes up" and still   believes it is the leader.

Without this step, a temporarily unreachable Node A that resumes after election has completed can still accept writes its own clients send it, producing two leaders' worth of writes that were never coordinated with each other — the delayed, and often more damaging, version of the split-brain problem.

A more adaptive alternative to a fixed timeout#

A fixed timeout treats every network the same regardless of its actual, observed behavior. Phi accrual failure detectors — used by systems like Cassandra and documented in Akka's implementation — instead track the recent history of heartbeat arrival times and output a continuous suspicion level rather than a binary alive-or-dead verdict, adapting automatically to a network that is normally slightly jittery without needing that jitter hand-tuned into a fixed timeout value.

Where this shows up in a system already covered on this site#

Kafka's consumer group protocol runs on exactly this trade-off: session.timeout.ms controls how long the broker waits without a heartbeat before declaring a consumer dead and triggering a rebalance — too short, and a consumer doing a slow batch of work gets kicked out of the group and reassigned mid-task; too long, and a genuinely crashed consumer's partitions sit unprocessed for that entire window before anyone picks them up.

Common mistakes#

Tuning the timeout without understanding what it trades away#

A shorter timeout is not simply "better" or "safer" — it is a deliberate choice to detect failures faster at the cost of more false positives. Changing it without accounting for both directions of the trade-off usually just moves the pain from one failure mode to the other.

Detecting a failure without fencing the old leader#

An election that produces a new leader is not, by itself, a guarantee that only one node will act as leader — without fencing, a node that was merely slow, not actually dead, can resume writing after losing the election it never knew was happening.

Assuming a "healthy" heartbeat means the node can do real work#

A heartbeat only proves the process can respond to a small, cheap request. A node deep in a slow disk operation can still answer heartbeats promptly while being effectively unable to make progress on anything else — heartbeat health and actual capacity to do work are related, not identical.