Engineering Atlas › System Design Fundamentals

Avoiding split brain

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

How it happens

Take a primary and a standby database with a heartbeat link. The link drops. The standby sees no heartbeats for 10 seconds, promotes itself, and the virtual IP or DNS record moves. But the old primary never crashed. Clients with cached connections keep writing to it. Orders now land on two databases with overlapping IDs, and merging them is manual and lossy.

The root cause is that "I cannot reach the leader" was treated as "the leader is dead". With two nodes there is no tiebreaker: each side sees the same silence.

Quorum: only one side may win

With a majority quorum, a node becomes leader only if more than half the cluster votes for it in a given term. Two disjoint majorities cannot exist, so at most one leader is elected per term. Raft and ZooKeeper's Zab work this way. A two-node pair can get the same property by adding a third vote: a witness node, a quorum disk, or a cloud lock service.

Quorum guarantees one elected leader per term. It does not stop the previous leader from still acting. That old leader may be paused, partitioned, or simply slow to notice it lost.

Leases: the old leader steps down

A leader holds a lease, for example 10 seconds, renewed by acks from a majority. If it cannot renew, it must stop serving before the lease ends. Followers wait longer than the lease before electing a replacement. This works only if clocks advance at roughly the same rate and the process is not frozen. A 15-second GC pause right after a successful renewal breaks it: the process wakes up still believing it holds the lease.

Fencing: the resource rejects the stale leader

Every time leadership changes, the coordinator issues a monotonically increasing number: a term, epoch or fencing token. The leader attaches it to every write. The storage layer remembers the highest token it has accepted and rejects anything lower. A paused leader that wakes with token 33 is refused once the new leader has written with token 34. Fencing moves the safety check to the place where damage happens, so it holds even when clocks and pauses misbehave (Kleppmann).

When the resource cannot check tokens, clusters use STONITH: power off the old node out of band before promoting the new one (Pacemaker docs).

Fencing writes is not the whole job. A deposed primary can keep answering reads from its stale copy until it next tries to write. HDFS keeps a fencing method configured even though its JournalNodes already refuse a second writer, for exactly this reason (HDFS docs).

Down the stack

Layer 1The contract

A coordination service promises less than people assume. An etcd lease promises that if no LeaseKeepAlive arrives within the TTL, the lease is revoked and every key attached to it is deleted, and that every change gets a cluster-wide, strictly increasing revision. ZooKeeper promises that an ephemeral znode disappears when its session expires and that every transaction gets a unique, ordered zxid. Neither promises that the old holder learns it lost the lock before someone else gets it. The holder finds out asynchronously, on its next round trip.

The contract that actually prevents split brain lives at the resource: a write carrying token t is applied only if t is at least the highest token accepted so far, and that comparison is atomic with the write. Without that, the lock is advisory. Jepsen showed this is not academic: etcd's lock let two holders in at once even on a healthy cluster, and about 18% of acknowledged updates to a lock-guarded set were lost; the fix it recommends is to use the lock key's revision as the fencing token (Jepsen).

Layer 2The mechanism

Raft makes election safe with two persisted fields, currentTerm and votedFor (Raft paper). A node grants at most one RequestVote per term, and only to a candidate whose log is at least as up to date (compared by lastLogTerm, then lastLogIndex). Every RPC carries the sender's term, and any node that sees a higher term immediately reverts to follower. That gives one leader per term; the term itself is the fencing token inside the cluster.

Real systems push the same idea outward. HDFS NameNode HA uses the Quorum Journal Manager: a newly active NameNode asks a majority of JournalNodes to promise a new, higher epoch, and each JournalNode persists its last promised epoch and rejects edit-log writes from any lower one. Kafka attaches a leader epoch to replication fetches, and transactional producers get a producer epoch so a zombie instance fails with a fenced error. Since Kafka 4.0 (March 2025) the controller is itself a Raft quorum (KRaft) whose leader epoch fences stale controllers, and ZooKeeper is no longer used at all (Apache Kafka). ZooKeeper's 64-bit zxid packs the leader epoch in the high 32 bits and a counter in the low 32, so ordering by zxid also orders by leadership (ZooKeeper internals). Pacemaker clusters that cannot fence at the resource use STONITH agents that cut power through IPMI or a cloud API before promotion.

Layer 3The machine

Leases fail because a process can stop while its view of time does not update. Stop-the-world GC on a large JVM heap can pause for seconds. A container hitting its CFS CPU quota is throttled until the next period (100 ms by default) and can be throttled repeatedly (kernel docs). Swapping, SIGSTOP, and hypervisor pauses during VM live migration add more. Clock drift is a smaller effect: a crystal oscillator off by 50 ppm drifts only 0.5 ms over a 10 s lease. So the lease holder should measure with CLOCK_MONOTONIC (read through the vDSO in tens of nanoseconds, immune to NTP steps), start its lease clock before sending the renewal, and stop serving some margin before expiry.

Votes and epochs must hit disk before the reply. If a node replies "yes" and crashes before fsync, it can restart and vote again in the same term, producing two leaders. That makes fsync latency part of election latency: tens of microseconds on an SSD with power-loss protection, several milliseconds on many cloud volumes. etcd's guidance is to keep p99 WAL fsync under 10 ms, because disk latency is part of leader liveness: slow disks delay heartbeats and trigger spurious elections (etcd FAQ).

Why it's built this way

  1. A node cannot distinguish a dead peer from a partition or a pause, so leadership is always decided on incomplete information.
  2. Majority votes stop two leaders being elected in one term, but cannot stop an old leader that has not yet heard the news.
  3. Pauses and clock behaviour are outside the process's control, so time-based leases can be broken by events the leader cannot detect.
  4. The only component guaranteed to see both writes is the resource itself, so the final check, a monotonic token compared atomically, belongs there.

Diagram

sequenceDiagram
    participant L1 as "Old leader"
    participant Z as "Lock service"
    participant L2 as "New leader"
    participant S as Storage
    L1->>Z: acquire lease
    Z-->>L1: granted, token 33
    Note over L1: 15 s GC pause, lease expires
    L2->>Z: acquire lease
    Z-->>L2: granted, token 34
    L2->>S: write (token 34)
    S-->>L2: ok, max token = 34
    Note over L1: wakes, still thinks it leads
    L1->>S: write (token 33)
    S-->>L1: rejected, stale token

Worked example

A job scheduler runs 3 instances and must have exactly one active, which dispatches about 400 billing jobs per minute. Leadership uses a lease in a coordination service: TTL 10 s, renewed every 3 s.

Without fencing: the leader pauses for 12 s. At 10 s its lease expires and instance B takes over with a new lease. At 12 s the old leader resumes and, in the 3 s before its next renewal fails, dispatches 20 jobs that B also dispatches. Twenty customers are billed twice.

With fencing: the coordination service returns a revision number with each lease grant. On takeover, B first raises a single fence row to its token, UPDATE fence SET token = 34 WHERE token < 34. Every dispatch then runs in one transaction that locks the fence row (SELECT token FROM fence FOR SHARE), aborts if the stored token is higher than its own, and claims the job with UPDATE dispatch SET owner_token = :t WHERE job_id = :id AND owner_token IS NULL. The old leader wakes with 33, sees 34 in the fence and stops. A per-row check such as WHERE owner_token < :t would not be enough: a job the old leader claimed first would later be overwritten by B's higher token and dispatched twice. Duplicates fall to zero, at the cost of one short transaction per job.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Majority quorum electionAt most one leader per termNeeds 3+ voters; minority side stallsAny replicated state machine
Witness or quorum diskGives two nodes a tiebreakerWitness becomes a dependencyTwo-site or two-node clusters
Time-based leaseFast local reads on leaderUnsafe under pauses or clock jumpsCombined with fencing
Fencing tokensSafe even if old leader runsResource must check tokensStorage you control
STONITHWorks with dumb resourcesNeeds out-of-band power controlShared disks, legacy HA pairs

Pitfalls in production

Leadership lens

Check yourself

  1. Why does a majority quorum not, on its own, prevent two nodes from acting as leader?

    Show answer

    It ensures only one leader is elected per term, but the previous leader may not know it was replaced and can keep acting until it learns the new term.

  2. What makes fencing tokens safe when leases are not?

    Show answer

    The check happens at the resource, which compares a monotonically increasing number and rejects stale ones. It does not depend on the old leader's clock or on it noticing its lease expired.

  3. A two-node database pair loses its heartbeat link. Why can't either node decide safely on its own?

    Show answer

    Each sees the same silence and cannot tell a dead peer from a broken link. Without a third vote, both promoting or both waiting are equally likely outcomes.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →