Choosing a leader among nodes

13 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

Leader election picks one node to do work that must not be done twice at once: own a shard, run a scheduler, accept writes for a log.
In practice you rarely run an election algorithm yourself. You take a lease from a strongly consistent store (etcd, ZooKeeper, Consul, a database row) and renew it.
No election can guarantee that only one node believes it is leader. Pauses and partitions make overlap possible, so the resource must reject the old leader using a fencing token (a term or epoch number).
Tuning is a trade between failover speed and false failovers: shorter leases detect death faster but fire on GC pauses and network blips.

How it works

Why you need a leader

A single leader turns a coordination problem into a local decision. The cost is a throughput ceiling and a failover gap when it dies.

Classic algorithms

The Bully algorithm has each node with a known ID; a node that suspects the leader is dead messages all higher-ID nodes, and if none reply it declares itself leader. The ring algorithm passes an election message around a logical ring collecting IDs, then announces the highest. Both assume reliable failure detection, so a partition gives each side its own leader. They are teaching tools, not production designs. Google's SRE book tells the production version of this story: a pair of file servers that picked a master by heartbeat timeout and STONITH ended up either both active or both down when the network degraded, because they tried to solve leader election with simple timeouts (Google SRE book).

Consensus-based election

Raft and ZooKeeper's Zab elect leaders by majority vote within a numbered term. Two leaders in the same term are impossible because each voter votes once per term and majorities overlap. A deposed leader from an older term can still be alive and acting, but any write it sends will be rejected by a majority that has moved to the newer term.

Lease-based election on a coordination service

Most applications borrow consensus. Each candidate tries to create a key such as /services/billing/leader with a TTL of, say, 10 s, using a compare-and-set that succeeds only if the key is absent. The winner renews every 3 s. Others watch the key and try again when it disappears. ZooKeeper does the same with ephemeral sequential znodes: the lowest sequence number leads, and each node watches only its predecessor to avoid a thundering herd. Kubernetes controllers use a Lease object with leaseDurationSeconds and renewTime (Kubernetes docs). Kubernetes also shows that "who leads" can be a policy, not a race: coordinated leader election (beta since v1.33, off by default) has candidates register LeaseCandidate objects, and the only built-in strategy picks the candidate with the oldest emulation version, so the leader during a rolling upgrade is always the one every peer is compatible with (Kubernetes docs).

Fencing

Suppose the leader stalls in a 15 s GC pause, which is not far-fetched: stop-the-world pauses have been known to last minutes (Kleppmann). Its 10 s lease expires, another node takes over, then the old leader wakes up and finishes a write it started. The lease alone cannot stop this. Fix it by issuing a monotonically increasing token with each lease (etcd's revision, ZooKeeper's zxid, or a counter you bump). Every write carries the token, and the storage layer rejects any token lower than the highest it has seen.

Self-demotion

A leader should stop acting before its lease could expire at the store. If the lease is 10 s, step down at about 7 s without a successful renewal. The margin absorbs clock rate drift and message delay. Amazon's guidance adds two habits: check the remaining lease time right before any side effect outside the leader, and do not renew from a background thread that keeps the lease alive while the work thread is stuck (AWS Builders' Library).

Down the stack

Layer 1The contract

There are two different promises, and mixing them up causes most incidents. Consensus election promises at most one leader per term; it never promises that only one process believes it is leader, because a deposed leader may not have heard yet. A lease promises that the store grants the key to one holder at a time, and that the holder can safely act for ttl - margin measured on its own monotonic clock, provided clock rates drift within a bound and the process is not paused. Only fencing gives a promise that holds with no timing assumption: the protected resource never accepts a write from an older term. Liveness: after the leader dies, a new one is in place within about one lease duration plus one retry period.

Layer 2The mechanism

ZooKeeper's own election (Zab's fast leader election) has each server vote for the peer with the highest (epoch, zxid, serverId), so the winner always has the most complete history. A zxid is 64 bits: the high 32 bits are the leader epoch and the low 32 a counter, so every transaction id also encodes which leadership produced it, which is why it works as a fencing token (ZooKeeper internals). Applications usually elect with the recipe above: ephemeral sequential znodes under a parent, lowest sequence wins, each candidate watching only its predecessor. The ephemeral node is bound to a session that the client keeps alive with pings; the server clamps the requested session timeout to between 2 and 20 times tickTime, which the sample config sets to 2 s (ZooKeeper admin guide).

etcd's concurrency.Election follows the same pattern with leases: each candidate writes a key under a prefix attached to its lease, and the key with the lowest create revision leads. Keepalives travel over a long-lived gRPC stream. Kubernetes' client-go elector stores holder identity and renewTime in a Lease object and updates it with an optimistic-concurrency write on resourceVersion. Controller-manager's defaults are a 15 s lease, a 10 s renew deadline and a 2 s retry period, so the leader gives up after 10 s without a renewal, leaving 5 s of margin (kube-controller-manager reference). Google's Chubby handed out sequencers, an early form of fencing token, and told a newly elected primary to confirm with CheckSequencer() that it was still primary (Chubby paper).

Layer 3The machine

Leases must be timed with CLOCK_MONOTONIC, never wall-clock time; Amazon's leases rely only on locally measured elapsed time for this reason (AWS Builders' Library): NTP can step the wall clock by seconds, while the monotonic clock only drifts with the crystal, typically tens of parts per million, or a few hundred microseconds over a 10 s lease. Drift is not the real threat. Pauses are: a stop-the-world GC on a large heap, a VM live migration, a swap storm or a SIGSTOP can freeze a process for seconds, and when it resumes the next instruction may be a write. No pre-write check closes that window.

The storage side enforces fencing cheaply: a conditional update such as UPDATE jobs SET ... WHERE fence <= $token is one indexed comparison inside a write that already happens. Each renewal is a small RPC but also a consensus write in etcd or ZooKeeper, so it costs a WAL fsync on a quorum. Thousands of short leases renewed every second can saturate the coordination cluster's disks.

Why it's built this way

Machines cannot tell a dead peer from a slow one, so any election is driven by timeouts and will sometimes produce two processes that think they lead.
Electing by majority prevents two leaders in one term, but it cannot stop an old leader acting late, so a term number must travel with every write.
Running consensus in every application is costly and error-prone, so applications borrow it from a small coordination cluster and hold leases there.
Leases need time, and only local monotonic durations are trustworthy, so expiry is measured locally with a safety margin and backed by fencing at the resource.

Diagram

sequenceDiagram
    participant A as Node A
    participant S as Lock store
    participant B as Node B
    participant D as Storage
    A->>S: acquire lease, TTL 10s
    S-->>A: granted, token 41
    A->>D: write with token 41
    Note over A: GC pause 15s
    B->>S: acquire lease
    S-->>B: granted, token 42
    B->>D: write with token 42
    A->>D: late write with token 41
    D-->>A: rejected, stale token

Worked example

A billing reconciliation job runs on 3 replicas; exactly one should run at a time. You use an etcd lease with TTL 10 s and renew every 3 s.

Leader crash: the lease expires within 10 s, followers see the key deletion through a watch in a few ms, and the next leader starts in roughly 10.1 s. Worst-case gap is the TTL.
Network blip of 4 s between leader and etcd: one or two renewals fail, but the lease survives because 4 s is less than 10 s. No failover.
Lowering TTL to 2 s would cut failover to about 2 s, but a 2.5 s stop-the-world GC on a 32 GB heap would now cause a spurious failover, plus the work of resuming the job on another node.

If the job writes to a Postgres table, add a leader_epoch column check: UPDATE ... WHERE id = $1 AND $token >= (SELECT max_epoch FROM fence). Now a stale leader's late write affects zero rows.

At scale, prefer one lease per shard rather than one global leader. With 1,000 shards and 50 workers, each worker holds about 20 leases, and a single failure moves 20 shards, not everything.

In the wild

GitHub, 2012: Pacemaker failover loop and split brain. Migration load made the MySQL primary fail Percona Replication Manager health checks, so Pacemaker moved the active role to a node with a cold InnoDB buffer pool, which failed the same checks. The next day a Pacemaker segfault partitioned the cluster and two master elections were attempted without coordination. The result was 1 h 46 min of downtime plus an hour of degradation, and 16 private repositories were visible to outsiders for seven minutes. Afterwards, failover of the active role required an operator to start it (GitHub).
Stripe, 2019: a shard that could not elect a primary. A database minor-version upgrade three months earlier carried a failover fault that only showed when several nodes stalled. On 10 July one shard could not elect a new primary and the API degraded for 27 minutes; rolling back the election protocol then clashed with a recent config change and starved CPU, for another 93 minutes. Stripe added alerts for nodes that stop reporting replication lag, circuit breakers per cluster and limits on client retries (Stripe).

Trade-offs

Option	Strengths	Weaknesses	Choose when
Lease in etcd/ZooKeeper/Consul	Battle-tested, watches, fencing tokens	Another tier-0 dependency	Most services needing a single active worker
Database row lease	No new infrastructure	Polling, DB failover affects leadership	You already have a strongly consistent DB
Embedded Raft	No external dependency, leader is also the data owner	You operate consensus yourself	You are building a replicated data store
Bully / ring	Simple, no external service	Unsafe under partitions	Teaching, or fixed clusters with fencing elsewhere
Cloud primitive (e.g. Kubernetes Lease)	Native to the platform	Bound to the API server's availability	Controllers and operators on Kubernetes

Pitfalls in production

Split brain after a pause: the old leader keeps writing. Pass a fencing token on every side-effecting call and enforce it at the resource.
Leases tuned below GC or I/O pause times cause flapping. Measure pause p99.9 and set TTL to several times it.
Using wall-clock timestamps from different machines to judge lease expiry. Use the store's TTL and local monotonic timers only.
Leadership that is not released on graceful shutdown adds a full TTL to every deploy. Revoke the lease in the shutdown hook. Kubernetes itself only added this for kube-controller-manager as an alpha gate in v1.36, ControllerManagerReleaseLeaderElectionLockOnExit (Kubernetes docs).
Automatic failover that fires on load, not death. A busy primary fails a health check, the standby takes over cold, and fails the same check. Require a human or a damping rule before a second failover.

Leadership lens

First question: does this work truly need a single leader, or can it be partitioned so each shard has its own owner? Sharded ownership scales and limits blast radius.
Ask what happens when two leaders act for 30 s. If the answer is "corrupt data", fencing is mandatory, not optional.
Expose metrics for leader changes and lease renewal latency, and alert on churn, which is the early warning of overload.
Document the failover time budget (TTL plus startup) and agree it with the service's SLO owners.

Check yourself

Why does a lease not by itself guarantee a single active leader?
Show answer
A paused or partitioned leader cannot know its lease has expired and may keep acting. Only the resource checking a fencing token can reject its late writes.
Why should a leader step down before its lease TTL elapses?
Show answer
Its local timer and the store's timer can drift and messages take time, so a safety margin keeps it from acting after another node has been granted the lease.
Why does ZooKeeper's recipe have each node watch only its predecessor?
Show answer
If all nodes watched the leader's znode, every one would wake on its deletion and race, creating a herd. Watching the predecessor wakes only the next in line.
What failure does the Bully algorithm not handle?
Show answer
A network partition: each side sees the other as dead and elects its own leader.

Sources and further study

Primary

Ongaro and Ousterhout, In Search of an Understandable Consensus Algorithm (2014): section 5.2 is the term-numbered, randomized-timeout election most systems now copy.
Gray and Cheriton, Leases (SOSP 1989): where time-bounded grants and their clock-drift assumption come from.

Systems docs

Kubernetes: Coordinated Leader Election: LeaseCandidate objects and version-aware leader choice.
kube-controller-manager flags: the --leader-elect-* lease, renew and retry settings.
ZooKeeper internals: zxid layout and what a new leader must have seen.

Books

Database Internals, Alex Petrov, chapter 10 "Leader Election": Bully, next-in-line failover, invitation and ring algorithms.
Designing Distributed Systems (2nd edition), Brendan Burns, chapter 10 "Ownership Election": whether you need a leader at all, the basics of election, and handling concurrent data manipulation.
Site Reliability Engineering, Beyer et al. (eds.), chapter 23 "Managing Critical State": why leader election by timeout alone causes split brain (read free).

Other curricula

Azure Architecture Center: Leader Election pattern: a blob-lease mutex with renewal; names Bully and Raft but does not discuss fencing.
roadmap.sh System Design: lists Leader Election among its cloud design patterns as a single node to look up.

Watch

Distributed Systems 6.1: Consensus (Martin Kleppmann): why electing a leader is itself a consensus problem.
UMass CS677 Lecture 14: Leader Election, Distributed Locks (UMass OS): Bully and ring algorithms alongside lock services.
Who's in Charge? Failure and leader election in databases (Ben Dicken): failure detection and election as covered in Database Internals.

Free resources

Leader election in distributed systems (AWS Builders' Library): how Amazon uses leases and why fencing matters.
The Chubby lock service: Google's paper on coarse-grained locks used mainly for election.
ZooKeeper recipes: the ephemeral sequential znode election recipe.
Kubernetes Leases: how controllers elect leaders on the API server.
Generation Clock (martinfowler.com): the epoch number pattern behind fencing.

Open in the interactive app: practice, quiz, mock interview →