Engineering Atlas › System Design Fundamentals

Copying data across nodes with replication

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Single-leader replication

All writes go to the leader. It records each change in a log (PostgreSQL's WAL, MySQL's binlog) and followers stream and replay that log in order. Followers serve reads, so read capacity scales with replica count while write capacity does not. Log shipping can be physical (byte-level page changes, needs identical versions) or logical (row-level changes, works across versions and feeds change data capture).

Synchronous, semi-synchronous, asynchronous

With asynchronous replication the leader commits locally and acknowledges the client; followers catch up later. Writes are fast, but if the leader dies, anything not yet shipped is lost. With synchronous replication the leader waits for a follower to confirm durability. No acknowledged write is lost, but one slow follower stalls every write, so systems usually wait for any one of several followers (semi-synchronous, or "any 1 of 3"). Read the fine print on what happens when no follower answers. MySQL's semi-synchronous source waits rpl_semi_sync_source_timeout (10 seconds by default) and then quietly reverts to asynchronous replication until a replica catches up (MySQL docs), so the "no loss" promise lapses exactly when things go wrong. PostgreSQL instead blocks commits until a synchronous standby returns. Consensus-based systems (Raft in etcd, CockroachDB, Spanner's Paxos groups) wait for a majority, which tolerates a minority of failures without losing data or availability. In Raft, an entry is committed once the leader has stored it on a majority, and a candidate cannot win an election unless its log is at least as up to date as a majority's, which is what makes promotion safe (Ongaro and Ousterhout).

Lag and its symptoms

Replica lag is the gap between the leader's log position and what a follower has applied. It is usually milliseconds and spikes to minutes under heavy writes, long-running replica queries, or network trouble. Visible symptoms:

These are the session guarantees Terry and colleagues named in 1994: read your writes, monotonic reads, writes follow reads, and monotonic writes (Terry et al.). Fixes include routing a user's reads to the leader for a few seconds after a write, tracking the write's log position and reading only from replicas that have applied it, or pinning a sequence of related reads to one replica so time never runs backwards.

Failover

When the leader fails, a follower is promoted. The hard parts are detection (a timeout too short causes needless failovers, too long extends downtime), choosing the most up-to-date follower, and fencing the old leader so it cannot keep accepting writes after a network partition heals. Two nodes both believing they are leader is split brain, and it corrupts data.

Multi-leader and leaderless

Multi-leader setups accept writes in several regions and exchange changes asynchronously, so concurrent writes to the same row conflict and need a resolution rule (last-writer-wins, merge functions, CRDTs). Leaderless systems (Dynamo-style, Cassandra) write to W of N replicas and read from R; with R + W > N a read overlaps the latest write, though edge cases around sloppy quorums and concurrent writes remain. Amazon's Dynamo paper describes a common setting of N=3, R=2, W=2, with hinted handoff and Merkle-tree anti-entropy to repair replicas that missed writes (DeCandia et al.).

Down the stack

Layer 1The contract

PostgreSQL makes the commit contract explicit with synchronous_commit. With local, a commit is acknowledged once the primary's WAL is flushed; standbys may lag, and failover can lose anything not yet shipped. With remote_write, a standby has received the WAL and written it to its OS, so it survives a standby process crash but not a standby power loss. With on (when synchronous standbys are configured), a standby has flushed it to disk. With remote_apply, a standby has replayed it, so a read on that standby immediately after commit sees the write. Each step up adds latency and narrows what a failover can lose. These levels only bite when synchronous_standby_names is set; with it empty, every non-off value behaves like on locally (Postgres docs).

Layer 2The mechanism

Changes are WAL records stored in segment files of 16 MB by default, set at initdb time (Postgres docs). On the primary, a walsender process per standby streams records; on the standby, walreceiver writes them and the startup process replays them. The standby reports three positions back: written, flushed, and replayed LSN, which the primary exposes in pg_stat_replication as write_lsn, flush_lsn, replay_lsn and the matching lag times. synchronous_standby_names = 'ANY 1 (s1, s2)' makes commits wait for whichever of two standbys confirms first, while FIRST 1 (s1, s2) waits for the highest-priority one that is connected (Postgres docs). Replication slots stop the primary from deleting WAL a standby still needs, which protects the standby but can fill the primary's disk if a standby disappears. Since Postgres 13, max_slot_wal_keep_size caps that retention; its default of -1 still means unlimited, so set it. Postgres 17 added sync_replication_slots, which copies logical failover slots to a physical standby so change-data-capture consumers can continue after a promotion. MySQL's equivalent is the binlog with GTIDs. Its semi-synchronous plugin (renamed rpl_semi_sync_source and rpl_semi_sync_replica; the old master and slave names are deprecated) waits for one replica to acknowledge receipt. By default it waits after the binlog is synced but before the storage engine commits (AFTER_SYNC), so no other session can see a write the replica has not received.

Layer 3The machine

A synchronous commit costs a local WAL fsync (roughly 50 µs to 1 ms on SSDs), a network round trip (about 0.5 ms in one availability zone, 1 to 2 ms across zones, 60 to 150 ms across regions), and the standby's own fsync. WAL volume spikes after each checkpoint because the first change to any 8 KB page writes the full page image, so lag often follows the checkpoint cycle. Replay in PostgreSQL runs in a single startup process (Postgres docs), so a primary writing with 32 cores can produce WAL faster than one core on the standby replays it. Since Postgres 15, recovery_prefetch (default try) reads ahead in the WAL and issues read-ahead advice for pages not yet in memory, which hides some disk latency but not the single-core limit (Postgres docs). Across regions, TCP throughput is limited by the bandwidth-delay product: filling 1 Gbps at 70 ms needs about 8.75 MB in flight, and smaller socket buffers silently cap the stream.

Why it's built this way

  1. Disks and machines fail, so data must exist on more than one node.
  2. Concurrent writers on several nodes create conflicts, so most systems use one leader and ship its log.
  3. Shipping the log the database already writes for crash recovery is cheaper than shipping data, so replication reuses the WAL.
  4. Waiting for a remote acknowledgement adds at least one round trip per commit, so asynchronous is the default and synchronous is opt-in.
  5. Two leaders would diverge, so promotion needs fencing or a majority vote.

Diagram

sequenceDiagram
    participant C as Client
    participant L as Leader
    participant F1 as Follower 1
    participant F2 as Follower 2
    C->>L: write x = 5
    L->>L: append to log (LSN 101)
    L->>F1: stream LSN 101
    L->>F2: stream LSN 101
    F1-->>L: applied 101
    L-->>C: commit ok (semi-sync, 1 ack)
    F2-->>L: applied 101 (later)
    C->>F2: read x with min LSN 101

Worked example

An e-commerce database takes 4,000 writes per second at about 1 KB of WAL each, so 4 MB per second of log. It has one leader and three followers in the same region, plus one cross-region follower 70 ms away.

With async replication, normal lag is under 50 ms locally. During a nightly bulk import of 30 GB, WAL volume rises to 60 MB per second, and the cross-region follower, limited to 40 MB per second of bandwidth, falls 20 minutes behind. Had the region failed then, failing over to that replica would lose 20 minutes of orders.

The team makes three changes: semi-synchronous commit with "any 1 of the 2 local followers", which adds about 1 ms per commit and guarantees zero acknowledged loss on single-node failure; throttling the import to 20 MB per second; and an alert when any replica's lag exceeds 30 seconds. Reads that must reflect the user's own writes carry the commit LSN in the session and are routed to a replica at or past it.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Async single-leaderFast writes, simpleData loss on failover, stale readsSome loss tolerable, read scaling needed
Semi-sync (any 1 of N)No acknowledged loss on one failureExtra round trip per commitLosing committed writes is unacceptable
Consensus (majority)Automatic, safe failoverHigher write latency, needs 3+ nodesStrong consistency with high availability
Multi-leaderLocal writes in each regionConflicts, complex resolutionMulti-region writes, offline clients
Leaderless quorumNo failover, tunable R and WWeaker guarantees, repair neededHigh write availability, simple data model

Pitfalls in production

Leadership lens

Check yourself

  1. Why does synchronous replication to a single named follower reduce availability?

    Show answer

    Every commit waits for that follower, so if it is slow or down, writes stall even though the leader is healthy.

  2. A user edits a post and sees the old version on refresh. What is happening and how do you fix it?

    Show answer

    The read hit a lagging replica; route that user's reads to the leader briefly, or to a replica that has applied the write's log position.

  3. Why does R + W > N not make a leaderless store linearizable on its own?

    Show answer

    Concurrent writes, sloppy quorums, and partial write failures can still let reads see different values, so extra mechanisms are needed for strict ordering.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →