Engineering Atlas › System Design Fundamentals

Finding and removing single points of failure

13 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Finding the SPOFs

Draw the request path from the user to the data, then ask of every box and arrow: if this disappears for an hour, does the request still succeed? Repeat for the paths that keep it running: deploys, secrets, monitoring, paging. In graph terms, a SPOF is a node that every path from the user to the data passes through.

Removing them

There are three standard moves. Replicate the component, so there are N copies and the system needs only some of them (N+1 for stateless services, a quorum of 3 or 5 for stateful ones). Partition it, so a failure takes out a slice of users rather than all of them (cells, shards). Decouple from it, so the system can keep working without it for a while (caches, queues, static stability where the data plane keeps running on the last known config when the control plane is down).

Replicas only help if the survivors can carry the load. Two nodes at 60% each become one node at 120% when either fails (Azure Well-Architected). AWS sizes services across three zones with 50% extra capacity, so each zone runs at about two-thirds of its tested limit and losing one needs no new launches (AWS Builders' Library). N+1 means the remaining N can carry the full peak, not N+1 copies that all run hot.

Independence is the whole game

Redundancy assumes the copies fail independently. They often don't. Two replicas in one rack share a top-of-rack switch and a power feed. Two regions running the same binary share the same bug and fail together on the same bad deploy. Two DNS providers configured by one script share the script. Correlated failure is why a team can have "three of everything" and still go down. Spread copies across failure domains (host, rack, availability zone, region, vendor, release train) chosen to match the failures you actually expect.

The switch is a SPOF too

Redundancy needs something that notices failure and moves traffic: a health check, a floating IP, a leader election. That mechanism can fail, flap, or make the wrong call, and a failover that has never been exercised usually fails when first needed. Test it on purpose, regularly.

The problem shape

This is the partial failure shape: one part breaks and you must decide whether the whole stops. In payments, a single acquiring bank is a SPOF for checkout, so merchants add a second acquirer. In AI agent systems, one model provider is a SPOF for every agent; a fallback model plays the role of the standby replica.

Down the stack

Layer 1The contract

For components in series (all required), system availability is the product: A = A1 x A2 x ... x An. For N independent replicas where any one suffices, A = 1 - (1 - a)^N. A SPOF-free design promises that no single failure in a stated set of failure domains (one host, one rack, one zone) drops availability below the target, and that failover completes within a stated time, such as 30 seconds, without losing acknowledged writes (RPO 0) or with a bounded loss.

Layer 2The mechanism

Stateless tiers use health checks: HAProxy, by default, probes every 2 seconds, marks a server down after 3 consecutive failures and back up after 2 successes (HAProxy docs). A pair of load balancers shares a virtual IP with VRRP (keepalived). The current spec, VRRPv3 in RFC 9568 (2024), replaced RFC 5798 and renamed the "master" to the Active Router: it advertises every second by default, and a backup takes over after three missed adverts plus a priority-based skew. Stateful tiers need consensus to avoid two primaries: Patroni holds a leader key in etcd with a default TTL of 30 seconds (Patroni docs). The replica that wins the key runs an optional pre_promote fencing script and then pg_ctl promote (Patroni source), and a watchdog resets the old leader's machine if Patroni hangs, timed to fire before the TTL lets anyone else take over (Patroni watchdog). DNS failover uses health-checked records (Route 53 failover routing) whose effect is bounded by the record's TTL. Route 53 checks every 10 or 30 seconds from many locations and calls an endpoint healthy while more than 18% of checkers agree, so one isolated checker cannot fail it over (Route 53 docs). Health checks run on Route 53's data plane, while editing records is a control-plane call, so AWS advises failover designs that never need a record change mid-incident (AWS blog). At the edge, anycast advertises one prefix from many sites, so BGP withdraws a dead site within seconds.

Layer 3The machine

Hardware SPOFs are concrete. A server with one PSU, one NIC and one uplink has three. Bonded NICs (LACP, still often called 802.3ad, although the standard moved to IEEE 802.1AX in 2008 (Wikipedia)) and dual PSUs on separate A and B power feeds remove them. A disk with an annual failure rate near 1% sounds safe until you have 10,000 of them: that is about two failures a week, which is why RAID and replicated storage exist. When a floating IP moves, the new holder sends a gratuitous ARP so switches update their tables; until they do, packets go to the dead host. Clients keep pooled TCP connections to the old address and only notice after a timeout or a TCP keepalive (Linux waits tcp_keepalive_time, 2 hours by default, then sends 9 probes 75 seconds apart (kernel docs)), so failover time is often set by client timeouts rather than by the failover system itself.

Why it's built this way

  1. Every physical part eventually fails, and at fleet scale "eventually" means daily, so failures must be routine events rather than outages.
  2. Availability multiplies along a chain, so the weakest required part sets the ceiling, and that part gets the redundancy budget first.
  3. Copies only help if their failures are independent, so they are spread across racks, zones and regions that do not share power, network or software rollout.
  4. Two copies of state that both accept writes will diverge, so stateful failover is gated by consensus and fencing rather than by a simple health check.
  5. Detection takes time and moving clients takes more, so systems that need fast recovery keep the standby warm and the clients' timeouts short.

Diagram

flowchart LR
    U["Users"] --> DNS["DNS (2 providers)"]
    DNS --> LB1["LB A (VIP owner)"]
    DNS -.-> LB2["LB B (standby)"]
    LB1 --> APP1["App 1 (zone a)"]
    LB1 --> APP2["App 2 (zone b)"]
    LB1 --> APP3["App 3 (zone c)"]
    APP1 --> DB1["Primary DB"]
    APP2 --> DB1
    APP3 --> DB1
    DB1 -->|"sync replication"| DB2["Standby DB"]
    ETCD["etcd (3 nodes)"] -->|"leader key"| DB1
    ETCD -.->|"promote on expiry"| DB2

Worked example

A checkout service needs DNS (99.99%), a load balancer (99.99%), an app tier, and a Postgres database. The app runs on three instances, each 99.5% available. The database is a single instance at 99.9%.

The app tier is three parallel copies: 1 - 0.005^3 = 0.999999875. The chain is 0.9999 x 0.9999 x 0.999999875 x 0.999 ≈ 0.99880, or 99.88%. That is about 10.5 hours of downtime a year, and almost all of it is the database. Adding a fourth app instance changes nothing visible; the app tier was never the problem.

Now add a synchronous standby with Patroni. Two independent 99.9% databases give a theoretical 1 - 0.001^2 = 0.999999, but real failovers take about 30 seconds and a small fraction fail and need a human. Measured over a year, the team sees the database tier at about 99.99%. The chain becomes 0.9999 x 0.9999 x 0.9999 ≈ 0.9997, roughly 2.6 hours a year. Now DNS and the load balancer are the biggest contributors, so the next step is a second DNS provider and a VRRP pair of load balancers.

A later game day finds one more: the app reads its database password from a secrets service at startup, so during a secrets outage autoscaling cannot start new instances. Caching the secret locally with a long expiry fixes it.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Active-active (N+1)No failover step, capacity is usedNeeds stateless design or multi-writer conflict handlingStateless services, read paths
Active-passive (hot standby)Simple writes, fast failoverStandby capacity idle, failover can failSingle-writer databases, queues
Cold standby or backup restoreCheapRecovery takes minutes to hoursInternal tools, non-critical data
Cells or shardsLimits blast radius of any failure, including bad deploysRouting layer, cross-cell operations, costLarge multi-tenant platforms
Multi-region or multi-vendorSurvives regional and provider outagesCost, latency, data consistency, operational complexityRevenue-critical paths with strict targets

Pitfalls in production

Leadership lens

Check yourself

  1. Three replicas each 99% available sit behind a single 99.95% load balancer. What is the approximate availability, and what should you improve first?

    Show answer

    The replicas give 1 - 0.01^3 = 0.999999, so the total is about 99.95%, set entirely by the load balancer. Make the load balancer redundant first.

  2. Why might two replicas in two regions still count as a SPOF?

    Show answer

    If they run the same code and receive the same deploy at the same time, a bad release takes out both. Independence requires staggered rollouts, not just separate locations.

  3. A database fails over in 10 seconds, but users see errors for 5 minutes. What is the likely cause?

    Show answer

    Clients are still holding connections or cached DNS pointing at the old primary. Client timeouts, keepalives and TTLs bound the real recovery time.

  4. Why is a simple health check not enough to promote a database standby?

    Show answer

    A partition can make a healthy primary look dead, and promoting the standby then gives two writers. Promotion needs consensus and fencing so only one primary can accept writes.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →