Designing for failing components

13 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

In a large system something is always broken: a disk, a host, a zone, a dependency, a deploy. The design question is not "will it fail" but "what does the user see when it does".
Survival rests on a few mechanisms used together: redundancy (more than one of everything, in independent failure domains), detection (health checks, timeouts), isolation (bulkheads, circuit breakers, cells) and graceful degradation (fallbacks and reduced functionality).
Most outages are not the first failure but the reaction to it: retry storms, cascading timeouts, failovers that overload the survivors. Good designs bound the reaction.
Static stability is the goal: when a component fails, the rest keeps working with what it already has, without needing to make new decisions or call a control plane.
In interviews, the interviewer will pick a box in your diagram and say "this dies, now what?" You should have an answer for every box.

How it works

Redundancy in independent failure domains

Run at least N+1 (often N+2) instances of every stateless service across availability zones, sized so losing one zone leaves enough capacity. Stateful components need replication: a database primary with synchronous or semi-synchronous replicas in other zones, a consensus group of 3 or 5 nodes for coordination data, object storage with erasure coding across zones. Redundancy only helps if failures are independent, so watch for shared dependencies: one DNS provider, one config service, one certificate, one deploy pipeline. Pre-provisioning the spare capacity, rather than launching it after the failure, is what AWS calls static stability: the surviving zones need no control-plane call to cope (AWS Builders' Library).

Detection

A failure you do not detect becomes a hang. Every network call needs a timeout set from the dependency's measured latency (for example p99.9 plus margin), not a library default. Load balancers run active health checks and also passive outlier detection, ejecting hosts that return consecutive errors. Health checks should test what matters (can this instance serve a request) without depending on shared dependencies, or a database blip will mark the whole fleet unhealthy at once. Kubernetes warns that a careless liveness probe can cause cascading restarts under load, and puts back-end dependency checks in the readiness probe, not the liveness one (Kubernetes docs).

Isolation and containment

Circuit breakers stop calling a dependency that is failing, fail fast, and periodically probe to see if it has recovered (Fowler).
Bulkheads give each dependency its own concurrency budget so one slow call cannot exhaust all threads.
Retry budgets cap retries to a fraction of normal traffic, so retries cannot multiply load during an outage. Google's clients allow at most three attempts per request and retry only while retries stay under 10% of a client's requests (Google SRE book).
Cells split the fleet into independent copies serving subsets of customers, so a bad deploy or poison request hurts one cell.

Degradation

Decide per dependency what happens without it. Recommendations fail: show popular items from a static list. Reviews fail: hide the reviews panel. Pricing fails: this is critical, so fail the request clearly. Precompute fallbacks so that the fallback path does not itself depend on something that might be down. Amazon goes further and avoids fallback logic where it can, because a rarely run path is the least tested code in the system; it prefers making the primary path more reliable or pushing data out ahead of time (AWS Builders' Library).

Failover for stateful parts

Leader-based stores elect a new leader when the old one stops renewing its lease. Failover must fence the old leader (with an epoch or fencing token) so a paused, not dead, primary cannot keep accepting writes. Asynchronous replication means failover can lose the last few seconds of writes; say so and decide whether that is acceptable.

How it shows up in interviews

Every design question has a reliability phase, and some are explicitly about it: design a highly available payment service, a distributed job scheduler, a key-value store, a notification system with delivery guarantees. A strong answer lists failure domains, gives each component a redundancy and failover story, sets timeouts and retries with budgets, adds circuit breakers and fallbacks, and discusses detection time and data loss on failover. Interviewers probe: what if the database primary dies mid-write, what if a dependency is slow rather than down, what if a whole zone goes, and how do you know a node is dead rather than slow.

Down the stack

Layer 1The contract

State availability and durability targets per component and per user journey: for example 99.95% success for checkout over 30 days, with recovery time under 60 seconds and zero acknowledged-write loss for orders. A circuit breaker promises that while open, calls fail within microseconds instead of waiting for a timeout, and that it will let a limited number of probe calls through after a cool-down. A health-checked load balancer promises to stop routing to a failed host within its detection window (for example 3 failed checks at 5-second intervals, so about 15 seconds). Synchronous replication promises an acknowledged write survives a single node loss; asynchronous replication does not.

Layer 2The mechanism

Envoy's outlier detection tracks per-host counters such as consecutive_5xx (5 by default) and ejects a host for base_ejection_time (30 seconds by default) multiplied by the number of prior ejections, capped by max_ejection_percent, 10% of the cluster by default, so it never ejects the whole cluster (Envoy docs). Envoy also supports retry budgets (retry_budget with budget_percent, 20% of active requests by default, Envoy docs), and gRPC clients have a retry throttling policy: each failure subtracts 1 from a token count that starts at maxTokens, each success adds tokenRatio, and retries stop while the count is at or below half of maxTokens (gRFC A6). A circuit breaker is a three-state machine: closed (count failures), open (reject until a timeout elapses), half-open (allow a probe; success closes, failure reopens). Leader election in etcd or ZooKeeper uses leases or ephemeral nodes; a client that loses its session must stop acting as leader, and the store hands the new leader a higher revision number usable as a fencing token. Kubernetes separates liveness probes (restart a wedged process) from readiness probes (stop sending traffic), which matters: failing liveness on a dependency outage restarts every pod at once.

Layer 3The machine

Without application timeouts, the kernel's defaults are slow. A connection attempt to a dead host retransmits SYNs with exponential backoff; with Linux's default tcp_syn_retries of 6, connect takes about 131 seconds to fail on current kernels. On an established connection, unacknowledged data is retransmitted up to tcp_retries2 (15 by default), a timeout of at least 924.6 seconds, about 15 minutes, before the socket errors. TCP keepalive defaults to 2 hours of idleness before the first probe (Linux kernel docs). A process paused by a long garbage collection, a VM live migration or a stalled disk can look dead for seconds and then resume, still believing it holds a lease, which is why fencing is needed. Disks fail in ways that are not crashes: an fsync that returns an error may have dropped dirty pages, and Postgres changed its behaviour to panic on fsync failure instead of retrying, after the 2018 "fsyncgate" discussion showed that a retry could falsely report success. The change shipped in February 2019 in 11.2 and the older supported branches, with data_sync_retry to restore the old behaviour (Postgres 11.2 release notes).

Why it's built this way

Hardware and networks fail independently and often at scale, so every component needs a peer in a separate failure domain. A slow component is indistinguishable from a dead one from the outside, so detection relies on timeouts tuned to real latency. Reactions to failure add load at the worst time, so retries and failovers are budgeted and damped. Paused processes can return, so leadership is enforced with fencing rather than assumed from silence.

Diagram

stateDiagram-v2
    [*] --> Closed
    Closed --> Closed: success or failure below threshold
    Closed --> Open: consecutive failures reach threshold
    Open --> Open: calls rejected fast
    Open --> HalfOpen: reset timeout elapsed
    HalfOpen --> Closed: probe succeeds
    HalfOpen --> Open: probe fails

Worked example

A checkout service runs 30 instances across three zones and calls pricing, inventory, payments and a recommendations widget. Each instance handles 100 requests per second.

Zone loss: losing one zone removes 10 instances. The remaining 20 must carry 3,000 requests per second, 150 each. The team sizes instances to run at 50% CPU normally, so 150 per instance is about 75%, survivable. Without that headroom, the survivors overload, latency climbs, health checks fail and the outage spreads to all zones.

Slow dependency: recommendations latency jumps from 30 ms to 4 seconds. The checkout call to it has a 150 ms timeout and a bulkhead of 20 concurrent calls per instance. Within a second, the breaker sees 20 consecutive timeouts and opens; for the next 30 seconds calls fail in microseconds and the page renders with a static "popular items" list. Every 30 seconds one probe tests recovery.

Retries: payments has a retry budget of 10%. During a 2-minute payments brownout where 40% of calls fail, naive single retries would add 40% more load to a struggling service; the budget caps the extra at 10%, and further failures surface as "please try again" with idempotency keys so user retries cannot double-charge.

Database failover: the orders primary uses semi-synchronous replication to one replica in another zone. When the primary's host dies, the orchestrator promotes the replica in about 20 to 30 seconds and bumps the epoch. Writes during that window fail and are retried by clients with idempotency keys.

In the wild

Google Cloud, 2025: no fail-open, no backoff. On 12 June a policy record with blank fields replicated worldwide within seconds and hit new Service Control code with no error handling and no feature flag, so the quota-checking binary crash-looped in every region. The kill switch rolled out within 40 minutes, but in us-central1 restarting tasks overloaded their Spanner table because they lacked randomized exponential backoff, adding up to about 2 hours 40 minutes. Google committed to making Service Control fail open and to auditing backoff (Google Cloud).
AWS, 2021: retries that would not back off. In us-east-1 an automated scaling action set off a surge of connections from internal clients that overwhelmed the devices linking AWS's internal and main networks. A latent bug stopped clients from backing off, so retries sustained the congestion for about seven hours, and the monitoring that would have shown it ran on the same congested network. AWS fixed the back-off bug and rebuilt its health dashboard (AWS).
HubSpot, 2024: readiness ejections cascade. HubSpot's Envoy routing pods were packed unevenly onto Kubernetes nodes. At a morning traffic spike, several pods on an overloaded node failed readiness together and were pulled from service, pushing their load onto the rest until those failed too, two waves of errors, roughly 5:35 to 6:09 AM and 9:14 to 9:44 AM on 25 September. Fixes included even pod spread and load balancers that avoid removing too many pods at once under high load (HubSpot).

Trade-offs

Option	Strengths	Weaknesses	Choose when
Active-active replicas	No failover delay, uses all capacity	Needs headroom for loss, harder for stateful parts	Stateless services
Active-passive failover	Simple for stateful components	Detection plus promotion delay, idle standby	Single-writer databases
Synchronous replication	No acknowledged-write loss	Higher write latency, availability tied to replica	Money, orders
Asynchronous replication	Low latency	Loses recent writes on failover	Feeds, analytics, caches
Circuit breaker with fallback	Fast failure, protects dependency	Tuning, fallback may be stale	Optional or degradable features

Pitfalls in production

Default timeouts. Library defaults of 30 seconds or none at all let a dead dependency hold every thread. Set explicit timeouts per call from measured latency.
Retry amplification. Three attempts at each of four layers turn one failure into 81 calls at the bottom. Retry at one layer, with backoff, jitter and a budget.
Health checks that test dependencies. A shared database blip fails every instance's check and the load balancer removes the whole fleet. Keep liveness local; use readiness carefully.
Failover without fencing. A paused old primary wakes up and writes, causing split brain. Use epochs or fencing tokens checked by the storage layer.
Untested fallbacks. The fallback path is rarely exercised and breaks when needed. Exercise it with fault injection and game days.
No capacity for loss. Running at 80% across three zones means a zone loss overloads the rest. Plan N+1 at the zone level.

Leadership lens

For each box in the architecture, ask: what is its failure domain, how is failure detected, how long until recovery, and what is lost.
Review timeouts, retries and breaker settings as configuration owned by the calling team, with dashboards.
Run game days and fault injection regularly; untested failover is a hypothesis.
Track the cost of headroom explicitly. Reliability is bought with idle capacity, and finance should see why.

Check yourself

Why can retries make an outage worse?
Show answer
They add load to a dependency that is already failing, and retries stacked across layers multiply, which can keep it from recovering.
Why does a half-open state exist in a circuit breaker?
Show answer
It lets a small number of probe calls test whether the dependency has recovered without sending full traffic to it.
Why is a health check that queries the shared database risky?
Show answer
A brief database problem makes every instance fail its check simultaneously, so the load balancer removes the entire fleet.
What does a fencing token prevent?
Show answer
It stops a paused former leader from writing after a new leader has taken over, because storage rejects writes with an older token.

Sources and further study

Primary

gRFC A6: gRPC retry design (gRPC proposals): retry policy, hedging and the token-based throttle that switches retries off.
IP sysctl reference (Linux kernel docs): the SYN, retransmission and keepalive defaults behind "the kernel will time out eventually".

Systems docs

Outlier detection (Envoy API docs): every ejection knob with its default.
Liveness, readiness and startup probes (Kubernetes docs): what each probe does, and the warning that a careless liveness probe can cascade.
Fsync errors (PostgreSQL wiki): the history of fsyncgate and how each OS reports lost writes.

Books

Release It! (2nd ed.), Michael T. Nygard, Ch 4 "Stability Antipatterns" and Ch 5 "Stability Patterns": chain reactions, cascading failures, timeouts, circuit breakers and bulkheads.
Site Reliability Engineering (1st ed.), Beyer et al., Ch 22 "Addressing Cascading Failures": overload, retries, slow startup and how to test for cascades.
Understanding Distributed Systems (2nd ed.), Roberto Vitillo, chapters "Downstream resiliency" and "Upstream resiliency": timeouts, retries and breakers on the caller side, load shedding and bulkheads on the callee side.

Other curricula

Circuit Breaker pattern (Azure Architecture Center): frames the breaker alongside the Retry and Bulkhead patterns, with guidance on which exceptions should trip it.
System Design Primer: Availability patterns (donnemartin): frames survival as fail-over (active-passive, active-active) and replication, with availability math for components in sequence and in parallel.

Watch

Mastering Chaos: A Netflix Guide to Microservices (InfoQ): fallbacks, breakers, cascading failures and regional evacuation in practice.
Close Loops and Opening Minds (AWS re:Invent 2018): designing control loops and systems that stay stable during failures.

Free resources

Static stability using Availability Zones (AWS Builders' Library): pre-provisioned capacity so zone loss needs no reaction.
Addressing cascading failures (Google SRE book): how overload spreads and the mechanisms that stop it.
Timeouts, retries and backoff with jitter (AWS Builders' Library): setting timeouts and avoiding retry amplification.
Circuit Breaker (martinfowler.com): the canonical description of the pattern.
Avoiding fallback in distributed systems (AWS Builders' Library): why rarely used fallback paths can make things worse.

Open in the interactive app: practice, quiz, mock interview →