Engineering Atlas › System Design Fundamentals

Designing for partial failure

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Failure models

A crash-stop node halts and never returns. A crash-recovery node halts and later restarts, possibly having lost in-memory state, so anything promised must be on disk first. Omission failures drop messages. Timing failures respond, but too late to matter. Byzantine failures send wrong or conflicting data; outside blockchains and safety-critical systems, we usually defend against these only with checksums and validation.

The most damaging real failures are grey: a node that is slow, not dead. It passes health checks, keeps taking traffic, and drags down every caller's p99.

Detect: timeouts and health checks

Every remote call needs a timeout. Set it from the dependency's measured latency, not from a round default like 30 seconds. AWS picks the false-timeout rate it can live with, such as 0.1%, and sets the timeout at that percentile of downstream latency, here p99.9 (AWS Builders' Library). Deadlines should propagate: if the user request has 800 ms left, downstream calls get less than that, not their own full budget. Health checks should be shallow for liveness (is the process alive) and deeper for readiness (can it serve), and deep checks should never fail a whole fleet at once because a shared dependency blinked.

Recover: retries done safely

Retry only operations that are idempotent or carry an idempotency key. Use exponential backoff with full jitter, so delay is a random value between zero and min(cap, base * 2^attempt); jitter spreads out clients that failed together. Cap attempts at 2 or 3. In a stack of 4 layers that each retry 3 times, one user request can produce 3^4 = 81 calls to the bottom service; AWS's own example of 5 layers gives 243x. Retry at one layer only, and use a retry budget. Google allows at most 3 attempts per request and stops a client retrying once retries pass 10% of its requests, which cuts worst-case growth from almost 3x to about 1.1x (Google SRE book). AWS SDKs build in the same idea: standard mode allows 3 attempts in total with full-jitter backoff, and a 500-token retry quota that each retry draws from stops a client retrying during a sustained outage (those numbers are from AWS's 2026 retry update, opt-in until it becomes the default) (AWS SDK docs).

Contain: breakers, bulkheads, shedding

A circuit breaker tracks recent failures to a dependency. Past a threshold it opens and fails calls immediately, then lets a few probe requests through after a cool-down. This gives the dependency room to recover and frees your threads. Bulkheads give each dependency its own connection pool or thread pool, so a stuck dependency exhausts only its own slots. Load shedding rejects work early, ideally by priority, when queue depth or latency crosses a limit, because serving 80% of requests well beats serving 100% badly. Be wary of clever fallbacks that switch to a different system under stress: AWS avoids them because they run rarely, hide bugs for years, and can push the extra load onto whatever is still standing (AWS Builders' Library).

Survive: redundancy and static stability

Run N+1 or N+2 capacity across availability zones so one zone can vanish without overload. Pre-provision rather than relying on scaling up during the incident. Cache config and credentials so the data plane keeps working when the control plane is down.

Down the stack

Layer 1The contract

Every remote call ends in one of three states: success, a definite failure, or unknown. The protocol tells you which. In gRPC, DEADLINE_EXCEEDED means unknown: the docs warn it can come back even when the operation succeeded. UNAVAILABLE is the code for a transient condition worth retrying with backoff, but the docs still say retrying non-idempotent calls on it is not always safe (gRPC status codes). In HTTP, RFC 9110 defines GET, HEAD, PUT, DELETE, OPTIONS and TRACE as idempotent and POST as not; a 503 or 429 with Retry-After is the server telling you when to come back. Payment APIs such as Stripe accept an Idempotency-Key header so a retried POST returns the stored first result (Stripe docs).

A well-behaved client therefore promises its own caller three things you can test: total latency never exceeds the deadline it was given, attempts never exceed N, and retry traffic never exceeds a fixed fraction of normal traffic.

Layer 2The mechanism

Deadlines travel on the wire. gRPC sends the remaining budget in the grpc-timeout header (for example 250m for 250 ms) (gRPC over HTTP/2), and in Go the server side surfaces it as a context.Context whose cancellation propagates to every downstream call made with that context.

Retry budgets are small counters. gRPC's retry throttling keeps a token bucket per server: each failure subtracts 1 token, each success adds tokenRatio (for example 0.1), and retries are allowed only while the bucket is above half of maxTokens; maxAttempts above 5 is treated as 5 (gRFC A6). Envoy expresses the same idea as a retry budget, by default 20% of active requests with a floor of 3 concurrent retries, alongside num_retries and a per_try_timeout (Envoy docs).

Breakers and bulkheads are mostly counting. A Resilience4j-style breaker keeps a sliding window, either the last N call outcomes in a ring buffer or per-second buckets, and opens when the failure rate crosses a threshold. Envoy's "circuit breakers" are really bulkheads: hard caps on connections, pending requests and active requests per upstream cluster, with overflow rejected immediately. Its outlier detection ejects a single host after 5 consecutive 5xx responses by default, for a 30 s base ejection time that grows on repeat offences, and by default ejects at most 10% of the cluster at once (Envoy docs).

Layer 3The machine

A hung dependency consumes concrete resources. By Little's law, in-flight work equals arrival rate times latency: 2,000 RPS held for 10 s is 20,000 concurrent requests. With thread-per-request each one pins a thread, typically 1 MB of reserved stack on a 64-bit JVM, plus a socket, a file descriptor and kernel send and receive buffers. Goroutines start with a stack of a few KB, so Go survives more stuck calls, but it still runs out of file descriptors and connection slots.

The kernel's own failure detection is too slow to rely on. A connect() to a host that silently drops SYNs keeps retransmitting SYN and, with the Linux default tcp_syn_retries = 6, fails with ETIMEDOUT only after about 131 s. Kernels since mid-2023 (Linux 6.5) send the first 4 retries 1 s apart (tcp_syn_linear_timeouts = 4) and only then back off exponentially; older kernels doubled from the start and gave up after about 127 s (kernel docs). A closed port on a live host replies with RST and fails in one round trip as ECONNREFUSED. That asymmetry is why "down" is cheap and "unreachable" is expensive, and why clients must set explicit connect and request timeouts.

Timers themselves are cheap if built well. Event loops pass the nearest deadline as the timeout to epoll_wait, and runtimes use timer wheels or heaps so that tracking a hundred thousand pending timeouts costs O(1) or O(log n) per operation. Timeouts should be measured with a monotonic clock so an NTP step cannot fire them early or late.

Why it's built this way

  1. A remote node's silence has no definite meaning, so every call needs a client-side deadline and a plan for the unknown outcome.
  2. Unknown outcomes force retries, and retries are only safe with idempotency keys or idempotent operations.
  3. Retries add load exactly when a dependency is weakest, so they need jitter to spread out and budgets to cap amplification.
  4. Stuck calls hold threads, sockets and memory, which are finite per host, so bulkheads and breakers cap how much of that capacity one dependency can take.

Diagram

stateDiagram-v2
    [*] --> Closed
    Closed --> Closed: success, or failures under threshold
    Closed --> Open: failure rate over 50% in 10 s
    Open --> Open: calls fail fast
    Open --> HalfOpen: after 30 s cool-down
    HalfOpen --> Closed: probe requests succeed
    HalfOpen --> Open: a probe fails

Worked example

A checkout service handles 2,000 requests per second and calls a recommendations service (p99 120 ms) and a payment service (p99 300 ms). Each instance has 200 worker threads.

Recommendations starts hanging. With a 10 s timeout and a shared pool, each stuck request holds a thread for 10 s. At 2,000 RPS, all threads are stuck within a fraction of a second and checkout, including payment, goes down. A non-critical feature has taken out revenue.

Now apply the toolkit. Recommendations gets a 250 ms timeout, its own 40-thread bulkhead, a circuit breaker that opens at 50% failures, and a fallback of an empty list. When it hangs, at most 40 threads are involved, the breaker opens within seconds, and checkout continues at full rate with no recommendations shown. Payment keeps a 1 s timeout, 1 retry with jitter, and an idempotency key so a retried charge is never applied twice.

In the wild

Trade-offs

TechniqueStrengthsWeaknessesChoose when
Retry with backoffAbsorbs transient errorsAmplifies load; needs idempotencyShort, rare failures
Circuit breakerFails fast, protects dependencyTuning; can open on noiseRemote calls that can hang
BulkheadLimits blast radiusLower utilisation per poolMixed critical and optional deps
Load sheddingKeeps latency boundedSome users get errorsOverload is possible
FallbackFeature degrades gracefullyRarely tested pathOptional data

Pitfalls in production

Leadership lens

Check yourself

  1. Why is a slow dependency often worse than a dead one?

    Show answer

    A dead one fails fast and trips breakers; a slow one holds threads and connections until timeouts fire, exhausting caller resources and spreading latency upstream.

  2. Why add jitter to exponential backoff?

    Show answer

    Clients that failed together would otherwise retry together in waves. Randomising the delay spreads retries out so the recovering service sees smooth load.

  3. What does a circuit breaker's half-open state protect against?

    Show answer

    Sending full traffic to a dependency that may still be broken. It lets a few probes test recovery before closing again.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →