Engineering Atlas › Low-Level Design

Code a resilient HTTP client

17 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

Requirements

Functional:

Non-functional:

Out of scope: connection pooling, TLS, request bodies and streaming, hedged requests, and per-host breaker registries. Extensibility returns to the last two.

Entities and relationships

Class diagram

classDiagram
    class Client {
        <<interface>>
        +send(request, timeoutMs) Response
    }
    class Transport {
        +send(request, timeoutMs) Response
    }
    class RetryingClient {
        -int maxAttempts
        -long baseMs
        -long capMs
        -long perTryMs
        +send(request, timeoutMs) Response
    }
    class CircuitBreaker {
        -State state
        -int failures
        -long openedAt
        -bool probing
        +send(request, timeoutMs) Response
    }
    class RetryBudget {
        -double tokens
        -double maxTokens
        -double tokenRatio
        +onSuccess()
        +tryWithdraw() bool
    }
    class Clock {
        <<interface>>
        +now() long
        +sleep(ms)
    }
    Client <|.. Transport
    Client <|.. RetryingClient
    Client <|.. CircuitBreaker
    RetryingClient o-- Client : wraps
    CircuitBreaker o-- Client : wraps
    RetryingClient --> RetryBudget
    RetryingClient --> Clock
    CircuitBreaker --> Clock

Key flows

One call through RetryingClient(CircuitBreaker(Transport)), where the first attempt gets a 503:

sequenceDiagram
    participant App
    participant R as RetryingClient
    participant B as CircuitBreaker
    participant T as Transport
    participant C as Clock
    App->>R: send(GET /items, 2000)
    R->>R: deadline = now + 2000
    R->>B: send(req, min(perTry, left))
    B->>T: send(req, timeout)
    T-->>B: 503
    B->>B: record failure
    B-->>R: 503
    R->>R: idempotent, attempts left, delay in deadline
    R->>R: budget.tryWithdraw
    R->>C: sleep(random x min(cap, base x 2^n))
    R->>B: send(req, min(perTry, left))
    B->>T: send(req, timeout)
    T-->>B: 200
    B->>B: record success
    B-->>R: 200
    R->>R: budget.onSuccess
    R-->>App: 200

The breaker sits inside the retry loop. Each attempt is one observation of the dependency's health, so the breaker sees real failures and not whole calls. An open breaker raises an open-circuit error, and the retry layer passes that error straight up: retrying a fast failure only burns attempts and sleeps.

The breaker is a small state machine:

stateDiagram-v2
    [*] --> Closed
    Closed --> Open : failures reach threshold
    Open --> HalfOpen : cool-down elapsed
    HalfOpen --> Closed : probe succeeds
    HalfOpen --> Open : probe fails

In Closed the breaker counts consecutive failures and resets the count on any success. At the threshold it moves to Open and records the time. While Open it rejects calls without touching the network. After the cool-down the next call moves it to HalfOpen and goes through as a probe; every other call is rejected until the probe finishes. A successful probe closes the circuit and a failed one reopens it with a fresh cool-down. Martin Fowler's description of the pattern has the same shape: the breaker trips once failures reach a threshold, and further calls return an error without reaching the protected call.

Design patterns used and why

Concurrency and edge cases

The client is shared by every thread in a process, so its mutable state needs care. The breaker's counters and the budget's token count are small read-modify-write sequences, like a rate limiter's bucket: guard each with a mutex. The rule that matters is never hold the lock across inner.send. Take the lock to decide whether to admit the call, release it, make the network call, then take it again to record the outcome. Holding it across the call serialises every request behind the slowest one.

The half-open probe needs one extra flag. Without it, a hundred threads waking after the cool-down all see "half-open" and all send traffic to a dependency that may still be down. A probing flag, set under the lock when a thread claims the probe, lets exactly one through.

Edge cases worth naming:

In the wild

Down the stack

Layer 1The contract

send makes between one and maxAttempts calls to the inner client. It returns the first response whose status is not retryable, or the last response or error when attempts, budget or deadline run out. It never returns later than timeoutMs after entry, to within one inner call that honours its own timeout. It never retries a non-idempotent request, and never retries an open-circuit error. The breaker guarantees that while Open it makes zero inner calls, and that in HalfOpen at most one call is in flight.

Layer 2The mechanism

Backoff for attempt n (counting from 1) is random() × min(cap, base × 2^(n-1)). This is "full jitter" from the AWS Architecture Blog's analysis of backoff and jitter: the delay is uniform from zero to the exponential ceiling, which removes the correlated waves that plain exponential backoff keeps producing when many clients fail together. The cap bounds worst-case sleep and prevents integer overflow of the exponent.

The retry budget follows the same accounting as gRPC's client retry design (A6): a per-server token count starts at maxTokens, each failure removes one, each success adds tokenRatio (0.1 in the example config), and retries stop while the count is at or below half of maxTokens. With a ratio of 0.1, retries are limited to about 10% of the successful traffic in the long run. The Google SRE book's chapter on cascading failures recommends a similar server-wide retry budget and gives 60 retries per minute per process as an example. Envoy's circuit-breaking docs likewise prefer retry budgets over static limits. The practice version is a simplified form that withdraws a whole token per retry and refuses when less than one remains.

Deadlines are absolute. A deadline computed once at entry survives retries and sleeps, and in a multi-service call it should be propagated: the SRE book's example has server A, after spending 7 of its 30 seconds, call B with a 23-second deadline. gRPC propagates deadlines for exactly this reason. An HTTP client can pass the remaining time in a header, and the server should drop work whose caller has already given up.

Idempotency is defined by RFC 9110: PUT, DELETE and the safe methods have the same intended effect when repeated. For POST, the client sends a unique key and the server stores the first outcome, as in Stripe's idempotent requests. An in-progress IETF draft for an Idempotency-Key header standardises this.

Layer 3The machine

A timeout is a timer in the networking stack. The client library arms a deadline on a socket (in Go, http.Client.Timeout covers dial, redirects and reading the body), and when it fires the client closes the connection. The server may still be running the handler: the TCP FIN or RST arrives after the work started, and some servers only notice on their next write. That is the physical reason a timeout leaves the outcome unknown.

Connecting costs one round trip for TCP (about 0.2 ms in a data centre, 20-150 ms across regions) plus one or two more for TLS, so a retry on a fresh connection is not free; a retry on a kept-alive connection avoids that. Under the retry loop, a sleep is a parked thread or goroutine. A goroutine parked in time.Sleep costs about 2-8 KB of stack and no CPU; a blocked OS thread costs around 1 MB of reserved stack and a scheduler slot, which is why high retry volume exhausts thread pools in thread-per-request servers. The breaker's state is a few words, and the check on the hot path is one mutex acquire (about 20 ns uncontended) and a comparison.

Why it's built this way

  1. Networks fail briefly and often, so a failed call is worth repeating. Repeating adds load.
  2. Added load is most dangerous exactly when the server is already slow, so retries are capped per request and metered against a shared budget that shrinks when success shrinks.
  3. Synchronised retries make the load arrive in waves, so delays are randomised, and exponential so the total load falls as the outage lasts.
  4. A dependency that is down gets nothing from more attempts, and each attempt ties up a caller's thread for the full timeout, so the breaker fails fast and probes sparingly.
  5. A caller only has so long to wait, so one deadline bounds everything, and the retry layer has to know what is safe to repeat, since a timeout hides whether the work happened.

Extensibility

Leadership lens

Check yourself

  1. A client retries a failed POST /payments after a timeout, and the customer is charged twice. What went wrong, and what are two fixes?

    Show answer

    The timeout did not mean the server had not processed the request, and POST is not idempotent, so the retry repeated a side effect. Fix it by sending a unique idempotency key that the server uses to replay the first outcome, or by not retrying non-idempotent requests automatically.

  2. Why does full jitter use random() × ceiling and not ceiling ± a small random amount?

    Show answer

    A small random offset leaves clients clustered near the same delay, so retries still arrive in waves. Drawing uniformly from zero to the ceiling spreads them across the whole window, which flattens the load on the recovering server.

  3. A service calls a dependency with 3 attempts per call, and the dependency's own client also makes 3 attempts to a database. The database slows down. How much extra load can it see, and how do you reduce it?

    Show answer

    Up to 9 calls per user request instead of 1, since attempts multiply across layers. Retry at one layer only, and add a retry budget so retries cannot exceed a fraction of successful traffic.

  4. Why must the breaker release its lock before calling the inner client, and what extra state does the half-open phase need?

    Show answer

    Holding the lock across a network call makes every request wait behind the slowest one. Half-open needs a probing flag set under the lock so that exactly one caller sends the probe and all others are rejected until it finishes.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →