Engineering Atlas › System Design Fundamentals

Isolating failures with bulkheads

11 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

The failure it prevents

A service has one shared pool of 200 threads for all outbound calls. One dependency slows from 20 ms to 5 seconds. Threads waiting on it accumulate until all 200 are stuck, and calls to perfectly healthy dependencies queue behind them. A failure in one feature becomes an outage of every feature. Timeouts and circuit breakers reduce how long this lasts; bulkheads cap how much it can take (Azure Architecture Center).

Bulkheads inside a process

Give each dependency its own concurrency limit. Two common implementations:

Sizing uses Little's law: concurrency needed equals request rate times latency. At 300 calls per second and 50 ms, you need about 15 concurrent slots; give 2 to 3 times headroom, so 30 to 45. When a call is rejected, the caller falls back exactly as with an open breaker.

Bulkheads across a system

The same idea applies at larger scale. Separate connection pools for interactive and batch queries to the same database. Separate instance groups for public API traffic and internal admin traffic. Separate queues and consumers per tenant tier. At the largest scale, cell-based architecture splits the whole stack into independent cells, each serving a subset of customers, so a bad deploy or poison request hurts one cell. AWS says its service teams have built this way for more than a decade, and its Well-Architected guidance presents cells as the bulkhead practice applied to a whole workload (AWS whitepaper). Shuffle sharding assigns each customer a small set of workers from a larger pool, so two customers rarely share all of their workers, and one bad customer affects very few others. Even 8 workers in pairs give 28 distinct shards; Route 53 splits capacity into 2,048 virtual name servers and gives each domain 4, about 730 billion combinations. Route 53 also assigns them so that no two domains share more than two name servers (AWS Builders' Library).

Interaction with other patterns

Bulkheads limit concurrency; timeouts bound how long a slot is held; circuit breakers stop sending to a dependency that is clearly broken. Use all three together.

Down the stack

Layer 1The contract

A bulkhead promises that at most N units of work run concurrently in its partition, that excess work either waits for a bounded time or is rejected immediately, and that saturation of one partition does not reduce the capacity available to others. At the infrastructure level, a CPU or memory bulkhead promises a ceiling: a workload cannot use more than its share, even if that means it is throttled or killed.

Layer 2The mechanism

In process, a semaphore bulkhead is a counter with tryAcquire(timeout): Resilience4j exposes this as maxConcurrentCalls (default 25) and maxWaitDuration (default 0, so a full bulkhead rejects at once). A thread-pool bulkhead is a fixed pool with a bounded queue and a rejection policy; Resilience4j sizes it from the CPU count with a 100-entry queue by default (Resilience4j docs). Defaults are a starting point, not a sizing: 25 slots is far too many for a 2-slot dependency and far too few for one that needs 200. Connection pools (HikariCP's maximumPoolSize, Envoy's max_requests per cluster) are bulkheads too. At the OS level, Kubernetes resource limits become cgroup v2 settings: cpu.max with a quota and period such as 200000 100000 (two CPUs' worth per 100 ms period; the default is max 100000, no limit) (kernel docs) and memory.max, beyond which the kernel's OOM killer acts inside that cgroup only. Cells and shuffle sharding apply the same idea one level up, with a routing layer mapping each customer to its partition.

Layer 3The machine

Partitions exist because the underlying resources are physical. Each Postgres connection is a separate backend process, forked when the client connects (Postgres docs), using several MB of memory, so a database supports hundreds, not tens of thousands, of connections. Each thread costs stack memory and a context switch of a few microseconds when the scheduler moves between them. CPU quotas are enforced by the CFS bandwidth controller per period, and a group that spends its quota stays throttled until the next period refills it (kernel docs): a container that burns its quota in the first 40 ms of a 100 ms period is throttled for the remaining 60 ms, which shows up as tail latency even when average CPU looks low. Queueing theory adds the last piece: as utilisation of any resource approaches 100%, waiting time grows without bound, so a capped pool that rejects early keeps latency predictable.

Why it's built this way

  1. Threads, connections, CPU, and memory are finite and shared, so one runaway consumer can starve all others.
  2. Queues near full utilisation have exploding latency, so concurrency is capped and excess is rejected rather than queued indefinitely.
  3. The kernel can enforce limits cheaply per cgroup, so isolation is pushed down to the OS where possible.
  4. Deploys, configs, and poison requests are shared resources too, so large systems extend bulkheads to whole cells.

Diagram

flowchart LR
    REQ["Incoming requests"] --> SVC["Product page service"]
    SVC --> B1{"Pricing slots free? (20)"}
    SVC --> B2{"Reviews slots free? (15)"}
    SVC --> B3{"Recs slots free? (20)"}
    B1 -->|"yes"| P["Pricing service"]
    B2 -->|"yes"| R["Reviews service (slow)"]
    B3 -->|"yes"| RC["Recommendations"]
    B2 -->|"full"| F["Fallback: hide reviews"]

Worked example

A product page service handles 1,500 requests per second. Each request calls pricing (required, 15 ms), reviews (optional, 30 ms), and recommendations (optional, 40 ms), in parallel. It runs on 10 instances, so each instance handles 150 requests per second.

Per instance, Little's law gives concurrency of 150 x 0.015 = 2.25 for pricing, 4.5 for reviews, and 6 for recommendations. The team sets semaphore limits of 20, 15, and 20, a generous multiple of normal load, with no waiting queue for optional calls and a 50-slot queue for pricing.

When the reviews database degrades and latency climbs to 3 seconds, reviews concurrency would need 450 slots per instance. Instead it hits its limit of 15 immediately; the other 135 review calls per second are rejected in microseconds and the page renders without reviews. Pricing and recommendations keep their full slots and normal latency. Without the bulkhead, the shared HTTP client's 200 connections would fill with review calls in under 2 seconds and pages would fail entirely.

At the system level, the same company moved its 20 largest enterprise customers onto shuffle-sharded worker sets of 4 out of 64 workers. The number of possible 4-worker combinations is over 635,000, so the chance that a poison request from one customer takes out all of another customer's workers is tiny.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Shared poolBest utilisation, simplestOne slow dependency can exhaust everythingFew dependencies, all equally critical
Semaphore per dependencyCheap, async-friendly, strong isolation of concurrencyCaller thread still blocks in sync codeMost service-to-service calls
Thread pool per dependencyFrees caller even on hangs, per-pool queuesMore threads, memory, and context switchingBlocking clients that cannot time out reliably
Cells or shuffle shardingContains bad deploys and poison tenantsCost, routing layer, cross-cell operationsLarge multi-tenant platforms

Pitfalls in production

Leadership lens

Check yourself

  1. A dependency handles 400 calls per second at 25 ms. What concurrency does it need, and how would you set the bulkhead?

    Show answer

    About 10 slots by Little's law, so a limit of roughly 20 to 30 gives headroom for spikes without letting a slowdown consume the whole process.

  2. Why might a thread-pool bulkhead be chosen over a semaphore?

    Show answer

    If the client library can block without honouring timeouts, a separate pool keeps the caller's threads free even when the dependency hangs.

  3. What does shuffle sharding add over simple sharding?

    Show answer

    Each customer gets a different random subset of workers, so a customer that breaks its workers overlaps fully with very few others, shrinking the blast radius.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →