Engineering Atlas › System Design Interviews

Surviving sudden bursts of load

13 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Where spikes come from

Spikes are rarely random. Marketing pushes to 20 million phones, which then open the app within a minute. A ticket on-sale starts at exactly 10:00. Cron jobs across thousands of clients fire at the top of the hour. A dependency recovers and every client retries at once. Knowing the cause tells you whether the spike is predictable (pre-scale), self-inflicted (add jitter), or truly unknown (buffer and shed).

Buffering with a queue

For work that does not need an immediate answer (sending emails, processing uploads, recording orders after a fast acknowledgement), put a durable queue between the front end and the workers. The front end only has to enqueue, which is cheap. Workers drain at their own rate. If the spike is 10x for 2 minutes and workers run at 2x normal, the backlog grows by 8 normal-minutes of work each minute, peaks at 16, and drains at the 1x of spare capacity in about 16 more minutes. This is queue-based load levelling (Azure Architecture Center). It works when the user accepts "we got it, processing" and when the total work in the spike fits within your patience window.

Shedding load

For synchronous requests, the server must decide quickly whether it can serve each one. Load shedding rejects work at the edge with a cheap error (HTTP 503 or 429 with Retry-After) when a signal crosses a threshold: concurrency in flight, queue wait time, CPU. Rejecting costs microseconds; serving a request that will time out anyway costs the full work and helps nobody. AWS calls the useful part of throughput goodput: requests answered without error and fast enough for the client to use, which collapses once latency passes the client timeout (AWS Builders' Library). Good shedding is prioritised: checkout beats browse, interactive beats batch, a user's first attempt beats their fifth retry.

Shaping the arrival

Pre-provisioning

For predictable events, scale before they start, warm caches, and pre-split database partitions. Google's SRE book warns that a service whose capacity depends on a cache cannot carry its normal load while that cache is empty, so new capacity should take a small share of traffic first (Google SRE book). Autoscaling still matters for the long tail of growth, but treat it as a slow loop, not an emergency brake.

How it shows up in interviews

Triggers: design Ticketmaster, a flash-sale system, a voting system for a live TV show, a notification service, a URL shortener that might go viral, or any prompt that says "traffic is 50x at peak." A strong answer quantifies the peak, separates synchronous from asynchronous work, places a queue where async is acceptable, sheds and prioritises where it is not, and mentions client-side jitter and pre-scaling. Interviewers probe: what happens to the queue if the spike never ends, how do you choose what to drop, how does the user experience a rejection, and how do retries make it worse.

Down the stack

Layer 1The contract

A buffered path promises durability on acknowledgement and eventual processing, with a latency envelope that grows with backlog: if 600,000 messages are queued and workers drain 2,000 per second, the newest one waits about 5 minutes. A shedding front end promises that accepted requests meet their latency objective and rejected ones fail fast, ideally in under a millisecond, with a retry hint. A waiting room promises ordering and a bounded admission rate, not admission. Make the contract explicit, because "the queue will absorb it" hides an unbounded wait.

Layer 2The mechanism

A token bucket stores tokens and last_refill; each request refills rate * (now - last_refill) up to capacity, then spends one token or is rejected. Envoy's local rate limit filter implements exactly this and answers 429 by default (Envoy docs), as do many API gateways. For queue management, Facebook adapted CoDel to server queues: if a queue has not been empty at any point in the last N milliseconds, newly queued requests get a short timeout of M milliseconds instead of the normal one, and the server switches to adaptive LIFO so the freshest requests, whose users are still waiting, are served first (summary of Maurer, "Fail at Scale"). Classic network CoDel uses a 5 ms target and a 100 ms interval (RFC 8289). Concurrency limiters such as Netflix's concurrency-limits library start from Little's law (limit equals request rate times latency) and adjust the limit with TCP-like algorithms such as Vegas or Gradient2, shrinking it when latency rises (Netflix). Kafka or SQS provide the durable buffer; consumer lag (records-lag-max in Kafka) is the backlog signal you scale workers on.

Layer 3The machine

Even without application queues, the kernel has its own. A listening socket has an accept queue sized by the listen() backlog, capped by net.core.somaxconn (4096 by default since Linux 5.4, 128 before). When it fills, the kernel drops incoming handshake packets unless tcp_abort_on_overflow is set, and the client retransmits its SYN. Older kernels back off exponentially (about 1, 3 and 7 seconds after the first try); kernels since 2023 retry the first four SYNs at one-second intervals (tcp_syn_linear_timeouts, default 4) before backing off (Linux kernel docs). Either way, a full accept queue turns into seconds of tail latency invisible to application metrics. Inside the process, each queued request holds memory: at 20 KB of buffers per request, 100,000 queued requests are 2 GB of heap, which in a garbage-collected runtime raises GC pause times exactly when you can least afford them. New instances take time: VM boot, container pull, JIT warm-up and cache warming can each take tens of seconds, so a fresh server can be slower than its neighbours for its first minute.

Why it's built this way

Capacity costs money every hour, so systems are sized near typical peak, not worst case. Capacity cannot appear instantly because machines must boot and warm, so short spikes must be absorbed in time (queues) or refused (shedding). Queues hold memory and add delay, so they must be bounded. Users abandon slow responses, so under overload serving fresh requests first and refusing stale ones maximises useful work.

Diagram

flowchart LR
    U["Users (50x spike)"] --> CDN["CDN for static"]
    U --> WR["Waiting room or rate limiter"]
    WR --> FE["Front end"]
    FE -->|"sync, high priority"| SH{"Concurrency below limit?"}
    SH -->|"yes"| API["API servers"]
    SH -->|"no"| R429["429 with Retry-After"]
    FE -->|"async work"| Q["Durable queue"]
    Q --> W["Workers (autoscaled on lag)"]

Worked example

A ticketing site normally handles 2,000 requests per second. A major concert goes on sale at 10:00, and the team expects 1.5 million users in the first five minutes, each refreshing and searching, roughly 80,000 requests per second at the peak.

The backend (seat inventory database and payment) can safely complete about 1,500 checkouts per second. The team designs three layers. First, a waiting room in front of the sale page: users get a random queue position at 10:00 (not first-come, which rewards bots refreshing at 09:59:59). It admits about 600 users per second into the shopping flow, so the API sees a bounded population of tens of thousands of active shoppers instead of 1.5 million. Second, seat browsing is served from a cache refreshed every second; only "hold this seat" touches the database. Third, the hold endpoint has a concurrency limit of 400 in flight per instance and sheds the rest with a 429 and a retry-after of 2 seconds plus jitter.

Pre-scaling at 09:30 doubles the API fleet. Payment processing after a hold goes through a queue; users see "confirming your order" for a few seconds. Even if every admitted user checks out, 600 per second stays well under the 1,500-per-second checkout ceiling, and the 50,000 available tickets are gone within a few minutes. The waiting room then tells the remaining users that inventory is gone instead of letting them hammer the API.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Durable queueSmooths bursts, decouples producersAdds latency, backlog can grow without boundWork is async and spikes are finite
Load sheddingProtects latency for accepted workSome users get errorsSynchronous APIs under overload
Rate limiting per clientFairness, stops one abuserDoes not cap total load from many clientsMulti-tenant APIs
Waiting roomPredictable admission, good UXExtra component, users waitKnown surges like sales or launches
Pre-provisioningNo warm-up lagCost of idle capacityScheduled events

Pitfalls in production

Leadership lens

Check yourself

  1. Why does a queue not help a synchronous API under sustained overload?

    Show answer

    Arrival rate exceeds service rate, so the queue grows without limit and every request waits longer until clients time out, wasting the work.

  2. Why serve the newest requests first when overloaded?

    Show answer

    Old requests are likely abandoned by their clients, so serving them wastes capacity; fresh requests still have someone waiting.

  3. A spike is 10x normal for 3 minutes and workers can run at 2x normal. How long does the backlog take to clear?

    Show answer

    The backlog builds at 8x normal for 3 minutes, so 24 minutes of normal-rate work; draining at 1x spare capacity takes about 24 more minutes.

  4. Why randomize queue positions in a waiting room?

    Show answer

    First-come ordering rewards bots and refresh storms right before opening; random positions remove the incentive to hammer the site.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →