Engineering Atlas › System Design Fundamentals

Scaling a system up and out

13 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Load, capacity and the scaling curve

Start by naming the load parameter that matters. A chat server breaks on open connections, a feed service on fan-out per post, a ledger on writes to one account. Capacity is the highest load served within the latency objective; past it, queues form and latency climbs sharply. Werner Vogels' definition is the one most curricula reuse: a service is scalable if adding resources raises performance in proportion (Vogels). Scalability is the shape of the capacity curve as you add resources: linear is the ideal, sublinear is normal, and retrograde (more nodes, less throughput) is the failure to design against.

Vertical scaling

A bigger machine is often the best first move. The largest cloud instances now reach 32 TiB of memory and up to 1,920 vCPUs (AWS EC2 U7i), and many companies never outgrow a well-tuned Postgres primary. It keeps transactions local and needs no code changes. Its limits are a hard ceiling, price that rises faster than capacity at the top end, and a single failure domain.

Horizontal scaling

Horizontal scaling adds machines behind a router. It is cheap for stateless services: if any instance can serve any request, you add instances until the next tier saturates. Making a service stateless means moving sessions to a shared store or signed token, uploads to object storage, and treating local caches as optional accelerators.

State is the hard part. Three moves cover most designs:

A related map is the scale cube from The Art of Scalability: clone instances (X axis), split by function into services (Y axis), and split by data (Z axis) (microservices.io). Large systems use all three.

Why the curve bends

Amdahl's law caps speedup when part of the work is serial: with 5% serial work the limit is 20x. Distributed systems also pay a coherency cost for messages that agree on membership, invalidate caches or take locks, and it grows with the number of node pairs, roughly N squared. Neil Gunther's Universal Scalability Law combines both:

X(N) = λN / (1 + σ(N − 1) + κN(N − 1))

Here λ is single-node throughput, σ is the contention fraction and κ is the coherency cost. Gunther's own pages write the same law with α for contention and β for coherency (Gunther, USL); Azure's scale-out guidance uses the same two words, contention and coherency (Azure Architecture Center). With κ above zero, throughput peaks at about √((1 − σ)/κ) nodes and then falls. That is why a cluster can get slower when you add nodes: a global lock, a gossip protocol that sends N squared messages, or a cache invalidation that hits every peer.

The problem shapes behind scalability

Two shapes recur. Hot spots: sharding by user spreads load until one celebrity account takes 30% of writes. Overload: autoscaled web servers exhaust the database's connection limit. An AI agent platform hits the same two, when one popular prompt template hammers a single cache key and parallel tool calls exceed the model provider's rate limit. The fixes carry across: better partition keys, key splitting, request coalescing, queues and backpressure.

Down the stack

Layer 1The contract

A scalability claim must be testable: "From N to 2N nodes, throughput at p99 under 200 ms rises at least 1.8x, and cost per million requests does not rise." State the data shape too (key distribution, payload size, read/write mix), because a system that scales on uniform keys can collapse on skewed ones. Say what happens past the breaking point: shed load with 429s, queue, or fall over.

Layer 2The mechanism

Stateless tiers use a load balancer plus an autoscaler. Kubernetes' Horizontal Pod Autoscaler runs every 15 seconds, computes desiredReplicas = ceil(currentReplicas × currentMetric / targetMetric), ignores changes within a 10% tolerance, and waits out a 300-second stabilization window before scaling down. Scale-up has no window by default: it may add 100% of current pods or 4 pods, whichever is more, every 15 seconds. Older guides say the tolerance is cluster-wide only; since Kubernetes 1.33 each HPA can set its own under behavior.scaleUp.tolerance and behavior.scaleDown.tolerance, and the field is stable as of 1.37 (Kubernetes docs). Data tiers use a partition map: key ranges or hash slots assigned to nodes, held in etcd or ZooKeeper or gossiped (Cassandra's token ring), with consistent hashing or many virtual partitions so a new node takes about 1/N of the data. Little's law sizes everything: 20,000 requests per second at 50 ms means 1,000 requests in flight, which sets thread, connection and queue limits.

Layer 3The machine

Scaling limits are usually physical and boring. A Postgres connection is its own backend process, spawned by the postmaster for each client (Postgres docs) and using several MB, so 5,000 connections from autoscaled pods can exhaust RAM before CPU; PgBouncer multiplexes them onto a few hundred. Linux's default ephemeral port range, 32768 to 60999 or about 28,000 ports (kernel docs), caps outbound connections from one IP to one destination address and port, and TIME_WAIT holds each port for 60 seconds. Inside a box, a contended lock bounces its cache line between cores at roughly 50 to 100 ns per transfer across sockets, so a "fast" global counter incremented by 64 cores serializes the whole machine. A 25 Gbps NIC moves about 3 GB per second, which is only 300,000 requests per second at 10 KB each.

Why it's built this way

  1. One machine has finite cores, memory and NIC bandwidth, so large systems must use many machines.
  2. Many machines cannot share memory cheaply, so state must be split or copied, and every split adds routing and every copy adds coordination.
  3. Coordination costs grow with the number of participants, so designs keep the coordinating group small (a Raft group of 3 or 5) and partition everything else.
  4. Load is skewed and bursty, so partition keys, caches and admission control matter more than raw node count.

Diagram

flowchart LR
    C[Clients] --> LB[Load balancer]
    LB --> A1[App 1]
    LB --> A2[App 2]
    LB --> A3["App N (stateless)"]
    A1 --> CA[Cache]
    A2 --> CA
    A3 --> CA
    CA -->|miss| R{Partition map}
    R --> S1["Shard 1 leader"]
    R --> S2["Shard 2 leader"]
    S1 --> S1R[Shard 1 replicas]
    S2 --> S2R[Shard 2 replicas]

Worked example

A checkout API runs on one 16-core node at 1,000 requests per second with p99 of 120 ms. Black Friday needs 8,000. The team load-tests at 1, 2, 4, 8 and 16 nodes and gets 1,000, 1,900, 3,440, 5,690 and 8,040 requests per second. Fitting the USL gives σ ≈ 0.05 and κ ≈ 0.001. The model predicts a peak of about 9,040 requests per second at 31 nodes and a decline after that, so 16 nodes only just meets the target with no headroom, and no amount of autoscaling will ever reach 10,000.

The traces explain both coefficients. σ comes from a single inventory row per SKU locked with SELECT ... FOR UPDATE; a popular item serializes all its buyers. κ comes from a cache invalidation that each node broadcasts to every other node. The fixes target the coefficients, not the node count: split hot inventory into 10 sub-counters that are reserved independently (σ falls toward 0.01), and replace peer broadcast with a single pub/sub channel (κ falls toward 0.0001). The refitted model predicts a peak near 99 nodes and about 13,600 requests per second at 16 nodes, so the same hardware now has 70% headroom over the target.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Vertical scalingNo code changes, local transactions, low latencyHard ceiling, steep price at the top, one failure domainEarly, or the workload fits one large box with headroom
Stateless horizontal (X axis)Near-linear, easy autoscalingPushes load onto shared state tiersWeb and API tiers
Functional split (Y axis)Teams and tiers scale independentlyNetwork hops, distributed failuresDistinct workloads with different load shapes
Data partitioning (Z axis)Scales writes and storageCross-shard queries, rebalancing, hot keysWrite volume or data size exceeds one node
Read replicas and cachesCheap read capacityStaleness, invalidationRead-heavy workloads tolerant of lag

Pitfalls in production

Leadership lens

Check yourself

  1. Why can adding nodes reduce total throughput?

    Show answer

    Coherency costs such as cache invalidation, membership gossip or distributed locks grow roughly with the square of node count, so beyond the peak each new node adds more coordination work than useful capacity.

  2. Your API tier is stateless and CPU usage is 30%, yet throughput stopped rising. Where do you look?

    Show answer

    At shared tiers and serial points: database connections and locks, a single queue partition, a rate-limited dependency, or a hot key. Idle CPU with flat throughput means requests are waiting, not working.

  3. Why does sharding by user ID not guarantee even load?

    Show answer

    Hashing spreads keys evenly, not traffic. A few users can generate a large share of requests, so a single shard can still saturate.

  4. What does Little's law tell you about a service doing 5,000 requests per second at 40 ms?

    Show answer

    About 200 requests are in flight at any moment, which sets the minimum concurrency (threads, connections, pool slots) the service needs to sustain that rate.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →