Engineering Atlas › System Design Fundamentals

What system design is and why it matters

15 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Two halves: what and how much

A system has functional requirements, the things it does ("a user can upload a photo"), and non-functional requirements, the qualities it must have while doing them ("p99 upload under 2 seconds", "no acknowledged payment is ever lost"). Code mostly serves the first half. System design is mostly about the second.

The non-functional list is longer than speed and uptime. It also covers how easy the system is to operate and change, how secure it is, and what it costs to run. AWS's review checklist, for example, groups these into six pillars, including cost optimization and sustainability (AWS Well-Architected). Write down the constraints too: team size, deadline, compliance rules and the stack you already run. They rule out options as surely as a latency target does.

A photo app for a 40-person club runs on one server with a SQLite file. The same features for 50 million daily users need a CDN, object storage, sharded metadata, thumbnail workers and a plan for losing a data centre. Nothing in the feature list changed. The numbers did.

The method

A repeatable way to work through any design. Most curricula teach a version of it; the System Design Primer, for one, uses four steps: use cases and constraints, high-level design, core components, then scaling (System Design Primer).

  1. Clarify scope. List the three to five features that matter and say out loud what is out of scope.
  2. Put numbers on it. Estimate daily users, reads and writes per user, object sizes and retention. Turn those into peak QPS, storage and bandwidth. The Estimation Cheatsheet lesson collects the constants.
  3. Define the interface. Write the main API calls and the data shape. This exposes access patterns, which decide indexes and stores.
  4. Draw the simplest design that could work. Client, load balancer, stateless service, one database. Then ask where it breaks at the numbers from step 2.
  5. Fix the bottlenecks one at a time. Add a cache where reads dominate, a queue where work can be deferred, partitioning where one machine cannot hold the data, replicas where one machine cannot be allowed to fail.
  6. Walk the failure modes. What happens when a node dies, a zone goes away, a dependency gets slow, or a key gets hot? This often sends you back to step 4.

The building blocks, and where each is covered

The problem shapes underneath

Most design problems are a few recurring shapes: overload, hot spots, partial failure, contention (many writers, one record), fan-out (one event, many recipients), duplicates, ordering and finding things fast. A ticket sale and a viral post are both contention plus hot spots; a chat message and a push notification are both fan-out. Recognise the shape and you already know the shortlist of tools.

Down the stack

Layer 1The contract

The contract of a system is its service level objectives, stated so they can be tested. Google's SRE book separates three terms: the indicator (SLI) is the measured quantity, the objective (SLO) is the target for it, and the agreement (SLA) adds consequences such as refunds when the target is missed (SRE book, Ch 4). "Fast and reliable" is not a contract. "99% of GET /feed calls return in under 200 ms, measured at the load balancer over 28 days" is. So is "an acknowledged write survives the loss of any one availability zone."

Each clause constrains the design. Surviving a zone loss means synchronous cross-zone replication before acknowledging, a 1 to 2 ms round trip on every write. AWS places zones up to about 100 km apart, close enough for synchronous replication at single-digit millisecond latency (AWS fault isolation whitepaper). Writing the contract first lets you refuse complexity that no clause requires.

Layer 2The mechanism

Each building block delivers part of the contract through a concrete mechanism. A load balancer keeps a health-check state machine per backend. Redis keeps a hash table and, once you set maxmemory and an LRU policy, evicts by sampling a few keys (5 by default) and dropping the least recently used; on 64-bit builds there is no memory limit unless you set one (Redis docs). Postgres keeps a write-ahead log and B-trees, and replicates by shipping WAL records. Kafka keeps an append-only log per partition with consumer offsets. etcd runs Raft, committing an entry once a majority has it, so a 5-node cluster keeps working with 2 nodes down (Raft).

System design is choosing which mechanisms sit on each path. A feed read might go CDN, L7 proxy, service, Redis, and only on a miss a database partition chosen by consistent hashing. The write might go to the database leader, then through change data capture into Kafka, which fans out to caches and search.

Layer 3The machine

Everything above is bounded by physics. Rough numbers worth memorising (the classic table is Latency numbers every programmer should know; its SSD row of 150 us is from older drives):

  • A main memory reference costs about 100 ns.
  • An NVMe SSD random read is about 50 to 100 us; a durable fsync on a good SSD is roughly 0.1 to 1 ms, and on a spinning disk 5 to 10 ms.
  • A round trip inside one data centre is about 0.2 to 0.5 ms. Across zones in a region it is 1 to 2 ms. Across a continent it is 30 to 70 ms, and across an ocean 70 to 150 ms.
  • A 10 Gbit/s NIC moves at most about 1.2 GB/s. A 25 Gbit/s NIC moves about 3 GB/s.

These numbers explain most designs. Caches exist because memory is about 1,000 times faster than an SSD read. Batching exists because fsync and round trips have fixed costs worth amortising. Regions exist because the speed of light makes a 150 ms round trip unavoidable across an ocean.

Why it's built this way

  1. One machine has hard limits on cores, memory, disk throughput and NIC bandwidth, and it fails as a unit. So large systems spread work and data across many machines.
  2. Machines talk over a network that is slow and can drop, delay or reorder messages, and a node cannot tell a crashed peer from a slow one (Kleppmann's lecture notes). So designs minimise round trips and plan for partial failure.
  3. Data that lives in several places can disagree. So every design picks a consistency model per piece of data, and pays for stronger ones in latency or availability.
  4. Hardware and people cost money. So the target is enough scale with headroom, and the simplest design that meets the contract wins.

Diagram

flowchart TD
    R["Requirements: features + SLOs"] --> E["Estimates: QPS, storage, bandwidth"]
    E --> A["API and data model"]
    A --> H["Simplest high-level design"]
    H --> B{"Breaks at target load?"}
    B -->|yes| F["Fix bottleneck: cache, queue, shard, replicate"]
    F --> H
    B -->|no| X{"Survives node, zone, dependency failure?"}
    X -->|no| F
    X -->|yes| D["Document trade-offs and decisions"]

Worked example

A team is building a link-preview service: given a URL, return the title, description and image of the page. Product expects 5 million daily active users, each triggering about 20 previews a day.

Numbers. 100 million previews a day is about 1,160 per second on average; plan for 4 times that at peak, about 4,600 per second. A sample shows 85% of requests hit a URL already fetched in the last day.

Simplest design. A stateless service fetches the URL, parses the HTML and returns JSON. A fetch takes 300 ms to 3 seconds plus about 20 ms of CPU to parse. At 4,600 per second that is 92 cores of parsing, thousands of open outbound connections, and a service that hammers popular sites.

Fix 1: cache. Put a shared cache keyed by normalised URL, with a 24-hour TTL, in front. 85% of requests now return in about 1 ms from Redis. Only 690 per second go out to fetch, which needs about 14 cores of parsing at peak.

Fix 2: decouple the slow work. On a miss the service returns a placeholder and enqueues the URL; workers fetch, parse and fill the cache. Duplicate misses collapse into one job keyed by the URL hash.

Fix 3: failure walk. If Redis is lost, every request misses: the queue absorbs the burst and previews degrade to placeholders. If one target site gets slow, a per-host concurrency limit (a bulkhead) stops it tying up every worker.

Fleet size. With 8 ms of CPU per cache-hit request at 16 cores per machine and a 60% utilisation target, the front tier needs ceil(4,600 × 8 / (16,000 × 0.6)), which is 4 machines, plus 2 spares so it survives a machine failure during a deploy. The practice task below builds exactly this calculator, checking CPU, memory and network to find which one binds.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Single server with a local databaseCheapest, simplest to debug, no network failures between partsOne failure takes everything down; hard ceiling on scalePrototypes, internal tools, low stakes
Stateless services behind a load balancer, one databaseEasy horizontal scaling of compute, simple consistencyDatabase becomes the bottleneck and the single point of failureMost products in their first years
Add caches and read replicasLarge read capacity, lower latencyStale reads, invalidation bugs, cache stampedesRead-heavy workloads with tolerable staleness
Partition data and add queuesScales writes and storage, isolates slow workCross-partition queries and transactions get hard; more moving partsData or write rate exceeds one machine

Pitfalls in production

Leadership lens

Check yourself

  1. Two products have identical features but one has 1,000 times the users. Which parts of the design change and why?

    Show answer

    The non-functional side: capacity, partitioning, caching, redundancy and failure handling. Features drive the API; the numbers decide machines, where state lives and what breaks.

  2. Why do caches appear in almost every large design?

    Show answer

    Memory is around a thousand times faster than an SSD read, and real workloads repeat reads. Serving repeats from memory cuts latency and protects the slower store.

  3. A requirement says "acknowledged writes survive the loss of a zone." What does that force on the write path?

    Show answer

    The write must be durable in at least two zones before the client gets an acknowledgement, which means synchronous cross-zone replication and a 1 to 2 ms round trip added to every write.

  4. Why can the fleet size be bound by network or memory rather than CPU?

    Show answer

    Each resource is a separate limit. Large responses saturate the NIC before the CPU, and long-lived connections hold memory per request for the whole latency, so whichever resource runs out first sets the count.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →