A map of core system design ideas
Open in the interactive app: practice, quiz, mock interview →In one minute
- Thirty concepts cover most of what comes up when designing or reviewing a large system. Each has its own lesson in this course; this page is the map.
- They fall into five families: moving traffic, storing data, communicating between services, surviving failure, and coordinating machines.
- The concepts are not independent. A single user request passes through a dozen of them, and each adds latency, a failure mode and a cost.
- The habit to build is composition: knowing that a page with 30 parallel backend calls is only as fast as the slowest one, and that availability multiplies down a chain of dependencies.
How it works
The map
| # | Concept | The idea in one line | Go deeper |
|---|---|---|---|
| 1 | Client-server | Clients ask, servers own the data and answer | Client-Server Architecture |
| 2 | Scalability | Grow capacity by bigger machines (vertical) or more machines (horizontal) | Scalability |
| 3 | Availability | The fraction of time the system answers correctly; nines multiply down a chain | Availability |
| 4 | Reliability | Doing the right thing consistently, measured as failure rate and time between failures | Reliability |
| 5 | Single point of failure | Any component whose loss stops the whole system | Single Point of Failure (SPOF) |
| 6 | Latency, throughput, bandwidth | Time per request, requests per second, and the size of the pipe | Latency vs Throughput vs Bandwidth |
| 7 | Load balancing | One address in front of a pool, routing around dead backends | What are Load Balancers?, Load Balancing Algorithms |
| 8 | DNS | Names to addresses, cached at every layer by TTL | Domain Name System (DNS), DNS Load Balancing |
| 9 | Proxies | Forward proxies act for clients; reverse proxies act for servers | Proxy vs Reverse Proxy |
| 10 | CDN | Serve bytes from a point of presence near the user | Content Delivery Network (CDN) |
| 11 | Caching | Keep repeated answers in fast memory | What is Caching?, Caching Strategies |
| 12 | Eviction and invalidation | Decide what leaves the cache and when cached data is wrong | Cache Eviction Policies, Cache Invalidation, Cache Stampede |
| 13 | Database types | Pick the store by access pattern: relational, document, key-value, wide column, graph | Common Database Types, SQL vs NoSQL |
| 14 | ACID transactions | All-or-nothing, isolated, durable changes to several rows | ACID Transactions |
| 15 | Indexing | Extra structures that turn scans into lookups at a write cost | Indexing, B-Trees and B+ Trees, LSM Trees |
| 16 | Replication | Copies of data for read scale and survival | Database Replication |
| 17 | Partitioning and sharding | Split data so no machine holds all of it | Database Partitioning, Sharding |
| 18 | Consistent hashing | Place keys on a ring so adding a node moves few keys | Consistent Hashing |
| 19 | CAP theorem | During a partition, choose consistency or availability | CAP Theorem |
| 20 | Consistency models | What a reader may see: linearizable, causal, eventual and others | Consistency Models |
| 21 | Message queues | Buffer work so producers and consumers run at their own pace | Message Queues, Dead Letter Queues |
| 22 | Pub/sub and events | Publish once, many independent subscribers react | Pub/Sub, Event-Driven Architecture |
| 23 | API styles | REST, gRPC, GraphQL and when each fits | API Styles, REST API Design, gRPC Deep Dive |
| 24 | API gateway | One front door for auth, routing and quotas | API Gateways, API Gateway Pattern |
| 25 | Rate limiting | Cap how fast a client may call, to protect capacity and fairness | Rate Limiting |
| 26 | Idempotency | Repeating a request has the same effect as doing it once | Idempotency, Delivery Semantics |
| 27 | Monolith vs microservices | One deployable versus many independently owned services | Monolithic Architecture, Microservices Architecture |
| 28 | Resilience patterns | Timeouts, retries, circuit breakers and bulkheads stop failures spreading | Circuit Breaker Pattern, Bulkhead Pattern, Handling Failures in Distributed Systems |
| 29 | Consensus and leader election | Machines agree on one value or one leader despite failures | Consensus Algorithms, Raft Algorithm, Leader Election |
| 30 | Observability | Logs, metrics and traces that let you explain behaviour from outside | Three Pillars of Observability, Distributed Tracing |
Five families
Moving traffic (1, 2, 6 to 10, 24, 25) is about getting a request from a user to a machine that can answer it, quickly and without overloading anyone. Storing data (11 to 18) is about where bytes live, how many copies exist and how to find them fast. Communicating (21 to 23, 26, 27) is about how services talk: synchronously when the caller needs the answer now, asynchronously when it can wait. Surviving failure (3 to 5, 28, 30) is about the assumption that something is always broken. Coordinating (19, 20, 29) is about what happens when several machines must agree.
The families map onto recurring problem shapes: overload (load balancing, rate limiting, queues), hot spots (caching, CDNs, partition keys), partial failure (replication, timeouts, circuit breakers), duplicates (idempotency) and finding things fast (indexes). Name the shape and the table tells you which rows to read.
How the concepts interact
Each concept obliges you to handle another. Caching (11) forces invalidation (12) and consistency (20). Sharding (17) makes cross-shard transactions (14) expensive, pushing you to sagas and idempotent steps (26). Replication (16) helps availability (3) but CAP (19) then forces a choice during partitions. Retries (28) become a retry storm without rate limits (25). In a review, name the concept each box embodies and the one it now obliges you to handle.
Down the stack
Layer 1The contract
The concepts compose into the two numbers users feel: end-to-end latency and availability. For calls made one after another, latencies add. For calls made in parallel, the slowest one decides. For dependencies that must all work, availabilities multiply: five services at 99.9% each give about 99.5% (System Design Primer). For a request that fans out to n backends, if each is slow with probability p, the chance that at least one is slow is 1 - (1 - p)^n. At p = 1% and n = 100, that is 63%: a rare tail on one server becomes the common case for the page. This is the example Dean and Barroso use to open their paper on tail latency (summary of The Tail at Scale).
Layer 2The mechanism
Follow one request to a product page. DNS (8) resolves the name from a cache. The CDN (10) serves static assets. A load balancer (7) picks a gateway instance (24), which checks a token, applies a rate limit (25) and routes to the page service. That service calls auth, then fans out in parallel to pricing, stock and reviews, each reading through a cache (11) in front of a replicated, sharded database (16, 17). Every call carries a deadline and a trace ID (30).
gRPC makes the budget concrete. The client's deadline travels as a grpc-timeout header, a number of at most 8 digits plus a unit such as m for milliseconds (gRPC HTTP/2 protocol). When a server makes its own outgoing call, gRPC can pass on the deadline minus the time already spent, so downstream services stop working on requests whose caller has given up. Two details catch teams out: gRPC sets no deadline by default, so a call can wait forever, and propagation is on by default in Java and Go but must be turned on in C++ (gRPC deadlines guide).
Layer 3The machine
Each hop has a hardware floor. A cached DNS answer takes under 1 ms; a full recursive lookup 20 to 100 ms. A new TLS 1.3 connection costs one extra round trip on top of the TCP handshake, about 1 ms in a region and 50 to 150 ms across continents, which is why connection pooling matters. TLS 1.3 also allows a zero round-trip resumption mode, but data sent that way can be replayed, so it suits only idempotent requests (RFC 8446). A Redis GET across the data centre takes 0.2 to 0.5 ms, mostly network, epoll_wait and socket copies rather than Redis. A Postgres page miss to NVMe adds about 100 us, to a network block device 0.5 to 1 ms. A 10 to 50 ms garbage-collection pause shows up as tail latency on every caller.
Why it's built this way
- Users notice latency above roughly 100 ms, lose their flow of thought after about 1 second and their attention after about 10 (Nielsen Norman Group). So each request gets a budget, split across hops.
- Hops are expensive and the network is unreliable. So systems cache near the user, pool connections and fan out in parallel instead of in series.
- Parallel fan-out makes tail latency dominant. So services set per-call timeouts, hedge slow requests, and treat p99 rather than the average as the target.
- Every added component multiplies failure probability. So each must earn its place, and the critical path is kept as short as the requirements allow.
Diagram
flowchart LR
U["User"] --> D["DNS (8)"]
U --> C["CDN (10)"]
C --> LB["Load balancer (7)"]
LB --> G["Gateway + rate limit (24, 25)"]
G --> P["Page service"]
P --> A["Auth"]
P --> PR["Pricing"]
P --> S["Stock"]
P --> R["Reviews"]
PR --> K["Cache (11)"]
K --> DB["Sharded replicas (16, 17)"]
S --> Q["Queue (21)"]Worked example
A product page has a 300 ms p99 target at the load balancer. The team draws the call tree and assigns budgets.
The gateway spends 5 ms on token checks and routing. The page service calls auth (p99 20 ms), then calls pricing (p99 60 ms), stock (p99 35 ms) and reviews (p99 120 ms) in parallel, then spends 25 ms rendering. The critical path is gateway, auth, reviews, render: 5 + 20 + 120 + 25 = 170 ms. That fits, with 130 ms of headroom.
Then reviews adds a recommendation call at p99 150 ms, in series after its 120 ms read. Reviews becomes 270 ms and the page 320 ms. Tuning pricing or stock cannot help: they are off the critical path. Options: run the two reviews calls in parallel (page 200 ms), load recommendations asynchronously, or cap reviews at 100 ms with an empty fallback.
The tail compounds too. Reviews reads 10 shards in parallel; if each is slow 1% of the time, 1 - 0.99^10 is about 9.6% of pages waiting on a slow shard. Hedged requests, a second copy to another replica once the first has been outstanding longer than the p95 latency, cut that for about 5% extra load, because by definition only one request in twenty waits that long (summary of The Tail at Scale). The practice task builds this calculator.
In the wild
- Slack, 2022: a cache miss becomes a fan-out to every shard. A Consul agent upgrade rolled to 25% of the fleet, and restarting agents deregistered memcached nodes, which came back empty. On a cache miss, client boot ran a query across every database shard, so database load grew faster than the miss rate. Timeouts stopped the cache from refilling and client retries held the cascade in place. Slack throttled boot requests, then raised limits step by step, and moved the query to a table sharded by channel (Slack).
- Google Cloud, 2025: retries without jitter slow the recovery. A quota policy row with blank fields replicated worldwide within seconds and hit a new Service Control code path that had no error handling and no feature flag, so API frontends crash-looped everywhere. A kill switch went out in about 40 minutes, but in us-central1 restarting tasks with no randomized backoff overloaded the Spanner table behind them. Google committed to failing open, rolling out global data incrementally and auditing backoff (Google Cloud).
- Canva, 2024: CDN, gateway and thundering herd in one chain. After a deploy, one JavaScript chunk stalled on a lossy network path, and CDN request coalescing held more than 270,000 requests on the same fetch. When it finally loaded, every waiting client hit the API at once, about 1.5 million requests per second or 3 times peak, and gateway tasks ran out of memory faster than autoscaling could replace them. canva.com was down for about 52 minutes. Canva blocked traffic at the CDN, then let it back gradually, starting with Australian users under strict rate limits, and listed extra load shedding as a follow-up (Canva).
Trade-offs
| Option | Strengths | Weaknesses | Choose when |
|---|---|---|---|
| Sequential calls | Simple, each call can use the last one's result | Latencies add; one slow call delays everything | Calls genuinely depend on each other |
| Parallel fan-out | Total latency is the slowest call, not the sum | Tail latency dominates; more load per request | Independent data for one page |
| Async via queue | Removes the work from the user's path | Result is not ready immediately; needs status handling | Work the user does not need to wait for |
| Degrade with a timeout | Bounded latency, page still renders | Partial results, more client logic | Optional parts of a response |
Pitfalls in production
- Using the 30 as a checklist. Adding every concept to every design makes it fragile and expensive. Each component should answer a stated requirement.
- Ignoring composition. Teams set each service to 99.9% and are surprised the product is at 99.5%. Compute end-to-end availability and latency from the call graph.
- Timeouts that do not shrink. Every hop uses a fixed 5-second timeout, so a request the user abandoned keeps consuming capacity three services deep. Propagate deadlines.
- Retries at every layer. Three layers making three attempts each turns one failure into 27 calls; AWS's example of five such layers reaches 243 times the load on the database. Retry at one layer, with jitter and a token-bucket retry budget (AWS Builders' Library).
- Averages in dashboards. The mean hides the tail that users actually feel on fan-out pages. Track p99 and p99.9 per dependency (SRE book, Ch 4).
Leadership lens
- In design reviews, ask for the call graph with a latency and availability number on each edge, and the computed end-to-end result.
- Have owners of critical-path dependencies publish SLOs, and plan for them like features.
- Budget complexity explicitly: every new box adds on-call load, upgrades and cost, so ask which requirement it serves.
Check yourself
Why does a page that fans out to 100 backends see tail latency far more often than any single backend?
Show answer
The page waits for the slowest call. If each backend is slow 1% of the time, the chance that at least one of 100 is slow is
1 - 0.99^100, about 63%.A service is on the page's critical path at 120 ms; another runs in parallel at 60 ms. Which one should you optimise first?
Show answer
The 120 ms one. Speeding up the parallel 60 ms call changes nothing until it becomes the slowest branch.
Why do sharding and idempotency often appear together in a design?
Show answer
Sharding makes multi-row transactions span machines, so designs use sagas or retried steps instead, and retried steps are only safe when each one is idempotent.
Why propagate deadlines rather than set a fixed timeout at each hop?
Show answer
A shrinking deadline lets downstream services stop work the caller has already abandoned, which saves capacity during overload and keeps the total within the user's budget.
Sources and further study
Primary
- RFC 8446: TLS 1.3: the one round-trip handshake and the replay limits of zero round-trip data.
- gRPC over HTTP/2 protocol: the wire format of
grpc-timeoutand what a server does when it is missing.
Systems docs
- Envoy: outlier detection: how a proxy ejects failing hosts and caps how many it ejects.
- Redis: key eviction: the eviction policies behind row 12, and why Redis approximates LRU.
Books
- Site Reliability Engineering (2016), Beyer et al., Ch 22 "Addressing Cascading Failures": how overload, retries and cold caches chain together, with the fixes (read free).
- Release It! (2nd ed., 2018), Michael Nygard, Ch 4 "Stability Antipatterns" and Ch 5 "Stability Patterns": integration points, chain reactions, timeouts, circuit breakers and bulkheads.
- Designing Data-Intensive Applications (2nd ed., 2026), Kleppmann and Riccomini, Ch 2 "Defining Nonfunctional Requirements": percentiles and how tail latency compounds across calls.
Other curricula
- Azure Architecture Center: Cloud design patterns: a catalog of about 45 patterns, each tagged with the quality it serves, and advice on pairing them, such as Retry with Circuit Breaker.
- roadmap.sh: System Design: the same concepts laid out as a learning path rather than a single map.
Watch
- 20 System Design Concepts Explained in 10 Minutes (NeetCode): a quick visual pass over many of the concepts on this page.
- Achieving Rapid Response Times in Large Online Services (Jeff Dean, Velocity 2014): tail latency under fan-out and the techniques Google uses to tame it.
- How to start with distributed systems? (Gaurav Sen): how the building blocks fit together as a system grows.
Free resources
- The Tail at Scale (Dean and Barroso): the paper behind fan-out tail amplification and hedged requests.
- Timeouts, retries and backoff with jitter (AWS Builders' Library): how to set timeouts and retries without causing overload.
- Deadlines (gRPC docs): how deadlines propagate across service hops.
- Fallacies of distributed computing (Wikipedia): the eight false assumptions that most of these concepts exist to handle.
- The System Design Primer (Donne Martin): a broad index of the same concepts with further links.