Engineering Atlas › System Design Fundamentals

Building systems that keep working correctly

14 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Faults, errors and failures

A fault is a defect or event: a disk sector goes bad, a bug ships, an operator runs a command on the wrong cluster. An error is the wrong internal state the fault produces. A failure is when the system visibly stops meeting its specification. Reliability engineering lives in the gap between the first and the last: tolerate faults, detect errors, and stop them before they become failures.

Faults come in three families. Hardware faults (disks, memory, power supplies, NICs) are mostly independent and random. Software faults (bugs, leaks, bad config, a leap second) are correlated: the same bug fires on every node at once. Human faults, mostly configuration changes, cause a large share of major outages in published postmortems. Each family needs a different defence: redundancy for hardware, staged rollouts and isolation for software, and guard rails plus automation for humans.

Measuring reliability

The constant-rate model is a simplification. Hardware follows a bathtub curve: high early failure (infant mortality), a flat middle, then wear-out. Software has its own version: most incidents follow a change. Google's SRE team puts it at roughly 70% of outages (Google SRE book), which is why change management is a reliability tool.

Treat MTTR as a rough guide, not a precise score. A team has a handful of incidents a quarter and their durations vary wildly, so one long outage moves the mean more than any real improvement. A Monte Carlo study from Google found MTTR-style metrics "poorly suited for decision making or trend analysis" for incidents (Google SRE). DORA has also replaced its old "time to restore service" key with failed deployment recovery time, which counts only recovery from bad changes (DORA). Use the formula to reason about design, and judge a team on SLO impact and incident trends instead.

MTBF versus MTTR

You can make failures rarer or recovery faster. Rarer failures come from better components, testing and review, and they hit diminishing returns quickly; you will never make a fleet of disks stop failing. Faster recovery comes from detection, automation and rollback, and it scales. Even preparation counts: Google reports roughly a 3x MTTR improvement when on-call engineers have a written playbook instead of improvising (Google SRE book). A service that has an incident every 30 days and takes 4 hours to recover is at 99.45%. Cut recovery to 30 minutes and it reaches 99.93% with the same number of incidents.

Reliability as a budget

An SLO turns reliability into a number the team can spend. At 99.9% over 30 days, the error budget is 0.1% of requests. While budget remains, the team ships; when it is gone, the team spends its time on reliability work (Google SRE book). This makes the trade-off between velocity and stability explicit instead of a recurring argument. The Alert Monitoring lesson covers how to page on budget burn.

Trust but verify

The problem shape behind reliability is trust but verify: assume every layer can silently corrupt or drop data, and check. A payments system reconciles its ledger against the bank's settlement file every day. A storage system checksums every block and scrubs disks in the background to find rot before a read does. An AI coding agent's output is run against tests and reviewed before merge, because "the tool call returned success" is not evidence that the change is correct. The pattern is the same everywhere: define an invariant, check it independently of the component that claims it, and alert on divergence.

Down the stack

Layer 1The contract

Reliability contracts separate kinds of promise. Correctness: an acknowledged write is never lost and reads return what was written. Durability: for example, S3 is designed for 99.999999999% annual durability of objects, a design target rather than a refund-backed SLA (AWS docs). Service level: "99.9% of requests succeed within 300 ms over 30 days." A good contract also states failure semantics: on timeout, is the operation retried, and is it safe to retry? Without that, callers cannot build reliable systems on top.

Layer 2The mechanism

Detection is built from checksums and heartbeats. HDFS stores a CRC32C checksum for every 512 bytes of a block and verifies it on every read; a background block scanner re-reads every block, by default once every 504 hours (three weeks), and re-replicates any corrupt replica (HDFS defaults). ZFS stores each block's checksum in its parent block pointer, so a block cannot vouch for itself, and zpool scrub walks the whole tree (Zhang et al., FAST 2010). PostgreSQL keeps a 16-bit pd_checksum in each 8 KB page header and reports mismatches instead of returning garbage. Older guides say you must opt in with initdb --data-checksums; since PostgreSQL 18, initdb enables checksums by default and --no-data-checksums turns them off (PostgreSQL 18 release notes). Containment uses idempotency keys so retries are safe, bulkheads so one failing dependency cannot take all threads, and automated rollback when canary error rates rise. Recovery uses replicas, point-in-time backups and runbooks that are exercised, not just written.

Layer 3The machine

Hardware fails more often and more quietly than people expect. Consumer drives specify an unrecoverable read error rate around 1 in 10^14 bits, about one bad read per 12.5 TB, so rebuilding a large RAID array can hit one. Google's 2.5-year study of DRAM in its fleet found more than 8% of DIMMs affected by errors each year, orders of magnitude above lab estimates (Schroeder et al., 2009); ECC (SECDED) corrects single-bit errors and detects double-bit ones, which is why servers use it. The 2018 "fsyncgate" discovery showed that when fsync fails on Linux, the kernel may drop the dirty pages and clear the error, so a retry of fsync "succeeds" while the data is gone; PostgreSQL changed to panic and recover from the WAL instead of retrying, first in version 12 and backpatched as far as 9.4 (PostgreSQL wiki). Silent corruption also comes from NICs, hosts and switches: in Internet traces, about 1 packet in 30,000 failed the TCP checksum, and the authors estimated the 16-bit checksum lets roughly 1 bad packet in 200 million through (Stone and Partridge, SIGCOMM 2000). At petabyte scale that is a steady stream, so systems that care add end-to-end checksums at the application layer.

Why it's built this way

  1. Components fail at a fixed rate, and fleets are large, so failure is a continuous condition rather than an exception.
  2. Lower layers can fail silently, so each layer that cares about correctness checks data end to end instead of trusting the layer below.
  3. Making failures rarer has limits, so the main investment goes into fast detection and automated recovery.
  4. Most outages follow a change, so changes are staged, canaried and reversible.
  5. Perfect reliability is infinitely expensive, so an explicit budget decides how much reliability work is enough.

Diagram

stateDiagram-v2
    [*] --> Healthy
    Healthy --> Faulty: fault occurs
    Faulty --> Detected: checksum, heartbeat or SLO alert
    Faulty --> Failure: not detected in time
    Detected --> Mitigated: failover, rollback, retry
    Failure --> Mitigated: incident response
    Mitigated --> Repaired: replace or fix root cause
    Repaired --> Healthy

Worked example

A storage service runs 2,500 servers with 12 disks each, 30,000 disks in total. At a 1.2% annual failure rate, close to the 1.36% Backblaze measured across about 344,000 drives in 2025 (Backblaze), it loses about 360 disks a year, roughly one per day, and servers with a 4-year MTBF add about 1.7 server failures a day. Each object has 3 replicas on different racks.

The risk is not a single failure but a second and third before repair. If re-replicating one dead 16 TB disk takes 10 hours at 450 MB per second, the window in which another replica of the same data can fail is short. Spreading each disk's data across many peers (so 500 disks each copy 32 GB in parallel) cuts the window to about 2 minutes, and the chance of losing all three replicas drops by orders of magnitude. The same thinking drives the software side: the team found that 70% of its incidents in the past year followed a deploy, with a median recovery of 50 minutes. Adding automated canary analysis with rollback in under 5 minutes did more for its numbers than any hardware change.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Invest in MTBF (testing, review, better parts)Fewer incidents and less customer painDiminishing returns, slows deliveryFailures are costly and hard to undo, such as data loss
Invest in MTTR (detection, rollback, automation)Scales, improves every incidentUsers still see brief failuresMost online services
End-to-end checksumsCatches silent corruptionCPU and storage overheadAny system storing data you cannot regenerate
Redundancy across failure domainsSurvives independent hardware faultsCost, no help with correlated bugsHardware and zone failures
Staged rollouts and canariesContains software and config faultsSlower rolloutEvery change to production

Pitfalls in production

Leadership lens

Check yourself

  1. Why does adding replicas protect against disk failures but not against most outages?

    Show answer

    Replicas cover independent hardware faults. Most large outages come from software or configuration changes that hit every replica at once, which needs staged rollouts and fast rollback instead.

  2. A service fails once a month and takes 2 hours to recover. Which is cheaper to improve, and why?

    Show answer

    Usually recovery time. Better detection, runbooks and automated rollback can cut 2 hours to minutes, while halving incident frequency needs broad changes across code and process.

  3. Why does ZFS store a block's checksum in the parent pointer rather than in the block itself?

    Show answer

    A misdirected or phantom write could replace the block and its checksum together, still looking valid. Keeping the checksum in the parent lets an independent record vouch for the data.

  4. What is the chance that a component with a 1,000-hour MTBF survives a 100-hour job, assuming a constant failure rate?

    Show answer

    e^(−100/1000) ≈ 90.5%, so roughly one run in ten will see a failure and needs checkpointing or retry.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →