How a Computer Runs Your Code: CPU, Memory and Disk

In one minute

• A CPU does one simple thing very fast: fetch an instruction, decode it, execute it, repeat, billions of times a second. Every program, from a shell script to a model server, ends up as that loop.
• Data lives in a hierarchy: registers, CPU caches, main memory (RAM), then disk or SSD, then the network. Each step down is bigger and cheaper and roughly 10 to 1000 times slower.
• Most "slow code" is not slow arithmetic. It is waiting: for memory, for disk, for the network. Knowing the rough latency numbers lets you guess where time goes before you profile.
• Caches work because programs reuse data (temporal locality) and touch nearby data (spatial locality). Code that walks memory in order is often several times faster than code doing the same work in a scattered order.
• The same shape repeats at every level of the stack: a small fast store in front of a big slow one. CPU caches, a page cache, Redis in front of a database and a CDN in front of an origin are one idea in four domains.

How it works

From source code to instructions

You write Python, TypeScript or Go. Something turns it into machine instructions the CPU understands. Go is compiled ahead of time to a native binary. TypeScript compiles to JavaScript, which a runtime such as V8 interprets and then just-in-time compiles hot functions to machine code. CPython compiles your file to bytecode and runs it in an interpreter loop, which is itself a compiled C program. The paths differ, but at the bottom a CPU core is executing machine instructions such as "load this address into a register", "add two registers", "jump if zero".

The fetch-decode-execute loop

A core keeps a program counter: the address of the next instruction. It fetches the instruction from memory, decodes what it means, executes it (an arithmetic operation, a memory load or store, or a branch), and moves on. Modern cores overlap many instructions in a pipeline, run several at once, guess which way branches will go and execute ahead speculatively. A 3 GHz core can finish several instructions per nanosecond when its data is ready. When the data is not ready, it waits.

The memory hierarchy

The exact numbers vary by hardware and year; the ratios are what matter. The figures above match the interactive table by Colin Scott, which extrapolates a classic list forward by year and holds the data-centre round trip fixed (Colin Scott's interactive latency table). The older 2012 list that many people quote has L1 at 0.5 ns, L2 at 7 ns, a 4 KB SSD read at 150 µs and a disk seek at 10 ms (Bonér's latency gist). Treat both as orders of magnitude and measure your own machine. If an L1 hit were one second, a RAM access would be about two minutes, an SSD read a few hours to a day and a cross-continent round trip two to five years.

Caches and locality

The CPU never fetches a single byte from RAM. It fetches a cache line, usually 64 bytes, and keeps it in cache (Drepper, What Every Programmer Should Know About Memory, section 3). If the next access falls in the same line, it is almost free. If the program jumps around memory, every access can miss and pay the RAM price. Two kinds of locality make caches work:

• Temporal locality: recently used data is likely to be used again soon (a loop counter, a hot config object).
• Spatial locality: data next to recently used data is likely to be used soon (the next element of an array).

Arrays and slices are laid out contiguously, so walking them in order is cache friendly. Linked lists, maps of pointers and objects scattered across the heap are not. Hardware prefetchers also spot simple sequential or fixed-stride access and fetch lines before the program asks for them, which is why sequential code often looks nearly free; Drepper devotes section 6.3 to prefetching and shows in section 6.2.1 how reordering a matrix multiply to walk memory sequentially cuts its run time sharply.

Disk, files and the page cache

Disks and SSDs store data in blocks of a few KB. The operating system keeps recently read file blocks in RAM in the page cache, so the second read of a file is often a memory read, not a disk read. Writes usually land in the page cache first and reach the disk later, when kernel flusher threads write out data that has been dirty long enough or when dirty memory passes a threshold (Linux vm sysctl docs). fsync blocks until the file's data and metadata are on the device, including flushing the drive's own write cache, but it does not make the file's directory entry durable: for a newly created file you must also fsync the directory (fsync(2)). That is why databases call fsync on commit, and why "it was written" and "it survives a power cut" are different claims.

A failed write-back is the nasty case. After a buffered write fails at the hardware level, Linux filesystems typically drop the affected pages and mark them clean, so a later fsync can report success for data that never reached the disk (LWN: PostgreSQL's fsync() surprise). Error reporting improved in Linux 4.13 and later, and PostgreSQL now defaults data_sync_retry to off, so a failed flush crashes the server and recovers from the write-ahead log instead of retrying (PostgreSQL docs).

The same shape everywhere

A small, fast, expensive store in front of a big, slow, cheap one, with a policy for what to keep: that is the L1 cache, the OS page cache, an application cache, Redis, and a CDN. Hit rate, eviction, invalidation and warm-up are the same questions at every level. Learn them once here and you will recognise them in every system design interview.

Down the stack

Diagram

flowchart LR
  CPU["CPU core: registers"] --> L1["L1 cache, about 1 ns"]
  L1 -->|miss| L2["L2 cache, about 4 ns"]
  L2 -->|miss| L3["L3 cache, tens of ns"]
  L3 -->|miss| RAM["RAM, about 100 ns"]
  RAM -->|page fault| SSD["SSD, tens of µs"]
  SSD -.->|remote data| NET["Network, 0.5 ms and up"]

Worked example

A service computes a daily report by summing a 10,000 by 10,000 grid of 8-byte numbers (800 MB) stored row by row. A new engineer writes the loop with the column index outside and the row index inside. It takes about 9 seconds. Swapping the loops so the code walks each row in order brings it under 1 second, with the same arithmetic.

Why: in row order, each 64-byte cache line holds 8 consecutive numbers, so one RAM fetch serves 8 additions, and the hardware prefetcher sees the sequential pattern and fetches ahead. In column order, consecutive accesses are 80,000 bytes apart, so every access touches a new line (and a new 4 KB page, which also strains the TLB), and by the time the loop comes back to a line it has often been pushed out of L1 and sometimes out of the whole cache. Almost every addition pays a deep-cache or RAM latency. The 9 and 1 second figures are illustrative; the gap on real hardware depends on cache sizes and associativity, so measure rather than trust them. Drepper's matrix example in section 6.2.1 shows the same effect with measured numbers (Drepper).

The same team later finds the report reading 2 GB of input files. The first run of the day takes 40 seconds; reruns take 6. The difference is the page cache: the first run reads from SSD, later runs read file blocks already in RAM. The fix is to read the files once, in large sequential chunks, rather than to buy a faster disk.

A back-of-envelope check helps here: 100 million random RAM accesses at about 100 ns each is 10 seconds; 100 million sequential cache-friendly reads is a fraction of a second. You can make that estimate before writing any code.

In the wild

• PostgreSQL community, 2018: fsync errors that Linux swallowed. In March 2018 a report to the pgsql-hackers list showed that PostgreSQL could lose data with no error: its checkpointer often opened files before calling fsync, and on Linux a failed write-back could be dropped as clean pages or consumed by another caller, so a later fsync returned success. The thread led to Linux error-reporting changes and to PostgreSQL treating a failed flush as a reason to crash and replay the WAL (LWN, PostgreSQL docs).
• NATS JetStream, 2025: acknowledged before durable. Jepsen found that JetStream 2.12.1 acknowledged publishes immediately but fsynced only every two minutes by default. A simulated power failure lost 131,418 of 930,005 messages, roughly 30 seconds of writes. NATS documented the hazard rather than changing the default (Jepsen).

Trade-offs

Pitfalls in production

• Assuming the database or network call is fast because it is fast on your laptop. Fix: know the latency numbers and measure in the real environment.
• Calling a remote service inside a loop (the N+1 pattern). Fix: batch the requests; one round trip instead of a thousand.
• Benchmarking with a warm cache and deploying to a cold one. Fix: measure both cold and warm runs.
• Trusting that a write is durable before fsync, or that a successful fsync covers the new file's directory entry. Fix: use a database, or fsync both the file and its directory, and treat an fsync error as fatal rather than retrying.
• Running out of RAM so the OS starts swapping and latency jumps a thousand times. Fix: set memory limits and alert on swap and page faults; Brendan Gregg's Systems Performance (2nd ed., Ch 7: Memory) covers the memory methodology and the Linux tools for it.

Leadership lens

• Ask "where does the time go: CPU, memory, disk or network?" before anyone optimises code.
• Teach the latency table; a team that knows the ratios makes better design choices without a profiler.
• Treat every cache, from CPU to CDN, as the same pattern with the same review questions: hit rate, eviction, invalidation, cold start.
• Prefer batching and sequential access in designs; they are the cheapest performance wins.
• Ask what a storage acknowledgement means: "written" to the page cache, to the drive cache, or to stable media. Many data-loss stories are a gap between those.

Check yourself

1. Why is iterating over an array usually faster than iterating over a linked list of the same length?

   Show answer

   The array is contiguous, so each cache line fetched from RAM holds several elements and the prefetcher can load ahead; list nodes are scattered, so each step can be a cache miss costing about 100 ns.

2. Roughly how much slower is a RAM access than an L1 cache hit, and a data-centre round trip than a RAM access?

   Show answer

   About 100 times for RAM versus L1, and about 5,000 times for a 0.5 ms round trip versus a 100 ns RAM access.

3. Why does the second run of a file-processing job often finish much faster than the first?

   Show answer

   The OS page cache kept the file blocks in RAM after the first read, so the second run reads from memory instead of disk.

4. What do a CPU cache and a CDN have in common?

   Show answer

   Both are small fast stores in front of a big slow one, rely on locality of access, and face the same questions of hit rate, eviction, invalidation and cold start.

Sources and further study

Primary

• Ulrich Drepper, What Every Programmer Should Know About Memory: cache lines, associativity, prefetching and the matrix-walk experiments (sections 3 and 6).
• Colin Scott, interactive latency numbers: the latency table by year, with its extrapolation assumptions visible in the page source.
• Bonér, Latency numbers every programmer should know: the 2012 list most people quote.

Systems docs

• Linux kernel vm sysctl documentation: dirty-page thresholds, flusher threads and swappiness.
• fsync(2) man page: what fsync guarantees, the directory caveat and error semantics.
• PostgreSQL error-handling settings: data_sync_retry and why retrying a failed flush is unsafe.

Books

• Operating Systems: Three Easy Pieces (Arpaci-Dusseau), Ch 13 Address Spaces, Ch 18 Paging, Ch 19 TLBs, Ch 21 to 22 Swapping, Ch 37 Hard Disk Drives, Ch 44 Flash-based SSDs: free online at the OSTEP site.
• Systems Performance, 2nd ed. (Brendan Gregg), Ch 6 CPUs, Ch 7 Memory, Ch 8 File Systems, Ch 9 Disks: methodology for finding which of these resources is the bottleneck.

Other curricula

• System Design Primer: lists latency numbers as an appendix and treats caching at the client, CDN, web server, database and application levels, which is the same hierarchy idea at larger scale.

Watch

• The Fetch-Execute Cycle: What's Your Computer Actually Doing? (Tom Scott): a clear walk through what a CPU does every cycle.
• Fetch Decode Execute Cycle in more detail (Computer Science Lessons): registers, the program counter and the cycle step by step.

Free resources

• Latency numbers every programmer should know (Jonas Bonér): the classic table of rough latencies.
• Interactive latency numbers by year (Colin Scott): how those numbers changed over time.
• Operating Systems: Three Easy Pieces: a free book on virtualisation, concurrency and persistence.
• What Every Programmer Should Know About Memory (Ulrich Drepper): the deep dive on caches and RAM.
• CPU cache (Wikipedia): cache lines, associativity and replacement.
• Teach Yourself Computer Science: a curated self-study path for the fundamentals.