Engineering Atlas › Problem Solving & Incidents (AI era)

First Principles and Mental Models

13 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Analogy versus first principles

Reasoning by analogy says "Company X uses Kafka for this, so should we" or "last time latency spiked it was the database". It is fast and often right, but imports assumptions you never checked. First-principles reasoning asks what must be true given the constraints: bytes, round trips, operations per second per component, and their costs. It is slower, so use it when stakes are high, analogies conflict, or the usual answer has failed.

A practical recipe:

  1. State the goal as a number (requests per second, latency, cost per month).
  2. List the physical steps the work requires (network round trips, disk reads, CPU work, serialisation).
  3. Put a cost on each from known numbers or a quick benchmark.
  4. Add them up and compare with the goal or the observed behaviour.
  5. Investigate the largest gap first.

Models worth carrying

Using models while debugging

A model gives you a falsifiable expectation. "The service handles 3,000 requests per second on 12 cores at 2 ms of CPU each, so CPU should be 50% busy." If CPU shows 95%, either requests cost more than you think (profile them) or something else is consuming CPU (GC, a noisy neighbour, a hot loop). Without the model, 95% is just a number.

Down the stack

Layer 1The contract

A good mental model promises a prediction within a stated error, under stated assumptions, that you can test. Little's law promises exactness for long-run averages of any stable system. An M/M/1 queue formula promises a good approximation only when arrivals are roughly random and service times roughly exponential; real services with heavy-tailed service times queue worse. The discipline is to state the model's assumptions alongside its answer, so the person reading your estimate knows when to stop trusting it.

Layer 2The mechanism

Queueing models are the workhorses. For c identical servers, arrival rate λ and mean service time S, the offered load is A = λS (in Erlangs) and utilisation is ρ = A / c. The Erlang C formula (see Erlang (unit))) gives the probability that an arriving request has to wait, and from it the mean wait Wq = C × S / (c − A). It is computed stably with the Erlang B recurrence B(k) = A·B(k−1) / (k + A·B(k−1)), then C = c·B / (c − A(1 − B)). Call centres have used this for a century; it applies equally to thread pools, database connection pools and Kubernetes replica counts. Amdahl's law (speedup = 1 / (s + (1 − s)/n)) and the Universal Scalability Law, C(N) = N / (1 + α(N−1) + βN(N−1)), cover parallel scaling. α is contention and β is coherency cost; with β = 0 it reduces to Amdahl's law, and with β > 0 throughput peaks near √((1−α)/β) and then falls (Gunther, USL). Gunther's own caution applies: when data diverge from the model, check the measurements before blaming the model.

Layer 3The machine

The numbers in the models come from hardware. Memory moves in 64-byte cache lines, so two threads writing different variables on the same line (false sharing) bounce it between cores at roughly 50 to 100 ns a time. A DRAM access that misses every cache costs about 80 to 120 ns, so pointer-chasing through 10 million nodes takes on the order of a second while a sequential scan of the same bytes takes milliseconds, thanks to the prefetcher. A 4 KiB random read on NVMe takes tens of microseconds; an fsync takes from tens of microseconds to several milliseconds depending on whether the drive has power-loss protection. A 10 Gbit/s NIC moves at most about 1.25 GB/s, and an in-region TCP round trip is around 0.1 to 0.5 ms. When a measurement breaks these orders of magnitude, suspect the measurement or a hidden step: a DNS lookup, a new connection per request, a retry.

Why it's built this way

  1. Systems are made of physical resources with fixed speeds, so cost can be estimated before code is written.
  2. Shared resources form queues, and queueing delay grows sharply near saturation, so utilisation targets sit well below 100%.
  3. Estimates are cheap and measurements are expensive, so a model tells you what to measure and what number to expect.
  4. Analogies carry hidden assumptions, so when the stakes are high, reasoning from constraints is the safer default.

Diagram

flowchart TD
    G["Goal or observation as a number"] --> D["Decompose into physical steps"]
    D --> C["Cost each step: ns, ms, bytes, ops/s"]
    C --> M["Combine with a model: Little, queueing, Amdahl"]
    M --> P["Prediction"]
    P --> X{"Matches measurement?"}
    X -->|"yes"| U["Model holds: act on it"]
    X -->|"no"| H["Hidden step or wrong assumption: investigate the gap"]
    H --> D

Worked example

A checkout service calls a payment provider with a mean latency of 120 ms. Traffic peaks at 400 requests per second. The team's HTTP client pool is sized at 40 connections, and during peak, p99 checkout latency jumps from 300 ms to 4 s. An agent suggests raising timeouts.

Little's law first: 400 × 0.12 s = 48 connections busy on average. A pool of 40 cannot serve that at all; utilisation would be 120%, so the queue grows without bound until timeouts shed load. That fully explains the 4 s. Raising timeouts would make it worse by holding requests longer.

How big should the pool be? Erlang C with A = 48: at 50 connections (96% busy) about 69% of requests wait, 42 ms on average; at 56 (86%) about 19% wait, 3 ms on average; at 64 (75%) about 2% wait. The team picks 64, then checks the provider's concurrency limit (say 100 per account) so they don't trade one queue for rejected calls. Peak p99 returns to around 320 ms. The analysis took 15 minutes and needed no profiler. Note the model assumes exponential service times; the provider's real tail is heavier, which is one more reason to stay at 75% rather than 86%.

In the wild

Trade-offs

ApproachStrengthsWeaknessesChoose when
Reason by analogyFast, uses team experienceImports unchecked assumptionsFamiliar problems with low stakes
First principles with napkin mathCatches impossible designs early, cheapCan miss second-order effectsCapacity, cost and latency questions
Formal model (queueing, simulation)Quantifies tails and saturationAssumptions may not hold; takes timeExpensive capacity decisions, SLO commitments
Measure firstGround truthWithout a model, numbers lack meaningAlways, but with a prediction written down beforehand

Pitfalls in production

Leadership lens

Check yourself

  1. A service receives 1,200 requests per second and each holds a database connection for 25 ms. What is the minimum pool size, and why go higher?

    Show answer

    Little's law gives 30 connections busy on average. Running at 100% utilisation means unbounded queueing, so size well above 30 (for example 40 to 45) using a queueing model or load test.

  2. CPU sits at 85% and latency is bad, but a teammate says there's 15% headroom. What do you say?

    Show answer

    Queueing delay grows like ρ/(1−ρ), so at 85% waiting is already several times the service time, and bursts push it higher. Utilisation headroom isn't latency headroom.

  3. Why can a pointer-chasing loop be 100 times slower than a sequential scan over the same data?

    Show answer

    Each dependent pointer load can miss cache and wait about 100 ns for DRAM, while sequential access lets the prefetcher stream cache lines at full memory bandwidth.

  4. When should you prefer reasoning by analogy?

    Show answer

    When the problem is familiar, the decision is cheap to reverse and the analogy's assumptions plainly hold; first principles cost more time and are best spent on high-stakes or novel decisions.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →