Latency, throughput and bandwidth compared

14 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

Latency is how long one operation takes, from request to response. It is a distribution, not a number, so describe it with percentiles (p50, p99, p99.9).
Throughput is how many operations complete per unit of time: requests per second, messages per second, bytes per second actually delivered.
Bandwidth is the maximum rate a channel can carry, such as a 10 Gbps link. Throughput can never exceed it and is usually well below it.
Little's law ties them together: work in flight = throughput x latency. To get more throughput at the same latency, you need more concurrency.
As utilisation approaches 100%, queues form and latency climbs steeply. Most systems trade a little throughput for predictable latency by running with headroom.

How it works

Three different questions

On a highway, bandwidth is the number of lanes, latency is how long one car takes from on-ramp to exit, and throughput is how many cars pass the exit per hour. Adding lanes raises the throughput ceiling but makes no single car faster.

Little's law

For any stable system, the average number of items inside it equals the arrival rate times the average time each spends inside: L = λ x W. A service handling 2,000 requests per second at 50 ms average latency has about 100 requests in flight at any moment. This is why thread pools, connection pools and TCP windows all have to be sized from both numbers. If the pool holds only 50, the service cannot exceed 1,000 requests per second at that latency, no matter how idle the CPU looks.

Utilisation and queueing

A server that takes 10 ms per request can, in theory, do 100 per second. In practice, requests arrive in bursts, and when one arrives while the server is busy, it waits. For a simple single-server queue, average time in the system is service_time / (1 - utilisation). At 50% utilisation, 10 ms becomes 20 ms. At 90%, it becomes 100 ms. At 99%, a full second. Throughput rose by under a factor of two while latency rose fifty-fold. This curve is why capacity plans target 60 to 70% utilisation for latency-sensitive services.

Batching trades one for the other

Sending one message per network round trip wastes the fixed cost of each trip. Grouping 100 messages into one write raises throughput dramatically but makes the first message wait for the batch to fill. Kafka producers expose this directly with linger.ms and batch.size; databases do it with group commit; GPUs do it with inference batches. Batching does not always cost latency, though. Kafka 4.0 raised the default linger.ms from 0 to 5 ms because, under load, larger batches mean fewer requests queueing at the broker, so producer latency often stays the same or drops (Kafka docs). The right setting depends on whether the consumer cares about the time of each item or the total volume.

Tails dominate at scale

If a page fans out to 100 backends and each has a 1% chance of a slow response, the page is slow 1 - 0.99^100 ≈ 63% of the time (Dean and Barroso, 2013). The average backend latency is irrelevant; the p99 of each backend becomes the median of the page. The same paper shows the fix can be cheap: in a Bigtable test, sending a backup request after 10 ms cut the 99.9th-percentile latency from 1,800 ms to 74 ms for only 2% more requests. This is the fan-out shape. The same maths applies to an AI agent that calls ten tools in parallel and to a checkout page waiting on pricing, inventory and fraud checks: the slowest dependency sets the experience.

Down the stack

Layer 1The contract

Performance contracts are stated as a latency percentile at a throughput: "p99 under 200 ms at 5,000 requests per second, with error rate under 0.1%." A latency number without a load level is meaningless, because latency depends on load. A bandwidth number from a provider is a ceiling on raw bits, before protocol overhead (Ethernet, IP and TCP headers take about 3 to 5% at 1,500-byte packets) and before retransmissions.

Layer 2The mechanism

TCP throughput on one connection is bounded by window / RTT. The receiver advertises a window (scaled by the window-scale option, up to 1 GiB (RFC 7323)), and the sender keeps a congestion window that starts small (10 segments, about 14.6 KB, as specified in RFC 6928 and used by Linux) and grows each round trip during slow start. To fill a link, the window must cover the bandwidth-delay product: 1 Gbps x 80 ms = 10 MB in flight. Linux caps this with net.ipv4.tcp_rmem and tcp_wmem. The receive maximum stayed at 6 MB from 2012 until Linux 6.16 raised it to 32 MB, because receivers on long paths were capped by it (kernel commit); the send maximum still defaults to at most 4 MB (kernel docs). Congestion control (CUBIC by default, now a Standards Track spec in RFC 9438, which replaced RFC 8312; or BBR) decides how fast to grow and how hard to back off on loss. Above TCP, HTTP/2 multiplexes many requests on one connection to avoid repeated handshakes.

Layer 3The machine

Light in fibre travels about 200,000 km per second, so every 1,000 km adds about 5 ms one way. New York to London (about 5,600 km) can never be faster than roughly 56 ms round trip (High Performance Browser Networking). Serialisation delay is the time to put bits on the wire: a 1,500-byte packet takes 12 µs at 1 Gbps and 1.2 µs at 10 Gbps. Inside a machine, an L1 cache hit costs about 1 ns, main memory about 100 ns, an NVMe read about 20 to 100 µs. The classic 500 µs for a round trip within a data centre is a 2000s figure: the fabric itself now takes a few microseconds, and the kernel, scheduler wake-ups and RPC stack add tens of microseconds or more on top (Barroso et al., 2017). NICs coalesce interrupts (ethtool -C rx-usecs) and the kernel merges packets with GRO, which raises throughput by doing less work per packet at the cost of a few microseconds of added latency per packet: the batching trade-off again, in hardware.

Why it's built this way

Distance has a floor set by the speed of light, so latency between regions cannot be engineered away, only avoided with caches and replicas closer to users.
Every operation has a fixed cost (a round trip, a syscall, a disk seek), so systems batch work to spread that cost, trading latency for throughput.
A pipe is only full if enough data is in flight to cover its delay, so windows and pools are sized by Little's law and the bandwidth-delay product.
Queues grow without bound as utilisation nears 100%, so systems keep headroom and shed load rather than run hot.

Diagram

sequenceDiagram
    participant C as Client
    participant S as Server (RTT 80 ms)
    C->>S: SYN
    S-->>C: SYN-ACK (80 ms gone)
    C->>S: Request + first 10 segments
    Note over C,S: Slow start: window doubles each RTT
    S-->>C: ACKs (160 ms)
    C->>S: 20 segments
    S-->>C: ACKs (240 ms)
    C->>S: 40 segments ... until window covers 10 MB in flight
    Note over C,S: Steady state: throughput = window / RTT, capped by bandwidth

Worked example

A team copies a 1 GB backup from a data centre in Virginia to one in Frankfurt. The link is 1 Gbps and the round trip is 80 ms. Their naive estimate is 8 seconds.

With an old 64 KB TCP window and no window scaling, one connection can only have 64 KB in flight per 80 ms, so throughput is 64 KB / 0.08 s = 800 KB/s, about 6.4 Mbps. The copy takes over 20 minutes on a link that is 99% idle. The link's bandwidth was never the limit; latency times window was.

The bandwidth-delay product is 1 Gbps x 0.08 s = 80 Mbit = 10 MB. Raising tcp_rmem and tcp_wmem maximums to 16 MB lets the window grow to fill the pipe. On Linux 6.16 or later the receiver may already allow 32 MB, but the sender's 4 MB default still caps the flow at about 50 MB/s, so check both ends. Slow start needs about 10 round trips (under a second) to reach 10 MB, and the copy then finishes in roughly 9 seconds. A tool that opens 8 parallel connections achieves the same with smaller windows, which is why aws s3 cp and most bulk-transfer tools use multipart, parallel uploads (the AWS CLI defaults to 8 MB parts and 10 concurrent requests (AWS CLI docs)).

The same reasoning applies to the API in front of that system: at 3,000 requests per second and 40 ms average latency, Little's law says about 120 requests are in flight. With a database pool of 64, it plateaued near 1,600 requests per second while CPU sat at 35%, because requests queued for a connection.

In the wild

Slack, 2021: Little's law in an outage. On the first workday of 2021 an AWS Transit Gateway did not scale fast enough, and packet loss sharply raised latency from the web tier to its backends. With each request taking longer, web servers needed more Apache workers in flight for the same throughput; CPU fell because threads were waiting, which first triggered automated downscaling. Then worker utilisation hit its limit, autoscaling tried to add 1,200 servers, and Slack was down or degraded from about 7:00 to 10:40 PST until AWS raised the gateway's capacity (Slack).
Discord, 2020: tail latency every two minutes. The Read States service kept a huge LRU cache in Go. Go forces a garbage collection at least every 2 minutes, and scanning the cache produced regular latency and CPU spikes even though average latency looked fine. A Rust rewrite removed the spikes and let each server hold 8 million entries (Discord).
Canva, 2024: a stalled fetch becomes a burst. A stale CDN route sent traffic over a lossy path, and request coalescing held about 270,000 browsers on one slow JavaScript file. When it finally loaded, they all hit the API together at about 1.5 million requests per second, three times the normal peak, and the gateway fell over for about 52 minutes. An accidental batch turned a latency problem into a throughput problem (Canva).

Trade-offs

Lever	Improves	Costs	Choose when
Batching	Throughput, cost per item	Latency of each item	Logs, analytics, bulk writes
More concurrency (pools, connections)	Throughput	Memory, contention, more load on dependencies	Latency is fine but throughput is capped by in-flight limits
Caching and replicas near users	Latency	Staleness, cost, invalidation work	Read-heavy, geographically spread users
Running with headroom	Tail latency	Idle capacity, money	User-facing, latency-sensitive paths
Hedged requests	Tail latency	Extra load (typically 2 to 5%)	Fan-out reads against replicated data
More bandwidth	Ceiling on throughput	Money	Measured utilisation is near the link limit

Pitfalls in production

Reporting averages. A 30 ms average hides a 2-second p99. Report percentiles, and aggregate them from histograms, not by averaging per-host percentiles; precomputed quantiles cannot be combined (Prometheus docs).
Coordinated omission. Load generators that wait for a response before sending the next request stop sending during stalls and under-count slow requests. Use open-loop load tests at a fixed arrival rate that measure from when each request should have been sent (wrk2).
Buying bandwidth for a latency problem. A bigger link does nothing for a transfer limited by window size or round trips. Compute the bandwidth-delay product first.
Pools sized by habit. A pool smaller than throughput x latency caps throughput and adds queueing delay. Size from Little's law and check under load.
Running hot. At 90% utilisation, small bursts cause large latency spikes. Alert on utilisation and saturation, not only on errors.
Batch settings tuned once. A linger.ms that suits peak traffic adds pointless delay at night. Use size-or-time triggers and revisit them when traffic patterns change.

Leadership lens

Insist that every performance target names a percentile, a load level and an error budget; "fast" is not a requirement.
Ask which resource is the bottleneck before approving spend on bandwidth, bigger instances or more nodes. Often it is a pool limit or round trips.
Decide per workload whether latency or throughput matters more, and make the batching and headroom choices explicit in the design doc.
Treat tail latency as a product decision: hedging and headroom cost money, and the business should choose the trade knowingly.

Check yourself

A service handles 500 requests per second with 200 ms average latency. How many requests are in flight, and what happens if the worker pool has 50 slots?
Show answer
About 100 are in flight by Little's law. With 50 slots, throughput caps at 250 per second at that latency and the rest queue, so latency rises.
Why can't a single TCP connection fill a 10 Gbps link across a 100 ms round trip with a 1 MB window?
Show answer
Throughput is window divided by RTT, so 1 MB / 0.1 s = 10 MB/s, about 80 Mbps. Filling the link needs a window of about 125 MB or many parallel connections.
A page calls 50 backends in parallel, each slow 1% of the time. Roughly how often is the page slow?
Show answer
1 - 0.99^50 ≈ 39% of the time, so the backends' p99 becomes the page's typical experience.
Why does batching raise throughput but hurt latency?
Show answer
It spreads a fixed per-operation cost over many items, but each item waits for the batch to fill or for the flush timer before it is sent.

Sources and further study

Primary

Dean and Barroso, The Tail at Scale (CACM 2013): fan-out maths and hedged, tied and canary requests.
Barroso et al., Attack of the Killer Microseconds (CACM 2017): why microsecond-scale networks and storage are wasted on millisecond-era software.
RFC 7323: TCP Extensions for High Performance: window scaling and timestamps for long fat networks.
RFC 9438: CUBIC for Fast and Long-Distance Networks: the default congestion control, now on the Standards Track.

Systems docs

Linux kernel: IP sysctl: tcp_rmem, tcp_wmem and the other knobs that bound per-connection throughput.
Apache Kafka: producer configs: linger.ms and batch.size and why the default changed.
Prometheus: Histograms and summaries: how to record latency so percentiles stay correct after aggregation.

Books

High Performance Browser Networking, Ilya Grigorik, chapter 2 "Building Blocks of TCP": slow start, the bandwidth-delay product and head-of-line blocking (read free).
Systems Performance (2nd ed.), Brendan Gregg, chapter 2 "Methodologies": latency, utilisation and saturation as working terms, with the USE method in appendix A.
Designing Data-Intensive Applications (2nd ed.), Kleppmann and Riccomini, chapter 2 "Defining Nonfunctional Requirements": the "Describing Performance" section on percentiles and tail latency.

Other curricula

donnemartin/system-design-primer: defines latency and throughput in two lines and advises maximal throughput with acceptable latency; no queueing or bandwidth-delay maths.
roadmap.sh System Design: has a "Latency vs Throughput" node that links to short introductory articles and an MIT lecture.

Watch

Latency vs Throughput: System Design Essentials (Be A Better Dev): the two metrics with examples of when each matters.
Latency Numbers Programmer Should Know (ByteByteGo): the order-of-magnitude costs of memory, disk and network operations.

Free resources

Primer on Latency and Bandwidth (High Performance Browser Networking): the components of network delay and why latency, not bandwidth, usually limits the web.
The Tail at Scale (Google Research): why fan-out makes rare slow responses common, and hedged requests as a fix.
Latency numbers every programmer should know (Jonas Bonér): the classic table of operation costs.
Bandwidth-delay product (Wikipedia): why long, fat networks need large windows.
Little's law (Wikipedia): the relation between throughput, latency and concurrency.
The USE Method (Brendan Gregg): utilisation, saturation and errors as a checklist for finding the real bottleneck.

Open in the interactive app: practice, quiz, mock interview →