Performance Debugging: USE, RED and Flame Graphs

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

Performance debugging fails most often from lack of method, not lack of tools. Two checklists cover most ground: RED for services and USE for resources.
RED asks, per service: Rate of requests, Errors, and Duration (latency distribution). It tells you which service is slow and since when.
USE asks, per resource (CPU, memory, disk, network, locks, pools): Utilisation, Saturation (queued work), and Errors. It tells you which resource is the bottleneck.
A CPU flame graph, built from stack samples, shows where on-CPU time goes. Width is the share of samples; the x-axis is not time.
Latency is on-CPU time plus waiting. When the CPU profile looks innocent, measure off-CPU time: locks, I/O, scheduler queues and throttling.

How it works

Start with RED to find the slow thing

Tom Wilkie devised RED in 2015 because USE fits hardware better than request-driven services, and he recommends using both as two views of one system (Grafana Labs). Before profiling anything, establish which service and which operation regressed, and how. A p99 that rose while p50 stayed flat suggests contention, queueing or a slow subset of requests. A shift of the whole distribution suggests more work per request. Rising Rate with rising Duration suggests saturation from load; flat Rate with rising Duration suggests something changed in the code, the data or a dependency. RED on each hop of the call graph locates the slow service; the slowest hop's own breakdown says whether it is slow itself or waiting on its own dependency.

Then USE on that service's resources

For each resource the service uses, check three things (Gregg, USE method). Utilisation: how busy it is over an interval. Saturation: how much work is waiting, such as run-queue length for CPU, I/O queue depth for disks, pending acquires for a connection pool, or swap activity for memory. Errors: failed operations, like disk errors, dropped packets or rejected connections. Saturation is the most important column. A resource at 60% average utilisation can still be saturated in bursts, and saturation is what adds latency.

USE applies to software resources too: thread pools, connection pools, mutexes and file descriptors all have utilisation, a queue and failure modes.

Profile where the time goes

If the bottleneck is CPU, sample stacks. A sampling profiler interrupts each CPU at a fixed rate, records the current stack, and counts identical stacks. A flame graph draws those counts: each box is a function, its width is the fraction of samples in which it appears, and children sit above their callers. Look for wide plateaus at the top (functions burning CPU themselves) and for wide towers under unexpected callers (a logging call that takes 20% of CPU). Box order along the x-axis is alphabetical, so adjacent boxes are not related in time (Gregg).

Off-CPU time

A thread waiting on a lock, a disk read, a network response or a CPU quota is not on a CPU, so the CPU profile cannot see it. Off-CPU analysis records the stack when a thread blocks and the duration until it runs again. Together, on-CPU and off-CPU profiles account for all of a request's wall time.

Down the stack

Layer 1The contract

The human process promises coverage: every resource on the request path is checked for utilisation, saturation and errors before anyone declares "it's not the infrastructure". Testable: the investigation notes contain a USE row per resource with a number or "not measurable". The profiling tools have contracts too. A sampling profiler at 99 Hz for 30 seconds on 8 CPUs collects about 23,800 samples, so a function at 1% of CPU appears in roughly 240 of them, which is enough to trust its width to a few tenths of a percent. Anything that runs for less than one sample period per call is only visible statistically.

Layer 2The mechanism

On Linux, perf record -F 99 -a -g opens one perf event per CPU with the perf_event_open syscall, configured to overflow at 99 Hz on a clock or cycle counter. On overflow, the kernel captures the instruction pointer and walks the call chain, usually by following frame pointers (each frame saves the caller's frame pointer, forming a linked list on the stack). Records go into a ring buffer that perf reads through mmap. perf script prints one stack per sample, leaf first, and a fold step (Brendan Gregg's stackcollapse scripts) reverses each stack to root first, joins frames with semicolons and counts duplicates: main;handle;validate;regexp.match 812. The flame graph renderer draws those lines.

Stacks break when code is compiled without frame pointers, which many distributions did for years to free a register. That default has been reversing: Fedora was the first distribution to re-enable them and Ubuntu 24.04 LTS ships with them on, and Gregg's overhead tests found the cost usually under 1%, with one deep-stack Java application at about 10% (Gregg, 2024). The fix is to rebuild with -fno-omit-frame-pointer, use DWARF unwinding (--call-graph dwarf, which copies stack bytes and is much heavier), or hardware branch records. JIT runtimes need a symbol map, such as /tmp/perf-<pid>.map, written by the JVM agent or Node's --perf-basic-prof.

Off-CPU profiling usually uses eBPF: a program on the sched:sched_switch tracepoint records the time and stack ID when a thread leaves the CPU, and the time it returns, summing blocked time per stack in a BPF map. bcc's offcputime tool does this.

Layer 3The machine

99 Hz rather than 100 Hz is deliberate: it avoids sampling in lockstep with activity driven by 100 Hz timers. At this rate overhead is usually well under 1% of CPU. CPU saturation shows up as the run-queue length (r in vmstat 1, which counts both waiting and running threads on current Linux) above the CPU count (Gregg, USE for Linux), and on modern kernels as pressure stall information in /proc/pressure/cpu and each cgroup's cpu.pressure (kernel cgroup v2 docs). In containers, the CFS bandwidth controller enforces CPU limits per period, by default 100 ms (kernel scheduler docs); on cgroup v2 the knob is cpu.max, written as quota period in microseconds with default max 100000. A container limited to 2 CPUs that runs 8 threads can use its 200 ms of quota in the first 25 ms of a period and then sit throttled for 75 ms. Average utilisation looks fine while p99 latency jumps by the throttled time; nr_throttled and throttled_usec in the cgroup's cpu.stat reveal it (cgroup v1 reports throttled_time in nanoseconds instead; cgroup v2 docs). Throttling can also appear with usage far below the limit: Linux 4.18 to 5.3 era kernels expired each CPU's leftover quota every period, which was fixed in 5.4 and backported (see the Indeed write-up below).

Below the scheduler, hardware counters explain why code is slow even when it is on-CPU. Instructions per cycle (IPC) under about 1 usually means stalls on memory: a last-level cache miss costs on the order of 100 ns, during which a modern core could have retired hundreds of instructions. perf stat -e cycles,instructions,cache-misses reads these counters.

Why it's built this way

The space of possible performance causes is huge, so methods like RED and USE give a finite checklist that guarantees coverage.
Tracing every function call is too expensive in production, so profilers sample at a low fixed rate and rely on statistics.
Thousands of unique stacks are unreadable as text, so flame graphs merge them visually by frequency.
CPU sampling cannot see waiting, so off-CPU analysis hooks the scheduler to account for the rest of wall time.

Diagram

flowchart TD
    R["RED: which service and operation regressed?"] --> U["USE per resource: utilisation, saturation, errors"]
    U --> C{"Bottleneck on CPU?"}
    C -->|"yes"| F["perf record at 99 Hz, fold stacks, flame graph"]
    C -->|"no"| O["Off-CPU: eBPF on sched_switch, lock and I/O waits"]
    F --> H["Hot path in code"]
    O --> W["Blocking cause: lock, disk, quota, dependency"]

Worked example

A profile service's p99 rose from 80 ms to 400 ms after a release; p50 went from 20 ms to 35 ms. RED: Rate flat at 1,200 requests per second, Errors flat, Duration up. So more work per request, not more requests.

USE on CPU: the pods show 55% average utilisation of their 2-CPU limit, which looks fine. But cpu.stat shows nr_throttled rising in 40% of periods, and throttled_usec grows by about 300 ms per second. The average hides bursts that exhaust the quota early in each 100 ms period.

A 30-second CPU profile across one pod (99 Hz, 2 CPUs, about 5,900 samples) folded into a flame graph shows 36% of samples under validateEmail, almost all in regex compilation. The release moved regexp.MustCompile inside the request handler, so every request compiles the same pattern, costing about 0.6 ms of CPU each. At 600 requests per second per pod that is 360 ms of CPU per second, close to a fifth of the pod's quota, spent on repeated work. Moving the compile to package initialisation drops CPU per request back to normal, throttling stops, and p99 returns to 85 ms.

The CPU profile found the hot code; USE explained why a modest CPU increase produced a five-fold tail latency increase.

In the wild

Cloudflare, 2019: one regex pegs every CPU. A new WAF rule contained a regular expression that backtracked catastrophically and shipped to the whole network in seconds. About 80% of traffic was lost for 27 minutes with 502s everywhere. A CPU safeguard had been removed in an earlier refactor; afterwards Cloudflare restored it, added CPU profiling to its tests, and moved toward RE2 or Rust's regex engine (Cloudflare).
Indeed, 2019: throttled while idle. Indeed saw worst-case response latency above two seconds in an application that was throttled even though it used far less CPU than its limit (one mostly idle app was allocated 4.1 CPUs and stayed under 0.5). The cause was a 4.18 kernel commit that expired each CPU's leftover quota every period, so on many-core hosts the stranded milliseconds added up. After the fix, worst-case latency in one application fell to 30 ms, and the fix is in mainline 5.4 and stable backports such as 4.14.154 and 4.19.84 (Indeed Engineering, part 2).

Trade-offs

Tool	Strengths	Weaknesses	Choose when
RED dashboards	Fast localisation across services	Says where, not why	First minutes of any latency problem
USE checklist	Systematic, catches saturation	Needs metrics for every resource	Suspected infrastructure bottleneck
CPU flame graph (`perf`, pprof)	Precise hot code, low overhead	Blind to waiting, needs symbols and stacks	CPU-bound regressions
Off-CPU profile (eBPF)	Shows locks, I/O and throttling	Higher overhead at high switch rates, needs privileges	High latency with low CPU
Distributed tracing	Per-request breakdown across services	Sampled, coarse inside a process	Cross-service latency

Pitfalls in production

Trusting average utilisation. Fix: look at saturation (run queue, PSI, throttling, pool waits) and short-interval data.
Broken stacks from missing frame pointers or JIT symbols, giving a flame graph of [unknown]. Fix: build with frame pointers and enable perf maps for JITs.
Profiling only on-CPU when the problem is lock contention or I/O. Fix: pair every CPU profile with an off-CPU or mutex profile.
Comparing p99 across hosts by averaging percentiles. Fix: aggregate histograms, then compute quantiles.
Benchmarking on a laptop with a different CPU, core count and no cgroup limits. Fix: profile in production or in an environment with the same limits.

Leadership lens

Make continuous profiling (low-rate always-on sampling) standard; regressions are found by diffing profiles between versions.
Require RED dashboards per service and USE metrics for every pool and queue you own, not just hosts.
Review container CPU limits with throttling data. Many tail-latency problems are a quota set too tight.
When an agent suggests a performance fix, ask for the before and after profile and the latency distribution, not only the mean.

Check yourself

p99 latency is up 5x while average CPU is 50%. What do you check first, and why?
Show answer
Saturation, especially CFS throttling and run-queue length. Averages hide bursts, and queueing during bursts drives tail latency.
Why does the x-axis of a flame graph not represent time?
Show answer
Stacks are merged by identity and sorted alphabetically at each level, so width shows frequency and position carries no temporal meaning.
A handler takes 300 ms but its CPU flame graph shows only 20 ms of work. Where is the rest?
Show answer
Off-CPU: waiting on locks, I/O, downstream calls or CPU quota. An off-CPU or tracing view is needed.
Why sample at 99 Hz rather than 100 Hz?
Show answer
To avoid aliasing with periodic activity at 100 Hz, which would bias samples toward or away from that work.

Sources and further study

Primary

Gregg, The Return of the Frame Pointers (2024): which distributions re-enabled frame pointers and what it costs.
Indeed Engineering, Unthrottled: Fixing CPU Limits in the Cloud: a worked case of throttling at low usage, found with a reproducer and git bisect.

Systems docs

Linux kernel: CFS bandwidth control: period, quota, slice and the throttling counters.
Linux kernel: cgroup v2: cpu.max, cpu.stat and cpu.pressure.
Gregg, USE Method: Linux checklist: the concrete saturation metrics per resource.
Grafana Labs: The RED Method: Wilkie's definition and how it pairs with USE.

Books

Systems Performance, Brendan Gregg (2nd ed.), chapter 2 "Methodologies" and chapter 6 "CPUs": USE, workload characterisation and CPU analysis.
Site Reliability Engineering, Beyer et al., chapter 6 "Monitoring Distributed Systems": the four golden signals that overlap with RED.

Other curricula

donnemartin/system-design-primer: its "benchmark and profile" paragraph points at load tests and the slow query log, with no CPU or off-CPU method.

Watch

Blazing Performance with Flame Graphs (Brendan Gregg, USENIX LISA13): how flame graphs work and how to read them.
Linux Systems Performance (Brendan Gregg, USENIX LISA19): methodologies and tools including USE, perf and eBPF.
Linux Performance Tools, part 1 (Brendan Gregg): a tour of observability tools by layer of the stack.

Free resources

The USE Method (Brendan Gregg): the method and a Linux checklist per resource.
Flame Graphs (Brendan Gregg): types of flame graphs and how to generate them.
Off-CPU Analysis (Brendan Gregg): measuring blocked time with scheduler tracing.
perf wiki: the Linux perf tool's documentation and tutorial.
Profiling Go Programs (Go blog): pprof CPU and memory profiles applied to a real program.

Open in the interactive app: practice, quiz, mock interview →