Debugging Distributed Systems

In one minute

• In a distributed system, a symptom in one service often has its cause in another. The first job is to find where the problem is, before working out what it is.
• Follow the request, not the server: a trace ID propagated through every hop lets you reconstruct one request's path, timing and errors across dozens of services.
• Expect partial and gray failures: a dependency that is slow rather than down, one bad host out of fifty, a network path that drops 1 percent of packets. Averages hide them; break metrics down by host, zone, version and dependency.
• Retries and timeouts change the shape of failures: they can turn a brief slowdown into a retry storm that keeps the system down after the trigger is gone (a metastable failure).
• Clocks disagree across machines, so do not trust cross-host timestamps for ordering within milliseconds; use trace parent-child structure and logical ordering instead.

How it works

Where before what

A user sees a slow checkout. The checkout service's latency is up, but checkout calls inventory, pricing, payments and a fraud check, each of which calls others. Debugging checkout's code first is often wasted time. Localise first, as Google's troubleshooting chapter advises: stabilise the service, then ask what changed recently and bisect the system by checking the communication paths on each side of a split (SRE book, ch. 12). Which hop added the latency or produced the error? Distributed tracing answers this directly; without it, compare each dependency's latency and error metrics from the caller's side and the callee's side.

Traces

A trace is the tree of spans for one request. Each span records a service, an operation, start and end times, status and attributes, plus its parent span. The W3C Trace Context standard defines the traceparent header, four dash-separated fields (version, a 16-byte trace ID, an 8-byte parent span ID and flags), that carries them across HTTP calls (W3C Recommendation, 2021); message queues carry the same in message headers. Google's Dapper paper (2010) built its tracing around sampling and a small set of instrumented common libraries, which kept overhead low enough to run everywhere (Dapper). Two analyses matter most: self time (time a span spent in its own code, excluding its children) and the critical path (the chain of spans that determined the total duration; speeding up anything off it does not help).

Partial and gray failures

Distributed systems rarely fail cleanly. A Microsoft Research paper (HotOS 2017) named this "gray failure" and traced it to differential observability: failure detectors may not notice problems that applications plainly suffer (Gray Failure). Examples: a host whose disk is slow, a service that answers health checks but times out on real work. Signs: errors or latency concentrated on one host, zone, version or client. Always split your metrics: by host, availability zone, deploy version, dependency and customer. A 2 percent error rate overall might be 100 percent on one of fifty hosts.

Timeouts, retries and metastability

Timeouts and retries protect against transient failures but change system behaviour under stress. If every layer retries three times, a failure at the bottom of a five-layer stack can multiply load by 3^5 = 243 times. A brief slowdown causes timeouts, timeouts cause retries, retries add load, load keeps latency high, and the system stays down after the original trigger is gone. A HotOS 2021 paper called these metastable failures: a trigger pushes the system into a bad state that persists after the trigger is removed, because a sustaining effect, often work amplification, keeps goodput low. Retries are one of the most common sustaining mechanisms, and failures that end when the trigger ends, such as a plain DoS attack, are not metastable (Bronson et al.). Their example: a database that handles 300 queries a second, an app at 280 QPS with one retry per request, and a 10-second network blip. The retries double the offered load to 560 QPS, so the database never recovers until load drops or retries stop. The fixes are retry budgets (retry only if fewer than, say, 10 percent of recent requests were retries), exponential backoff with jitter, retrying at one layer only, and load shedding.

Clocks and ordering

Each machine's clock drifts; NTP typically keeps servers within a few milliseconds, sometimes worse. Two log lines from different hosts 2 ms apart may be in the wrong order. Use the trace's parent-child structure for causality, use per-key sequence numbers where order matters, and treat cross-host timestamp comparisons below the clock error as unknown.

The same shape everywhere

Localising a fault in a chain of dependencies is the same problem as finding the slow step in a business process or the bottleneck in a supply chain: measure each stage's own contribution, find the critical path, split by segment to find concentrated problems.

Down the stack

Diagram

flowchart TD
  C["checkout 420 ms"] --> I["inventory 30 ms"]
  C --> P["pricing 45 ms"]
  C --> PAY["payments 330 ms"]
  PAY --> F["fraud-check 300 ms"]
  F --> DB["fraud DB 290 ms"]
  DB -.->|"critical path"| R["root cause candidate"]

Worked example

Checkout p99 latency rises from 400 ms to 2.5 seconds at 14:10, and the error rate from 0.2 to 3 percent. No deploys in the last six hours. The checkout team's dashboards show nothing wrong in their own code.

Opening 20 slow traces, the critical path in 18 of them runs through payments, then fraud-check, where the fraud-check span shows 2 seconds on the payments (client) side but only 40 ms on the fraud-check (server) side. The time is between the services. Splitting fraud-check's client metrics by destination host shows it: 1 of 12 fraud-check hosts has p99 of 2 seconds; the other 11 are normal. That host's ss -ti output shows retransmissions on 4 percent of segments. The cloud provider later confirms a degraded network card.

Meanwhile, the error rate keeps climbing even after the bad host is removed at 14:25: payments retries fraud-check 3 times, and checkout retries payments 2 times, so slow calls produced up to 6 times the normal load on fraud-check, whose queues are now full. The team sheds non-essential traffic (fraud scoring for low-value orders falls back to rules) for 10 minutes, the queues drain, and latency recovers. Follow-ups: retry only at the payments layer, a 10 percent retry budget, outlier detection to eject slow hosts automatically, and tail-based sampling so every slow trace is kept.

In the wild

• AWS DynamoDB, 2015: retries sustaining an overload. A brief network disruption made some storage servers miss their window to fetch membership data from the metadata service, and responses had grown slow because rapid adoption of Global Secondary Indexes enlarged membership lists. Servers that timed out dropped out and kept retrying, which slowed responses for healthy servers too; errors reached about 55% by 2:37 AM PDT. Engineers only got relief by pausing metadata requests at 5:06 AM, and most customers recovered by 7:10 AM. AWS added metadata capacity, tightened monitoring of membership size, and lowered the request rate from storage nodes (AWS summary).
• Healthchecks.io, 2025: a safeguard that made it worse. When one load balancer ran short of connection slots, it was automatically removed from DNS rotation, which shifted its load onto the others and started a chain reaction. The lesson for debugging: check that a protective action reduces load rather than moving it (post-mortem).

Trade-offs

Pitfalls in production

• Debugging the service where the symptom appears. Fix: localise with traces and per-dependency metrics first.
• Averages across hosts. Fix: split by host, zone and version; look at the worst, not the mean.
• Retries at every layer. Fix: retry at one layer, with backoff, jitter and a retry budget.
• Trusting cross-host timestamps for ordering. Fix: use trace structure or sequence numbers.
• Trace context lost at queues and thread pools. Fix: propagate context through message headers and async boundaries, and test it.

Leadership lens

• Make trace propagation and per-dependency metrics part of the service template, so every new service gets them.
• Set a company-wide retry and timeout policy: who retries, how many times, with what budget and deadlines.
• Add outlier detection and automatic ejection of bad hosts in load balancers or service meshes.
• In postmortems, record how long it took to localise the fault; it is the metric observability work improves most.

Check yourself

1. A client span shows 2 seconds and the matching server span shows 40 ms. Where did the time go?

   Show answer

   Between the two: network delays or retransmissions, connection setup or pool waits, or queueing before the server started processing.

2. Why can retries at every layer turn a short outage into a long one?

   Show answer

   They multiply load on the failing component exponentially with depth, keeping it overloaded after the original trigger has gone.

3. Why follow the child that finished last when computing a critical path?

   Show answer

   The parent cannot finish before that child, so it is the one that determines the parent's end time; speeding up earlier-finishing children would not help.

4. What is a gray failure, and how do you spot it?

   Show answer

   A component that looks healthy to its own checks but is degraded for real work; you spot it by splitting metrics by host, zone or version and comparing callers' views.

Sources and further study

Primary

• AWS: DynamoDB service event, September 2015: metadata timeouts and retries in the provider's words.
• Healthchecks.io post-mortem, November 2025: connection-slot exhaustion and a harmful automatic safeguard.

Books

• Site Reliability Engineering (Google), Ch 12: Effective Troubleshooting, for the hypothesis-and-test loop and bisecting a system.
• Observability Engineering (Majors, Fong-Jones, Miranda), Ch 6: Stitching Events into Traces, and Ch 8: Analyzing Events to Achieve Observability.
• Release It!, 2nd ed. (Nygard), Ch 5: Stability Patterns (Timeouts, Circuit Breaker, Bulkheads), for the controls that stop retry amplification.

Other curricula

• Google's SRE book frames debugging as repeated hypothesis and test, starting from recent changes and bisecting the system, which is the order this lesson uses: localise, then diagnose.

Free resources

• Dapper, a Large-Scale Distributed Systems Tracing Infrastructure (Google): the paper behind modern tracing.
• Traces (OpenTelemetry): spans, context propagation and attributes.
• Trace Context (W3C): the traceparent header standard.
• Gray Failure: The Achilles' Heel of Cloud-Scale Systems (Microsoft Research): failures that health checks do not see.
• Metastable Failures in Distributed Systems (HotOS 2021, PDF): why systems stay down after the trigger is gone.
• Timeouts, retries, and backoff with jitter (AWS Builders' Library): retry budgets and jitter in practice.