Following a request across services

11 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

A trace records the path of one request through many services as a tree of spans. Each span is a timed operation with a parent, attributes, and status.
Context propagation carries the trace ID and parent span ID across process boundaries, usually in the W3C traceparent header or message metadata.
Instrumentation (OpenTelemetry SDKs and auto-instrumentation) creates spans; a collector batches, samples, and exports them to a backend such as Jaeger, Tempo, or a vendor.
Volume forces sampling. Head sampling is cheap but blind to outcome; tail sampling keeps errors and slow traces but needs buffering.
Traces answer "where did this request spend its time" and "which dependency failed".

How it works

Spans and the trace tree

A span has a 16-byte trace_id shared by the whole trace, an 8-byte span_id (both random by default in OpenTelemetry SDKs (OTel trace SDK spec)), a parent ID, a name like GET /cart, start and end times, attributes, and a status. Its kind is server, client, producer, consumer, or internal. A client span in service A and the matching server span in service B show network and queueing time as the gap between them.

Propagation

When service A calls B, the SDK injects a header:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

The fields are version, trace ID, parent span ID, and flags, where the lowest bit says whether this trace is sampled (W3C Trace Context). Trace Context Level 2, still a Candidate Recommendation draft, adds a second flag (02) declaring that the right-most 7 bytes of the trace ID are random, so downstream systems can sample or shard on them consistently (W3C Trace Context Level 2). B extracts it, creates a server span with that parent, and passes context to its own outgoing calls. For queues, the context goes into message headers, and the consumer span may link to the producer span rather than be its child, since one batch can contain messages from many traces. Any hop that drops the header breaks the trace into disconnected pieces.

Collection pipeline

SDKs batch spans and export them over OTLP (port 4317 for gRPC, 4318 for HTTP; the spec's default transport is http/protobuf (OTLP exporter spec)) to a collector tier, which enriches, scrubs, samples, and fans out to backends. Backends have converged on the same pipeline: Jaeger v2, released in November 2024, is built on the OpenTelemetry Collector and ingests OTLP natively (CNCF). Backends index spans by trace ID plus a few searchable fields such as service, operation, duration, and error.

Sampling

Head sampling decides at the root, for example keep 1 percent, and propagates the decision via the sampled flag so every service agrees. It is cheap and consistent but drops 99 percent of rare errors. Dapper ran Google-wide at one trace in 1,024 and found that enough for high-volume services but not for low-traffic ones, which pushed it toward adaptive sampling by target rate (Dapper paper). In OpenTelemetry the ratio sampler TraceIdRatioBased is now deprecated in favour of ProbabilitySampler, because its exact algorithm was never specified, so it is safe only at the root (OTel trace SDK spec). Tail sampling buffers all spans of a trace in a collector (keyed by trace ID, so all spans of a trace must reach the same collector instance) and decides after the trace completes: keep all errors, all traces over 2 seconds, and 1 percent of the rest. It costs memory and a load-balancing layer that routes by trace ID. The Collector's tail sampling processor waits 30 seconds (decision_wait) and holds 50,000 traces (num_traces) by default, and its docs prescribe a first tier with the load-balancing exporter in front of the sampling tier (tail sampling processor).

Down the stack

Layer 1The contract

Tracing promises far less than people assume. If every hop propagates context and the trace is sampled, the backend will eventually hold a tree of spans sharing one trace ID, with parent links and per-host timestamps. It does not promise completeness (SDKs drop spans when buffers fill), ordering across hosts (clocks differ), or timeliness (spans arrive seconds to minutes after the request). Instrumentation promises bounded overhead: a few microseconds per span and no blocking of the request path on export.

Layer 2The mechanism

In the OpenTelemetry SDK, starting a span reads the current context (thread-local in Java, contextvars in Python, context.Context passed explicitly in Go), generates a random 64-bit span ID, and records a monotonic start time. Ending the span hands it to a BatchSpanProcessor: a bounded queue (2,048 spans by default) drained by a background worker that exports up to 512 spans per batch every 5 seconds or when the batch fills, with a 30-second export timeout (OTel trace SDK spec). If the queue is full, new spans are dropped and a counter increments, which is how the SDK keeps the request path non-blocking. The propagator writes traceparent on outgoing requests and parses it on incoming ones; the sampler decision is the 01 flag. Export uses OTLP over gRPC or HTTP: a protobuf ExportTraceServiceRequest grouping spans by resource and instrumentation scope. Tail-sampling collectors keep a map from trace ID to buffered spans and a timer per trace; a load-balancing exporter in front hashes the trace ID onto a ring of sampler instances.

Layer 3The machine

Creating a span costs roughly 0.5 to 2 microseconds and allocates a few hundred bytes; a service handling 20,000 requests per second with 10 spans each allocates around 40 to 80 MB per second of short-lived objects, which in garbage-collected runtimes shows up as more frequent young-generation collections. Timestamps come from a monotonic clock read via vDSO in about 20 to 30 nanoseconds without a syscall. Export batches of 512 spans compress to tens of KB, so network cost is small, but each OTLP export is a gRPC call over HTTP/2 that the collector must decode, which makes protobuf parsing a large share of collector CPU. Cross-host timestamps rely on NTP, which typically keeps hosts within 1 to 10 milliseconds; that is why a 2 ms child span can appear to start before its parent.

Why it's built this way

Requests cross many processes, so identity must travel with the request in headers.
Tracing must never slow down or break the request, so spans are buffered, exported asynchronously, and dropped under pressure.
Recording every span of every request is too expensive to store, so sampling decides what to keep.
The interesting traces are known only after they finish, so tail sampling buffers whole traces at a cost in memory and routing.

Diagram

sequenceDiagram
  participant U as Client
  participant G as API gateway
  participant C as Cart service
  participant P as Pricing service
  participant X as Collector
  U->>G: GET /cart
  G->>C: traceparent trace T, parent g1
  C->>P: traceparent trace T, parent c2
  P-->>C: price list
  C-->>G: cart
  G-->>U: 200 OK
  G->>X: span g1 (root)
  C->>X: spans c1, c2
  P->>X: span p1
  X->>X: assemble trace T, tail sample

Worked example

An online store handles 8,000 requests per second at the edge. Each request fans out to about 25 spans across 9 services, so the fleet produces 200,000 spans per second. At roughly 500 bytes per span after OTLP encoding, that is 100 MB/s, about 8.6 TB per day.

Keeping everything for 7 days would mean 60 TB, too much for the budget. Head sampling at 2 percent cuts storage to about 170 GB per day but keeps only 2 percent of the 0.3 percent of requests that fail, so engineers rarely find the trace they need.

Tail sampling instead: a load-balancing collector tier routes spans by trace ID to 12 sampling collectors. Each waits 10 seconds after the last span of a trace before deciding. Buffered data is 100 MB/s times 10 seconds, about 1 GB spread across the tier. Policy: keep 100 percent of traces with an error, 100 percent over 1.5 seconds (about 1 percent of traces), and 2 percent of the rest. Kept volume is roughly 0.3 + 1 + 2 percent, about 3.3 percent, so 285 GB per day and 2 TB for a week, while every failing and slow request is available.

In the wild

Google, 2010: Dapper shows that sampling makes always-on tracing affordable. In a web search cluster, tracing every request added 16.3 percent to average latency, while sampling 1 in 16 or fewer was within measurement error, and the production default of 1 in 1,024 still gave enough data for high-volume services. The collection daemon never used more than 0.3 percent of one core. Low-traffic services missed events at that rate, so Google moved toward adaptive sampling by target traces per second (Dapper paper).
Uber, 2021: critical-path analysis over Jaeger traces. With a few thousand microservices and close to 100,000 operations, single trace views were unreadable, so Uber built CRISP to compute the critical path of each trace and aggregate it at p50, p95 and p99. One finding: an operation on an important critical path was about 15 percent faster at p50 on a 2.4 GHz CPU SKU than on a 2.2 GHz one, with roughly 1.5 times worse tail latency on the slower SKU (Uber).
Slack, 2020: traces as rows you can query. Slack flattened spans into a simple SpanEvent record (ID, parent, trace ID, timestamp, duration, tags) and loaded about 310 million traces and 8.5 billion spans a day into a real-time store and a data warehouse. Large services were sampled at 1 to 2 percent, some low-volume services at 100 percent, and a slash command could trace every request from one client for two minutes when debugging a customer issue (Slack).

Trade-offs

Option	Strengths	Weaknesses	Choose when
Head sampling	Simple, cheap, consistent across services	Misses rare errors and slow outliers	Very high volume, aggregate latency analysis
Tail sampling	Keeps interesting traces by outcome	Collector memory, trace-ID routing, decision delay	You need every error and slow trace
Keep everything	Complete data, no sampling bias	Storage and ingest cost	Low traffic or short retention

Pitfalls in production

Broken propagation. A proxy, thread pool, or async callback loses context and traces split into fragments. Use instrumented clients and executors, and test propagation in CI with a synthetic request.
Clock skew. Hosts disagree by milliseconds and children appear to start before parents. Trust durations within a host, and keep NTP healthy.
Sensitive data in attributes. SQL statements and URLs include emails or tokens. Scrub in the collector and set allow-lists for attributes.
High-cardinality span names. Naming spans GET /users/12345 defeats aggregation. Use route templates like GET /users/{id}.
Tail sampler overload. A burst exceeds collector memory and spans are dropped silently. Monitor dropped spans and scale the collector tier on throughput.

Leadership lens

Make propagation a platform responsibility: shared libraries, gateways, and message clients must carry context by default.
Decide the sampling policy with product and on-call input: which traces must never be lost?
Link signals: put trace IDs in logs and exemplars on latency histograms so alerts lead to traces in one click.

Check yourself

Why must all spans of a trace reach the same collector for tail sampling?
Show answer
The decision depends on the whole trace, such as any span erroring or total duration, so one instance must see every span before deciding.
What does the gap between a client span and its server span tell you?
Show answer
Time spent outside the callee's handler: network latency, connection setup, and queueing before the server started processing.
Why use span links rather than parent-child for batch consumers?
Show answer
A batch contains messages from many traces, and a span can have only one parent, so links connect the consumer span to each producer without forcing a false hierarchy.

Sources and further study

Primary

W3C Trace Context Level 2: the draft that adds the random trace-ID flag on top of Level 1.
OpenTelemetry trace SDK specification: ID generation, samplers (including the TraceIdRatioBased deprecation) and batch processor defaults.
OTLP exporter specification: transports, ports and environment variables every SDK honours.

Systems docs

OpenTelemetry tail sampling processor: policies, defaults and the two-tier scaling layout.
Jaeger v2 released (CNCF): why a major tracing backend rebuilt itself on the OpenTelemetry Collector.

Books

Observability Engineering, Majors, Fong-Jones and Miranda, chapter 6 "Stitching Events into Traces" and chapter 17 "Cheap and Accurate Enough: Sampling".
Building Microservices (2nd edition), Sam Newman, chapter 10 "From Monitoring to Observability": correlation IDs and tracing as the step after log aggregation.

Other curricula

Azure Architecture Center: monitoring and diagnostics: frames tracing as debugging operations across services, held together by correlation IDs in every record.
AWS Builders' Library: instrumenting distributed systems: the latency and status of every dependency call recorded in each request log, a log-based complement to traces.
donnemartin/system-design-primer: lists the Dapper paper but has no tracing section.

Watch

Why tracing is hard (Ben Sigelman): a Dapper and OpenTelemetry co-creator on the practical difficulties.
Keynote: OpenTracing and Containers (Ben Sigelman, CNCF): depth versus breadth in tracing and where it is heading.

Free resources

Dapper, a Large-Scale Distributed Systems Tracing Infrastructure: Google's paper that defined spans, trees, and low-overhead sampling.
W3C Trace Context: the standard traceparent and tracestate headers.
OpenTelemetry traces: spans, kinds, links, and the data model.
OpenTelemetry sampling: head versus tail sampling with configuration guidance.

Open in the interactive app: practice, quiz, mock interview →