Logs, metrics and traces together

In one minute

• Metrics, logs and traces store the same events with different amounts of pre-aggregation. Metrics drop identity to stay cheap, logs keep the detail of one process, traces keep the shape of one request across processes.
• Metrics tell you that something is wrong and how badly, traces tell you where in the request path, logs tell you why at the line of code.
• The value is in moving between them in one click. Shared resource attributes, a trace ID on every log line, and exemplars on latency histograms are the joins.
• Cost scales differently: metrics by series, logs by bytes, traces by spans kept after sampling.
• "Three pillars" is a useful inventory and a poor architecture. Treat them as views over well-structured events, not three tools owned by three teams.

How it works

One event, three shapes

Take one checkout request that fails because the payment provider timed out after 2 seconds. There are three classic shapes for recording it.

A metric turns it into increments: http_server_requests_total{route="/checkout",status="502"} goes up by one, and the 2.1 second duration lands in the le="2.5" bucket of a latency histogram. The request ID, the customer and the error message are gone. What remains is tiny and fast to query over months.

A log line keeps what the code chose to say at one point in one process: {"level":"error","msg":"payment authorise failed","provider":"acme","timeout_ms":2000,"trace_id":"4bf9..."}. It is rich but local, blind to the 11 other services on the same request unless they share an ID.

A trace keeps the request's shape: a tree of timed spans. It shows that checkout waited on payments.authorise, which waited 2,000 ms on an outbound call, while inventory and pricing finished in 15 ms. It is sampled, so most requests have no trace at all. Google's Dapper sampled about one trace in 1,024 and found that still gave enough data for high-volume services, while sampling every request added 16 percent to average latency in their test (Dapper paper).

A database analogy: logs are the raw table, metrics are a materialised view with GROUP BY route, status already applied, and traces are a precomputed join across services on trace_id. This is the finding things fast problem shape: every observability backend is an index chosen in advance for the queries you expect at 3 a.m.

What each pillar is good at

Metrics are for detection and trend: the only signal cheap enough to cover every request, so alerts, SLOs and capacity plans live here (see the Metrics and Instrumentation and Alert Monitoring lessons). Google's four golden signals, latency, traffic, errors and saturation, are all metrics (Google SRE book). They cannot explain an outlier, because it was summed into a bucket.

Traces are for localisation: which hop grew, whether calls ran in sequence or in parallel, where retries stacked up. The Distributed Tracing lesson covers propagation and sampling.

Logs are for explanation: the exception, the SQL that timed out, the flag value, the config version. They are also all most third-party software emits, so they never go away. Amazon's guidance is one request log entry per unit of work, carrying the timings and dependency results for that request, which makes a log line behave like a small wide event (AWS Builders' Library).

The joins are the product

Three disconnected tools give you an engineer comparing timestamps by eye. The links that matter:

• Resource attributes. Every series, log record and span from a process carries the same service.name, service.version, k8s.pod.name and deployment.environment.
• Trace context in logs. The logger reads the active span and stamps trace_id and span_id on every record, so one click from a span lists its log lines across services.
• Exemplars. A histogram bucket keeps one sample observation with its trace ID, so a dot on the p99 panel opens a real slow trace.
• Span-derived metrics. The collector's spanmetrics connector computes rate, errors and duration from spans before sampling, emitted as traces.span.metrics.calls and traces.span.metrics.duration, so traces and RED dashboards agree (spanmetrics connector).

Beyond the pillars

Charity Majors and Ben Sigelman argue that the pillars framing confuses data types with capabilities. Their alternative is wide structured events: one record per request per service with dozens of fields (route, user tier, build ID, cache hit, flag values) in a columnar store that can group by any field at query time. Metrics become queries over events and traces become events with parent IDs. That answers high-cardinality questions ("only beta-build customers in ap-southeast-1 on Safari?") that pre-aggregated metrics cannot. The cost is more bytes per request than a counter, so most organisations keep metrics for alerting and long retention, and sampled traces plus structured logs (ideally one wide line per request) for investigation. Majors later named the split-silo approach "observability 1.0" and the single store of wide events "observability 2.0", arguing that 1.0 costs grow as a multiple of traffic because every signal stores the event again (charity.wtf).

The list of signals is also growing. OpenTelemetry's continuous profiling signal entered public alpha in 2026, and profile samples carry trace_id and span_id, so a slow span can open a flame graph of the code that ran during it (OpenTelemetry blog).

The same trade appears in a bank: account balances (aggregate), a transaction ledger (raw), and fraud case files that stitch transactions across accounts (the trace). The questions decide the shapes; the IDs that join them decide whether investigation takes minutes or days.

Down the stack

Diagram

flowchart LR
  APP["Service + OTel SDK"] -->|"OTLP"| COL["Collector"]
  COL -->|"spanmetrics"| M["Metrics TSDB"]
  COL -->|"tail sample"| T["Trace store"]
  COL --> L["Log store"]
  A["Alert: p99 high"] --> M
  M -->|"exemplar trace_id"| T
  T -->|"trace_id filter"| L
  L -->|"build, flag, error"| FIX["Root cause"]

Worked example

A checkout flow runs at 5,000 requests per second. Each request touches 12 services (12 spans), and each service writes about 3 log lines per request.

Metrics: 40 pods times 1,500 series is 60,000 series. At one sample per 15 seconds and 1.3 bytes per sample, that is about 450 MB per day; thirteen months fits on one disk.

Logs: 5,000 times 12 times 3 lines times 500 bytes is 90 MB per second, 7.8 TB per day raw, about 1 TB compressed. Logs are the largest line item by far, and dropping debug lines plus the two chattiest info lines halves it.

Traces: 60,000 spans per second at 700 bytes is 42 MB per second unsampled. Tail sampling keeps all errors (0.4 percent of requests), all requests over 1 second (0.8 percent) and 1 percent of the rest: about 2.2 percent of traces, 80 GB per day.

The incident: the p99 alert fires at 1.9 seconds against an 800 ms SLO, only on /checkout, from 14:05. An exemplar in the 2.5 second bucket opens a trace where payments.authorise spends 1,850 ms in an outbound call with three retry spans. Filtering logs by that trace ID shows retry attempt=3 provider=acme status=503 and payments.timeout_ms=600 from a deploy at 14:04. Alert to cause: 6 minutes, against 50 minutes the previous quarter when the team grepped logs by timestamp across 40 pods.

In the wild

• Roblox, 2021: the telemetry depended on the system that failed. A 73-hour outage began in Roblox's Consul cluster, and the monitoring stack used that same Consul. The postmortem says the team lacked the telemetry that would have shown what was wrong, which badly slowed triage. Afterwards telemetry was made independent of the systems it watches, extended to cover Consul and BoltDB internals, and given targeted alerts for the same warning signs (Roblox).
• Anthropic, 2025: aggregate signals hid three overlapping bugs. A routing bug sent 0.8 percent of Sonnet 4 requests to the wrong server pool at first and 16 percent in the worst hour, and about 30 percent of Claude Code users hit it at least once. Two other bugs corrupted token selection. Evaluations and summary signals did not show the drop, privacy controls limited access to individual interactions, and each bug looked different on each platform. The fixes included continuous quality evaluations on production traffic and better tooling to debug from user reports (Anthropic).

Trade-offs

Pitfalls in production

• Logs without trace IDs because the code logs outside the request context. Fix: a logging filter or the OpenTelemetry log bridge that reads the active span, and a dashboard of the share of error logs carrying a trace ID.
• Different service names per signal (checkout, checkout-svc). Fix: set OTEL_SERVICE_NAME once in the deployment.
• Exemplars pointing at traces that sampling dropped. Fix: attach exemplars only for sampled spans, or tail-sample slow and failed traces.
• Logs used as metrics: counting "payment failed" lines per minute costs 100x a counter and breaks when the message is reworded. Fix: emit a counter alongside the log.
• User or order IDs added as metric labels "for debugging". Fix: keep them on spans and logs.
• A gateway collector with no queue that drops all three signals during the incident. Fix: per-node agents with persistent queues and a scaled gateway tier.

Leadership lens

• Budget by signal and by team. Showing each team its log and trace cost per million requests changes the debug-logging conversation.
• Make correlation a review standard: service.name, W3C trace context, trace IDs on logs. Cheap at birth, expensive to retrofit.
• Ask in design review: "When this fails at 3 a.m., which metric turns red first, and how many clicks to the log line that explains it?" If the answer involves grep and timestamps, the design is not done.
• Standardise instrumentation on OpenTelemetry even if backends differ, so vendor choice stays reversible.
• Agents writing instrumentation tend to log everything at info and put user IDs in metric labels. Put the house rules in the prompt and the lint.

Check yourself

1. Why can a metric not tell you which customer had the slowest request, even if it is a histogram with fine buckets?

   Show answer

   Observations are summed into bucket counters at write time, so identity is discarded. Only an exemplar or a log or span with that customer's ID can point back to the request.

2. Your p99 panel has exemplars, but clicking them often shows "trace not found". What is the most likely cause?

   Show answer

   The exemplar was recorded for a request whose trace was dropped by sampling. Record exemplars only for sampled spans, or bias sampling to keep the slow traces that exemplars land on.

3. Why do span-derived metrics have to be computed before tail sampling rather than after?

   Show answer

   After sampling, the remaining spans are biased toward errors and slow requests, so rates and percentiles computed from them would overstate both.

4. A team proposes logging every request at info level instead of adding tracing. What do they lose?

   Show answer

   Cross-service structure: parent-child timing, parallel versus sequential calls, and where time went. They also pay per byte for every request instead of sampling, and still need IDs to join lines across services.

Sources and further study

Primary

• Pelkonen et al., Gorilla (VLDB 2015): the time-series compression that makes metrics the cheapest signal, measured at 1.37 bytes per point.
• W3C Trace Context Level 2: the traceparent header that every join between signals depends on, now with a random-trace-ID flag.
• Charity Majors, Is it time to version observability?: the case for one store of wide events over three silos.

Systems docs

• Prometheus feature flags: exemplar storage and the OTLP delta-temporality options, with their current status.
• OpenTelemetry spanmetrics connector: how spans become request, error and duration metrics.
• OpenTelemetry Profiles enters public alpha: the fourth signal and how it links to traces.

Books

• Observability Engineering, Charity Majors, Liz Fong-Jones and George Miranda, chapter 1 "What Is Observability?" and chapter 5 "Structured Events Are the Building Blocks of Observability".
• Building Microservices (2nd edition), Sam Newman, chapter 10 "From Monitoring to Observability": how the problem changes as one service on one server becomes many services on many servers.

Other curricula

• Azure Architecture Center: monitoring and diagnostics: organises the topic by scenario (health, availability, performance, security, SLA, audit, usage) feeding one instrumentation pipeline.
• AWS Builders' Library: instrumenting distributed systems: Amazon's practice of one detailed request log per unit of work, from which metrics are derived.
• roadmap.sh System Design: lists monitoring as separate nodes (health, availability, performance, security, usage, instrumentation, visualisation and alerts) rather than by signal type.
• donnemartin/system-design-primer: has no monitoring or observability section; Dapper appears only in its list of papers.

Watch

• Three Pillars with Zero Answers: A New Observability Scorecard (Lightstep): Ben Sigelman's critique of treating the pillars as the goal instead of the questions they answer.
• Observability for Emerging Infra (Strange Loop): Charity Majors on high-cardinality events and why pre-aggregated metrics stop being enough.

Free resources

• OpenTelemetry signals (OpenTelemetry docs): the vendor-neutral definitions of traces, metrics and logs and how they share context.
• OpenTelemetry logs data model: the fields that let a log record carry trace and span IDs.
• OpenMetrics specification: the exposition format, including exemplar syntax on histogram buckets.
• Monitoring Distributed Systems (Google SRE book): the four golden signals and what deserves a page.
• Dapper (Google Research): the paper that made sampled, always-on tracing practical.
• Exemplars (Grafana docs): how a metric panel links straight to a trace in practice.