Collecting logs into one place

11 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

Log aggregation collects log lines from every process and host into a central system where they can be searched, filtered, and retained.
The pipeline has four stages: emit (structured logs to stdout or a file), collect (an agent on each node), buffer and process (a queue plus parsers), and store and query (an index or object store).
Two storage philosophies dominate: full-text indexing (Elasticsearch, OpenSearch) for fast arbitrary search, and label indexing plus brute-force scan (Loki, ClickHouse-style) for much lower cost.
Log volume grows faster than traffic, so sampling, levels, and retention tiers are what keep the bill under control.
Logs are for detailed, per-event forensics. Use metrics for trends and alerts, and traces for request flow.

How it works

Emit

Applications should write structured logs, one JSON object per line, with consistent fields: timestamp, level, service, trace_id, message, and event-specific keys. The OpenTelemetry log data model is a good target schema, since its records already carry severity, trace ID and span ID (OTel logs data model). Writing to stdout and letting the platform capture it keeps shipping and rotation out of the app (Twelve-Factor).

Collect

An agent (Fluent Bit, Vector, the OpenTelemetry Collector) runs on every node, tails container log files, adds metadata such as pod, namespace, and region, and ships batches. It must checkpoint its read position so a restart loses nothing, and buffer to disk when the backend is slow. Fluent Bit's tail input keeps inode-to-offset pairs in a small SQLite file and warns that a few records may be read again after a crash, so "neither loses nor resends" is really "loses nothing, may resend" (Fluent Bit docs). The agent is not optional on Kubernetes: the kubelet keeps at most five 10 MiB files per container by default, and kubectl logs shows only the newest one (Kubernetes logging).

Buffer and process

At scale, agents send to a durable queue such as Kafka, which absorbs bursts and decouples producers from indexers. Processors parse unstructured lines, drop noise, redact secrets and personal data, and route streams to different destinations by team or retention class.

Store and query

Inverted-index stores tokenize every field and build a term-to-document index. Searches for any word are fast, but indexing costs CPU and the index can be as large as the raw data, so storage commonly runs 1 to 1.5 times raw. Label-indexed stores index only a few low-cardinality labels (service, environment, level) and keep log content as compressed chunks in object storage. Queries select streams by label, then scan chunks in parallel (Loki overview). Ingest and storage are much cheaper, often 10 times compression on object storage, but searching for a rare string across a whole day means scanning everything.

Retention tiers

Recent days stay on hot storage; older data moves to cold tiers or object storage and is deleted on a schedule set by debugging and compliance needs (Elastic ILM). Which events you collect matters as much as how long you keep them: an investigation can only search the event types that were switched on before the attack, which is why security teams decide audit logging in advance rather than during an incident.

Down the stack

Layer 1The contract

A log pipeline usually promises at-least-once delivery from the node to the store: every line written to stdout is eventually searchable, possibly duplicated, within a stated delay such as 30 seconds at p99. Ordering is promised only per source stream, by timestamp, not globally. Under sustained overload it promises explicit behaviour, either backpressure with bounded disk buffering or counted drops, never silent loss. Retention promises that data is deleted after its class's period.

Layer 2The mechanism

In Kubernetes, the container runtime writes each container's stdout to /var/log/pods/NAMESPACE_POD_UID/CONTAINER/0.log in CRI format: timestamp, stream, a partial-line flag, and the message. The kubelet rotates these files at containerLogMaxSize (10 MiB by default), keeping containerLogMaxFiles (5). The agent tails them with inotify, records the inode and byte offset of each file in a small database, and handles rotation by continuing to read the old inode until EOF before switching. Partial-line flags let it reassemble lines the runtime split at 16 KB. Batches are compressed and shipped; acknowledgement from Kafka or the backend advances the stored offset. Elasticsearch indexes each document into a Lucene in-memory buffer, makes it searchable after a refresh (1 second by default) by writing a new segment, and merges small segments in the background. A shard that has not been searched for 30 seconds stops refreshing until the next search arrives, unless the interval is set explicitly (Elasticsearch index settings). Loki groups lines by their label set into a stream, appends to an in-memory chunk per stream, and flushes chunks with a target size of 1.5 MB (chunk_target_size 1,572,864 bytes) to object storage with a small index entry per chunk (Loki config). High-cardinality fields that people still filter on, such as trace_id or pod name, go in structured metadata: stored beside each line, not indexed, and available only with the tsdb index and schema v13, which replaced the deprecated boltdb-shipper (Loki docs).

Layer 3The machine

Tailing is cheap: inotify wakes the agent on writes, and reads hit the page cache because the data was just written. The expensive parts are compression and indexing. Gzip or Zstd at 100 to 500 MB/s per core means a node shipping 20 MB/s spends a few percent of a core. Lucene indexing, by contrast, tokenises every field and builds postings lists; a single core indexes on the order of 5,000 to 20,000 small log documents per second, so 300,000 lines per second needs tens of cores for indexing alone, plus disk writes for segment merges that rewrite data several times over. Loki avoids most of that CPU and write amplification, paying instead at query time by decompressing chunks at around 1 GB/s per core.

Why it's built this way

Applications should not block or crash because the log backend is slow, so they write to local files and an agent ships asynchronously.
Agents and networks fail, so offsets are checkpointed and delivery is at-least-once.
Log bursts are much larger than average traffic, so a durable queue absorbs them.
Indexing every token is CPU- and storage-heavy, so cheaper designs index labels only and scan compressed chunks when needed.

Diagram

flowchart LR
  APP["App: JSON to stdout"] --> AG["Node agent: tail, enrich, buffer"]
  AG --> KQ["Kafka buffer"]
  KQ --> PR["Processors: parse, redact, route"]
  PR --> HOT["Hot index (7 days)"]
  PR --> OBJ["Object storage archive"]
  HOT --> UI["Search UI"]
  OBJ -.->|"rehydrate on demand"| UI

Worked example

A company runs 2,000 pods across 40 services. Average log rate is 150 lines per second per pod at 400 bytes, so 300,000 lines per second, 120 MB/s, about 10 TB per day raw.

Option A: Elasticsearch with 7 days hot. Index size near raw size, times two for one replica, gives 20 TB per day and 140 TB on disk: about 18 data nodes at 8 TB usable SSD each, a count that indexing 300,000 documents per second needs for CPU anyway.

Option B: Loki-style storage. Logs compress about 10 times into 1 TB per day of chunks in object storage. Thirty days is 30 TB, about $700 per month in storage. Ingesters and queriers need maybe 10 to 15 nodes. A query like "errors from checkout in the last hour" selects one stream set and scans 2 GB in seconds. A search for one order ID across all services for 7 days scans 7 TB compressed and takes minutes, unless the order ID is also emitted to a cheaper index.

Reduction first: the team finds that 45 percent of volume is DEBUG from two services and health-check access logs. Dropping health checks and sampling successful-request access logs at 10 percent cuts volume to 5 TB per day before choosing any backend.

In the wild

Cloudflare, 2024: a blank config made the log pipeline fan out 40x. A bug in config generation handed Logfwdr an empty configuration; the change was reverted within five minutes. Logfwdr's failsafe treated "no config" as "send everything", so it forwarded logs for every customer. Buftee, which keeps a buffer per customer job, went from about 1 million buffers to 40 million, its overload protections had never been configured, and it needed a full restart. About 55 percent of customer logs were lost over roughly 3.5 hours. Fixes: alerts for this misconfiguration, a tested fix, and regular overload tests (Cloudflare).
US federal agency, 2023: the attack was visible only in a premium log. In mid-June 2023 an agency spotted MailItemsAccessed events from an application ID that never normally read mail, and that led to the discovery of the Storm-0558 intrusion into Microsoft-hosted email. CISA and the FBI advise enabling Purview Audit (Premium) logging, which requires G5/E5 licensing, and credit the agency's enhanced MailItemsAccessed logging for the detection, so organisations without that logging had no record to search (CISA AA23-193A).
Okta, 2023: the evidence was under a different event type. An attacker used a stolen service account to download customer support files, some of them HAR files with live session tokens, from 134 customers between 28 September and 17 October. Opening a file from the Files tab wrote a different log event from opening it through a case, and the first investigation searched only for the latter, which is part of why detection took about two weeks (Okta).

Trade-offs

Option	Strengths	Weaknesses	Choose when
Full-text index (Elasticsearch, OpenSearch)	Fast search on any term, rich aggregations	High CPU, RAM, and storage cost; cluster tuning	Security analytics, heavy ad hoc search
Label index plus scan (Loki)	Cheap ingest and storage on object storage	Slow needle-in-haystack queries over long ranges	Operational debugging scoped by service and time
Columnar store (ClickHouse)	Fast aggregations, good compression, SQL	Schema design effort, self-managed	Structured logs used like analytics events
SaaS (Datadog, Splunk Cloud)	No operations, polished UX	Cost per GB scales painfully	Small teams or until volume makes it unaffordable

Pitfalls in production

Unbounded volume. A retry loop logs a stack trace per attempt and one service generates 5 TB in an hour. Set per-service rate limits and quotas in the agent or processor, and alert on volume spikes.
Secrets and PII in logs. Tokens, emails, and card numbers end up searchable by everyone. Redact in the pipeline, use structured fields with allow-lists, and restrict access by team.
High-cardinality labels. Adding user_id as a Loki label creates millions of streams and overwhelms the index. Keep labels to low-cardinality dimensions and put IDs in the log body or in structured metadata.
Lost logs during incidents. Under load, agents drop lines just when you need them. Use disk buffers, a Kafka buffer, and monitor agent drop counters.

Leadership lens

Cost is the main design driver. Ask each team for their log volume and who reads it; much of it is never queried.
Define a logging standard: required fields, levels, trace IDs, and what must never be logged. Enforce it with a shared library.
Decide retention per class, for example 7 days for debug, 30 days for application, 1 year for audit, and get compliance to sign off.

Check yourself

Why is a label-indexed store much cheaper than a full-text index?
Show answer
It indexes only a few low-cardinality labels and stores compressed content in object storage, avoiding the CPU and disk cost of indexing every token.
What role does a Kafka buffer play between agents and the store?
Show answer
It absorbs bursts, decouples ingestion from indexing speed, and allows replay if the store is down or a processor has a bug.
Why put trace_id in every log line?
Show answer
It links logs to the distributed trace for the same request, so you can move from a slow span to the exact log lines it produced.

Sources and further study

Primary

OpenTelemetry logs data model: the vendor-neutral record shape that agents and backends increasingly share.
CISA and FBI advisory AA23-193A: a real investigation that turned on which audit events had been collected.

Systems docs

Kubernetes logging architecture: kubelet rotation defaults and the node-agent, sidecar and direct-push patterns.
Fluent Bit tail input: offset database, memory limits and pause-on-backpressure.
Grafana Loki structured metadata: where high-cardinality fields go when they cannot be labels.
Elasticsearch index settings: refresh interval and search-idle shards.

Books

Observability Engineering, Majors, Fong-Jones and Miranda, chapter 18 "Telemetry Management with Pipelines": pipeline anatomy and Slack's telemetry pipeline as a case study.
Designing Distributed Systems (2nd edition), Brendan Burns, chapter 5 "Adapters": the logging and monitoring adapter containers that normalise output before shipping.

Other curricula

Azure Architecture Center: monitoring and diagnostics: treats collection, storage and analysis as stages of one pipeline and separates audit data from operational data.
AWS Builders' Library: instrumenting distributed systems: frequent rotation, a separate partition, and agents that read from page cache.
donnemartin/system-design-primer: does not cover log aggregation.

Watch

Beginner's Crash Course to Elastic Stack, Part 1 (Official Elastic Community): how Elasticsearch indexes and searches documents.
Meet Grafana Loki (Techno Tim): a practical tour of label-based log aggregation.

Free resources

The Twelve-Factor App: Logs: the case for treating logs as event streams written to stdout.
Grafana Loki overview: label indexing, chunks, and why it is cheaper than full-text indexing.
Elasticsearch index lifecycle management: hot, warm, cold tiers and rollover in practice.
OpenTelemetry logs: the log data model and correlation with traces.

Open in the interactive app: practice, quiz, mock interview →