Streaming-only data pipelines

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

Kappa architecture uses one stream processing pipeline for everything. There is no separate batch layer.
The durable, replayable log (usually Kafka) is the source of truth. Any derived view is the result of running a job over that log.
To fix a bug or change logic, you reprocess: start a new version of the job from the beginning of the log (or a chosen offset), let it catch up into a new output table, then switch readers over.
It removes the dual-codebase problem of Lambda, at the price of long log retention and reprocessing throughput that must be planned.
It suits naturally incremental logic: aggregations, windowed joins, enrichment, materialised views.

How it works

The log as the system of record

Every event is appended to a partitioned, ordered log. Retention is long: days to forever, using tiered storage that moves old segments to object storage, or log compaction for keyed changelogs where only the latest value per key matters. The two do not mix: Kafka's tiered storage, declared production-ready in 3.9 (Kafka 3.9 release), still does not support compacted topics, so compacted changelogs stay on broker disks (Kafka docs). Consumers track their position as an offset per partition, so any consumer can rewind.

Jay Kreps named the pattern in 2014 as an answer to Lambda's two codebases: keep the raw input in the log, and recompute by running a second copy of the same stream job (Kreps). The Azure Architecture Center frames it the same way, with events mirrored to long-term storage so the stream can be recomputed when needed (Azure).

Stateful stream jobs

A job reads the log, maintains state (counters, windows, join tables) in a local store such as RocksDB, and writes results to an output: another topic, a database, or a search index. State is checkpointed with the input offsets, so after a crash the job restores state and resumes from the matching offset without losing or double counting.

Reprocessing instead of batch

When logic changes, you do not patch the old output in place. You deploy job v2 alongside v1, reading from offset zero (or from a snapshot point), writing to a new table views_v2. v2 processes history much faster than real time because it is reading stored data in bulk. When v2's lag reaches near zero, readers switch to views_v2 and v1 is stopped. Rollback is switching back.

Replaying from offset zero is not the only start point. If the change is small and the state is compatible, v2 can start from a savepoint of v1, a consistent image of its state and offsets. In Flink this only works if every stateful operator has a stable ID set with uid(); auto-generated IDs change when the program changes, and the restore fails (Flink docs). A logic bug that corrupted the state itself still needs a replay from before the bug.

Event time and ordering

Because historical replay races through months of data in hours, the job must use event time (when the event happened) and watermarks, not processing time (when the job sees it), to assign events to windows (Akidau). Otherwise a replay assigns every event to "now". Ordering is guaranteed only within a partition, so keys that need ordered processing must share a partition.

Down the stack

Layer 1The contract

The log promises that records within a partition are stored and delivered in offset order, that acknowledged records (with acks=all and min.insync.replicas=2) survive the loss of one broker, and that records remain readable until retention expires. acks=all and idempotence are producer defaults today (producer configs), but min.insync.replicas must still be raised from its default of 1 for that promise to hold. A consumer group promises each partition is assigned to one consumer at a time, and committed offsets mark where processing resumes. Together these let a job promise that its output is a deterministic function of the log prefix it has processed.

Layer 2The mechanism

Each Kafka partition is a directory of segment files, by default 1 GiB each (log.segment.bytes, Kafka configs), named by their base offset, with a sparse .index mapping offsets to byte positions and a .timeindex mapping timestamps to offsets. A consumer that seeks to offset 0 or to a timestamp uses these indexes to find the right segment and byte position in a binary search. With tiered storage, closed segments are copied to object storage and deleted locally after a short local retention; brokers fetch remote segments on demand when an old offset is requested. A fetch request serves remote data for only one partition at a time, which caps how fast one consumer can replay many partitions from the remote tier (Kafka wiki). Log compaction runs a cleaner thread that rewrites old segments keeping only the newest record per key, and treats a record with a null value (a tombstone) as a delete. Tombstones themselves are removed after delete.retention.ms, 24 hours by default, so a consumer rebuilding from offset 0 must reach the head within that time or it will miss deletes (Kafka design). Consumer offsets are themselves stored in the compacted internal topic __consumer_offsets, keyed by group, topic, and partition.

Layer 3The machine

Kafka relies heavily on the OS page cache. Producers append to the active segment with sequential writes, and recent reads are served from page cache by sendfile, which copies bytes from the cache to the socket inside the kernel without passing through user space (Kafka design). A live consumer reading the tail therefore costs almost no disk I/O. A replay is different: it reads cold segments from disk or object storage, pulling gigabytes through the page cache and evicting the hot tail that live consumers depend on. Their reads then fall through to disk, latency rises from sub-millisecond to tens of milliseconds, and producers can slow down too. This is the physical reason to throttle replays with quotas or to serve them from tiered storage instead of broker disks.

Why it's built this way

Maintaining two codebases for batch and stream is expensive, so Kappa keeps one.
One codebase needs a way to recompute history, so the log is retained long enough to replay.
Disks and networks are efficient for large sequential reads, so replaying a log is much faster than re-querying a database row by row.
Keeping years of segments on broker SSDs is costly, so old segments move to object storage, and keyed state uses compaction.

Diagram

flowchart LR
  P["Producers"] --> LOG["Kafka log (long retention)"]
  LOG -->|"live offsets"| J1["Job v1"]
  LOG -->|"replay from offset 0"| J2["Job v2"]
  J1 --> T1["Table views_v1"]
  J2 --> T2["Table views_v2"]
  T1 --> SW{"Read alias"}
  T2 -.->|"switch when caught up"| SW
  SW --> APP["Applications"]

Worked example

A marketplace computes seller metrics (orders, revenue, cancellation rate by day) from an order_events topic with 64 partitions. Traffic averages 20,000 events per second at 500 bytes, about 860 GB per day. Kafka keeps 7 days on broker disks and 2 years in tiered object storage, around 630 TB, costing roughly $14,000 per month at standard object storage prices.

The team discovers the cancellation rate double counts orders cancelled and then reinstated. They fix the logic and launch v2 from the start of retention.

Replay throughput: each of 64 Flink task slots reads one partition. Reading from tiered storage and updating RocksDB, each sustains about 40 MB/s, so the job reads 2.5 GB/s in aggregate. Two years of data, 630 TB, would take about 70 hours. That is too long. The metric is only displayed for 13 months, so v2 replays from 13 months ago, about 340 TB, in roughly 38 hours. Partition count caps read parallelism at 64, so going faster would mean replaying from a Parquet archive in the lake instead.

v1 keeps serving during replay. When v2's lag drops under a minute, the alias moves; v1's table stays a week for rollback.

In the wild

The New York Times, 2017: every article since 1851 in one Kafka topic. The publishing pipeline stores every published asset in the "Monolog", a single-partition topic kept forever so it stays totally ordered, under 100 GB at the time. New services build their view by replaying from the start, then keep reading live updates; some only start a few hours back (Confluent blog).
Uber, 2020: backfills without flooding Kafka. Uber rejected replaying days of trip and client logs back into Kafka because it would overwhelm the cluster and cause lag, and rejected running the streaming code as a Spark batch job because that ignored event-time windows and watermarks. Uber instead made a Hive table a streaming source for the same Spark job, relaxed the watermark from 10 seconds to two hours, and backfilled about nine days (10 TB) of Hive data; the production streaming job it reuses runs on 75 cores and 1.2 TB of memory (Uber).
Honeycomb, 2025: tiered storage in the way of recovery. An EU Kafka cluster lost redundancy, about a quarter of partitions had no leader, and brokers filled their disks until ingest was switched off. Turning off tiered storage freed space but made damage to internal metadata permanent. Customers saw about 6 hours of ingest outage, and the team evacuated to a new cluster, producers first (Honeycomb).

Trade-offs

Option	Strengths	Weaknesses	Choose when
Kappa	One codebase and semantics; reprocessing via replay	Long log retention cost; replay time for large histories	Logic is incremental and the log is the source of truth
Lambda	Batch can use heavy, non-incremental logic	Two implementations that drift	Batch-only computations are essential
Kappa with lake snapshots	Replay from a recent state snapshot plus log tail	Snapshot and log must stay consistent	Histories too large to replay from zero
Batch only	Cheap and simple	No freshness	Minutes or hours of latency are acceptable

Pitfalls in production

Retention shorter than needed. The fix requires 90 days of history but the topic keeps 7. Decide retention from the longest reprocessing horizon, and archive to object storage with tiered storage.
Replay overloads the cluster. A full replay saturates broker disks and slows live consumers. Use quotas, read from tiered storage, or replay from an archived copy.
Processing-time windows. A replayed job puts two years of events into the current hour. Use event time and watermarks everywhere.
Side effects during replay. Replaying a job that sends emails or charges cards repeats those actions. Separate pure view-building jobs from effectful consumers, and make effects idempotent with dedupe keys.
Rebuild misses deletes. A new job reads a compacted topic from offset 0 but takes three days to reach the head, while tombstones are removed after 24 hours, so keys deleted meanwhile survive in the rebuilt view. Raise delete.retention.ms above the worst-case rebuild time.
Schema evolution. Old events in the log use schema v1 and the new job expects v3. Use a schema registry with compatibility rules and code that can read all historical versions.

Leadership lens

Ask "how long would a full reprocess take?" in every design review. If the answer is weeks, the architecture is not really Kappa.
Treat the log as a product: schemas, compatibility rules, retention, and ownership must be explicit.
Budget for replay capacity, not just steady-state capacity. A job sized for live traffic will crawl through history.
Keep effectful consumers separate from derived-view jobs so reprocessing is always safe to run.

Check yourself

Why must Kappa jobs use event time rather than processing time?
Show answer
During replay, processing time is the moment of reprocessing, so windows and aggregations would be wrong; event time keeps results identical to a live run.
How does Kappa handle a logic bug that affected three months of output?
Show answer
Deploy a fixed job version that replays the log from before the bug into a new output, then switch readers once it has caught up.
What makes log compaction useful for Kappa?
Show answer
For keyed changelogs it keeps the latest value per key indefinitely, so new jobs can rebuild current state without storing every historical update.

Sources and further study

Primary

Akidau et al., The Dataflow Model (VLDB 2015): event-time windows, watermarks and triggers, the semantics that let one streaming job also do the batch job's work.
KIP-405: Kafka Tiered Storage: the design for moving closed segments to remote storage.
Kafka Tiered Storage GA release notes: the current limits, including no compacted topics and one remote partition per fetch.

Systems docs

Kafka design: persistence and compaction: page cache, sendfile, and the tombstone retention rule.
Flink savepoints: how to upgrade a stateful job without replaying, and why operator IDs matter.

Books

Designing Data-Intensive Applications (1st ed.), Martin Kleppmann, Ch 11 "Stream Processing": logs, change data capture, and reprocessing for application evolution.
Designing Data-Intensive Applications (2nd ed.), Kleppmann and Riccomini, Ch 13 "A Philosophy of Streaming Systems": unbundling the database and deriving state from an event log.

Other curricula

Azure Architecture Center: Big data architectures: presents Kappa as a single stream path with events mirrored to long-term storage for recomputation.
donnemartin/system-design-primer: covers queues and asynchronism but has no section on Lambda, Kappa, or replay.

Watch

Kafka: I Love Logs (Jay Kreps, SF Scala): the log-centric view of data systems that motivated Kappa.
Turning the database inside out with Apache Samza (Martin Kleppmann, Strange Loop): materialised views derived from a log, the core Kappa idea.

Free resources

Questioning the Lambda Architecture (Jay Kreps): the essay that proposed reprocessing by replay.
Kafka log compaction: how compacted topics keep the latest value per key.
Kafka Streams documentation: state stores, changelogs, and reprocessing in a library-based engine.
Streaming 101 (Tyler Akidau): why event time matters and how streaming can subsume batch.

Open in the interactive app: practice, quiz, mock interview →