Engineering Atlas › System Design Fundamentals

How stream processing engines work

11 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Dataflow graph and parallelism

A job is a graph of operators, each running as N parallel tasks. keyBy(user_id) hash-partitions records so every event for a key reaches the same task, which owns that key's state. Rescaling moves state in units of key groups, a fixed set of hash buckets assigned to tasks in ranges. The number of key groups is the job's max parallelism, which caps how far it can ever scale out without discarding state (Flink config).

Event time, windows, and watermarks

Events carry the time they happened, which differs from arrival time when phones go offline or partitions lag. A tumbling window groups fixed, non-overlapping intervals (every minute); a sliding window overlaps (5 minutes every minute); a session window closes after a gap of inactivity. A watermark W(t) is a claim that no more events with timestamps at or before t are expected. Sources usually generate it as "max seen timestamp minus allowed out-of-orderness", for example 30 seconds. An operator with several inputs takes the minimum of their watermarks (Flink docs). When the watermark passes a window's end, the window fires. Events arriving after that are late: they can update the result within an allowed-lateness period, go to a side output, or be dropped.

State and checkpoints

State lives in memory (Flink's default heap backend) or in an embedded RocksDB store on local disk, which handles terabytes per job. Flink 2.0 (March 2025) added a third option, the ForSt backend, which keeps its SST files on remote storage such as S3 and uses local disk only as a cache, so state is no longer bounded by a node's disk and rescaling moves less data (Flink 2.0). The docs still mark it experimental and warn about network latency when hot state is remote (Flink docs). Flink takes consistent snapshots using barriers: the source injects a marker into the stream; each operator, on receiving the barrier from all inputs, snapshots its state to durable storage and forwards the barrier. When all sinks acknowledge, the checkpoint is complete and includes source offsets. On failure, every task restores the last checkpoint and sources rewind to its offsets.

Exactly-once end to end

Replay alone produces duplicates in external systems. Sinks therefore either write idempotently (upsert by key) or use two-phase commit: write into a transaction during the checkpoint interval and commit only when the checkpoint completes. The latter adds output latency equal to the checkpoint interval.

Down the stack

Layer 1The contract

With checkpointing enabled and a replayable source, the engine promises that operator state reflects each input record exactly once after recovery. With a transactional or idempotent sink, that extends end to end. Window results are emitted once the watermark passes the window end, and the engine promises nothing about records that arrive after the allowed lateness except that they will not silently change already-final results. Latency is bounded by the watermark delay plus processing time, and with transactional sinks also by the checkpoint interval.

Layer 2The mechanism

In Flink, keyed state is split into key groups, the number set by maxParallelism. Unless you set it, Flink derives it as roughly 1.5 times the operator's parallelism rounded up to a power of two, with a floor of 128 and a ceiling of 32,768 (Flink docs); set it explicitly, because changing it later breaks state compatibility. Each key maps to murmurHash(key) % maxParallelism, and each task owns a contiguous range of key groups, so rescaling moves whole ranges. With the RocksDB backend, each state entry is stored under a composite key of key group, key, and namespace (the window), so all state for one window can be found and cleared by prefix. Timers for window ends live in a priority queue per task, ordered by timestamp; when a watermark arrives, the task pops every timer at or before it and fires those windows. A checkpoint barrier carries a checkpoint ID; an operator with several inputs aligns by buffering inputs that already delivered barrier n until all inputs have, then snapshots. Unaligned checkpoints skip that wait by including in-flight buffers in the snapshot. Incremental RocksDB checkpoints upload only new SST files since the last checkpoint, but they are off by default (execution.checkpointing.incremental: false) (Flink config), so large-state jobs must turn them on.

Layer 3The machine

RocksDB is a log-structured merge tree: writes go to an in-memory memtable (64 MB by default) and a write-ahead log, flush to immutable SST files, and are merged by background compaction. Point reads that miss the block cache cost an SSD read of about 100 microseconds, so a job doing random state reads at 100,000 per second per task needs either a large block cache or fast local NVMe; network-attached disks with 1 to 2 ms latency will backpressure the job. Snapshots are cheap because SST files are immutable: the checkpoint hard-links current SST files and uploads them asynchronously, so processing barely pauses. Records cross tasks through network buffers of 32 KB (taskmanager.memory.segment-size); when the receiver has no free buffers, credit-based flow control stops the sender, which is how backpressure propagates without dropping data.

Why it's built this way

  1. Events arrive out of order, so results are defined by event time and a watermark estimates completeness.
  2. State is too large for memory and must survive failure, so it lives in an embedded LSM store and is snapshotted to durable storage.
  3. Pausing the world to snapshot would add latency, so barriers flow with the data and snapshots are asynchronous.
  4. External systems cannot be rolled back, so end-to-end exactly-once needs idempotent or two-phase-commit sinks.

Diagram

sequenceDiagram
  participant S as Source
  participant O as Window operator
  participant K as Sink
  participant C as Checkpoint store
  S->>O: events with event time
  S->>O: watermark 12:00:30
  O->>K: fire window 11:59-12:00
  S->>O: checkpoint barrier n
  O->>C: snapshot state for n
  O->>K: forward barrier n
  K->>C: pre-commit transaction
  C-->>K: checkpoint n complete
  K->>K: commit transaction

Worked example

A ride-hailing service computes surge pricing per geohash cell every minute from driver and rider pings.

Input: 400,000 pings per second, keyed by roughly 250,000 active cells. Each cell keeps a one-minute window of about 200 bytes, so live state is around 50 MB. The real load is update rate: 400,000 updates per second over 48 tasks is about 8,300 per task, comfortable for RocksDB.

Lateness: 99 percent of pings arrive within 5 seconds, 99.9 percent within 20 seconds. With a 20-second out-of-orderness bound, a window ending at 12:01:00 fires around 12:01:20, so surge prices are about 20 seconds stale. A 5-second bound is fresher but makes 1 percent of events late; those go to a side output and prices are not revised.

Checkpoints every 30 seconds write about 200 MB incrementally to object storage in 2 seconds. A task failure costs at most 30 seconds of reprocessing plus restore, around 45 seconds total, during which prices hold their last value. The sink upserts by (cell, window_start), so replayed results overwrite rather than duplicate.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Apache FlinkEvent-at-a-time, rich windows, large state, mature checkpointsCluster to operate, steep learning curveComplex stateful jobs at high volume
Kafka StreamsA library in your service, no clusterKafka-only; slow state rebuild from changelogStateful microservices on Kafka
Spark Structured StreamingSame API as Spark batch; micro-batchesLatency typically 100 ms to seconds (Real-time Mode in 4.1 goes lower, stateless queries only); less flexible windowsTeams already on Spark, seconds-level latency fine
Managed (Google Dataflow, Amazon Managed Service for Apache Flink)No cluster ops, autoscalingCost, vendor lock-in, less tuning controlSmall platform teams

Amazon Managed Service for Apache Flink is the August 2023 name for what AWS called Kinesis Data Analytics; the APIs did not change, so older guides and IAM policies still use the old name (AWS).

Pitfalls in production

Leadership lens

Check yourself

  1. Why does one idle partition stop all windows from firing?

    Show answer

    The operator's watermark is the minimum across its inputs, and an idle partition never advances its watermark, so the minimum stays stuck.

  2. How do barriers produce a consistent snapshot without pausing the whole job?

    Show answer

    Each operator snapshots its state when the barrier passes, so every snapshot reflects exactly the events before that barrier, and together with source offsets they form a consistent cut.

  3. What does a shorter out-of-orderness bound trade away?

    Show answer

    Results arrive sooner, but more events become late and are either dropped, handled separately, or cause revisions to emitted results.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →