Engineering Atlas › System Design Fundamentals

Processing data in batches or streams

16 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Bounded versus unbounded

A batch job reads a fixed dataset, such as every order in dt=2026-10-15, and computes the result once. Because the input is finite, the engine can sort it, scan it twice, and take its time. If the job fails, rerun it; the input has not changed, so neither will the answer.

A streaming job never sees the end of its input. Events arrive from Kafka, Kinesis or a change-data-capture feed, and the job must emit something useful before it knows whether more relevant events are coming. It keeps per-key state (running counts, open sessions, join buffers) between events, and it needs a policy for deciding when a result is final enough to publish.

Two clocks

Every event has an event time (when it happened) and a processing time (when the job saw it). In batch the gap barely matters: late uploads are in the partition by the time the job runs, or they are not. In streaming it is the central problem. A user's phone goes offline in a tunnel, and its events arrive four minutes late. A streaming job that grouped by processing time would count them in the wrong minute; one that groups by event time has to keep the minute open, or accept that it will be wrong.

That is the trade every streaming design makes: completeness, latency and cost, pick a balance. Wait longer and results are more complete but later. Emit early and update later, and downstream systems must handle changing answers. The streaming engines lesson covers how watermarks and triggers express that choice; this lesson is about when to pay for it at all.

Batch is a special case

The Dataflow model (Google, 2015) frames batch as streaming over a bounded input where the watermark jumps to infinity at the end (Akidau et al.). Seen that way, the two are the same computation with a different completeness signal. Apache Beam, Flink and Spark all take this view: the same windowed aggregation code can run over a Kafka topic or a directory of Parquet files. Flink's batch mode makes it literal: it ignores user watermark generators and emits a single maximum watermark at the end of input, so every timer fires once (Flink docs). The engines still execute the two cases differently, which is where the cost difference comes from.

Micro-batch

Spark Structured Streaming chops the stream into small batches, by default starting the next one as soon as the previous one finishes (Spark docs), and runs each as a tiny batch job with state carried between them. Latency is the batch interval plus scheduling overhead, typically one to several seconds. In exchange it reuses the batch engine's optimiser, columnar execution and fault model. Record-at-a-time engines such as Flink and Kafka Streams reach tens of milliseconds but pay per-record overhead.

The line between the two is moving. Spark 4.1 (December 2025) added an opt-in Real-time Mode that keeps long-running tasks and streams records through them, with Databricks reporting single-digit-millisecond p99 for simple queries (Databricks). In 4.1 it covers stateless queries only, from Kafka to Kafka or foreach sinks, and the trigger duration becomes a checkpoint interval rather than a latency target (Spark docs). Choose an engine by the operators you need, not by the micro-batch label alone.

Problem shape: completeness versus timeliness

The same choice appears outside data pipelines. Accounting closes the month after all invoices land, while fraud scoring must judge a payment in 50 ms. An embedding index for retrieval rebuilt nightly is simple and consistent; one updated on every document change is fresh but must handle deletes, reordering and partial failures.

Down the stack

Layer 1The contract

A batch job promises a deterministic function of an immutable input snapshot: same input, same code, same output, and a failed run can be retried from scratch with no visible side effects if the output is written atomically (write to a temporary path, then rename or commit a table snapshot). A streaming job promises a result per key and window that is emitted once the watermark passes the window end, reflects every on-time event exactly once after recovery from a checkpoint, and treats events behind the watermark according to a stated lateness policy. The precise difference: batch's completeness is a fact about the input; streaming's completeness is an estimate, and the contract must say what happens when the estimate is wrong.

Layer 2The mechanism

Batch engines are built around the shuffle. Spark writes each map task's output sorted by partition into one data file plus an index file, and reducers fetch their byte ranges from every mapper; MapReduce did the same with spill files merged on disk. Grouping happens by sorting or hashing a whole partition at once. Flink's batch execution mode (execution.runtime-mode: BATCH) switches from pipelined to blocking shuffles, groups by sorting instead of keeping state in a key-value store (the configured state backend is ignored), skips checkpoints entirely, and recovers by rerunning failed stages, because the input can be read again (Flink docs).

Streaming engines are built around keyed state and timers. Each record updates its key's state in place; a timer per window fires when the watermark passes. Recovery restores the last checkpoint and replays the source from its saved offsets. Micro-batch engines sit in between: Spark records the offset range of each batch in a write-ahead log before running it and stores state in a versioned store per partition, so a failed batch reruns over the same offset range.

Layer 3The machine

Batch wins on hardware efficiency because it can do large sequential I/O. A Parquet file stores data in row groups, each holding one chunk per column: parquet-java writes 128 MB row groups by default (parquet-java) and the format's docs recommend 512 MB to 1 GB for larger sequential reads (Parquet docs). A query reading 3 of 40 columns touches under a tenth of the bytes, and those bytes arrive in multi-megabyte reads that saturate disks and object-store connections. Vectorised operators process batches of values per call (Spark's columnar readers default to 4,096 rows (Spark config)), keeping data in CPU caches and amortising function-call overhead. A sort-based shuffle turns a random regrouping into sequential writes plus sequential range reads.

Streaming pays per record. Each event is deserialised, routed through operator code, and usually triggers a state lookup; with an on-disk state store that read can cost 1 to 100 microseconds depending on cache hits. Records are batched into network buffers that flush when full or after a timeout (Flink's execution.buffer-timeout.interval, 100 ms by default; older guides call it execution.buffer-timeout (Flink config)), so latency and throughput trade directly against each other. Checkpoints write state snapshots to object storage every few seconds to minutes. And the capacity is always on: a streaming job sized for the evening peak burns the same cores at 4 a.m.

Batch has its own machine-level failure: burstiness. Google's SRE book describes thousands of workers of a periodic pipeline starting at once and swamping shared services (a thundering herd), and unrelated pipelines whose schedules drift into overlap and spike the same resource together (Google SRE book). Staggered start times, jittered retries and per-pipeline quotas are the batch equivalent of capacity planning.

Why it's built this way

  1. Data that is all available can be scanned sequentially and sorted, which is the cheapest thing hardware does, so batch engines organise everything around full scans and shuffles.
  2. Some decisions are worth more in seconds than tomorrow (fraud, alerting, personalisation), so streaming accepts per-record overhead to act early.
  3. Acting early means acting on incomplete data, so streams need event time, watermarks and a lateness policy.
  4. Keeping running answers across failures needs durable state and replayable sources, which adds checkpoints, log retention and operational weight.
  5. Because batch is cheaper and more certain while streaming is fresher, mature platforms keep both and reconcile them.

Diagram

flowchart LR
  SRC["Events (Kafka)"] --> ARC["Archive: hourly Parquet"]
  SRC --> ST["Stream job: keyed state"]
  ST --> WM{"Watermark past window end?"}
  WM -->|"yes"| FRESH["Provisional result (seconds)"]
  WM -->|"late event"| SIDE["Side output / correction"]
  ARC --> BJ["Nightly batch: scan + shuffle"]
  BJ --> FINAL["Final result (next day)"]
  FRESH --> CMP["Reconcile: diff stream vs batch"]
  FINAL --> CMP

Worked example

A ride-hailing company emits 2 billion trip and app events a day, about 1 KB each: 2 TB a day, averaging 23,000 events per second with an evening peak of 80,000. Two consumers want session metrics (how long riders browse before booking).

Batch. A nightly job reads the day's Parquet (compressed to about 400 GB), sessionises by rider, and writes results. With 200 executor cores reading at about 100 MB/s each, the scan takes around 20 seconds of I/O; the shuffle and sort of 2 billion rows dominates, and the job finishes in about 20 minutes. That is roughly 70 core-hours a day, which can run on spot capacity at a deep discount. Results are 1 to 25 hours old but include events from phones that came back online hours late.

Streaming. A Flink job sessionises in real time with a 10-minute inactivity gap and a 2-minute watermark delay. Each core handles about 10,000 events per second with state lookups, so the peak needs 8 cores; with 2x headroom for catch-up after a restart, 16 cores run 24 hours: 384 core-hours a day, about five times the batch cost, on on-demand capacity. Sessions are published about 12 minutes after they end. Measured on real traffic, 0.4% of events arrive more than 2 minutes late; those land in a side output.

The decision. The marketing dashboard reviewed weekly takes batch. The surge-pricing model, which needs to know that browse sessions in one district have doubled in the last 15 minutes, takes the stream, and its 0.4% loss is acceptable. The finance report on completed trips takes batch because it must match the payment ledger exactly. To catch drift, a daily reconciliation job compares the stream's session counts for yesterday with the batch result and alerts if they differ by more than 1%.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
BatchCheapest per record, complete input, easy reruns and backfillsHours of staleness, bursty cluster loadReporting, billing, training data, backfills
Micro-batchBatch engine and optimiser, seconds of latencyPer-batch scheduling overhead, not sub-secondNear-real-time ETL and dashboards
Record-at-a-time streamMilliseconds to seconds, continuous stateAlways-on cost, watermarks, harder opsFraud, alerting, live features, CDC fan-out
Stream plus batch reconcileFresh numbers and exact numbersTwo outputs to explain, drift to monitorMoney, compliance, anything audited
Stream with replay (Kappa)One codebase, reprocess by replaying the logLong log retention, slow full replaysLogic that is naturally incremental

Pitfalls in production

Leadership lens

Check yourself

  1. Why can a failed batch job simply be rerun, while a failed streaming job needs checkpoints?

    Show answer

    The batch input is an immutable snapshot, so a rerun from scratch produces the same output. A streaming job's accumulated state reflects input it may no longer be able to reread cheaply, so it must snapshot state together with source offsets to resume consistently.

  2. A streaming job with a 1-minute watermark delay and a batch job over the same day disagree by 0.3%. Which is more likely to be right, and why?

    Show answer

    Usually the batch job, because it ran after the data settled and saw events that arrived behind the stream's watermark. The gap is the late-data rate, which should be measured rather than guessed.

  3. When is micro-batching a better choice than a record-at-a-time engine?

    Show answer

    When seconds of latency are acceptable and the team benefits from the batch engine's SQL optimiser, columnar execution and simpler operations. Sub-second requirements push toward record-at-a-time.

  4. Why does the Dataflow model call batch a special case of streaming?

    Show answer

    A bounded input is a stream whose watermark jumps to infinity after the last record, so every window is known to be complete and fires exactly once with the full answer.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →