Kafka for system design interviews

13 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

Kafka is a distributed, replicated commit log. Producers append records to partitions of a topic; consumers read them by offset, at their own pace, as many times as they like until retention deletes them.
Ordering is guaranteed only within a partition. The partition count sets the ceiling on consumer parallelism within one group, and the key decides which partition a record lands in.
Each partition has a leader and followers that pull from it. With acks=all and min.insync.replicas=2 on replication factor 3, an acknowledged write survives any single broker failure.
It is fast because it does almost nothing per message: sequential appends to the page cache, batched and compressed records, and zero-copy transfer from file to socket.
Choose it for event streaming, change data capture, log and metrics pipelines, and decoupling services with replayable history. Avoid it for per-message routing, delays and priorities, request-reply, or tiny workloads where a managed queue is simpler. Share groups (production-ready in 4.2) add queue-style per-record acknowledgement, but not routing or delays.

How it works

Topics, partitions and offsets

A topic is split into partitions, each an ordered, append-only sequence of records numbered by offset. Producers choose a partition by hashing the key (murmur2 in the Java client), so all events for order-123 stay in order; keyless records stick to one partition until about batch.size bytes have gone there, then move on (producer configs). Retention is by time or size (retention.ms, 7 days by default), or by key with log compaction, which keeps the newest record per key (topic configs).

Consumer groups

Consumers sharing a group.id split a topic: each partition goes to exactly one member, so 12 partitions allow at most 12 active consumers per group. Progress is the committed offset of the next record to read, stored in the internal __consumer_offsets topic. Groups read independently, so one stream can feed an indexer, a fraud model and a warehouse.

Share groups (queues for Kafka)

Since 4.2, a share group is a production-ready alternative to a consumer group: its members read the same partitions together, each record is locked to one consumer (30 s by default) and then accepted, released for redelivery or rejected on its own; after 5 delivery attempts by default it is archived as a poison record (Kafka 4.2 upgrade notes, KIP-932). That lifts the "consumers no more than partitions" cap for job-style work, at the cost of per-partition ordering. It fits independent tasks, not ordered streams.

Delivery semantics

Process then commit gives at-least-once; commit then process gives at-most-once. Exactly-once within Kafka uses idempotent producers plus transactions that atomically write outputs and commit input offsets; external systems still need idempotent sinks.

Down the stack

Layer 1The contract

For a producer with acks=all, a successful response means the record is in the leader's log and in the log of every replica currently in the in-sync replica set (ISR), and that set had at least min.insync.replicas members; otherwise the write fails with NotEnoughReplicas. Records are visible to consumers only below the high watermark, the offset every ISR member has replicated, so a consumer never reads a record that a leader failover could erase. Since 3.0 the producer defaults are acks=all and enable.idempotence=true, up from acks=1 and no idempotence (KIP-679). Idempotent producers attach a producer ID and per-partition sequence numbers so retries do not duplicate or reorder. With acks=1, a leader crash can lose acknowledged records. Produce latency with acks=all is dominated by the follower fetch round trip and batching (linger.ms, whose default rose from 0 to 5 ms in 4.0 (upgrade notes)).

Layer 2The mechanism

Log layout. Each partition is a directory such as orders-3/. The log is split into segments, each named by its base offset: 00000000000000368771.log holds record batches, .index is a sparse map from relative offset to byte position (one entry every log.index.interval.bytes, 4 KB by default), and .timeindex maps timestamps to offsets. A segment rolls at segment.bytes (1 GiB) or segment.ms (topic configs). Retention deletes whole closed segments, never individual records, and never the active one. To read offset N, the broker binary-searches segment base offsets, then the sparse index, then scans forward a few KB.

Record batches. The unit of storage and transfer is a batch, not a record. A v2 batch header carries baseOffset, batchLength, partitionLeaderEpoch, magic, a CRC, attributes (compression codec, transactional, control), lastOffsetDelta, first and max timestamps, producerId, producerEpoch and baseSequence. Records inside store offset and timestamp as varint deltas from the header. The broker stores producer-compressed batches as is and hands them to consumers unchanged.

Replication. Followers replicate by sending Fetch requests to the leader, the same API consumers use, with the offset they want next. The leader learns each follower's log end offset from those fetches and advances the high watermark to the minimum across the ISR. A follower that has not caught up within replica.lag.time.max.ms (30 s) is dropped from the ISR (broker configs). When a leader fails, the controller picks a new leader from the ISR; with unclean.leader.election.enable=false, an out-of-sync replica is never promoted, which has been the default since 0.11 (Kafka design). Leader epochs stored in leader-epoch-checkpoint tell a returning replica exactly where to truncate divergent data.

Metadata. Since KRaft, a small Raft quorum of controllers stores cluster metadata (topics, partitions, ISR changes, configs) in the __cluster_metadata log, and brokers replay it. ZooKeeper mode was removed in 4.0, so 3.9 is the last line that can run it (upgrade notes), and failover of the controller takes about as long as a Raft election.

Groups. One broker is the group coordinator for each group, chosen by hashing the group ID onto a partition of __consumer_offsets. Assignment used to be a stop-the-world rebalance; cooperative rebalancing and the server-side consumer protocol (KIP-848, generally available since 4.0, opted into with group.protocol=consumer (rebalance protocol docs)) move partitions incrementally so most consumers keep working.

Layer 3The machine

A broker uses Java NIO: an acceptor thread, num.network.threads processor threads running selectors that read requests into a shared queue, and num.io.threads handler threads that append to logs or serve fetches. Requests that must wait, such as an acks=all produce or a fetch with fetch.min.bytes, park in a purgatory until satisfied or timed out, without holding a thread.

Writes go to the OS page cache with a plain write; Kafka does not fsync per message by default (flush.messages defaults to Long.MAX_VALUE) and relies on replication for durability, letting the kernel flush in the background (Kafka design). Reads of recent data are served from the same page cache, so consumers that keep up never touch disk. For plaintext connections, the broker uses sendfile (via FileChannel.transferTo) to move bytes from page cache to socket without copying through the JVM heap; TLS breaks this because bytes must be encrypted in user space, so the broker skips sendfile when SSL is on and pays noticeable CPU. The practical limits are NIC bandwidth (a 25 Gb/s NIC is about 3 GB/s, shared between producer ingress, replication and consumer egress) and disk throughput for consumers reading old data.

Partitions are not free: each has open file handles for its segments and indexes, replication fetch state, and metadata. Guidance has been a few thousand partitions per broker (hedge: it depends on version, hardware and KRaft), and very high partition counts slow controller failover and increase end-to-end latency.

Why it's built this way

Disks and SSDs are far faster at sequential I/O than random I/O, so each partition is an append-only log and reads are sequential scans.
Per-message overhead dominates at high rates, so producers batch and compress, and the broker stores and serves batches without unpacking them.
Copying data through the JVM costs CPU and GC, so Kafka leans on the page cache and sendfile instead of its own cache.
A log with offsets makes replay trivial, so consumers own their position and many independent groups can read the same data cheaply.
Ordering across a whole topic would need one sequencer, so order is only per partition, and partitions are the unit of parallelism, replication and placement.

Diagram

sequenceDiagram
    participant P as Producer (acks=all)
    participant L as Leader (broker 1)
    participant F1 as Follower (broker 2)
    participant F2 as Follower (broker 3)
    participant C as Consumer
    P->>L: Produce batch (offsets 100-149)
    L->>L: append to active segment (page cache)
    F1->>L: Fetch(offset=100)
    L-->>F1: batch 100-149
    F2->>L: Fetch(offset=100)
    L-->>F2: batch 100-149
    F1->>L: Fetch(offset=150)
    F2->>L: Fetch(offset=150)
    L->>L: all ISR at 150, HW = 150
    L-->>P: ack
    C->>L: Fetch(offset=100)
    L-->>C: records below HW (sendfile)

Worked example

A ride-hailing app publishes driver locations: 1 million drivers, one 200-byte update every 4 seconds, so 250,000 records and 50 MB/s.

With linger.ms=10 (the default is 5 since 4.0) and zstd at roughly 4 to 1 (hedge: payload-dependent), about 12.5 MB/s reaches brokers. RF 3 means 37.5 MB/s of disk writes and 25 MB/s of follower traffic, and three consumer groups read another 37.5 MB/s. 24 hours of retention is 12.5 MB/s x 86,400 s x 3, about 3.2 TB.

Matching needs per-driver order, so the key is driver_id. At about 10,000 records per second per matching consumer, 25 consumers are needed; the topic gets 48 partitions to grow without repartitioning, since changing the count remaps keys. Six brokers lead 8 partitions each, with headroom to absorb a broker loss.

With acks=all, min.insync.replicas=2 and RF 3, one broker can fail without losing writes or blocking producers. If two fail, shared partitions reject writes rather than silently lose them, the right trade here.

In the wild

Jepsen, 2013: the ISR shrinks to the leader alone. Under a network partition, Kafka 0.8 let the in-sync set shrink to just the leader, which kept acknowledging writes held only on its own disk; ZooKeeper then elected a lagging replica. In one run 520 of 987 acknowledged writes, about 53%, were lost. The analysis argued for a minimum ISR size and refusing writes over silent loss, which Kafka later shipped as min.insync.replicas and disabled unclean election by default (Jepsen).
Honeycomb, 2025: lost redundancy, full disks and tiered storage. On 5 December the EU cluster lost redundancy and about a quarter of event partitions had no leader. Unbalanced brokers filled until ingest was shut off with under 5% disk left, a full ingest outage of 6 hours 23 minutes. Disabling tiered storage freed space, but damaged metadata partitions left the control plane unrepairable, so the team rehearsed and then evacuated to a new cluster, producers first, then consumers (Honeycomb status).
Uber, 2018: retry topics instead of a stuck partition. A handler that kept failing never committed its offset, so newer records waited behind one bad message. Uber's insurance team moved failures through a chain of retry topics with growing delays, each with its own consumers, ending in a dead-letter topic that tooling can list, purge or merge back (Uber Engineering).

Trade-offs

Option	Strengths	Weaknesses	Choose when
Kafka	High throughput, replay, ordered partitions, huge ecosystem	Partition planning, operational weight, no per-message routing	Event streaming, CDC, analytics pipelines
RabbitMQ	Flexible routing, per-message ack, priorities and delays	Lower throughput, replay only with streams	Task queues and complex routing
SQS or Pub/Sub	Fully managed, scales without partitions	Limited ordering and replay, per-request cost	Simple decoupling in one cloud
Pulsar	Separate storage (BookKeeper), multi-tenancy, tiered storage	More moving parts, smaller community	Many tenants or topics, geo-replication needs

Pitfalls in production

Too few partitions. Parallelism is capped and adding partitions later breaks key ordering. Size for 2 to 3 years of growth.
Hot partitions. One huge tenant overloads one partition and consumer. Salt the key or give heavy keys their own topic.
Rebalance storms. Slow processing exceeds max.poll.interval.ms and the group thrashes. Bound per-poll work, use cooperative rebalancing and static membership.
acks=1 or min.insync.replicas=1. A leader crash loses acknowledged writes. Use acks=all with min.insync.replicas=2.
Lag past retention. A slow consumer silently loses data. Alert on lag measured in time.

Leadership lens

Treat topics as public APIs: define schemas (Avro or Protobuf with a registry), compatibility rules and ownership per topic.
Decide delivery semantics per consumer and require idempotent sinks; exactly-once inside Kafka does not cover your database.
Managed Kafka (MSK, Confluent Cloud, Aiven) removes most operational load; self-hosting needs real expertise in upgrades, rebalancing and disk planning.
Tiered storage moves old segments to object storage, which changes retention economics; it became production-ready in 3.9 (Kafka 3.9 announcement). Consider it before buying bigger disks, and rehearse recovery with it on.
Ask whether a simpler queue would do. Kafka is a strong default for streams and a heavy one for a few hundred jobs per minute.

Check yourself

Why can a consumer group not use more active consumers than partitions?
Show answer
Each partition is assigned to one consumer in the group to preserve order; extra consumers sit idle.
What does the high watermark protect against?
Show answer
Consumers reading records that are only on the leader; if that leader failed, those records could be truncated and consumers would have seen data that no longer exists.
Why does Kafka not fsync every write, and when is that unsafe?
Show answer
Replication across brokers provides durability more cheaply than per-write disk flushes; it is unsafe when all replicas can lose power at once, such as a single-rack cluster with RF 1 or with acks=1.
Why does increasing partitions on a keyed topic break ordering?
Show answer
The partition is hash(key) mod count, so changing the count moves keys to new partitions while their older records remain in the old ones.

Sources and further study

Primary

KIP-101: leader epochs for log truncation: why truncating to the high watermark could lose committed data, and the epoch fix.
KIP-932: Queues for Kafka: the design of share groups, acquisition locks and delivery counts.
KIP-848: next-generation consumer rebalance protocol: server-driven incremental assignment.

Systems docs

Kafka 4.3 topic configuration reference: current defaults for retention, segments, flushing and min ISR.
Kafka tiered storage operations: how remote log storage is configured and its limits.

Books

Designing Data-Intensive Applications, 2nd ed., Kleppmann and Riccomini, Ch 12 "Stream Processing" (Transmitting Event Streams: log-based message brokers).
Designing Distributed Systems, 2nd ed., Burns, Ch 12 "Event-Driven Batch Processing" (Publisher/Subscriber Infrastructure; Hands On: Deploying Kafka).

Other curricula

system-design-primer: treats Kafka under asynchronism and message queues, next to task queues and back pressure.
Azure Architecture Center: Publisher-Subscriber pattern: frames Kafka-style topics as a decoupling pattern with ordering, poison-message and idempotency concerns.

Watch

System Design: Why is Kafka fast? (ByteByteGo): sequential I/O and zero-copy in a few minutes.
Apache Kafka in 6 minutes (James Cutajar): a compact tour of topics, partitions, offsets and consumer groups.

Free resources

Apache Kafka documentation: design: the official explanation of persistence, efficiency, replication and delivery semantics.
Kafka: a Distributed Messaging System for Log Processing (NetDB 2011): the original LinkedIn paper.
KIP-500: Replace ZooKeeper with a self-managed metadata quorum: why and how KRaft replaced ZooKeeper.
Kafka internals course (Confluent Developer): free modules on the broker, replication, consumer groups and transactions.
Hands-free Kafka replication (Confluent blog): how ISR membership is decided by time lag.

Open in the interactive app: practice, quiz, mock interview →