Engineering Atlas › System Design Fundamentals

Systems that react to events

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Kinds of events

Not every "event" means the same thing. An event notification is thin ("Order 42 changed") and makes consumers call back for details, which reintroduces coupling and load. It also hides the business flow: Fowler's warning is that the flow is not written down in any one program, so you learn it by watching the running system (Fowler). Event-carried state transfer puts the needed data in the event so consumers can keep a local copy and never call back, at the price of many copies of the data (Fowler). A domain event records a business fact in the producer's language. Be explicit about which you are publishing, because it determines payload size, coupling, and replay value. Whatever the kind, give every event a standard envelope. The CNCF CloudEvents spec requires id, source, specversion and type, and says a producer must keep source plus id unique per distinct event, so consumers may treat a repeat of that pair as a duplicate (CloudEvents spec). That pair is your deduplication key.

Brokers: logs versus queues

A log (Kafka, Kinesis, Pulsar) appends events to partitions and keeps them for a retention period. Each consumer group tracks its own offset, so many independent consumers can read the same stream and new ones can replay history. Ordering is guaranteed only within a partition, so you choose a partition key (for example order_id) to keep related events in order. A queue (SQS, RabbitMQ) hands each message to one consumer and deletes it on acknowledgement. Queues are simpler for work distribution; logs are better for fan-out and replay. Plain pub/sub without a log (such as Azure Event Grid) sits apart from both: it pushes each event to current subscribers and does not keep it, so a subscriber added tomorrow sees nothing from today (Azure Architecture Center).

The line between logs and queues is blurring. Kafka's share groups let many consumers work on one partition with per-record acknowledgement and delivery counts, which is queue behaviour on top of a log; they arrived as early access in Kafka 4.0 and were declared production-ready in 4.2 in February 2026 (Apache Kafka).

Delivery guarantees

Brokers and networks fail, so the practical guarantee is at-least-once. A consumer that processes an event and crashes before committing its offset will see the event again. Make handlers idempotent: store processed event IDs, use upserts keyed by business ID, or compare versions. On the producer side, use a transactional outbox so the database change and the event are committed together. Broker-side producer idempotence helps but only covers retries of the same send: in Kafka it rejects a retried batch the broker already has, and it is on by default together with acks=all (Kafka producer configs). Kafka 3.0 made both the defaults, but a bug kept idempotence off unless set explicitly until 3.0.1, 3.1.1 and 3.2.0, so older clients may still be running without it (Kafka 3.0 upgrade notes). It does nothing about a handler that charges a card twice.

Choreography and failure handling

When many services react to each other's events, the business process is spread across them (choreography). That scales organisationally but makes it hard to answer "where is order 42 stuck?". For long processes with compensations, an orchestrator that emits commands and listens for results is easier to monitor. Messages that keep failing go to a dead-letter queue after a bounded number of attempts, with alerting and a replay tool. Put a correlation ID on every event from the start, so logs and traces can join one business transaction across producers, brokers and consumers; retrofitting it later is much harder (Azure Architecture Center).

Down the stack

Layer 1The contract

Using Kafka as the reference broker: a record acknowledged with acks=all (the producer default) is stored on every in-sync replica, and with min.insync.replicas=2 and replication factor 3 it survives the loss of one broker. Records in a partition are delivered in offset order. A consumer group gets each partition assigned to exactly one member at a time. Delivery is at-least-once: a consumer that processes records and crashes before committing its offset will receive them again. Ordering across partitions is not guaranteed.

Layer 2The mechanism

Producers group records into a RecordBatch per partition. The batch header carries baseOffset, producerId, producerEpoch, and baseSequence; with idempotence enabled the broker rejects duplicate sequence numbers, so producer retries do not create duplicates. Each partition is a directory of segment files: a .log file of batches, a sparse .index mapping offsets to file positions, and a .timeindex. Segments roll at 1 GiB by default and are deleted or compacted by retention. A leader appends, followers fetch, and the leader advances the high watermark when all in-sync replicas have the record; consumers only read up to it. Consumer groups coordinate through a group coordinator broker, and committed offsets are themselves records in the internal __consumer_offsets topic. The classic protocol used JoinGroup and SyncGroup with a stop-the-world barrier, and the group leader (a client) computed assignments. Kafka 4.0 (March 2025) made the replacement generally available: consumers send a single ConsumerGroupHeartbeat RPC, the coordinator computes the target assignment on the broker, and partitions move incrementally so members whose assignment does not change keep working (KIP-848). Clients opt in with group.protocol=consumer. Kafka 4.0 is also the first release that runs with no ZooKeeper at all; metadata lives in the KRaft controller quorum (Apache Kafka).

Layer 3The machine

Appends go to the OS page cache with a write call. Kafka does not fsync per message by default; it relies on replication to other machines for durability and lets the kernel flush in the background. Sequential writes to a log let a single NVMe drive sustain gigabytes per second, while random small writes would be limited by IOPS. Consumers reading recent data are served from page cache, and on plaintext connections the broker uses sendfile to copy file pages straight to the socket without entering user space; TLS prevents this unless kernel TLS is in use. Batching (batch.size default 16 KB, plus linger.ms) amortises syscalls, headers, and compression across many records. Kafka 4.0 raised the linger.ms default from 0 to 5 ms because bigger batches usually give similar or lower latency overall (KIP-1030, Kafka producer configs).

Why it's built this way

  1. Disks and networks are fast for sequential bytes and slow for random small operations, so the broker is an append-only log.
  2. Per-message fsync would cap throughput at device flush latency, so durability comes from replicating to other machines instead.
  3. Consumers run at different speeds and some need replay, so the broker retains data and each group tracks its own offset.
  4. Total order across all events would need a single writer, so order is promised only per partition and you choose the key.
  5. Networks and processes fail between processing and committing, so delivery is at-least-once and consumers must be idempotent.

Diagram

flowchart LR
    O["Orders service"] -->|"write + outbox (one tx)"| ODB[("Orders DB")]
    ODB --> R["Outbox relay"]
    R -->|"OrderPlaced"| T["Topic: orders (partitioned by order_id)"]
    T --> G1["Consumer group: payments"]
    T --> G2["Consumer group: email"]
    T --> G3["Consumer group: analytics"]
    G1 -->|"fails 5x"| DLQ["Dead-letter queue"]
    G1 -->|"PaymentCaptured"| T2["Topic: payments"]

Worked example

An e-commerce platform publishes OrderPlaced at a peak of 2,000 events per second, average 1.5 KB each, into a topic with 24 partitions keyed by order_id. Throughput is about 3 MB/s, trivial for a small Kafka cluster. Seven-day retention needs 3 MB/s x 604,800 s, about 1.8 TB per replica, so 5.4 TB with replication factor 3.

Each consumer group can run up to 24 consumers, one per partition. The email consumer takes 40 ms per event, so one consumer handles 25 events per second and 24 consumers handle 600 per second. At peak it falls behind by 1,400 events per second, and after a 30-minute sale it is 2.5 million events behind, about 70 minutes of catch-up at full speed after the peak. That is acceptable for receipts. The payment consumer must keep up, so its handler is kept under 5 ms and its lag is alerted at 10 seconds.

A bug causes a payment consumer to crash after charging but before committing offsets. On restart it re-reads 200 events. Because the handler checks a processed_events table keyed by event ID inside the same transaction as the charge record, no customer is charged twice.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Synchronous request/responseSimple, immediate answer, easy to traceTemporal coupling, cascading failuresThe caller needs the result to proceed
Event notificationSmall events, loose couplingConsumers call back, adding load and couplingConsumers need only a trigger
Event-carried state transferConsumers are autonomous and fastLarger events, duplicated data, schema couplingConsumers need data and must not depend on producer uptime
Orchestrated workflowVisible process state, clear compensationCentral coordinator to build and runLong, multi-step business processes

Pitfalls in production

Leadership lens

Check yourself

  1. Why do event consumers need to be idempotent even when the broker claims exactly-once?

    Show answer

    Exactly-once usually covers only the broker's own read-process-write path. Side effects in external systems, crashes before commit, and producer retries can still cause redelivery.

  2. You need all events for one order processed in order. How do you get that from a partitioned log?

    Show answer

    Use order_id as the partition key so all events for that order land in one partition, which is consumed sequentially by one consumer in the group.

  3. What is the risk of thin notification events?

    Show answer

    Every consumer calls back to the producer for details, creating load spikes and runtime coupling to the producer's availability.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →