Design a stock trading exchange

21 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

One thread owns each order book. Matching is a strictly sequential function of the order in which messages arrive, so each symbol's book lives on a single thread with no locks. Scale comes from partitioning symbols across engines, not from parallelism inside a book.
The sequencer defines reality. Every inbound message gets a global sequence number before it is matched. The book is a pure function of that sequenced log, so replaying the log rebuilds the exact book, every trade and every outbound message.
Replicate the input, not the state. Hot standbys receive the same sequenced log and run the same deterministic code. Failover means a follower that already has the identical book starts publishing, in milliseconds.
Market data is a sequenced broadcast. Every change to every book goes out on UDP multicast with gapless sequence numbers, so thousands of subscribers get the same bytes at the same moment and recover gaps on their own.
Latency is measured in microseconds and fairness is a product feature. Kernel bypass, pinned cores and pre-allocated memory keep the path short; equal cable lengths and deterministic ordering keep it fair.

Requirements

Functional

Members submit limit and market orders, cancel them, and reduce or replace them, over authenticated sessions.
Match by price-time priority: best price first, then earliest arrival at that price. Trades execute at the resting order's price.
Send each member an acknowledgement, fills and cancels for its own orders (execution reports).
Publish public market data: every add, execute, cancel and trade (full depth), plus best bid and offer.
Halt a symbol, enforce price bands, and let a member's risk officer kill all its orders at once.

Non-functional

Gateway-in to acknowledgement-out under 50 µs median and 200 µs at p99.
An acknowledged order or trade is never lost, even if the matching host dies the next microsecond.
Deterministic: replaying a day's input reproduces every output byte for byte, which regulators and dispute resolution rely on.
Fair: no member can see or act on information before others who paid for the same access.
99.99% availability during the trading session, which is about 2 seconds of downtime across a 6.5 hour day.

Out of scope: clearing and settlement (a central counterparty takes over after the trade; US equities have settled on T+1, one business day after the trade, since 28 May 2024 (SEC)), opening and closing auctions in detail, the brokerage apps that retail investors use, and options and futures pricing.

Back-of-envelope

Planning numbers for a mid-to-large equities exchange.

Symbols: 8,000 listed instruments; the busiest 100 carry about half the traffic.
Inbound messages: 1.5B per day (new orders, cancels, replaces; most orders are cancelled, not filled). A 6.5-hour session is 23,400 s, so the average is 1.5B / 23,400 ≈ 64,000 messages/s. The open and close run at about 20× average, roughly 1.3M/s, and microbursts in the first second are higher still. Size for 3M/s.
Trades: about 2% of messages execute: 30M trades per day.
Journal: binary inbound messages average about 50 bytes; with the sequence number, timestamp and session id the journal record is about 64 bytes. 1.5B × 64 B ≈ 96 GB/day.
Market data: each message produces one or two public events of about 40 bytes, say 2.5B × 40 B ≈ 100 GB/day.
Five-year storage: about 252 trading days a year: (96 + 100) GB × 252 × 5 ≈ 250 TB, kept compressed for regulatory replay.
Bandwidth: peak inbound 3M/s × 64 B ≈ 190 MB/s, about 1.5 Gbit/s across all gateways. Market data at peak, 5M events/s × 40 B ≈ 200 MB/s per feed, fits on 10 GbE with headroom; a second redundant feed doubles it.
Book memory: if a symbol has 5,000 resting orders on average at 64 bytes each, 8,000 symbols need 8,000 × 5,000 × 64 B ≈ 2.6 GB. A busy symbol's book fits in a core's L2 or L3 cache, which is what makes microsecond matching possible.

API

Members connect over TCP to a gateway using a compact binary protocol (Nasdaq's OUCH is a public example); slower members use FIX through a translating gateway. Field names below are illustrative.

-> EnterOrder   { token: "A17", symbol: "ACME", side: B|S, qty: 500,
                  price: 10125 (in 1/100 cents) | MARKET, tif: DAY|IOC }
<- Accepted     { token: "A17", order_ref: 884201, seq: 73310021, ts_ns }
<- Executed     { token: "A17", qty: 200, price: 10120, match_id: 551903 }
<- Rejected     { token: "A17", reason: "price_band" | "risk_limit" | "halted" | "dup_token" }

-> CancelOrder  { token: "A17", qty_remaining: 0 }
<- Canceled     { token: "A17", qty_canceled: 300 }

-> ReplaceOrder { old_token: "A17", new_token: "A18", qty, price }   (loses time priority
                                                                    unless only qty decreases)
Market data (UDP multicast, gapless seq per channel):
   AddOrder { order_ref, side, qty, symbol, price }    OrderExecuted { order_ref, qty, match_id }
   OrderCancel { order_ref, qty }   OrderDelete { order_ref }   Trade { ... }
Recovery: UDP unicast re-request to a retransmission server { session, from_seq, count }; book snapshots over TCP.

The member's token is unique per session per day, so a resent order after a disconnect is recognised as a duplicate instead of becoming a second order. Prices are integers in a fixed tick unit; floating point never touches the book. The recovery path mirrors Nasdaq's MoldUDP64, where a listener that sees a gap sends a small request packet (session, first sequence number, message count) to a re-request server and gets the missing packets back by UDP unicast (MoldUDP64 spec).

Data model

In memory, per symbol (the order book). Two sides. Each side maps price to a level; each level holds a FIFO doubly linked list of resting orders and a running total quantity. Equities trade on a fixed tick grid inside a daily price band, so a level index can be an array indexed by (price - band_low) / tick, with best bid and best ask tracked as indexes. A hash map from order_ref to the order node makes cancel O(1). An order node is about 64 bytes, one cache line: ref, member, remaining quantity, price, timestamp, and the two list pointers.

Input journal. Append-only, one per matching partition, keyed by sequence number. Each record is the message as the sequencer stamped it. This is the system of record; everything else can be rebuilt from it.

Output journal. Execution reports and market data events, also sequenced, so a gateway or a market data subscriber can ask for "everything after seq N".

Session state per member connection (next expected sequence, tokens seen today, risk counters) lives with the gateway and is rebuilt from the journals on restart.

Reference data, loaded before the open: symbol to partition, tick size, price bands, member permissions. Symbol is the partition key because matching never crosses symbols. Trades then flow to an ordinary database for clearing and surveillance, off the critical path.

High-level design

flowchart LR
  M["Member trading systems"] --> G["Order gateways (session, validation, pre-trade risk)"]
  G --> S["Sequencer (cluster leader)"]
  S --> J[("Replicated input journal")]
  S --> E["Matching engine, one thread per partition"]
  J --> F["Follower engines (hot standby)"]
  E --> O["Output sequencer"]
  O --> G
  O --> MD["Market data publisher"]
  MD --> UDP["Multicast feeds A and B"]
  UDP --> M
  O --> DC["Drop copy, clearing, surveillance"]

Write path (an order that trades).

A member sends EnterOrder to its gateway. The gateway checks the session, the token, the symbol's state and the price band, and runs pre-trade risk (order size and value limits, open exposure) from counters it keeps in memory.
The gateway forwards the message to the sequencer for that symbol's partition, which stamps a sequence number and a timestamp and appends it to the journal. The journal entry is replicated to followers.
Once a quorum has the entry, the matching thread processes it: it walks the opposite side from the best price while the order crosses, generating fills at each resting order's price, and rests any remaining quantity at the back of its price level's queue.
The engine emits Accepted, Executed for both sides, and public events (OrderExecuted, AddOrder for the remainder) into the output sequencer.
The gateway delivers execution reports to both members; the publisher multicasts the public events; drop copy and clearing receive the trades.

Read path (market data).

Subscribers join the A and B multicast groups for their channels and take whichever copy of each packet arrives first.
On a gap in sequence numbers, a subscriber requests the missing range from a retransmission server, or, if it is far behind, loads the latest book snapshot and replays from the snapshot's sequence.

Deep dives

1. The matching engine: one thread, one book

Options considered: a database with a row per order and transactions per match, a multithreaded engine with locks per price level, or a single thread per partition.

A database transaction per match costs milliseconds, three orders of magnitude over budget. Locked multithreading fails because every order touches the top of the book, the same few cache lines, so threads serialize anyway and pay for cache-line transfers between cores; worse, results now depend on thread scheduling.

I choose one thread per partition: read the next sequenced message, mutate the book in memory, write outputs, with no locks and no system calls. Martin Fowler's write-up of LMAX describes a single-threaded business logic processor handling about 6 million orders per second on a 2010-era dual-socket server, entirely in memory with no database on the hot path (Martin Fowler). Parallelism comes from partitioning: 8,000 symbols over, say, 32 engine threads, with the busiest symbols on threads of their own.

Inside the thread, the goal is to touch little memory. The best level comes from a tracked index, the level's FIFO gives time priority for free, and a fill against the first resting order touches about four cache lines. A replace that only lowers quantity keeps its queue position; any other change is a cancel plus a new order at the back.

2. Sequencing, determinism and failover

The options for high availability are a primary that writes state to a replicated database, active-active engines that each accept orders, or a replicated input log with deterministic replicas.

State replication is too slow for a book that changes millions of times a second, and active-active engines that order messages independently will disagree about who arrived first, which is fatal for an exchange.

I choose the replicated log. The sequencer is the cluster leader; it stamps each message and replicates it to two or more followers with a consensus protocol such as Raft (Aeron Cluster is an open-source Raft implementation that sequences many client connections into one replicated log and lists central limit order books among its typical uses (Aeron docs)). Every follower runs the same engine on the same entries and holds an identical book, provided the engine is deterministic: time comes only from the sequencer's stamp, no unordered hash-map iteration, no floating point, no dependence on thread timing.

Failover is a leader election: a follower with every committed entry takes over with a book that is already correct. Gateways resend unacknowledged orders and the token deduplicates them. Snapshots written by a follower, so the leader never pauses, bound restart to "load snapshot, replay a few minutes of log". Even the coarse version is fast: LMAX snapshotted nightly and reported a full restart, including replaying a day's journal, in under a minute (Martin Fowler).

The sequence number, not any timestamp a member claims, decides who was first. Exchanges also equalize cable lengths from every colocation cage, and IEX went further with a "speed bump" that delays inbound orders and messages by 350 µs while its own view of other venues' prices is not delayed. The coil was famously 38 miles of fiber; after IEX moved its primary data center in 2024 it lengthened the coil to 44 miles to keep the delay at 350 µs (IEX). The delay is the product, not the cable.

3. Market data and pre-trade risk at speed

Every book change goes to thousands of subscribers. TCP per subscriber would make the publisher's send order an unfair advantage and multiply bandwidth. UDP multicast sends each packet once and lets switches copy it, so every colocated subscriber gets the same bytes at nearly the same time. Reliability is added back by the protocol: gapless sequence numbers per channel (Nasdaq frames ITCH messages in MoldUDP64 packets, and each packet is transmitted once however many listeners there are (MoldUDP64 spec)), two identical feeds on separate paths, and a retransmission server plus snapshots.

Pre-trade risk sits on the gateway so it runs in parallel across members: per-order limits (quantity, notional, price band) and per-member exposure from counters fed by execution reports, with a margin because the counters lag the engine by microseconds. The kill switch makes the gateway reject everything from a member and injects mass-cancels into the sequenced stream, so they are ordered like any other message.

The same problem shape, one ordered log driving identical replicas, appears in Design Google Docs with operations on a document and in Raft itself, which is a replicated log with a deterministic state machine on top.

Down the stack

Layer 1The contract

Acknowledged means durable. Accepted or Executed is sent only after the input that caused it is on a quorum of journal replicas.
Priority. For any two resting orders on the same side, the one at the better price fills first; at the same price, the one with the lower sequence number fills first.
Determinism. Output for input sequence numbers 1 to n is a pure function of those inputs and the start-of-day reference data.
Market data. Within a channel, events carry consecutive sequence numbers; a subscriber that applies them in order holds the exact public book.
Idempotent entry. A second EnterOrder with a token already used that day in that session is rejected as a duplicate, never matched.

Layer 2The mechanism

The core is an event loop over a ring buffer, the structure the LMAX Disruptor made famous (Disruptor paper). The network thread writes sequenced messages into pre-allocated slots of a power-of-two ring; the matching, journal and replication threads each read the slots in order and publish their own sequence counter, which producers check before overwriting a slot. No queue nodes are allocated and no locks are taken.

Per message, the engine validates against book state (halted? order ref known?), matches while crossing, rests or discards the remainder by time-in-force, and emits events. Market data mirrors book operations one to one: ITCH's A (add), E (executed), X (partial cancel), D (delete) and U (replace) let a subscriber keep a full-depth copy of the book, and every message carries a timestamp in nanoseconds since midnight (Nasdaq TotalView-ITCH 5.0).

Layer 3The machine

Network. The kernel network path costs a few microseconds per packet in syscalls, copies and interrupts. Kernel-bypass stacks (DPDK, or Solarflare's OpenOnload and ef_vi) poll the NIC's receive ring from user space and cut that to around a microsecond. NIC hardware timestamps synchronized with PTP give nanosecond-resolution audit records.
CPU. The matching thread is pinned to an isolated core (isolcpus, interrupts steered elsewhere) and busy-spins, because waking a sleeping thread costs tens of microseconds. An L1 hit is about 1 ns and DRAM about 80 to 100 ns, so order nodes are one 64-byte cache line and hot books stay in cache.
Memory. Order nodes come from pre-allocated pools, so the steady state allocates nothing: no garbage collection pauses on the JVM, no malloc contention in C++ or Rust.
Storage. Durability comes from replication into other hosts' memory (10 to 30 µs round trips in a rack), with the journal written to disk in large sequential appends, not an fsync per message.
Physics. Light in fiber travels about 5 ns per metre, so 100 m of extra cable costs 0.5 µs.

Why it's built this way

Members compete on who is first, so the exchange must define one total order of events; that order is the sequencer's log.
A total order processed by many threads would need locks on the same top-of-book data, so one thread per book is both faster and simpler.
One thread per book is a single point of failure, so availability comes from replicas running the same deterministic code on the same log.
Replicas only agree if the code is deterministic, which forbids wall clocks, unordered iteration and floating point in the engine.
Microsecond budgets make every kernel crossing and cache miss visible, so the machine is tuned around the one hot thread: bypassed network, pinned core, pre-allocated memory.

Failure modes

10× load at the open or on news. Opening auctions absorb the queued orders in one batch match, but the batch match is itself a hot path to load-test. At the Facebook IPO in 2012, Nasdaq's cross took 20 ms instead of the usual 1 to 2 ms because members had entered over 496,000 orders, and cancels arriving during that window kept failing its validation check, so the cross looped (SEC). During continuous trading, per-session message throttles and order-to-trade ratio limits protect the engine, and gateways reject with "throttled" rather than queueing, so latency stays flat for everyone else.
Engine leader crash. A follower with all committed entries becomes leader within milliseconds to low seconds. Members resend unacknowledged orders with the same tokens. Market data continues from the next sequence number, so subscribers see a pause, not a gap.
Region or site outage. Exchanges run a primary data center and a disaster-recovery site that receives the journal asynchronously. A site failover is a declared event with a halt, a reconciliation of the last trades with members and clearing, and a reopening; trading does not silently resume elsewhere.
Hot symbol. A meme stock can generate a tenth of all messages. A single book cannot be split, so it gets a dedicated engine thread and core, and the partition map is rebalanced overnight from the previous day's traffic.
Runaway algorithm. A member's bug floods orders or trades through the book. Price bands and per-member limits at the gateway catch most of it; the kill switch stops the rest; market-wide circuit breakers halt trading when prices move too far, too fast, and the halt is itself market data (ITCH carries MWCB decline-level and status messages (Nasdaq TotalView-ITCH 5.0)).
Market data loss. A subscriber misses packets on both feeds. It detects the sequence gap immediately, requests retransmission, and marks its book stale until it catches up.

In the wild

Nasdaq, 2012: the Facebook IPO cross that would not finish. Members entered more than 496,000 orders into the opening cross, against test runs capped at 40,000. The calculation took 20 ms instead of 1 to 2 ms, a cancel arrived during it, the validation check failed and forced a recalculation, more cancels arrived, and the cross looped. Nasdaq completed it by failing over to a copy of the matching engine with the validation code removed, which made the exchange take the other side of the last-moment cancels and violated order-priority rules. The SEC fined Nasdaq $10 million (SEC).
Knight Capital, 2012: a member's runaway algorithm. A technician deployed new order-routing code to seven of eight servers. On the eighth, a reused flag woke retired "Power Peg" logic that no longer tracked fills, so it kept sending child orders: about 4 million executions in 154 stocks in roughly 45 minutes and a loss of more than $460 million. Removing the new code from the seven good servers made all eight run Power Peg. This is the failure that gateway limits and a member kill switch exist to contain (SEC).

Trade-offs

Option	Strengths	Weaknesses	Choose when
Single thread per book	No locks, deterministic, cache friendly	Hot symbol capped by one core	Matching engines, almost always
Locked multithreaded book	Uses more cores per symbol	Contention on top of book, nondeterministic	Rarely, if ever
Replicate input log, deterministic replicas	Fast failover, perfect audit replay	Engine must be strictly deterministic	Exchanges and other order-sensitive state machines
Replicate state to a database	Familiar tooling	Too slow for millions of changes per second	Post-trade systems, not matching
Multicast market data	Fair, one send for all subscribers	Needs gap recovery and redundant feeds	Colocated, latency-sensitive subscribers
TCP or WebSocket market data	Reliable, internet friendly	Per-subscriber cost, ordering unfairness	Retail and remote subscribers, via a separate distribution tier
Continuous matching	Immediate execution	Rewards raw speed	Most of the trading day
Frequent batch auctions	Removes speed races within a batch	Delay, unfamiliar to members	Opens, closes and some alternative venues

Leadership lens

Build first: the sequencer and replicated journal, the single-threaded engine with a replay harness that compares outputs byte for byte, and the gateway with token deduplication. Market data and risk features build on these.
Team ownership: a core team owns the sequencer, engine and journal with an exceptionally high bar for change; a connectivity team owns gateways and protocols; a market data team owns feeds and recovery; a market operations group owns halts, kill switches and reference data.
Cost drivers: low-latency hardware and colocation space, redundant network paths, and engineering time spent on latency regressions. Storage is cheap by comparison.
Rollout: run a new engine version as a shadow follower on the production log for weeks, diffing its outputs against the leader's; promote it only on a weekend with a full replay of recent trading days, and keep the previous version as a follower for rollback.
Review questions: what in this change could make two replicas diverge; what is the p99.9 latency at three times today's peak; what does a member see during failover, message by message.

Check yourself

Why use one thread per order book instead of a multithreaded engine with locks per price level?
Show answer
Every order touches the top of the book, the same few cache lines, so locked threads serialize anyway, pay for cache-line transfers between cores, and produce results that depend on thread scheduling. One thread per partition needs no locks or system calls and is deterministic; scale comes from partitioning symbols across engines.
Why replicate the sequenced input log rather than the book state or run active-active engines?
Show answer
State replication is too slow for a book that changes millions of times a second, and active-active engines would disagree about who arrived first. Deterministic followers replaying the same log hold an identical book, so failover is a leader election in milliseconds to low seconds.
The engine leader crashes right after receiving an order. Is anything lost or duplicated?
Show answer
No acknowledged order is lost, because Accepted is sent only after the input is on a quorum of journal replicas. Members resend unacknowledged orders with the same tokens, which deduplicate them, and market data continues from the next sequence number, so subscribers see a pause, not a gap.
Why can all order books be held in memory with room to spare, and why does it matter?
Show answer
With 8,000 symbols, about 5,000 resting orders each and 64-byte order nodes, the total is about 2.6 GB. A busy symbol's book fits in a core's L2 or L3 cache, which is what makes microsecond matching possible.

Sources and further study

Primary

OUCH 5.0 order entry specification (Nasdaq): a real binary order-entry protocol with tokens, replaces and cancels.
SEC order on Nasdaq and the Facebook IPO (Release 34-69655, 2013): a regulator's minute-by-minute account of an auction, a failover and the rule violations that followed.

Systems docs

IEX technology: the speed bump (IEX): how a fixed inbound delay is built and why its length changed in 2024.
Aeron Cluster overview (Aeron): Raft-sequenced replicated log with snapshots, used for order books.
Shortening the securities settlement cycle (SEC press release, 2023): the T+1 rule and its 28 May 2024 compliance date.

Books

System Design Interview, Volume 2 (Xu and Lam, 2022), Ch 13 "Stock Exchange": sequencer, order book, market data and event-sourced recovery at interview depth.
Designing Data-Intensive Applications (Kleppmann, 1st ed., 2017), Ch 9 "Consistency and Consensus": total order broadcast and state machine replication, the theory under the sequencer.
Systems Performance (Gregg, 2nd ed., 2020), Ch 6 "CPUs", Ch 7 "Memory" and Ch 10 "Network": the analysis behind pinned cores, cache-line-sized nodes and kernel bypass in Layer 3.

Other curricula

Event Sourcing pattern (Azure Architecture Center): frames the input journal as the system of record and the book as a rebuildable projection with snapshots.
Reliability and constant work (AWS Builders' Library): frames systems that do the same work regardless of load, the property a busy-spinning, pre-allocated engine aims for.

Watch

How to Build an Exchange (Jane Street): Brian Nigito on sequencers, deterministic state machines, multicast market data and failover in real exchanges.

Free resources

The LMAX Architecture (Martin Fowler): a single-threaded, in-memory business logic processor fed by an event-sourced input log, with replay for recovery.
LMAX Disruptor technical paper: the ring buffer and sequence-counter design behind lock-free handoff between threads.
Nasdaq TotalView-ITCH 5.0 specification: the real message types for a full-depth order book feed.
MoldUDP64 protocol specification (Nasdaq): how sequenced multicast packets and retransmission requests work.
Aeron Cluster overview: an open-source Raft-based replicated log for deterministic services such as matching engines.
Order book (Wikipedia): background on bids, asks, depth and matching rules.

Open in the interactive app: practice, quiz, mock interview →