Design a team collaboration chat tool

In one minute

• The channel is the unit of everything. Order, storage partitioning, fan-out and permissions all key off channel_id. Get that choice right and most of the design follows from it.
• Give every channel its own sequence numbers. Each message gets a dense, gap-free seq inside its channel, assigned in the same transaction that stores it. Clients then sync by cursor and can see when they missed something.
• Push is best-effort. The log is the truth. WebSocket delivery makes the product feel live, but a client that drops a frame repairs itself by fetching seq > cursor from the history API. Never design for "push must not lose anything".
• Fan out to gateways, not to users. A message in a 50,000-member channel goes to the few hundred gateway hosts that hold those members' sockets. It does not become 50,000 writes.
• Reconnects decide your peak load, not message sends. A deploy or a network blip that reconnects millions of clients at once is the load you plan capacity for.

Requirements

Functional

• Send messages to channels and DMs. Messages can be edited, deleted and threaded.
• Receive new messages in real time on every connected device.
• Open a channel and scroll its history. Show unread counts and mentions per channel.
• Start or reconnect a client and catch up on what was missed, without downloading everything.
• Push notifications go to users who are offline or idle.

Non-functional

• Send-to-receive latency: p50 under 200 ms and p99 under 1 s within a region.
• Ordering: everyone sees one channel in the same order. There is no order across channels.
• Durability: once the sender sees the message acknowledged, it is never lost.
• Availability: 99.99% for send and read. Losing presence or typing indicators is acceptable.
• Workspaces range from 5 members to more than 100,000, and some channels have tens of thousands of members.

Out of scope: huddles, file storage internals, search ranking, enterprise key management and Slack Connect federation.

Back-of-envelope

These are assumed round numbers, not Slack's published figures.

• DAU: 20M. Each user sends 40 messages a day, so 20M × 40 = 800M messages a day.
• Write QPS: 800M / 86,400 ≈ 9,300/s on average. Work hours bunch traffic into about a third of the day, so peak is about 3× that, roughly 30k/s.
• Deliveries: if each message reaches 30 online recipients on average, that is 9,300 × 30 ≈ 280k deliveries/s on average and about 1M/s at peak.
• Reads: each user switches channels or scrolls about 100 times a day, which is 2B history reads a day ≈ 23k/s on average and about 70k/s at peak. Most of these hit a cache of the newest page.
• Connections: if 60% of DAU are connected at peak, that is 12M WebSockets. At about 50k sockets per gateway host, you need around 240 hosts plus headroom, so call it 400.
• Storage: a message plus metadata and indexes is about 1 KB. 800M × 1 KB = 800 GB a day, and 800 GB × 365 × 5 ≈ 1.5 PB over five years. With 3 replicas that is about 4.4 PB. Files are stored separately in object storage.
• Bandwidth: 1M deliveries/s × about 1 KB ≈ 1 GB/s, or 8 Gbps leaving the gateways at peak. That is about 20 Mbps per gateway, so memory per socket is the real limit.

API

POST /v1/channels/{channel_id}/messages
  { "client_msg_id": "uuid-v4", "text": "ship it", "thread_root_seq": null }
  -> 200 { "channel_id": "C42", "seq": 18231, "message_id": "M9f..", "ts": 1760500000123 }

GET /v1/channels/{channel_id}/messages?after_seq=18200&limit=50
  -> 200 { "messages": [ { "seq": 18201, "author": "U7", "text": "...", "ts": ... } ], "has_more": false }

POST /v1/sync
  { "cursors": { "C42": 18200, "C77": 990 } }
  -> 200 { "channels": { "C42": { "head_seq": 18231, "unread": 31, "mentions": 1 },
                          "C77": { "head_seq": 990,   "unread": 0,  "mentions": 0 } } }

POST /v1/channels/{channel_id}/read   { "seq": 18231 }  -> 204

WS /v1/connect  (after the hello frame, server pushes events)
  <- { "type": "message", "channel_id": "C42", "seq": 18231, "message": { ... } }
  <- { "type": "message_changed", "channel_id": "C42", "seq": 18100, "edit_version": 2 }
  -> { "type": "ping", "id": 17 }

client_msg_id makes a send idempotent. If a phone retries after a timeout, the server returns the original seq and does not create a duplicate.

Data model

messages is partitioned by channel_id, with primary key (channel_id, seq). Columns: message_id (globally unique, time-sortable), client_msg_id, author_id, ts, text, thread_root_seq, edit_version, deleted. It has a unique secondary index on (channel_id, client_msg_id) so retried sends are deduplicated. This is the right partition key because history reads are always "one channel, a range of seq". A clustered primary key turns them into one sequential range scan on one shard.

channels is partitioned by channel_id and lives on the same shard as its messages. Columns: workspace_id, name, kind (public, private, dm), head_seq, member_count. head_seq is the per-channel counter. Keeping it on the same shard as the message rows lets one local transaction both increment it and insert the message.

memberships holds (user_id, channel_id) -> last_read_seq, notify_prefs. It is stored twice:

• partitioned by user_id, for the sidebar and sync
• partitioned by channel_id, for fan-out and permission checks

Both copies are written through an outbox, so they agree eventually.

Sharding: relational shards with Vitess-style routing on channel_id give range scans and transactions for the counter (Slack moved to Vitess between 2017 and 2020, partly so it could shard message data by channel instead of keeping a whole workspace on one shard, and by late 2020 Vitess served 99% of its MySQL traffic at 2.3M QPS peak, Slack). A wide-column store such as Cassandra or ScyllaDB also works with partition key (channel_id, time_bucket), but then seq must come from a separate counter service.

High-level design

flowchart LR
  C["Client (desktop/mobile)"] -->|HTTPS send / history| API["API tier"]
  C <-->|WebSocket| GW["Gateway servers"]
  API --> MS["Message service"]
  MS -->|"txn: head_seq+1, insert"| DB[("Channel shards")]
  MS -->|publish after commit| CS["Channel servers (hash by channel)"]
  CS -->|fan-out to subscribed gateways| GW
  MS --> OB["Outbox -> Kafka"]
  OB --> PN["Push notifications"]
  OB --> SR["Search indexer"]
  OB --> UR["Unread counters"]
  API --> HC["History cache (latest page)"]
  HC --> DB

Write path.

1. The client sends POST /messages with a client_msg_id.
2. The API tier checks membership against a cached membership set.
3. The message service runs one transaction on the channel's shard. It increments channels.head_seq, inserts the row with that seq, and writes an outbox row.
4. On commit it returns seq to the sender. The sender's client changes the message from "pending" to "sent" and moves it to its final position.
5. The message service publishes the event to the channel server that owns channel_id, chosen by consistent hashing.
6. The channel server looks up which gateways have members of this channel connected and sends one frame to each.
7. Each gateway writes the frame to the matching local sockets.
8. Separately, the outbox relay feeds Kafka. Kafka drives push notifications to offline users, search indexing, link unfurls and unread counters.

Read path.

• Opening a channel calls GET /messages?after_seq=... or fetches the newest page. The newest page is served from a cache keyed by channel_id that is invalidated on each write, so a hot channel costs one DB read per write rather than one per viewer.
• Startup or reconnect. The client calls /sync with its cursors. It gets back head sequences and unread counts, which are a few bytes per channel. It loads message bodies only for the channel on screen.

Deep dives

1. Ordering: who assigns the sequence number

Three options were considered.

• Server timestamps or Snowflake IDs. These need no coordination, but two API hosts with 5 ms of clock skew can produce IDs that do not match commit order. Clients also cannot tell "no message there" from "a message I missed", because the IDs are sparse. Slack's own public API works this way: a message is identified by its ts, which is guaranteed unique only within its channel (Slack docs), so gap detection needs something more than the ID.
• A single in-memory sequencer per channel on the channel server. This is fast, but the sequencer becomes stateful. It needs leases and fencing when it fails over, or two owners will issue the same seq.
• A counter row in the channel's shard, incremented in the same transaction as the insert. This is what I chose. The row lock on channels orders concurrent writers, and commit order matches seq order. The cost is that one row serialises every write to that channel. A row lock with group commit handles roughly 1k to 5k updates per second, and even a very busy channel sees tens. This gives dense, gap-free numbering, which is exactly what client-side gap detection needs.

2. Real-time fan-out

The naive design keeps a user -> socket map and loops over channel members. For a 50k-member channel that means 50k lookups, most of them for offline users.

Instead, the routing keys on channels. When a client connects, its gateway subscribes to every channel the user belongs to on the owning channel servers. A channel server then holds channel_id -> set(gateway_id). One message becomes one frame per gateway, at most a few hundred. Each gateway keeps its own local channel_id -> sockets index. This is how Slack describes its own system: channels map to in-memory channel servers by consistent hashing, about 16 million channels per host at peak, gateways in several regions subscribe to them asynchronously after the client connects, and a message crosses the world in about 500 ms (Slack).

The follow-on problem is subscription churn: a user in 300 channels creates 300 subscriptions per connect, so gateways batch them per channel server and stay subscribed while any local socket needs the channel. For announcement channels, also drop typing indicators and coalesce reactions into one update every 1 to 2 seconds.

3. Client sync and gap detection

For each channel, the client stores cursor, the highest seq it holds contiguously. When a push event arrives:

• If seq == cursor + 1, the client applies it and moves the cursor forward. It also drains any buffered events that are now contiguous.
• If seq > cursor + 1, the client buffers the event, records the missing range, and asks for after_seq=cursor.
• If seq <= cursor, the event is a duplicate and the client drops it.

On reconnect the client sends all of its cursors to /sync. A user with 300 channels sends about 300 × 16 bytes ≈ 5 KB.

The server answers with heads only. A client whose cursor equals the head has nothing to fetch. One that is behind shows an unread badge and fetches when the user opens the channel. If it is more than a page behind, it jumps to the newest page and fills older messages as the user scrolls, so it never replays thousands of messages.

Edits bump edit_version on the original row instead of taking a new seq, because they do not change the order of messages.

Down the stack

UPDATE channels SET head_seq = head_seq + 1 WHERE id = ?
SELECT head_seq
INSERT INTO messages ...
INSERT INTO outbox ...
COMMIT

Failure modes

• Load at 10×, usually a reconnect storm. A gateway deploy or a regional network event triggers millions of /sync calls at once. Mitigations: - clients reconnect with exponential backoff and full jitter, which spreads retries far better than backoff alone (AWS Builders' Library) - sync returns only heads, served from a cache of per-channel head_seq - gateways drain slowly during deploys, about 1% of sockets per second - the boot payload is served from an edge cache; Slack's Flannel cut the startup payload 7× on a 1,500-user team and 44× on a 32,000-user team (Slack)
• Hot channel. A 100k-member #general announcement causes a burst of reads plus thousands of reactions. Mitigations: - the newest-page cache absorbs the reads - reactions are counted asynchronously and coalesced - posting permission is restricted
• Shard primary failure. Writes to the channels on that shard stall for the failover window, typically 10 to 30 seconds. Mitigations: - clients keep a durable outbox of unsent messages keyed by client_msg_id and retry, so failover never produces duplicates - reads continue from replicas
• Region outage. Each workspace has a home region. The secondary region holds asynchronous replicas that may be a few seconds behind, and promoting them can lose those last seconds of acknowledged messages unless replication is synchronous across regions. Most teams accept that small RPO for chat and document it.
• Channel server crash. Its channels are reassigned and gateways resubscribe. Slack reports a replacement channel server is ready in under 20 seconds (Slack). Clients repair pushes lost in the gap through gap detection or the periodic heartbeat sync.

In the wild

• Slack, 2021: the first Monday back is a reconnect storm. On 4 January 2021, clients returning from the holidays with cold caches pulled more data than usual at the 7 a.m. peak. AWS Transit Gateways did not scale fast enough and dropped packets, autoscaling thrashed while trying to add about 1,200 servers, and Slack's own monitoring sat behind the same gateways. Slack was degraded or down for about 3.5 hours and set itself a reminder to ask AWS for pre-emptive gateway upscaling before the next holiday season (Slack).
• Slack, 2022: a cache miss becomes a query to every shard. A gradual Consul upgrade emptied memcached nodes. Group DM membership was cached by channel but the Vitess table was sharded by user, so each miss turned into a scatter query across all shards; the database overloaded, could not refill the cache, and client retries piled on, so many users could not boot Slack. The fix moved the query to a channel-sharded table, which is the same partition-key argument this design makes (Slack).
• Discord, 2017: fan-out to 30,000 sessions. Each Discord guild is one Erlang process that sends every event to every connected session. In its largest guilds, with up to 30,000 concurrent users, each send cost 30 to 70 µs, so one publish took 900 ms to 2.1 s. Discord's Manifold groups recipients by node and sends one batch per node, the same "fan out to gateways, not users" idea (Discord).

Trade-offs

Leadership lens

• Build first: the channel log with seq, idempotent send and the /sync cursor protocol. A product with polling and correct catch-up beats a fast one that loses messages. Add the gateway and channel-server layer second, with push treated as an optimisation from day one.
• Team ownership: - the messaging core team owns the message service, shards and the seq contract - a real-time infrastructure team owns gateways and channel servers - client teams own the cursor and gap-repair logic, with shared test vectors - the notifications and search teams consume Kafka
• Cost drivers: gateway RAM per connection and how long sockets stay connected, five-year message storage (consider moving cold history to cheaper storage after a year), and cross-region replication traffic.
• Rollout: shadow-write the new store, compare per channel, move workspaces in cohorts, and keep a kill switch that falls back to client polling.

Check yourself

1. Why assign seq from a counter row in the channel's shard instead of using Snowflake IDs?

   Show answer

   Snowflake IDs need no coordination, but clock skew between hosts can produce IDs that do not match commit order, and sparse IDs stop clients from telling a gap from a missed message. A counter incremented in the same transaction as the insert gives dense, gap-free numbering in commit order, at the cost of capping writes per channel at a few thousand per second.

2. A gateway drops a push frame. How does the client end up with every message anyway?

   Show answer

   Push is best-effort and the log is the truth. When a later event arrives with seq > cursor + 1, the client buffers it, records the missing range and fetches after_seq=cursor from the history API, so combining push with fetch yields every message in order.

3. Why is a reconnect storm, not message sends, the load you plan capacity for, and how is it kept affordable?

   Show answer

   A deploy or network blip can trigger millions of /sync calls at once. Clients reconnect with exponential backoff and full jitter, sync returns only per-channel heads from a cache (about 5 KB for 300 channels), and gateways drain about 1% of sockets per second during deploys.

4. Why fan out to gateways instead of looping over channel members, and roughly how many gateways are there?

   Show answer

   Looping over a 50k-member channel means 50k lookups, mostly for offline users, while a channel server holding channel_id -> set(gateway_id) sends one frame per gateway, at most a few hundred. With about 12M sockets at peak and 50k per host, that is around 240 hosts, provisioned as about 400.

Sources and further study

Primary

• RFC 6455: The WebSocket Protocol: framing, masking, ping and pong, and close semantics the gateways rely on.
• Lamport, "Time, Clocks, and the Ordering of Events in a Distributed System" (1978): why wall-clock IDs cannot give an order every reader agrees on.

Systems docs

• MySQL 8.4: Semisynchronous replication: what the replica acknowledges and the silent fallback to asynchronous mode.
• Vitess documentation: Sharding: keyspaces, vindexes and how a sharding key like channel_id routes queries.
• Slack: Retrieving messages: the public history API, cursors and the per-channel ts message ID.

Books

• System Design Interview, Volume 1, Alex Xu, chapter 12 "Design A Chat System": per-channel message IDs, sync across devices and stateful chat servers.
• Designing Data-Intensive Applications, Martin Kleppmann, chapter 9 "Consistency and Consensus": ordering guarantees, sequence-number ordering and total order broadcast.

Other curricula

• AWS Builders' Library: Timeouts, retries and backoff with jitter: treats reconnect and retry storms as the load to design for, with jitter as the main defence.
• Azure Architecture Center: Publisher-Subscriber pattern: frames channel fan-out as pub/sub and lists the ordering and duplicate-message caveats.

Watch

• Keith Adams on the Architecture of Slack (InfoQ): Slack's chief architect on MySQL sharding, edge caching and the messaging server.
• Scaling Slack, Keith Adams (GOTO 2018): how the architecture changed as workspaces and channel sizes grew.

Free resources

• Real-time Messaging (Slack Engineering): Slack's own description of channel servers, gateway servers and how a message reaches clients.
• Scaling Datastores at Slack with Vitess (Slack Engineering): why Slack moved from workspace sharding to Vitess, and the migration.
• Flannel: An Application-Level Edge Cache (Slack Engineering): how lazy loading at the edge made client boot cheap for large workspaces.
• How Slack Built Shared Channels (Slack Engineering): what changes when one channel spans two organisations' data.
• How Discord Stores Trillions of Messages (Discord): a wide-column alternative, with time-bucketed partitions and request coalescing for hot channels.