Engineering Atlas › System Design Interviews

Design a push notification platform

19 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

Requirements

Functional

Non-functional

Out of scope: the content of marketing campaigns, audience segmentation queries, A/B testing of copy, and building our own SMS or email transport.

Back-of-envelope

Assumed numbers for a large consumer app.

API

POST /v1/notifications
  { "user_id": "U81", "template": "order_shipped", "category": "transactional",
    "priority": "high", "dedup_key": "order-5521-shipped",
    "vars": { "order_id": "5521", "eta": "Friday" }, "send_at": null }
  -> 202 { "notification_id": "N01J9...", "status": "accepted" }
  -> 200 { "notification_id": "N01J8...", "status": "duplicate" }   (same dedup_key within 24h)

POST /v1/campaigns
  { "template": "spring_sale", "segment_id": "S77", "category": "marketing",
    "window": { "start": "2026-04-01T09:00", "end": "2026-04-01T12:00", "local_time": true } }
  -> 202 { "campaign_id": "C12", "estimated_recipients": 41200000 }

GET  /v1/notifications/{id}        -> { status, channels: [ { channel, state, provider_id, updated_at } ] }
PUT  /v1/users/{id}/preferences    { categories: { marketing: ["email"], social: ["push", "email"] },
                                     quiet_hours: { start: "22:00", end: "07:00" }, time_zone: "Europe/Berlin" }
POST /v1/users/{id}/devices        { platform: "ios", token: "...", app_version: "8.4.1" }
POST /v1/providers/{name}/events   (webhooks from email and SMS providers: delivered, bounced, complaint, STOP)

Ingest returns 202 because sending is asynchronous. The dedup key is scoped per user, and a repeat returns the original notification ID so a producer's retry is harmless.

Data model

notifications in a wide-column store, partition key (user_id, month), clustering key notification_id (time-sortable). Columns: template, category, dedup key, and a map of channel to state. Reads are almost always one user's recent notifications, so partitioning by user keeps them on one node; the month bucket bounds partition size.

preferences in a relational or key-value store, keyed by user_id: opt-ins per category and channel, quiet hours, IANA time zone name, locale. Read on every send, so cached and invalidated on write.

device_tokens keyed by user_id: (token, platform, app_id, app_version, last_seen, invalid).

dedup in Redis: dd:{user_id}:{dedup_key} -> notification_id with a 24-hour TTL, written with SET NX EX so the check and the claim are one atomic step.

rate counters in Redis keyed (user_id, channel, window); small, short-lived, loss-tolerant.

suppression list keyed by address or phone number: hard bounces, spam complaints and SMS STOP replies. Checked before every email or SMS, because sending to these hurts reputation or breaks the law.

attempts keyed by (notification_id, channel): state (pending, sent, failed), provider_message_id, attempt_count. This is what makes a worker crash safe to recover from.

High-level design

flowchart LR
  P["Producer services"] --> API["Ingest API"]
  API --> D[("Dedup (Redis)")]
  API --> K["Kafka: critical / standard / bulk"]
  CS["Campaign service"] --> K
  K --> EL["Eligibility workers"]
  EL --> PR[("Prefs + rate counters")]
  EL -->|"quiet hours"| SCH["Delay scheduler"]
  SCH --> K
  EL --> R["Renderer (templates, locale)"]
  R --> PW["Push workers"] --> APNS["APNs / FCM"]
  R --> EW["Email workers"] --> ESP["Email provider"]
  R --> SW["SMS workers"] --> SMSP["SMS provider"]
  ESP -->|"webhooks"| FB["Feedback processor"]
  APNS -->|"410 / UNREGISTERED"| FB
  FB --> PR

Write path.

  1. A producer calls POST /notifications. The API validates the template and runs SET NX on the dedup key. A duplicate returns the earlier ID.
  2. The request is written to the Kafka topic for its priority, partitioned by user_id. The API returns 202 only after Kafka acknowledges with acks=all.
  3. An eligibility worker loads preferences and checks suppression. If the category is opted out, the notification is recorded as suppressed. If the user is in quiet hours and the category is not critical, it goes to the delay scheduler with a release time.
  4. The worker walks the user's channel order for this category and picks the first channel that is enabled, reachable (a valid token or verified address) and under its rate limit.
  5. The renderer fills the localised template. The channel worker writes a pending attempt row, calls the provider, then records sent with the provider's message ID.
  6. Provider feedback (delivery receipts, bounces, dead tokens) arrives asynchronously and updates status, the suppression list and the token table.

Read path. The in-app inbox reads notifications by (user_id, month) with a page cursor, and the unread badge comes from a per-user counter. Status lookups for support tools read the same partition.

Deep dives

1. The eligibility decision

The decision is a pure function of the request, the user's settings, their recent send history and the current time. Keeping it pure (clock injected) makes it testable and replayable, so you can explain why something was or was not sent.

Quiet hours are the trap. Storing a UTC offset is wrong for any region with daylight saving time, because the offset changes twice a year; store the IANA zone name and convert with a tz database. Windows that cross midnight (22:00 to 07:00) need a wrap-around comparison. The release time should be the end of the window plus a random spread of a few minutes, or every deferred notification in a time zone fires at exactly 07:00:00 and you build your own thundering herd. Firebase's own scaling guidance makes the same point from the provider side: ramp traffic gradually and spread requests across each quota window instead of front-loading them (FCM: sending at scale).

Frequency caps use a sliding window per user and channel, for example at most 3 marketing pushes per 24 hours and 2 SMS per hour. When the first channel is capped, the fallback order decides: a security alert might go push, then SMS, then email; a social notification might go push, then in-app only, and never SMS. Fallback is per category, chosen by product and finance together, because the cheapest-first and fastest-first orders differ.

2. Priority isolation and campaign fan-out

Option one is a single queue with a priority field. Kafka partitions are FIFO, so a passcode sits behind a million campaign messages already in the partition. Option two is one topic per priority with separate consumer groups and worker pools: critical, standard and bulk. I choose option two, and size the critical pool to stay mostly idle.

Campaigns are a fan-out problem (see the Fanout lesson). The campaign service pages through the segment and emits per-user messages into the bulk topic at a controlled rate, so the 41M-recipient campaign above becomes a steady stream instead of a spike. "Local time" campaigns are emitted per time zone as each zone reaches 09:00. Rate-shaping at the source also means a bad campaign can be paused mid-flight by stopping the emitter, the most valuable control in an incident.

3. Duplicates and partial failure

Duplicates come from three places. Producers retry an ingest call that timed out: the dedup key catches that. A worker crashes after calling the provider but before committing its Kafka offset: on restart it reprocesses the message. The attempt row catches most of these; if it says sent, skip. If it says pending, the worker cannot know whether the provider received the request, because APNs and FCM accept no idempotency key. For push, resend with the same replacement key so the device replaces the earlier notification instead of showing two: apns-collapse-id (at most 64 bytes) on iOS, and android.notification.tag on Android. FCM's collapse_key is a different thing: it only collapses messages that are still queued while the device is offline, at most four keys per device, and FCM ignores it for notification messages (FCM collapsible messages). For SMS, where a duplicate costs money and annoys people, prefer to mark it failed and let a reconciliation job check the provider's message log. The third source is upstream: two services both deciding to notify about the same event. A dedup key derived from the event (order-5521-shipped) rather than a random UUID handles that.

Down the stack

Layer 1The contract

  • Ingest. A 202 means the request is durable in Kafka on all in-sync replicas. A repeat with the same (user_id, dedup_key) within 24 hours returns the same ID and causes no second send.
  • Decision. For a given request, settings and time, the eligibility result is deterministic: suppressed, deferred until T, rate-limited, or sent on channel C.
  • Delivery. At-least-once to the provider for accepted, eligible notifications. Critical notifications reach the provider within 2 s p99. No guarantee that the device displays it; the OS and the user decide that.

Layer 2The mechanism

APNs is an HTTP/2 API. Each notification is one POST /3/device/{token} stream on a long-lived TLS connection, authenticated by a signed JWT provider token. The JWT must be refreshed no more than once every 20 minutes and no less than once every 60 minutes, so workers share one cached token instead of minting one per request (Apple). Headers carry apns-push-type, apns-priority (10 for immediate; 5 and 1 for power-friendly delivery that the device may batch, throttle or drop), apns-expiration and apns-collapse-id. A 200 response returns an apns-id. A 410 means the token is dead and must be deleted; the reason is either Unregistered or the newer ExpiredToken, so match on the status code, not one reason string (Apple). FCM's HTTP v1 API takes a JSON message with android.priority, android.ttl and android.collapse_key (and apns and webpush blocks for other platforms) and returns UNREGISTERED (HTTP 404) for dead tokens (FCM error codes). HTTP v1 is now the only option: Google deprecated the legacy HTTP and XMPP APIs in June 2023 and began shutting them down in July 2024, so designs that still mention server keys are out of date.

Email goes through a provider over SMTP or HTTPS. Deliverability depends on SPF, DKIM signatures (RFC 6376) and a DMARC policy, and large senders must offer one-click unsubscribe through the List-Unsubscribe and List-Unsubscribe-Post headers (RFC 8058). Since February 2024 Gmail enforces this: anyone sending more than 5,000 messages a day to Gmail accounts needs SPF, DKIM and DMARC, one-click unsubscribe on marketing mail, and a reported spam rate below 0.3% (Google sender guidelines). Bounces and complaints come back as provider webhooks and feed the suppression list.

Kafka topics are partitioned by user_id, so all of one user's notifications are decided by the same worker in order. That lets the worker keep a hot cache of that user's preferences and lets rate-limit counters be updated without cross-worker races for the same user.

Layer 3The machine

  • HTTP/2 multiplexing. One TLS connection to APNs carries many concurrent streams, up to the limit the server advertises in SETTINGS_MAX_CONCURRENT_STREAMS, which RFC 9113 recommends be at least 100 (RFC 9113). HPACK compresses repeated headers to a few bytes each. A push worker with a few dozen connections can sustain thousands of sends per second, bounded by round-trip time: at 50 ms RTT and 500 streams per connection, one connection supports about 10k requests per second.
  • Connection reuse. A TLS 1.3 handshake adds a round trip and key-exchange CPU, so workers keep pools of warm connections.
  • Redis checks. SET NX EX and INCR are single-threaded in-memory operations of a few microseconds; the cost is the network round trip, about 0.2 to 0.5 ms in a datacenter. Pipelining the dedup, rate and preference lookups for a batch of 100 messages turns 300 round trips into one.
  • Kafka. Producers batch messages (linger.ms of 5 to 10 ms; the default changed from 0 to 5 ms in Kafka 4.0 (KIP-1030)) so each disk write and replication round trip carries hundreds of records. Brokers append to page-cache log segments and serve consumers with sendfile.

Why it's built this way

  1. Sending is irreversible and users punish spam by disabling notifications entirely, so the system invests in a careful, replayable decision before any send.
  2. Providers do not support idempotency keys, so exactly-once is impossible at the edge; the system dedups everything it can upstream and uses collapse IDs to hide what remains.
  3. Critical and bulk traffic differ by four orders of magnitude in volume and urgency, so they get separate queues and workers.
  4. Providers throttle and fail independently, so each channel has its own worker pool, rate shaping and circuit breaker.
  5. SMS and email reputation have real costs, so feedback loops (suppression, dead tokens) are part of the core design, not an afterthought.

Failure modes

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
One topic per priorityPasscodes never wait behind campaignsMore topics and pools to operateMixed transactional and marketing traffic
Single queue with priority fieldSimpleHead-of-line blocking in FIFO partitionsLow volume, one kind of traffic
Dedup at ingest with Redis TTLCheap, stops producer retriesLost on cache failure; window is finiteAlmost always, as the first layer
Attempt record before sendSurvives worker crashesOne extra write per sendExpensive channels such as SMS
Collapse IDs at the providerHides residual duplicates on devicePush only; replaces rather than preventsPush notifications
Fixed fallback order per categoryPredictable cost and behaviourNot personalisedMost products
Learned channel selectionBetter engagementHard to explain, needs dataMature products with experimentation

Leadership lens

Check yourself

  1. A campaign to 80 million users starts and passcode latency jumps to 40 seconds. What design choice would cause that, and what fixes it?

    Show answer

    A single queue with a priority field, because Kafka partitions are FIFO and a passcode sits behind campaign messages already in the partition. The fix is one topic per priority with separate consumer groups and worker pools, plus a campaign emitter that sends into the bulk topic at a controlled rate.

  1. A push worker crashes after calling APNs but before recording success. What happens on recovery, and why can the system not simply guarantee exactly-once?

    Show answer

    The attempt row says pending, so the worker cannot know whether APNs received the request, and APNs and FCM accept no idempotency key. It resends with the same apns-collapse-id (or android.notification.tag on Android), so the device replaces the earlier notification instead of showing two.

  1. Using the lesson's numbers, why is SMS treated as a budget line, and what controls follow from that?

    Show answer

    SMS is only 1% of 2B notifications a day, but 20M messages at $0.005 to $0.05 each is $100k to $1M a day. That is why per-user, per-channel rate limits, a per-category fallback order chosen with finance, and per-team SMS budgets exist.

  1. Why must quiet hours be stored as an IANA time zone name, and what goes wrong if every deferred notification is released exactly at the window's end?

    Show answer

    A stored UTC offset is wrong in any region with daylight saving time, because the offset changes twice a year. Releasing everything at exactly 07:00:00 creates a thundering herd, so the release time adds a random spread of a few minutes.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →