Code a multi-channel notification module

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

As an object-design question, a notification system is the code inside one delivery worker: turn "order 812 shipped" into a rendered message, pick a channel, call a provider, and decide what to do when the provider says "try later" or "never". The distributed pipeline (queues, preference service, campaign fan-out) is covered in Design Notification Service.
Separate the three things that change at different speeds: templates (edited weekly by product), channels (added yearly: email, SMS, push, chat apps) and delivery policy (retries, fallbacks, tuned after incidents). Each gets its own abstraction.
A delivery is a small state machine: pending, sent, failed, with attempts and a next-attempt time. Retries back off exponentially per channel; a permanent error or exhausted retries moves to the next channel in a fallback chain.
Every send carries an idempotency key. Provider calls are slow, remote and sometimes ambiguous, so "send exactly once" is really "send at least once, suppress what you can recognise".
The problem shapes are partial failure (the provider times out after maybe sending) and duplicates (the producer retries the request). The same shape shows up in webhook delivery and in payment capture calls.

Requirements

Functional:

Producers (order service, auth service) request a notification with a template id, variables, a recipient and an ordered list of channels, for example push then SMS for a delivery alert.
Templates contain placeholders such as {{first_name}}; rendering fails loudly if a variable is missing rather than sending "Hi ,".
Each channel has its own sender (email via an SMTP or HTTP provider, SMS via a gateway, push via APNs or FCM).
Transient provider errors are retried with exponential backoff; permanent errors (invalid number, uninstalled app) and exhausted retries fall back to the next channel; the last failure marks the notification failed.
A request repeated with the same notification id does not send twice.
Status is queryable: pending on which channel, sent on which channel, or failed.

Non-functional:

A slow or broken provider must not block the other channels.
Retry timing is deterministic under test: time comes from an injected clock.
Templates and channels are added without editing the delivery engine.

Out of scope: user preferences and quiet hours, rate limits, campaign fan-out and analytics. The system design lesson owns those; here they would be one more filter before the engine.

Entities and relationships

NotificationRequest is what a producer submits: id (the idempotency key), recipient, template id, variables, channel list, priority. It is an immutable value.
Template has an id and a body per channel (a 160-character SMS body and an HTML email body are different texts). It knows how to render itself against variables.
Recipient carries the addresses the channels need: email, phone number, device tokens. Addresses come from a user directory, not from the producer, so a stale phone number is fixed in one place.
Channel is an interface with send(recipient, message) -> Result. EmailChannel, SmsChannel and PushChannel adapt specific providers.
Result is one of OK, RETRY (timeout, 429, 5xx) or FAIL (permanent: invalid address, unregistered token). Classifying provider errors into these three is the most important code in each channel.
Delivery is the mutable record: request id, rendered text, current channel index, attempts on that channel, next attempt time and state.
RetryPolicy decides the next attempt time; DeliveryEngine owns deliveries and runs due ones on each tick; Clock supplies time.

Relationships: the engine aggregates deliveries and holds a registry of channels and templates. A delivery references its request and template by id. Channels own nothing but their provider client.

Class diagram

classDiagram
    class NotificationRequest {
      +id: string
      +recipient: RecipientId
      +templateId: string
      +vars: Map~string, string~
      +channels: ChannelName[]
    }
    class Template {
      +id: string
      +bodies: Map~ChannelName, string~
      +render(channel, vars) string
    }
    class Delivery {
      +requestId: string
      +text: string
      +channelIndex: int
      +attempts: int
      +nextAt: int
      +state: DeliveryState
    }
    class Channel {
      <<interface>>
      +send(recipient, text) Result
    }
    class RetryPolicy {
      <<interface>>
      +nextAt(attempts, now) int
      +maxAttempts() int
    }
    class DeliveryEngine {
      -deliveries: Map~string, Delivery~
      +submit(request) bool
      +tick() Attempt[]
      +status(id) Status
    }
    class Clock {
      <<interface>>
      +now() int
    }
    DeliveryEngine "1" o-- "many" Delivery
    DeliveryEngine --> Channel
    DeliveryEngine --> Template
    DeliveryEngine --> RetryPolicy
    DeliveryEngine --> Clock
    Delivery --> NotificationRequest
    EmailChannel ..|> Channel
    SmsChannel ..|> Channel
    PushChannel ..|> Channel
    ExponentialBackoff ..|> RetryPolicy

Key flows

A delivery alert asks for push first and SMS as the fallback. Push times out twice, then the retry budget is spent, and SMS succeeds on the next tick.

sequenceDiagram
    participant O as OrderService
    participant E as DeliveryEngine
    participant T as Template
    participant P as PushChannel
    participant S as SmsChannel
    O->>E: submit(id n1, shipped, [push, sms])
    E->>T: render(vars)
    T-->>E: "Your order 812 has shipped"
    E->>E: Delivery n1 PENDING on push, due now
    E->>P: send (attempt 1)
    P-->>E: RETRY (timeout)
    E->>E: next attempt now + 10 s
    E->>P: send (attempt 2)
    P-->>E: RETRY
    E->>E: budget spent, fall back to sms, due now
    E->>S: send (attempt 1)
    S-->>E: OK
    E->>E: n1 SENT via sms

Design patterns used and why

Strategy (with Adapter inside). Each channel is a strategy behind Channel.send, and each implementation adapts one provider's API and error codes to the three-valued Result. Adding WhatsApp is one new class plus a registry entry.
Chain of Responsibility, as data. The fallback chain is an ordered list of channel names on the request, walked by the engine. A list beats linked handler objects here because the order is per request, not per deployment. See Chain of Responsibility.
State. A delivery moves pending, then sent or failed, with retries as self-transitions that bump attempts and nextAt. Keep it an enum plus guarded transitions in one method; the bugs come from transitions spread across callers.
Template Method, avoided on purpose. A tempting design is an abstract Notifier whose notify() calls format(), send() and log() hooks per channel subclass. It couples formatting to transport and grows a subclass per combination. Composition (template object plus channel object) does the same job with fewer classes.
Decorator for cross-cutting channel behaviour: a TimedChannel that enforces a deadline, a MetricsChannel that counts outcomes, a SandboxChannel that redirects every message to a test inbox in staging. Each wraps a Channel and is itself a Channel.
Observer at the edge: producers emit domain events, and a subscriber maps "OrderShipped" to a notification request. The engine never imports the order service.
Builder for requests with many optional fields (priority, collapse key, deep link, locale), so producers cannot create half-formed requests.

Concurrency and edge cases

Ambiguous sends. The provider times out after it may already have accepted the message. Retrying risks a duplicate, not retrying risks silence. Pass the notification id to providers that accept an idempotency key, collapse duplicates on the device with a collapse id for push, and accept that SMS may occasionally duplicate. Write the choice down per channel.
Duplicate submits. A producer retries submit after its own timeout. The engine keys deliveries by request id; a second submit is acknowledged and ignored. In a database this is a primary key on the id and an insert that tolerates conflict.
Two workers, one delivery. With several worker processes polling for due deliveries, two can pick the same row and both send. Claim rows with SELECT ... FOR UPDATE SKIP LOCKED or a conditional update that sets a lease, and only the claimant sends.
Render before queueing. Render at submit time and store the text, so a template edit between submit and retry does not change a message mid-flight, and a missing variable fails the producer's call instead of a background job at 3 a.m.
Slow channels. One provider at 30 seconds per call must not starve the others. Give each channel its own worker pool or concurrency limit (bulkhead) and a per-call timeout shorter than the retry interval.
Retry storms. If a provider is down for ten minutes, every delivery backs off on the same schedule and they all return together. Add jitter in production; the practice code omits it so tests stay exact.

Down the stack

Layer 1The contract

submit either rejects a bad request synchronously (unknown template, missing variable, unknown channel, empty channel list) or durably records exactly one delivery for that id; a repeat submit returns "duplicate" and changes nothing. Each tick attempts every delivery whose next-attempt time is at or before now, at most once per tick. After an attempt, a delivery is SENT on the channel that returned OK, still PENDING with a later time or the next channel, or FAILED when the last channel gives up. Delivery is at-least-once per channel attempt; the engine never sends on a later channel after an earlier one succeeded.

Layer 2The mechanism

In memory, deliveries live in a map by id plus a list or min-heap ordered by (nextAt, sequence) so a tick takes due items in a stable order. Backoff for attempt k on a channel is base * 2^(k-1), so with a 10-second base the waits are 10, 20, 40 seconds. In production the same state is a table with next_attempt_at indexed; workers run a claim query that skips locked rows, call the provider outside the transaction, then record the outcome with a conditional update on the lease they hold. Provider protocols matter for classification: APNs is HTTP/2 and answers each request with a status and a reason (a 410 means the token is no longer valid, which is permanent); SMTP uses 4xx reply codes for "try again" and 5xx for permanent rejection; SMS gateways return their own codes that the adapter must map.

Layer 3The machine

A provider call is a network round trip plus TLS, typically 50 to 500 ms, while the engine's bookkeeping is microseconds. A worker that sends synchronously therefore spends over 99% of its time waiting on sockets, which is why delivery workers are I/O-concurrent (an event loop, goroutines, or a thread pool far larger than the core count) and reuse connections: a new TLS 1.3 handshake costs at least one extra round trip, and APNs over HTTP/2 can multiplex many requests on one connection. Payload limits shape templates: Apple documents a 4 KB limit for a regular push payload, and an SMS segment holds 160 characters in the GSM 7-bit alphabet but only 70 once a single emoji forces UCS-2 encoding, so one emoji can double the cost of a message. Durable delivery state costs a database write per attempt, around a millisecond for a commit on SSD.

Why it's built this way

Providers fail independently and often transiently, so each attempt's outcome is classified and retried per channel rather than failing the whole notification.
Retrying a struggling provider immediately makes it worse, so retries back off exponentially (and, in production, with jitter).
Some failures are permanent and some users are reachable elsewhere, so a fallback chain moves on instead of retrying forever.
Producers and workers both retry, so ids make submits idempotent and leases make claims exclusive.
Templates, channels and policies change on different schedules and are owned by different people, so each sits behind its own interface.

Extensibility

New channel (WhatsApp, Slack). One Channel implementation with its error classification; producers opt in by naming it in their chain.
Preferences and quiet hours. A filter that edits the channel list before submit (drop SMS for users who opted out, defer non-urgent sends). The engine stays unchanged; Design Notification Service implements that filter.
Digests. Low-priority notifications go to a DigestBuffer keyed by user that flushes one combined message on a schedule; the flush is just another submit.
Localisation. Templates gain a body per locale per channel, chosen from the recipient's locale with a default fallback.
Read receipts and analytics. Channels that report opens or delivery receipts publish events to a status subscriber; the delivery record gains a delivered state set asynchronously.

Leadership lens

Build first: the delivery state machine and the error classification for your highest-volume channel, with tests for retry, fallback and duplicate submit. A wrong template is embarrassing; a retry loop that sends 40 SMS to one customer is a cost and trust incident.
Ask in review: which provider errors are retryable and who decided; what the idempotency key is; what happens if the worker crashes after the provider accepted the message; what the per-channel budget is per user per day.
SMS is the cost driver, often orders of magnitude more per message than push or email. Put fallback-to-SMS behind an explicit allowlist of notification types (security codes, delivery alerts), not a default.
Give template editing to product with a preview and a required-variables check, and keep channels and the engine with the platform team. That boundary is where most organisational friction appears.

Check yourself

Why render the template at submit time instead of at send time?
Show answer
Missing variables fail the producer's call where someone can fix them, and every retry and fallback sends the same text even if the template is edited meanwhile.
A push provider returns "token unregistered". Should the engine retry it?
Show answer
No. That is a permanent failure: retrying wastes the budget and delays the user. Move to the next channel and remove the stale token from the recipient's addresses.
Why does an idempotent submit not guarantee the user receives at most one message?
Show answer
The ambiguity is downstream: a provider can accept a message and then time out before replying, and the retry sends a second copy. Only provider-side keys or device-side collapse can remove that duplicate.
Why give each channel its own concurrency limit?
Show answer
So a slow provider exhausts only its own workers. With a shared pool, a stalled SMS gateway would hold every worker and delay push and email too.

Watch

Design A Scalable Notification System (Irtiza Hafiz): the distributed picture around the engine built here: queues, workers and storage.
The Chain of Responsibility Pattern Explained and Implemented (Geekific): the classic handler chain, to compare with fallback as an ordered list.
Strategy Pattern, Design Patterns ep 1 (Christopher Okhravi): the pattern behind pluggable channels.

Free resources

Timeouts, retries, and backoff with jitter (AWS Builders' Library): why retries need budgets and jitter, from a team that learned it in production.
Sending notification requests to APNs (Apple): the HTTP/2 protocol, headers such as collapse ids, and the error responses a push channel must classify.
GSM 03.38 (Wikipedia): the SMS character set and why one emoji shortens a segment from 160 to 70 characters.
Chain of Responsibility (Refactoring.Guru): structure and variants of the fallback idea.
SELECT, including FOR UPDATE SKIP LOCKED (PostgreSQL docs): the locking clause workers use to claim due deliveries without blocking each other.

Open in the interactive app: practice, quiz, mock interview →