Design a webhook delivery service

In one minute

• Promise at-least-once, never exactly-once. A delivery succeeds only when the customer answers 2xx, and a lost response looks identical to a lost request. So duplicates are certain, and the contract is: every event carries a stable ID, and receivers deduplicate on it.
• Isolate every endpoint. One slow or dead customer must never delay another. That means a logical queue, a concurrency cap and a failure state per endpoint, scheduled fairly across endpoints.
• Retry with exponential backoff, full jitter and a deadline. Retries stretch over hours or days. Jitter and a per-endpoint ramp keep a recovering customer from being flattened by its own backlog.
• Sign every request, and bind the signature to a timestamp and an ID. An HMAC over id.timestamp.body proves origin and integrity. The timestamp bounds replay, and secret rotation lets two secrets be valid for a while.
• Keep the events, not only the deliveries. Replay needs a retention window (30 days is common), and ordering is best effort. Say so in the contract, and give receivers an API to fetch current state.

Requirements

Functional

• Tenants register endpoints (HTTPS URL, event types, secret). Producers publish events, and each event is delivered to every matching endpoint.
• Retry failed deliveries on a schedule. Mark endpoints unhealthy, disable them after sustained failure and tell the tenant.
• Sign each delivery. Support secret rotation.
• Show a delivery log per endpoint (attempts, status, latency, response snippet). Support manual and bulk replay of events within the retention window.

Non-functional

• First attempt within 5 seconds at p99 for healthy endpoints, measured from event accepted to request sent.
• No accepted event is lost, even if a region or a queue node fails after acceptance. Duplicates are allowed and documented.
• Isolation: a failing endpoint adds no more than a few percent to anyone else's latency.
• Scale to 2M endpoints, 1B events per day and bursts of 5x.

Out of scope

• Building the producing systems, a customer-facing UI, transformation or routing languages, and non-HTTP sinks (queues, functions) beyond naming them as extra delivery adapters.

Back-of-envelope

• Events: 1B/day divided by 86,400 s is about 11.6k events/s. Peak at 5x is about 58k/s. Average fan-out is close to 1 delivery per event, because most events have one subscriber and a few have dozens.
• Attempts: with about 20% extra from retries, that is about 14k attempts/s average and about 70k/s at peak.
• Concurrency (Little's law): healthy endpoints answer in about 300 ms, so 70k/s means 21k requests in flight. If 1% of attempts hit a 10 s timeout, that is 700/s times 10 s, another 7k connections held open doing nothing. So size for about 30k to 40k concurrent connections, or about 20 workers at 2k each.
• Event storage: 1B events at 2 KB is 2 TB per day. A 30-day replay window is 60 TB before replication, so payloads live in a cheap blob or wide-column store, not in the queue.

API

POST /v1/endpoints
{ "tenant_id": "t_91", "url": "https://shop.example.com/hooks",
  "event_types": ["order.paid", "order.refunded"], "ordered": false }
-> 201 { "endpoint_id": "ep_7f3", "secret": "whsec_..." }

POST /v1/events                      (producers; Idempotency-Key header required)
{ "tenant_id": "t_91", "type": "order.paid", "data": { "order_id": "o_55", "version": 12 } }
-> 202 { "event_id": "evt_01J9..." }

GET  /v1/endpoints/ep_7f3/deliveries?status=failed&since=2026-10-01T00:00:00Z
POST /v1/endpoints/ep_7f3/replay     { "from": "2026-10-16T09:00:00Z", "to": "2026-10-16T11:00:00Z", "only_failed": true }
POST /v1/endpoints/ep_7f3/rotate-secret

What the customer receives follows the shape of the open Standard Webhooks specification (Standard Webhooks spec):

POST /hooks HTTP/1.1
webhook-id: msg_2KWPBg        # stable across retries and replays
webhook-timestamp: 1760695200 # seconds, set on every attempt
webhook-signature: v1,K5oZfzN95Z9UVu1Esf...   # base64 HMAC-SHA256 of "id.timestamp.body"
content-type: application/json

{"type":"order.paid","data":{"order_id":"o_55","version":12}}

Any 2xx is success. 429 and 503 with Retry-After are honoured. Everything else, including redirects and timeouts, is a failure.

Data model

• endpoints, key (tenant_id, endpoint_id): URL, event-type filter, ordered flag, up to two active secrets with valid_until, state (active, degraded, paused, disabled), first_failure_at. Small and read-heavy, so it is cached in memory and invalidated by version.
• events, partition key (tenant_id, day), clustering by event_id (time-sortable ULID): payload and metadata. Immutable. This is the source for replay and is the table that holds the 30-day retention window.
• deliveries, partition key endpoint_id, clustering by event_id: status (pending, in_flight, succeeded, retrying, dead), attempt, next_attempt_at, last_status. This is the state machine. Partitioning by endpoint makes "all pending work for this endpoint" a single-partition read, which is what fairness and replay need.
• attempts, partition key (endpoint_id, day): one row per try, with latency and status. Append-only, TTL 30 days.

Committing the events row and the deliveries rows together (a transactional outbox) is what makes "accepted means not lost" true.

High-level design

flowchart LR
  P[Producer] -->|"POST /events"| ING["Ingest API"]
  ING --> ES[("Event store")]
  ING --> FO["Fan-out (match subscriptions)"]
  FO --> DS[("Delivery state")]
  FO --> Q["Partitioned queue (key = endpoint_id)"]
  Q --> D["Dispatcher (per-endpoint scheduler)"]
  D -->|"signed POST"| EG["Egress proxy"]
  EG --> C["Customer endpoint"]
  D -->|"result"| DS
  DS -->|"retry due"| T["Timer / delay store"]
  T --> Q
  D -->|"disable, alert"| N["Notifier"]
  R["Replay API"] --> ES
  R --> FO

Write path.

1. The ingest API authenticates the producer, deduplicates on the idempotency key, and commits the event and its delivery rows together. It answers 202.
2. Fan-out looks up matching endpoints (cached) and publishes one message per delivery to a partitioned queue, keyed by endpoint_id. All work for one endpoint lands on one partition, so one dispatcher owns that endpoint's state.
3. The dispatcher keeps, per active endpoint, a small in-memory queue, an in-flight count, a failure streak and a circuit state. A fair scheduler picks the next endpoint with capacity. It loads the payload, signs it with the current secret and sends it through the egress proxy.
4. On 2xx it marks the delivery succeeded. On failure it records the attempt, computes the next time with jittered backoff, and writes next_attempt_at. A timer service re-publishes the delivery to the queue when it is due.

Read path. The delivery log and replay API read deliveries and attempts by endpoint. Replay creates new delivery rows for stored events with the same webhook-id, so receivers can deduplicate across copies.

Deep dives

1. Retries, backoff and the recovery stampede

Options. Fixed interval retries are simple, but every failed delivery comes back in lockstep. Exponential backoff spreads them out. Exponential backoff with full jitter spreads them further, because retries from many senders are randomised across the whole window, which AWS's simulations found reduces contention compared with no jitter (AWS Architecture Blog).

Choice. Delay is random(0, min(cap, base * 2^(attempt-1))), with a long schedule. Real services retry for hours to days. Stripe retries live-mode events for up to three days with exponential backoff (Stripe docs), and Svix's default schedule is immediately, 5 s, 5 min, 30 min, 2 h, 5 h, 10 h and 10 h more (Svix docs). Shopify retries 8 times over 4 hours (Shopify docs).

The part people forget. Jitter handles retries of the same age. It does not handle a customer that was down for six hours and comes back with 400k queued events. If you release them all, you knock the endpoint over again. So each endpoint has a concurrency limit that adapts like TCP congestion control: start at 1 to 4 in flight, add one per success window, halve on timeout or 5xx, honour Retry-After. The backlog drains at whatever rate the customer can take. Waiting deliveries sit in a timer store (a table indexed on next_attempt_at), not in worker memory.

2. Fairness, slow endpoints and circuit breaking

The failure to prevent. A shared pool of workers and a shared FIFO means one customer whose endpoint takes 10 s to time out can hold all workers. Every other customer then waits behind it. This is head-of-line blocking plus a noisy neighbour, a problem AWS describes for multi-tenant systems.

Options. One physical queue per endpoint isolates perfectly, but 2M mostly empty queues are a management problem. One shared queue with a per-tenant rate limit is cheap but guards against volume, not slowness. The middle path is a logical queue per endpoint multiplexed on shared partitions, with a per-endpoint scheduler in the dispatcher.

Choice: the middle path. The scheduler is round robin over endpoints that have work and capacity (the same shape as the practice problem). Each endpoint has a hard cap on in-flight requests and a tight connect and total timeout (GitHub asks receivers to answer within 10 seconds and cuts the connection after that, GitHub docs). A circuit breaker, in the sense of Fowler's pattern, opens after a failure streak: the endpoint stops getting traffic except a single probe delivery, and goes half-open when the probe succeeds.

Dead endpoints. After a configured period of continuous failure (say 3 days), mark the endpoint disabled, keep the events, email the tenant and offer one-click replay. Shopify instead deletes an Admin-API subscription after 8 consecutive failures, which is harsher than keeping events.

3. Ordering, idempotency and replay

Ordering. Strict ordering and high availability conflict here. To deliver event 2 after event 1 you must hold event 2 until event 1 succeeds, so one failing event blocks the whole endpoint. Stripe states it does not guarantee delivery in the order events are generated (Stripe docs). The default should be unordered with a per-resource version or created field in the payload. For tenants that need order, offer an ordered endpoint: a single in-flight delivery per endpoint (or per ordering key, such as the order ID), at the price of head-of-line blocking.

Idempotency. Duplicates come from the sender (a timeout after the customer already processed the request), from replay, and from the queue (redelivery after a dispatcher crash). The sender cannot remove them. It can make them recognisable: webhook-id stays the same across attempts and replays. The receiver stores processed IDs and ignores repeats, as Stripe's advice to track event IDs and the Standard Webhooks advice to treat webhook-id as an idempotency key both say.

Replay. Replay is a new set of deliveries rows over stored events, with a filter (time range, event type, only failed). It reuses the original webhook-id, gets a new timestamp and signature, and goes through the same per-endpoint scheduler, so a bulk replay of 5M events is still rate limited by the endpoint's adaptive concurrency. Retention bounds what can be replayed: Stripe lets you resend an event for up to 30 days (Stripe docs).

Down the stack

Worked example

A tenant, Acme, has one endpoint subscribed to order.paid and receives 40 events/s. Its server goes down for 5 hours (18,000 s). The dispatcher uses base 5 s, cap 10 h, full jitter, and caps the probe interval at 5 minutes while the breaker is open.

• At the start, attempts fail fast (connection refused in about 20 ms). The breaker opens after 5 consecutive failures, about 125 ms in. From then on the dispatcher sends about one probe per 5 minutes instead of 40 requests/s. Over the outage that avoids roughly 720k doomed requests.
• Events pile up as retrying: 40 x 18,000 = 720k deliveries, about 1.4 GB of payload in the event store and a modest number of timer rows. No other tenant sees a change, since only Acme's logical queue grew.
• The server returns. The next probe succeeds, the breaker closes, and concurrency starts at 2 and grows by one per 100 successes. Acme's server can handle about 200 requests/s, and the ramp reaches that in about a minute.
• Draining at 200/s while 40/s keep arriving nets 160/s, so 720,000 / 160 = 4,500 s, or 75 minutes, until Acme is current.
• A few events arrive twice, because a timeout fired after the handler committed. Acme deduplicates on webhook-id.

Failure modes

• Producer burst at 10x. Ingest is cheap and durable, so accept up to a per-tenant quota (shed the rest with 429). Delivery is protected by per-endpoint caps, so a burst for one tenant makes only that tenant's backlog longer.
• Customer outage with a huge backlog. The stampede on recovery is the real risk. Mitigate with the adaptive concurrency ramp, Retry-After, a per-endpoint rate limit tenants can lower, and replay that runs through the same scheduler.
• Slow but not dead endpoint. Responses at 9.5 s never trip a failure breaker but hold connections. Track a latency signal (p95 near the timeout) and treat it as degraded: lower its concurrency cap rather than waiting for failures.

In the wild

• GitHub, 2018: a backlog of five million webhook events. During the 21 October 2018 incident, MySQL clusters became inconsistent after a network partition. To protect data integrity, GitHub paused webhook delivery and Pages builds on purpose. When processing resumed there were over five million hook events and 80 thousand Pages builds queued. About 200,000 payloads had outlived an internal TTL and were dropped. GitHub paused, raised the TTL and finished the backlog at 23:03 UTC on 22 October. The lesson for this design is that queue retention must outlast your worst outage (GitHub incident analysis).
• Shopify: a published failure policy. Shopify asks receivers to answer within five seconds, retries 8 times over 4 hours, and deletes an Admin-API subscription after 8 consecutive failures. It also tells apps to build a reconciliation job that fetches what they may have missed, because delivery is not a complete record (Shopify docs).
• Stripe: documented guarantees and their limits. Stripe retries live events for up to three days with exponential backoff, offers manual resend for up to 30 days, says plainly that event order is not guaranteed, and tells receivers to track event IDs to catch duplicates (Stripe docs).

Trade-offs

Leadership lens

• Rollout. Ship delivery, retries and signing first, with the delivery log from day one, because support tickets will be "did you send it?" Add the adaptive limiter and breaker before the first large customer, and replay in the second release.
• Cost drivers. Event retention (60 TB at 30 days), attempt logs and the connection fleet. Retention is a product knob, so price it.
• Design-review questions. What happens to events when an endpoint is down for a week? What is the cap on in-flight requests per endpoint? Who is told when we disable an endpoint? Can a tenant point us at 169.254.169.254?

Check yourself

1. A customer's endpoint returns 200 and then the connection drops before your service reads the response. What does your service do, and what must the receiver have done to make that safe?

   Show answer

   It sees a failure and retries, so the receiver gets a duplicate. The receiver must deduplicate on the stable webhook-id (store processed IDs), because the sender cannot tell a lost response from a lost request.

2. Why use full jitter on the retry delay, and why is jitter alone not enough after a six-hour customer outage?

   Show answer

   Jitter spreads retries of similar age so they do not arrive in lockstep. After a long outage, the backlog itself is the stampede, so you also need a per-endpoint adaptive concurrency limit that ramps up as the receiver proves it can take load.

3. Your pipeline is a single shared worker pool and a FIFO queue. One customer's endpoint starts timing out at 10 s. What breaks, and what is the minimum fix?

   Show answer

   Workers fill up with 10 s waits, so every tenant's deliveries queue behind them. The minimum fix is a per-endpoint in-flight cap with a short timeout, a failure breaker, and fair scheduling across endpoints.

4. With 1B events/day and 20% retry overhead, roughly how many attempts per second are made on average and how many connections are open at 300 ms latency?

   Show answer

   1.2B / 86,400 is about 14k attempts/s. By Little's law, 14k x 0.3 s is about 4.2k open requests, before adding slow timeouts and the 5x peak.

Sources and further study

Primary

• Standard Webhooks specification: the webhook-id, webhook-timestamp and webhook-signature headers, signed content format and verification guidance.
• RFC 2104: HMAC: the keyed-hash construction every webhook signature scheme uses.
• RFC 9110: HTTP Semantics: status code classes and the Retry-After field that govern retry decisions.

Systems docs

• Stripe: webhooks: retry window, resend limit, event ordering and duplicate handling.
• Stripe: signature verification: timestamped signatures and the replay tolerance.
• GitHub: best practices for webhooks: the 10-second response limit, redelivery and X-GitHub-Delivery.
• Shopify: HTTPS webhook delivery: retry count, automatic subscription removal and reconciliation advice.
• Svix: retry schedule: a concrete backoff schedule and endpoint disabling.

Books

• Release It!, 2nd edition, Michael Nygard, chapter 5 "Stability Patterns": circuit breaker, bulkhead and timeouts for calls to systems you do not control.
• Designing Data-Intensive Applications, 1st edition, Martin Kleppmann, chapter 11 "Stream Processing": acknowledgements, redelivery and what exactly-once really means.

Other curricula

• Martin Fowler: Idempotent Receiver: frames deduplication as a property of the receiver, keyed by a client-supplied ID.
• AWS Builders' Library: timeouts, retries and backoff with jitter: the sender-side view of retry storms.

Watch

• Webhook Architecture Design with Svix Founder & CEO Tom Hacohen (Svix): a conversation with a webhook-infrastructure founder about how such a service is architected.
• Webhook handling: building rock-solid Stripe integrations (Stripe Developers): the receiver's side of the same contract.

Free resources

• Exponential backoff and jitter (AWS Architecture Blog): simulations comparing full, equal and decorrelated jitter.
• Fairness in multi-tenant systems (AWS Builders' Library): quotas, admission control and shuffle sharding for noisy neighbours.
• Validating webhook deliveries (GitHub): HMAC verification with code samples in several languages.
• Implementing Stripe-like idempotency keys in Postgres (Brandur Leach): a transactional approach to the duplicate problem.
• Circuit Breaker (Martin Fowler): states and trip conditions in a short page.