Engineering Atlas › System Design Fundamentals

Pushing events to other systems with webhooks

16 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

The basic exchange

A customer registers https://shop.example.com/hooks/payments with a payment provider and receives a signing secret. When a charge succeeds, the provider sends:

POST /hooks/payments HTTP/1.1
Content-Type: application/json
webhook-id: evt_01J9ZQ4K7M
webhook-timestamp: 1760601600
webhook-signature: v1,K5oZfzN95Z9UVu1EsfQmfVNQhnkZ2pj9o9NDN/H/pI4=

{"type":"charge.succeeded","data":{"charge_id":"ch_88","amount":4200}}

The receiver checks the signature, records the event id, enqueues the work and returns 200 within a second or two. Anything other than a 2xx, or no answer before the sender's timeout, counts as a failure, and the sender will try again later. Timeouts are short: GitHub drops the connection after 10 seconds (GitHub docs), and the Standard Webhooks spec suggests 15 to 30 (spec). A redirect is a failure too: senders do not follow a 302 to a new URL, so a moved endpoint must be re-registered (Stripe docs).

The header names above follow the Standard Webhooks convention. Providers differ in details (Stripe packs t= and v1= into one Stripe-Signature header and tells receivers to ignore every scheme except v1 to block downgrade attacks; GitHub sends X-Hub-Signature-256 over the body alone) but the ideas are the same. The spec also asks payloads to stay small, usually under 20 KB, with large data linked rather than embedded.

Retries, and what they imply

Senders retry on a schedule that grows: seconds, then minutes, then hours. Stripe keeps trying for up to three days in live mode but only three times over a few hours in a sandbox (Stripe docs). Not every sender retries at all. GitHub does not redeliver failed deliveries automatically; the receiver has to list failures through the API and ask for redelivery (GitHub docs). Read each provider's policy before assuming one. Where retries exist, delivery is at-least-once, which brings two problem shapes with it.

Duplicates: a receiver that processed an event but timed out before replying will get it again. The fix is a unique constraint on the event id in the same transaction as the side effect, so the second delivery is a no-op. Dedupe on the id, never on the signature: Stripe signs each retry afresh with a new timestamp. Stripe also warns that it occasionally emits two separate events for one change, so for money movements key the check on the object id plus the event type as well.

Ordering: retries reorder events. subscription.canceled can arrive before the subscription.updated that preceded it. Receivers either compare a version or timestamp in the payload and ignore older ones, or treat the webhook as a hint and fetch the current state from the provider's API. The second style, often called a thin event, is more robust and is why many providers send only ids. Stripe's newer event destinations now recommend thin payloads for new integrations (Stripe docs), and the Standard Webhooks spec points out a one-way door: a thin payload can later grow into a full one without breaking anyone, but a full one can never shrink.

Email has worked this way for forty years. An SMTP server that cannot reach the destination queues the message and retries for days, then bounces it; the SMTP standard suggests waiting at least 30 minutes between tries and giving up after four to five days (RFC 5321). Webhook delivery systems rediscover the same design: a durable queue, backoff, a give-up point and a report to the sender.

The sender's side

A provider sending millions of webhooks a day is a fan-out and partial-failure system. Events are written to an outbox in the same transaction as the business change, so an event is never lost or invented. Delivery workers read the outbox, sign, POST, and record each attempt. Each endpoint gets its own queue or concurrency limit, so one customer whose server takes 30 seconds to respond does not delay everyone else's deliveries. Endpoints that fail for days are disabled with a notice (and at once if they answer 410 Gone, per the Standard Webhooks spec), and every attempt is visible in a dashboard with a "redeliver" button, because customers will ask.

Down the stack

Layer 1The contract

The sender promises: every event will be attempted until it is acknowledged with a 2xx or the retry budget runs out; each request carries a unique event id, a timestamp and a signature over both plus the raw body; and events are not guaranteed to arrive in order or only once. The receiver promises to answer quickly and to treat duplicates as no-ops.

The security contract is precise: a request is authentic if the HMAC-SHA256 of id.timestamp.body under the shared secret matches one of the signatures in the header, compared in constant time, and the timestamp is within a tolerance of the receiver's clock. Stripe's libraries default to five minutes, and a tolerance of 0 there switches the check off rather than making it strict (Stripe docs). The tolerance bounds how long a captured request can be replayed; the id lets the receiver reject replays inside that window.

Layer 2The mechanism

On the sender, a delivery is a small state machine: pending, in flight, delivered, or retry scheduled at a time, ending in delivered or dead. A scheduler keeps a priority queue (often a database index on next_attempt_at, or a Redis sorted set) and hands due deliveries to workers. After a failure, the next attempt time is now + backoff(attempt), where backoff doubles from a base up to a cap and adds jitter so that a receiver recovering from an outage is not hit by every queued retry in the same second (AWS Builders' Library).

Signing uses HMAC because both sides already share a secret and HMAC is cheap. Some providers (and the Standard Webhooks spec, under the v1a prefix) also allow asymmetric signatures with Ed25519, so receivers hold only a public key. Secret rotation works by sending several space-separated signatures during the overlap: v1,OLD_SIG v1,NEW_SIG. Receivers accept a match against any of them. Stripe, for example, keeps the old secret signing for up to 24 hours after a roll.

On the receiver, the critical detail is signing the raw bytes. A framework that parses JSON and re-serializes it changes whitespace or key order, and the signature no longer matches. Verification must run on the body exactly as it came off the socket, before any parsing.

Layer 3The machine

Each delivery to a new host costs a DNS lookup, a TCP handshake (one round trip), and a TLS 1.3 handshake (one more), so 2 to 3 round trips before the first byte of the POST. Across the internet that is 50 to 300 ms of setup per cold delivery. Senders keep per-host connection pools, but with thousands of distinct customer hosts most connections go cold between events.

Slow receivers are the expensive case. A worker holding a socket open for a 30-second timeout ties up a file descriptor, kernel buffers and a worker slot for the whole time. At 2,000 deliveries per second, if 5 percent of endpoints hang until timeout, that is 100 new hung requests per second, each held for 30 seconds: 3,000 concurrent sockets doing nothing. That is why timeouts are short and why per-endpoint concurrency is capped.

The signature itself is cheap. SHA-256 with CPU extensions runs at 1 to 2 GB/s per core, so an HMAC over a 2 KB payload takes one or two microseconds. Clocks matter more: receivers on hosts with drifting time and no NTP reject valid deliveries as stale, which looks like a provider bug.

Webhook URLs are attacker-supplied. A sender that POSTs to http://169.254.169.254/ or http://10.0.0.5:6379/ from inside its network is doing server-side request forgery for the attacker. Senders resolve the hostname, reject private, loopback and link-local addresses, connect to the IP they checked (to defeat DNS rebinding), and send through an egress proxy. The check must run at delivery time, not only when the URL is registered. Plane validated webhook IPs at creation, then let the delivery task resolve DNS again unchecked, and that gap earned a CVSS 7.6 advisory in 2026 (CVE-2026-104973).

Why it's built this way

  1. The receiver is someone else's server, often down, slow or redeploying, so delivery must be asynchronous and retried rather than part of the sender's request path.
  2. Retries after timeouts cannot know whether the first attempt succeeded, so duplicates are unavoidable and the protocol carries an id for receivers to deduplicate.
  3. Anyone can POST to a public URL, so requests carry a signature; signatures can be replayed, so they cover a timestamp and the receiver enforces a tolerance.
  4. Retry storms after an outage can knock a recovering receiver over again, so retries back off exponentially with jitter and give up eventually.
  5. One bad endpoint must not stall everyone else, so delivery is isolated per endpoint and failed endpoints are disabled rather than retried forever.

Diagram

sequenceDiagram
    participant A as "Provider app"
    participant O as "Outbox table"
    participant W as "Delivery worker"
    participant R as "Receiver endpoint"
    participant Q as "Receiver queue"
    A->>O: write event in same transaction
    W->>O: claim due delivery
    W->>R: POST signed event (attempt 1)
    R-->>W: timeout
    W->>O: schedule retry at now + backoff
    W->>R: POST signed event (attempt 2)
    R->>R: verify signature, timestamp, dedupe id
    R->>Q: enqueue work
    R-->>W: 200 OK
    W->>O: mark delivered

Worked example

A SaaS billing platform sends 40 million webhooks per day to 25,000 customer endpoints. That is about 460 per second on average and roughly 2,000 per second at the hourly invoice-run peak.

With a median delivery time of 150 ms, 2,000 per second needs about 300 concurrent requests. Plan for the bad case: if 2 percent of endpoints are down and time out at 10 seconds, those 40 requests per second hold 400 extra slots. A fleet with 2,000 slots and a per-endpoint cap of 20 concurrent deliveries keeps one broken customer from consuming more than 1 percent of capacity.

Retry volume: a 1 percent failure rate at the peak means 20 new retries per second, each retried up to 15 times over three days. With exponential backoff most attempts happen in the first hour, so retries add roughly 10 to 20 percent to delivery load, not 15 times. Storage for the attempt log at 300 bytes per attempt is about 13 GB per day, kept for 30 days for the customer dashboard.

The receiver side, in Python:

def handle(req):
    body = req.raw_body                      # bytes, before JSON parsing
    if verify(req.headers, body, SECRET, now()) != "ok":
        return 401
    with db.transaction() as tx:
        if not tx.insert_ignore("seen_events", id=req.headers["webhook-id"]):
            return 200                       # duplicate, already handled
        tx.enqueue("billing-jobs", body)
    return 200                               # fast; real work happens later

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
WebhooksNear real-time, no polling cost, works across companiesReceiver must be public, at-least-once, ordering not guaranteedNotifying external systems of state changes
Polling an events APIReceiver controls pace, works behind firewalls, easy to resume by cursorLatency equals the poll interval, wasted callsReceivers that cannot expose an endpoint, or as a backfill for missed webhooks
Fat events (full payload)Receiver needs no extra callStale or out-of-order data can be applied, larger payloadsImmutable facts such as "invoice paid"
Thin events (ids only)Receiver always fetches current state, ordering matters lessExtra API call per event, load on the provider's APIMutable resources that change often
Managed event bus (EventBridge, Pub/Sub push)Retries, DLQ and filtering built inVendor coupling, both sides on one platformInternal or same-cloud integrations

Pitfalls in production

Leadership lens

Check yourself

  1. A receiver verifies signatures and deduplicates by event id. Why does it still need a timestamp tolerance?

    Show answer

    The dedupe table may be pruned or the attacker may replay to a different environment that never saw the id. The tolerance bounds the replay window so the id store only needs to remember a few minutes of ids.

  2. Why do many providers send only ids and expect the receiver to call back for details?

    Show answer

    Retries reorder events, so applying full payloads can overwrite newer state with older data. Fetching current state makes every event a safe "go look" hint.

  3. Your receiver returns 200 only after updating three downstream systems, and customers report double charges. What is happening?

    Show answer

    Slow processing exceeds the sender's timeout, the sender retries, and the second delivery runs the side effects again. Acknowledge quickly after durably recording the event, and make processing idempotent by event id.

  4. Why sign id.timestamp.body rather than just the body?

    Show answer

    Signing only the body lets an attacker replay it later with a fresh timestamp or a different id. Binding all three means none can be changed without invalidating the signature.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →