Design a Webhook Delivery System

Deliver 3 billion signed webhooks a day to a million customer endpoints, retrying for three days, without one slow endpoint delaying anyone else.

Last updated 2026-10-27. Difficulty: hard. Patterns: webhooks, retries, isolation, multi-tenancy, ssrf. Reported at Stripe, Shopify, Twilio, Zapier, Atlassian, Slack.

Functional requirements

• Register endpoints and subscribe to event types. An HTTPS URL, a list of event types (payment.succeeded, order.paid) or all of them, a signing secret per endpoint, and an optional maximum rate. The destination is checked at registration for fast feedback, and again on every send.
• Deliver every event to every subscribed endpoint, at least once. One delivery per (event, endpoint). A 2xx inside the deadline is delivered; anything else is retried. Duplicates are possible and documented; losing a committed event is not.
• Sign every request. HMAC-SHA256 over a timestamp and the raw body with the endpoint’s secret, so receivers can reject forgeries and replays. Rotation keeps two secrets valid for a window.
• Retry failures for up to 3 days. Exponential backoff with jitter on timeouts, connection errors, 5xx, 408 and 429. Other 4xx responses fail that delivery without retries, but stay one click from replay.
• Dashboard of attempts. Per endpoint: every attempt with time, status, latency, error and the first bytes of the response, searchable for 30 days, plus success rate and backlog.
• Manual replay and backfill. Replay one event, or every failed delivery in a time window, to an endpoint. Paced so a recovered endpoint is not knocked over by its own backlog.
• Warn, then disable, failing endpoints. Email the owner after a day without a success, disable after three days, and let them re-enable and backfill once fixed.
• Events API. List the last 30 days of events, so a customer can reconcile after an outage instead of trusting that every webhook arrived.
• Out of scope. Producing the business events themselves (each service owns its outbox), the receiving side (see Zapier), the system that sends the warning emails, ordered delivery, and transports other than HTTPS such as pushing to a customer’s queue.

Non-functional requirements

• Scale (2 B events/day · 3 B deliveries · 100 k/s peak). An average fan-out of 1.5 endpoints per event over 1 M registered endpoints. Most endpoints are nearly idle; a few receive hundreds a second.
• First attempt for healthy endpoints (p99 < 5 s from commit). Merchants fulfil orders off these events, so the pipeline from the producer’s commit to the first POST must stay short even at peak.
• Isolation (a slow endpoint costs only itself). The requirement that drives the hardest trade-off: with 1 M endpoints, some are always slow, down or hostile, and none of them may hold capacity or queue position that healthy endpoints need.
• No committed event lost (at least once · duplicates expected). If the payment committed, its event is delivered or visibly failed after 3 days. Exactly once is not on offer; stable event ids make duplicates harmless.
• Safe to call untrusted URLs (zero requests to private addresses). Customers choose the URL and the DNS behind it. The fleet must never become a proxy into our own network or cloud metadata.
• Producers never wait on delivery (accept 99.99 %). A broken delivery pipeline delays webhooks; it must never slow a payment. The outbox and the log decouple them completely.
• Debuggable by customers (attempts visible in seconds, kept 30 days). Most support tickets are “did you send it?”. The attempt log answers that without us, if it is fresh and complete.

Back-of-envelope estimates

• Deliveries per second: ~35 k avg · 100 k peak. 2 B events/day × 1.5 endpoints each = 3 B deliveries/day ÷ 86 400 ≈ 35 k/s. Peaks (US business hours, month-end billing runs) reach about 3× that, 100 k/s. Events alone are ~23 k/s; the fan-out multiplies them.
• Requests in flight at peak: ~30 k. Little’s law: in flight = rate × time. 100 k/s × 0.3 s mean for a healthy endpoint = 30 k open requests. The largest endpoint hanging until a 10 s deadline at its peak of 1 k/s would want 10 k more on its own, which is why every endpoint’s share is capped.
• Retries if a big endpoint is down a day: ~300 M attempts. The largest endpoint gets 1 % of traffic: 30 M deliveries a day. Down for 24 h, with each pending delivery on its own backoff, about 10 attempts each that day is 300 M doomed attempts, ~3.5 k/s. If it hangs rather than refusing, that is 35 k sockets held for 10 s each, more than the whole healthy fleet. A per-endpoint breaker turns it into ~150 probes.
• Attempt log: ~1 TB/day · ~3 TB stored. 3 B first attempts plus ~10 % retries ≈ 3.3 B attempts × ~300 B (ids, time, IP, status, latency, error class, first bytes of the response) ≈ 1 TB/day, 30 TB for 30 days raw. A columnar store compresses it ~10× to ~3 TB, which is what makes a 30-day dashboard affordable.
• Event payload store: ~4 TB/day · 120 TB for 30 days. 2 B events × ~2 KB average payload (capped at 256 KB) = 4 TB/day. Kept 30 days for retries, replay and the events API: 120 TB raw, ~30 TB compressed. Stored once per event, never per delivery or per attempt.
• Egress bandwidth: ~2.4 Gbit/s peak · ~9 TB/day. About 3 KB on the wire per delivery (2 KB body, signed headers, TLS framing). 100 k/s × 3 KB = 300 MB/s ≈ 2.4 Gbit/s at peak, and the 35 k/s average is ~9 TB a day out of the egress proxies. Bytes are not the problem; connections are.
• Dispatcher hosts: ~30 at peak · ~40 provisioned. Budget 60 k sockets at peak: 30 k healthy plus room for slow endpoints inside their caps. Async I/O holds ~2 000 concurrent requests per host comfortably (~64 KB of TLS state each), so 30 hosts, ~40 to survive a zone loss. New TLS handshakes, for the ~40 % of deliveries that find no warm connection, cost ~40 k/s × 1 ms ≈ 40 cores across the fleet.
• Endpoint index: ~1 GB. 1 M endpoints × ~1 KB (URL, event types, encrypted secrets, limits, state) = 1 GB. Small enough to hold in memory on every fan-out and dispatcher host, so neither makes a database read per delivery.

Components

• Service DB + outbox (Postgres · CDC relay): The producing service’s own database (payments, orders, repositories). The business change and an outbox row holding the event commit in one transaction, so an event exists if and only if the change does. A CDC relay tails the write-ahead log and publishes outbox rows; the producer never calls the webhook system.
• Event log (Kafka · keyed by event id): Every event from every producing service, ~23 k/s on average, retained 3 days so fan-out can be re-run after a bug. Keyed by event id for an even spread: fan-out needs no ordering, and one huge account must not make a hot partition.
• Fan-out (Kafka transactions): For each event: store the payload once, look up the endpoints of that account subscribed to that type in the in-memory endpoint cache, and write one thin delivery task per endpoint. Consuming the event and producing its tasks commit in one Kafka transaction, so a crash neither loses nor duplicates tasks.
• Delivery tasks (Kafka · keyed by endpoint): One ~200-byte task per (event, endpoint): delivery id, event id, endpoint id, attempt and lane (live, replay, backfill). Keyed by endpoint id, so every endpoint lives in exactly one partition and has exactly one owner. Retention is a day, because anything not resolved quickly moves to the delivery store.
• Delivery store (backlogs · due-time index): Only the unhappy minority: retries with their next due time, in an index partitioned by (shard, due minute) so each dispatcher reads only its own; per-endpoint backlogs of parked tasks, oldest first; and checkpoints of each endpoint’s breaker and limit. The ~95 % of deliveries that succeed first time never touch it.
• Event store (KV by event id · 30 days): Each event’s exact payload bytes, stored once by event id: 4 TB a day for 30 days. Read by dispatchers on every send (through a local cache, since an event goes to 1.5 endpoints on average), by retries and replays, and by the events API customers use to reconcile.
• Dispatcher (owns partitions · async HTTP): Each host owns ~32 task partitions and so every endpoint in them, which makes per-endpoint limits, token buckets and breakers local counters. It schedules endpoints that have work by deficit round robin under an adaptive per-endpoint concurrency limit, parks what is over the limit instead of waiting on it, signs, sends with a 10 s deadline, and writes due times for failures.
• Egress proxy (SSRF filter · pinned IP): The only way out of the network for webhook traffic. The proxy resolves each CONNECT itself, refuses it if any address is private, loopback, link-local, metadata or reserved, and connects to the address it checked, so DNS rebinding has nothing to race. TLS runs end to end from the dispatcher; customers can allowlist the proxies’ published IPs.
• Customer dashboard: Where customers register endpoints and pick event types, watch each endpoint’s attempts and health, open a failed attempt, rotate secrets, and replay one event or backfill a window. The same actions are in the API for automation.
• Webhooks API (endpoints · attempts · replay): The public API behind the dashboard: create and update endpoints (checking HTTPS and the destination at registration), rotate secrets, list events and attempts, request replays and backfills. Replays become ordinary tasks on the delivery-tasks topic, so they obey the same limits as live traffic.
• Endpoint cache (in memory · ~1 GB): A full in-memory copy of the endpoint registry on every fan-out and dispatcher host: (account, event type) to endpoints, plus each endpoint’s URL, encrypted secrets, limits and state. Loaded at start and kept current from the registry’s changelog within seconds, so a disable or a rotation takes effect without a database read per delivery.
• Customer endpoints (HTTPS · 1 M of them): Customers’ HTTPS receivers. Most answer 200 in a few hundred milliseconds; some are slow, some are down for days, some are misconfigured, and a few are hostile. A 2xx inside the deadline is the only thing counted as delivered.
• Endpoint registry (Postgres · changelog): The source of truth for endpoints: account, URL, event types, signing secrets (encrypted under a KMS-wrapped key, two during a rotation), rate and concurrency ceilings, and state (enabled, warned, disabled). Every change is published to a compacted changelog topic that feeds the caches.
• Endpoint health (every minute): Aggregates the attempt log per endpoint: success rate, latency, backlog age and time since the last success. Warns the owner after 24 h without a success, disables the endpoint at 72 h, and flags endpoints that are slow or falling behind. It only writes endpoint state; dispatchers react through the changelog.
• Attempt log (ClickHouse · 30 days): One row per attempt: 3.3 B a day, ~1 TB raw and ~100 GB compressed, kept 30 days in daily partitions that are dropped whole. Ordered by (endpoint, time), so “latest attempts for this endpoint” is one range read and per-endpoint success rates are a cheap aggregate.

User flows

1. A payment event reaches a healthy endpoint. The main path: an event committed with the payment, fanned out to one task per subscribed endpoint, signed and posted through the egress proxy, and recorded. Nothing on it waits for anything slow, which is how the first attempt lands in about a second.
   1. The payment service commits the charge and an outbox row with the event, in one transaction. The event id is generated inside the transaction, so every later copy of the event carries the same id. A rolled-back charge leaves no event; a committed one cannot lose its event, whatever crashes next. The producer’s latency is one extra row insert.
   2. A CDC relay tails the database log and publishes the row to the event log. Tailing the write-ahead log (Debezium-style) costs the producer nothing and preserves commit order. The relay is at least once: after a crash it re-publishes a few rows, which the deterministic ids downstream absorb. Lag is typically 100 to 500 ms.
   3. Fan-out stores the payload and looks up the account’s endpoints for this event type. The payload is written once, by event id, for retries, replays, the dashboard and the events API. Subscriptions come from the in-memory endpoint cache (1 GB for 1 M endpoints), so fan-out does no database read. Disabled endpoints are skipped; the event is still stored and listable.
   4. Fan-out writes one thin delivery task per endpoint, keyed by endpoint. Tasks carry references, not payloads: about 200 bytes, so 35 k/s of them is 7 MB/s. The delivery id is a hash of (event, endpoint), and the consumed offset and the produced tasks commit in one Kafka transaction, so fan-out never loses or doubles a task.
   5. The dispatcher that owns the partition checks the endpoint’s limits and fetches the payload and secret. Because one host owns every task for this endpoint, its in-flight count, token bucket and breaker are plain local counters: no Redis round trip, no race. A free slot, a token and a closed breaker mean send now. The payload comes from a local cache in front of the event store.
   6. It signs the request and posts it through an egress proxy, which resolves the host, vets the IP and connects. The signature covers the timestamp and the exact body bytes. The proxy checks the address it is about to connect to, not the name, so a rebinding DNS answer cannot slip through. Connect within 3 s, the whole attempt within 10 s, redirects never followed.
   7. The endpoint answers 200 in 180 ms; the dispatcher records the attempt and moves on. Any 2xx is success and the body is ignored beyond the first kilobyte we keep. A first-attempt success writes nothing to the delivery store, only an attempt record, batched into the attempt log within a second or two. Commit to 200 took about a second.
2. A customer debugs and replays a failure. The read path: an endpoint’s attempts from the attempt log, the payload from the event store, a replay that goes through the normal delivery path, and a secret rotation that reaches every dispatcher in seconds.
   1. The customer opens their endpoint and lists its recent failed attempts. The attempt log is ordered by (endpoint, time), so the newest 50 for one endpoint is one range read even with billions of rows a day. Attempts appear within a few seconds of being made, which is what stops “did you send it?” tickets.
   2. They open one failed attempt: the response we received and the payload we sent. The response is the first kilobyte, stored and shown as escaped text, never rendered. The payload is the exact bytes from the event store, so the customer can re-run their signature check locally against what we signed.
   3. The health panel shows the endpoint’s success rate, latency and breaker state for the day. One aggregate over the endpoint’s rows by minute. A 62 % success rate with all failures on one event type points at a handler bug; a cliff to 0 % points at a deploy or DNS. Showing which it is saves the customer an hour.
   4. After fixing their handler, they replay the event. A replay is an ordinary task with the original event id, a new delivery id and lane "replay", written to the endpoint’s partition. It gets no special path, so it obeys the same limits, signing and egress checks as live traffic.
   5. The owning dispatcher sends it and the new attempt appears in the list. Same Webhook-Id, fresh timestamp and signature. If the receiver had in fact processed the event before, its dedupe on the event id makes the replay a no-op, which is the right outcome: replay is for events it failed to handle.
   6. They rotate the signing secret, keeping the old one valid for 24 hours. The registry stores both secrets with the old one’s expiry; the changelog reaches every cache in seconds. Until expiry each request carries two v1 signatures, so the receiver can deploy the new secret whenever it likes without a single rejected request.
3. A big endpoint slows to eight-second responses. The scale-breaking case: the largest endpoint (350 deliveries a second) still answers, but in 8 s. Nothing fails, so no breaker trips; only the concurrency limit and fair scheduling stand between it and everyone else.
   1. The customer’s handler starts doing slow work inline; responses take 8 s but still return 200. Little’s law: 350/s × 8 s = 2 800 requests in flight to keep up, 27 times what it needed at 300 ms, and 8 k at its daily peak of 1 k/s. In a shared pool with no caps, that one endpoint would take a quarter of the fleet.
   2. Its concurrency limit caps it at about 100 in-flight requests and stops growing. The limit only grows while p95 latency is under 2 s and errors are low, and halves on timeouts, 5xx or 429. Nothing errors here, so it holds at ~100: at 8 s each that is 12.5 deliveries a second. The endpoint pays for its slowness in its own throughput, and nobody else pays anything.
   3. Its tasks keep arriving at 350 a second; past 500 queued in memory they are parked in its backlog. The partition reader never waits on one endpoint, so the ~1 000 other endpoints sharing the partition see no head-of-line blocking. Offsets commit past parked tasks because parked is durable. The backlog grows by about 1.2 M an hour.
   4. Deficit round robin keeps the host’s sockets shared among endpoints that have work. The host has ~2 000 sockets; the slow endpoint holds at most its limit and every other ready endpoint gets its turn. Healthy endpoints on the same host keep a first-attempt p99 near a second. This is the requirement, and it is met without any special case.
   5. The health job flags the endpoint as falling behind and tells the owner why. Backlog 1.2 M and growing, oldest 58 minutes, p50 latency 8 s. The message says what to change: acknowledge with 2xx first and do the work asynchronously. Raising the ceiling would only move the cost onto everyone else.
   6. Once the handler is fixed, latency drops, the limit climbs and the backlog drains behind live traffic. Live tasks get their share first so the customer’s system becomes current; the backlog uses the rest, oldest first, reading payloads from the event store. At its ceiling of 400 and 200 ms that is up to 2 000/s, 1 650/s beyond live traffic, so 1.2 M clears in about 12 minutes.
4. An endpoint is down for twenty hours. The failure path: per-delivery retries until the pattern is clearly the endpoint, then one probe timer instead of millions of attempts, then a ramped drain when it comes back.
   1. The endpoint’s servers go down; connections are refused and attempts fail fast. A refused connection costs milliseconds; a filtered port that drops packets costs the 3 s connect timeout, which is why connect has its own short limit. Every failure is in the attempt log with its error class.
   2. Each failed delivery gets a due time with exponential backoff and jitter in the delivery store. No worker sleeps. The owner polls its own shard of the due index once a second, so a retry costs a row write and a read when it is due. Jitter spreads retries that failed together, so they do not return together.
   3. Failures cross the threshold, the breaker opens, and new tasks park in the endpoint’s backlog. Opened at half of the last 100 attempts failing with at least 20 failures. From now on the endpoint costs one timer, not 350 new attempts a second plus every retry: about 150 probes a day instead of ~300 M doomed attempts.
   4. The owner probes with the oldest pending delivery, every minute at first, backing off to every ten. A probe is a real delivery, so success delivers something and proves the actual handler works, not just that the port is open. Twenty hours of probing is about 120 requests.
   5. The endpoint returns; a probe gets 200 and the breaker half-opens. Half-open means sending resumes at a concurrency of 2, not at the old limit. A recovering service often has cold caches and a fraction of its capacity, and 25 M parked deliveries at full speed would knock it straight back over.
   6. The dispatcher drains the backlog oldest first, ramping the limit while latency stays healthy, with live events first. Additive increase every second, halved on rising p95 or 5xx. Twenty hours at 350/s is ~25 M parked; at a ceiling of 400 and ~200 ms that is up to 2 000/s, 1 650/s beyond live traffic, so about 4 hours to clear. Anything past 72 h old would be expired, not sent.
5. Failing for three days: warn, disable, backfill. The background path: the health job, not the dispatchers, decides when an endpoint has failed long enough to warn and then disable; the owner fixes it, re-enables, and backfills the gap at a pace their server can take.
   1. An endpoint’s domain lapses; every probe fails with a DNS error. The breaker opened within minutes and probes keep going. Error class nxdomain is recorded, which is what the warning email will quote, because “your domain does not resolve” is fixable in a way “delivery failed” is not.
   2. After 24 hours without a success, the health job marks the endpoint warned and the owner is emailed. The job runs every minute over per-endpoint aggregates, so it is cheap. The email goes through the notification system with the error class, the first failure time and a link to the endpoint. Stripe documents emailing owners of failing endpoints; the thresholds here are mine.
   3. At 72 hours it disables the endpoint; the caches drop it and the dispatcher stops probing. Fan-out stops creating tasks for it, and its backlog is expired as failed, which is also the end of the 3-day retry promise. Events keep being stored and listable for 30 days, so nothing is unrecoverable.
   4. The owner fixes DNS, re-enables the endpoint and asks for a backfill of the failed window. Re-enabling resets the health state and the breaker to half-open, so live traffic resumes slowly. The backfill is a separate, explicit request because some customers would rather reconcile from the events API than receive four days of old events.
   5. The API expands the window from the event store into backfill tasks, which the dispatcher parks and drains at a ramped pace. The API lists the account’s events in the window that matched the endpoint’s subscriptions, and writes tasks with lane "backfill". Backfill tasks always park, so they drain only from the capacity live traffic leaves, oldest first, with a progress bar and a cancel button.

Deep dives

The theory behind it

• The circuit breaker pattern: an interactive playground. Make the same calls to a failing service with and without a circuit breaker, and watch it trip, fail fast, probe half-open and close again.
• Fair queuing: an interactive playground. Share eight workers between three tenants while one floods the queue, and compare a FIFO queue, per-tenant caps and fair round-robin queuing on waits and finish times.
• Webhook delivery: an interactive playground. Send webhooks to an endpoint that goes down for 30 minutes, 12 hours or 4 days, and compare dropping, per-event backoff and a per-endpoint queue with a breaker on lost events, wasted calls and ordering.
• Designing webhooks: How to deliver webhooks reliably: event payloads, signing and verification, retries with backoff, ordering, idempotency on the receiver, per-customer isolation, and replay, as Stripe and GitHub do it.
• At-most-once, at-least-once and exactly-once delivery: What message delivery guarantees really mean: why exactly-once delivery is impossible over a network but exactly-once processing is achievable, acknowledgements and retries, idempotent consumers, deduplication, transactional processing in Kafka, and ordering.
• Rate limiting and resilience: Token bucket, leaky bucket, fixed and sliding windows, distributed rate limiters, plus the resilience toolkit: timeouts, retries with backoff and jitter, circuit breakers, bulkheads, load shedding and graceful degradation.

Related

• Design Zapier
• Design a Distributed Logging System
• Design a RAG System
• Design a CDN
• All 83 system design interview questions
• Run this question for a friend: the interviewer kit