Design a Webhook Delivery System

• Register endpoints and subscribe to event types: An HTTPS URL, a list of event types (payment.succeeded, order.paid) or all of them, a signing secret per endpoint, and an optional maximum rate. The destination is checked at registration for fast feedback, and again on every send.
• Deliver every event to every subscribed endpoint, at least once: One delivery per (event, endpoint). A 2xx inside the deadline is delivered; anything else is retried. Duplicates are possible and documented; losing a committed event is not.
• Sign every request: HMAC-SHA256 over a timestamp and the raw body with the endpoint’s secret, so receivers can reject forgeries and replays. Rotation keeps two secrets valid for a window.
• Retry failures for up to 3 days: Exponential backoff with jitter on timeouts, connection errors, 5xx, 408 and 429. Other 4xx responses fail that delivery without retries, but stay one click from replay.
• Dashboard of attempts: Per endpoint: every attempt with time, status, latency, error and the first bytes of the response, searchable for 30 days, plus success rate and backlog.
• Manual replay and backfill: Replay one event, or every failed delivery in a time window, to an endpoint. Paced so a recovered endpoint is not knocked over by its own backlog.
• Warn, then disable, failing endpoints: Email the owner after a day without a success, disable after three days, and let them re-enable and backfill once fixed.
• Events API: List the last 30 days of events, so a customer can reconcile after an outage instead of trusting that every webhook arrived.
• Out of scope: Producing the business events themselves (each service owns its outbox), the receiving side (see Zapier), the system that sends the warning emails, ordered delivery, and transports other than HTTPS such as pushing to a customer’s queue.

• Scale (2 B events/day · 3 B deliveries · 100 k/s peak): An average fan-out of 1.5 endpoints per event over 1 M registered endpoints. Most endpoints are nearly idle; a few receive hundreds a second.
• First attempt for healthy endpoints (p99 < 5 s from commit): Merchants fulfil orders off these events, so the pipeline from the producer’s commit to the first POST must stay short even at peak.
• Isolation (a slow endpoint costs only itself): The requirement that drives the hardest trade-off: with 1 M endpoints, some are always slow, down or hostile, and none of them may hold capacity or queue position that healthy endpoints need.
• No committed event lost (at least once · duplicates expected): If the payment committed, its event is delivered or visibly failed after 3 days. Exactly once is not on offer; stable event ids make duplicates harmless.
• Safe to call untrusted URLs (zero requests to private addresses): Customers choose the URL and the DNS behind it. The fleet must never become a proxy into our own network or cloud metadata.
• Producers never wait on delivery (accept 99.99 %): A broken delivery pipeline delays webhooks; it must never slow a payment. The outbox and the log decouple them completely.
• Debuggable by customers (attempts visible in seconds, kept 30 days): Most support tickets are “did you send it?”. The attempt log answers that without us, if it is fresh and complete.

• Deliveries per second: ~35 k avg · 100 k peak. 2 B events/day × 1.5 endpoints each = 3 B deliveries/day ÷ 86 400 ≈ 35 k/s. Peaks (US business hours, month-end billing runs) reach about 3× that, 100 k/s. Events alone are ~23 k/s; the fan-out multiplies them.
• Requests in flight at peak: ~30 k. Little’s law: in flight = rate × time. 100 k/s × 0.3 s mean for a healthy endpoint = 30 k open requests. The largest endpoint hanging until a 10 s deadline at its peak of 1 k/s would want 10 k more on its own, which is why every endpoint’s share is capped.
• Retries if a big endpoint is down a day: ~300 M attempts. The largest endpoint gets 1 % of traffic: 30 M deliveries a day. Down for 24 h, with each pending delivery on its own backoff, about 10 attempts each that day is 300 M doomed attempts, ~3.5 k/s. If it hangs rather than refusing, that is 35 k sockets held for 10 s each, more than the whole healthy fleet. A per-endpoint breaker turns it into ~150 probes.
• Attempt log: ~1 TB/day · ~3 TB stored. 3 B first attempts plus ~10 % retries ≈ 3.3 B attempts × ~300 B (ids, time, IP, status, latency, error class, first bytes of the response) ≈ 1 TB/day, 30 TB for 30 days raw. A columnar store compresses it ~10× to ~3 TB, which is what makes a 30-day dashboard affordable.
• Event payload store: ~4 TB/day · 120 TB for 30 days. 2 B events × ~2 KB average payload (capped at 256 KB) = 4 TB/day. Kept 30 days for retries, replay and the events API: 120 TB raw, ~30 TB compressed. Stored once per event, never per delivery or per attempt.
• Egress bandwidth: ~2.4 Gbit/s peak · ~9 TB/day. About 3 KB on the wire per delivery (2 KB body, signed headers, TLS framing). 100 k/s × 3 KB = 300 MB/s ≈ 2.4 Gbit/s at peak, and the 35 k/s average is ~9 TB a day out of the egress proxies. Bytes are not the problem; connections are.
• Dispatcher hosts: ~30 at peak · ~40 provisioned. Budget 60 k sockets at peak: 30 k healthy plus room for slow endpoints inside their caps. Async I/O holds ~2 000 concurrent requests per host comfortably (~64 KB of TLS state each), so 30 hosts, ~40 to survive a zone loss. New TLS handshakes, for the ~40 % of deliveries that find no warm connection, cost ~40 k/s × 1 ms ≈ 40 cores across the fleet.
• Endpoint index: ~1 GB. 1 M endpoints × ~1 KB (URL, event types, encrypted secrets, limits, state) = 1 GB. Small enough to hold in memory on every fan-out and dispatcher host, so neither makes a database read per delivery.

• Service DB + outbox (Postgres · CDC relay): The producing service’s own database (payments, orders, repositories). The business change and an outbox row holding the event commit in one transaction, so an event exists if and only if the change does. A CDC relay tails the write-ahead log and publishes outbox rows; the producer never calls the webhook system.
• Event log (Kafka · keyed by event id): Every event from every producing service, ~23 k/s on average, retained 3 days so fan-out can be re-run after a bug. Keyed by event id for an even spread: fan-out needs no ordering, and one huge account must not make a hot partition.
• Fan-out (Kafka transactions): For each event: store the payload once, look up the endpoints of that account subscribed to that type in the in-memory endpoint cache, and write one thin delivery task per endpoint. Consuming the event and producing its tasks commit in one Kafka transaction, so a crash neither loses nor duplicates tasks.
• Delivery tasks (Kafka · keyed by endpoint): One ~200-byte task per (event, endpoint): delivery id, event id, endpoint id, attempt and lane (live, replay, backfill). Keyed by endpoint id, so every endpoint lives in exactly one partition and has exactly one owner. Retention is a day, because anything not resolved quickly moves to the delivery store.
• Delivery store (backlogs · due-time index): Only the unhappy minority: retries with their next due time, in an index partitioned by (shard, due minute) so each dispatcher reads only its own; per-endpoint backlogs of parked tasks, oldest first; and checkpoints of each endpoint’s breaker and limit. The ~95 % of deliveries that succeed first time never touch it.
• Event store (KV by event id · 30 days): Each event’s exact payload bytes, stored once by event id: 4 TB a day for 30 days. Read by dispatchers on every send (through a local cache, since an event goes to 1.5 endpoints on average), by retries and replays, and by the events API customers use to reconcile.
• Dispatcher (owns partitions · async HTTP): Each host owns ~32 task partitions and so every endpoint in them, which makes per-endpoint limits, token buckets and breakers local counters. It schedules endpoints that have work by deficit round robin under an adaptive per-endpoint concurrency limit, parks what is over the limit instead of waiting on it, signs, sends with a 10 s deadline, and writes due times for failures.
• Egress proxy (SSRF filter · pinned IP): The only way out of the network for webhook traffic. The proxy resolves each CONNECT itself, refuses it if any address is private, loopback, link-local, metadata or reserved, and connects to the address it checked, so DNS rebinding has nothing to race. TLS runs end to end from the dispatcher; customers can allowlist the proxies’ published IPs.
• Customer dashboard: Where customers register endpoints and pick event types, watch each endpoint’s attempts and health, open a failed attempt, rotate secrets, and replay one event or backfill a window. The same actions are in the API for automation.
• Webhooks API (endpoints · attempts · replay): The public API behind the dashboard: create and update endpoints (checking HTTPS and the destination at registration), rotate secrets, list events and attempts, request replays and backfills. Replays become ordinary tasks on the delivery-tasks topic, so they obey the same limits as live traffic.
• Endpoint cache (in memory · ~1 GB): A full in-memory copy of the endpoint registry on every fan-out and dispatcher host: (account, event type) to endpoints, plus each endpoint’s URL, encrypted secrets, limits and state. Loaded at start and kept current from the registry’s changelog within seconds, so a disable or a rotation takes effect without a database read per delivery.
• Customer endpoints (HTTPS · 1 M of them): Customers’ HTTPS receivers. Most answer 200 in a few hundred milliseconds; some are slow, some are down for days, some are misconfigured, and a few are hostile. A 2xx inside the deadline is the only thing counted as delivered.
• Endpoint registry (Postgres · changelog): The source of truth for endpoints: account, URL, event types, signing secrets (encrypted under a KMS-wrapped key, two during a rotation), rate and concurrency ceilings, and state (enabled, warned, disabled). Every change is published to a compacted changelog topic that feeds the caches.
• Endpoint health (every minute): Aggregates the attempt log per endpoint: success rate, latency, backlog age and time since the last success. Warns the owner after 24 h without a success, disables the endpoint at 72 h, and flags endpoints that are slow or falling behind. It only writes endpoint state; dispatchers react through the changelog.
• Attempt log (ClickHouse · 30 days): One row per attempt: 3.3 B a day, ~1 TB raw and ~100 GB compressed, kept 30 days in daily partitions that are dropped whole. Ordered by (endpoint, time), so “latest attempts for this endpoint” is one range read and per-endpoint success rates are a cheap aggregate.

1. A payment event reaches a healthy endpoint: The main path: an event committed with the payment, fanned out to one task per subscribed endpoint, signed and posted through the egress proxy, and recorded. Nothing on it waits for anything slow, which is how the first attempt lands in about a second.
   1. The payment service commits the charge and an outbox row with the event, in one transaction.
   2. A CDC relay tails the database log and publishes the row to the event log.
   3. Fan-out stores the payload and looks up the account’s endpoints for this event type.
   4. Fan-out writes one thin delivery task per endpoint, keyed by endpoint.
   5. The dispatcher that owns the partition checks the endpoint’s limits and fetches the payload and secret.
   6. It signs the request and posts it through an egress proxy, which resolves the host, vets the IP and connects.
   7. The endpoint answers 200 in 180 ms; the dispatcher records the attempt and moves on.
2. A customer debugs and replays a failure: The read path: an endpoint’s attempts from the attempt log, the payload from the event store, a replay that goes through the normal delivery path, and a secret rotation that reaches every dispatcher in seconds.
   1. The customer opens their endpoint and lists its recent failed attempts.
   2. They open one failed attempt: the response we received and the payload we sent.
   3. The health panel shows the endpoint’s success rate, latency and breaker state for the day.
   4. After fixing their handler, they replay the event.
   5. The owning dispatcher sends it and the new attempt appears in the list.
   6. They rotate the signing secret, keeping the old one valid for 24 hours.
3. A big endpoint slows to eight-second responses: The scale-breaking case: the largest endpoint (350 deliveries a second) still answers, but in 8 s. Nothing fails, so no breaker trips; only the concurrency limit and fair scheduling stand between it and everyone else.
   1. The customer’s handler starts doing slow work inline; responses take 8 s but still return 200.
   2. Its concurrency limit caps it at about 100 in-flight requests and stops growing.
   3. Its tasks keep arriving at 350 a second; past 500 queued in memory they are parked in its backlog.
   4. Deficit round robin keeps the host’s sockets shared among endpoints that have work.
   5. The health job flags the endpoint as falling behind and tells the owner why.
   6. Once the handler is fixed, latency drops, the limit climbs and the backlog drains behind live traffic.
4. An endpoint is down for twenty hours: The failure path: per-delivery retries until the pattern is clearly the endpoint, then one probe timer instead of millions of attempts, then a ramped drain when it comes back.
   1. The endpoint’s servers go down; connections are refused and attempts fail fast.
   2. Each failed delivery gets a due time with exponential backoff and jitter in the delivery store.
   3. Failures cross the threshold, the breaker opens, and new tasks park in the endpoint’s backlog.
   4. The owner probes with the oldest pending delivery, every minute at first, backing off to every ten.
   5. The endpoint returns; a probe gets 200 and the breaker half-opens.
   6. The dispatcher drains the backlog oldest first, ramping the limit while latency stays healthy, with live events first.
5. Failing for three days: warn, disable, backfill: The background path: the health job, not the dispatchers, decides when an endpoint has failed long enough to warn and then disable; the owner fixes it, re-enables, and backfills the gap at a pace their server can take.
   1. An endpoint’s domain lapses; every probe fails with a DNS error.
   2. After 24 hours without a success, the health job marks the endpoint warned and the owner is emailed.
   3. At 72 hours it disables the endpoint; the caches drop it and the dispatcher stops probing.
   4. The owner fixes DNS, re-enables the endpoint and asks for a backfill of the failed window.
   5. The API expands the window from the event store into backfill tasks, which the dispatcher parks and drains at a ramped pace.

Ask: One customer’s endpoint starts taking eight seconds to answer. Why doesn’t everyone else’s webhook get late?

Good answers name: Shared log keyed by endpoint; one owner per partition enforces adaptive per-endpoint limits, schedules by deficit round robin, and parks overflow in a durable per-endpoint backlog, One shared FIFO queue and a shared worker pool, A physical queue per endpoint (SQS queue or Redis list), consumed while non-empty, Pools by health: fast lane, slow lane, quarantine.

Our pick: Delivery tasks go to a Kafka topic with 1,024 partitions keyed by endpoint id, so each endpoint has exactly one owner. A dispatcher host owns ~32 partitions (about 30 k endpoints), reads them continuously into small per-endpoint queues, and sends with an async HTTP client under three limits: the endpoint’s concurrency limit (adaptive, starting at 10, additive increase while p95 latency and errors are healthy, halved on timeouts, 5xx or 429, between 1 and a ceiling of 100 by default and up to 400 for the biggest customers), its token bucket (the customer’s maximum rate), and the host’s ~2 000-socket budget, shared across endpoints with ready work by deficit round robin. When an endpoint has more than 500 tasks waiting in memory, or its breaker is open, further tasks are written to its backlog in the delivery store and the reader moves on. Offsets commit up to the oldest unresolved task, where resolved means a 2xx, a retry written with its due time, or a durable park. With that in place, a hanging endpoint holds at most its limit, 100 sockets, instead of 10 k. Stripe and GitHub do not publish how they do this internally, so this is my design.

1. The dispatcher that owns a partition crashes. What happens to its endpoints?
   The consumer group gives its partitions to other hosts within seconds (cooperative rebalancing moves only those partitions). The new owner resumes from the last committed offset, so tasks that finished after that commit are sent again: up to about 10 s of that partition’s traffic, harmless to receivers that dedupe on the event id. Breaker state and adaptive limits are checkpointed to the delivery store when they change, so the new owner does not hammer an endpoint the old one had already found dead. Parked backlogs and due retries are durable and simply have a new reader.
2. One endpoint receives 5 k deliveries a second. Isn’t a single owner a bottleneck?
   At 200 ms that is 1 000 requests in flight, half of one host’s budget, which works but is uncomfortable, and deficit round robin keeps it from starving its neighbours. For the handful of endpoints that big, key their tasks by (endpoint, hash(event id) mod k) so k owners share it, each enforcing limit ÷ k. The cost is that the limit becomes a sum of shares to rebalance as traffic shifts, so it is a list of named exceptions rather than the default.
3. Why not a Redis counter per endpoint, so any worker can take any task?
   The counter is easy, about 200 k operations a second at peak. The hard question is what a worker does with a task whose endpoint is at its limit: put it back and it spins or reorders, hold it and it blocks, so you end up parking anyway. Counters also leak when a worker dies holding a slot, which needs leases with expiry. Ownership by partition makes the limit exact and local and makes parking a decision one process takes.
4. Fair per endpoint: can a customer game it with 200 endpoints?
   Yes, 200 endpoints get 200 turns, and if they all point at one server that server gets 200 times the limit. So limits also apply per destination host and per account, at a higher ceiling, because what actually overloads is a server, and what we are being fair to is a customer. The per-host limit also blunts someone using our fleet to flood a third party.
5. How would you know isolation is working?
   Measure what the requirement says: first-attempt latency from commit, at p99, for endpoints that are themselves healthy, broken down by dispatcher host and partition. If it rises on a host while its endpoints are fine, isolation failed. Run a canary endpoint that hangs, and another that returns 500, permanently in production, and alert if any partition’s healthy p99 moves when they are switched on.

Ask: Some customers’ handlers take twenty seconds. Why not give every endpoint a thirty-second timeout?

Good answers name: Connect within 3 s, whole attempt within 10 s, at most 64 KB of response read, no redirects, A 30 s timeout, An adaptive timeout per endpoint from its own latency, Fire and forget: do not wait for a response.

Our pick: Every attempt has one deadline of 10 s from its start: DNS, TCP and TLS must finish within 3 s, then we write the request and read the status line and at most 64 KB of body, keeping the first 1 KB for the dashboard. Any 2xx inside the deadline is success; a 3xx is a failure with a clear error, because following a redirect would bypass the egress checks. Payloads are capped at 256 KB; bigger objects are sent thin, with an id and a URL to fetch. Fleet sizing follows directly: 30 k healthy sockets at peak plus room for slow endpoints, each bounded by its limit, so a 60 k budget, about 30 hosts at ~2 000 each, 40 provisioned. The docs tell customers to return 2xx within a few seconds and process asynchronously, and the health job emails the owners of endpoints whose median is close to the deadline. GitHub documents a 10 s window and caps payloads at 25 MB, as far as I know; the numbers here are mine.

1. A receiver does its work and answers after 12 seconds. What happens?
   We time out at 10 s, record "timeout_total", and retry on the normal backoff, so the receiver gets the event a second time. Its dedupe on the event id makes that harmless, and the dashboard shows "timed out after 10 s", which tells them exactly what to fix. If it happens often, the health job emails them the endpoint’s latency distribution next to the deadline.
2. What does a slowloris endpoint do to you?
   It answers one byte every few seconds, so a per-read timeout never fires and the socket stays open forever. The overall 10 s deadline is what stops it, which is why the deadline is on the whole attempt rather than on each read. The 64 KB read cap stops the opposite trick, a server that streams gigabytes back at us.
3. Walk me through the 30 hosts.
   Little’s law gives 30 k healthy sockets at 100 k/s and 300 ms; doubling it allows slow endpoints within their limits, so 60 k. An async client holds ~2 000 concurrent TLS connections per host at ~64 KB each, about 128 MB, so 30 hosts. CPU is mostly TLS handshakes: the long tail of quiet endpoints means ~40 % of deliveries open a new connection, ~40 k handshakes a second at ~1 ms each, about 40 cores. HMAC signing a 2 KB body is microseconds.
4. Why does connect get its own, shorter timeout?
   Because connect failures are the most common sign of a down endpoint, and a firewall that silently drops packets would otherwise cost the full 10 s per attempt. A healthy server anywhere completes TCP and TLS in well under a second, so 3 s is generous. Failing in 3 s rather than 10 s also lets the breaker see the pattern three times faster.
5. Why cap payloads at 256 KB?
   At 100 k/s, a small fraction of multi-megabyte payloads would dominate egress, memory in the dispatchers and the event store. Most payloads are a few KB, so the cap only affects outliers, which get a thin event and fetch the object through the API. Rejecting at the producer, not at send time, means the limit is found in development rather than in production.

Ask: An endpoint is down for a day. What does your system do every minute of that day?

Good answers name: Due times in a scheduled-time index owned by the endpoint’s shard, per-delivery exponential backoff with jitter, and a per-endpoint breaker that turns an outage into one probe timer and a parked backlog, Workers sleep and retry in place, Kafka retry topics with fixed delays (1 min, 10 min, 1 h), Retry everything every 5 minutes for 3 days.

Our pick: Per delivery: after failure n, the next attempt is due 1 min × 2^(n−1) later, capped at 6 h, with ±50 % jitter, which is about 20 attempts over 72 hours counted from the event’s creation. The due time goes into the delivery store’s due index, partitioned by (shard, due minute) where the shard is the endpoint’s task partition, so the owning dispatcher reads its own bucket every second and nothing waits in memory. Per endpoint: the owner opens the breaker when half of the last 100 attempts failed with at least 20 failures. While it is open, new and due tasks park in the endpoint’s backlog and a single timer probes with the oldest pending delivery, after 1 minute and then doubling to every 10 minutes. A successful probe half-opens it: concurrency starts at 2 and grows by one a second while p95 latency and errors stay healthy, live tasks get their share first, and the backlog drains oldest first; deliveries older than 72 h are expired, not sent. The health job emails the owner at 24 h without a success and disables the endpoint at 72 h. Stripe documents retrying for up to three days with exponential backoff; the exact schedule here is mine.

1. The endpoint returns 500 for one event type and 200 for everything else.
   That is sporadic from the breaker’s point of view, because most attempts succeed, so it stays closed and those deliveries retry on their own backoff for 72 hours. The health panel shows "invoice.paid failing 100 %", which is the clue the customer needs. The breaker counts each delivery’s latest outcome rather than every retry, so one poison event retried 20 times cannot open it on its own.
2. It recovers with 25 M deliveries pending. How fast do you drain?
   As fast as it stays healthy: the limit ramps from 2 while p95 latency and error rates hold, up to the endpoint’s ceiling. At a ceiling of 400 and ~200 ms that is ~2 000 a second, of which 350 are live, so ~1 650 a second of backlog and about four hours. The customer can lower the maximum rate, or skip the backlog and reconcile from the events API instead.
3. What about 429 with Retry-After?
   Treat it as backpressure rather than failure: halve the endpoint’s limit and rate, honour Retry-After (capped at an hour) for that endpoint’s next sends, and do not count it towards the breaker or the disable clock. A receiver telling us to slow down is working correctly, and punishing it would be perverse.
4. Our own dispatchers were down for 20 minutes. What happens when they return?
   About 42 M tasks are waiting in Kafka (20 minutes at 35 k/s). Nothing special is needed: per-endpoint limits and the host budgets pace the catch-up, so each endpoint sees at most its usual concurrency, and the fleet’s spare capacity, ~200 k/s of sockets at 300 ms against 35 k/s of live traffic, clears the lag in a few minutes. Alert on consumer lag in seconds, because first-attempt latency is the SLO.
5. Why count 72 hours from the event’s creation, not from the last attempt?
   It makes the promise bounded and easy to state: a delivery is pending for at most three days, after which it is failed and replayable for the rest of the 30 days. Counting from the last attempt would let a delivery that is probed late live forever. It also matches how customers think about it: “I was down from Tuesday to Thursday”.

Ask: A customer asks: will I get each event exactly once, in order, and how do I know it came from you?

Good answers name: At least once and unordered; event id and object version in every payload; HMAC-SHA256 over timestamp and body with a per-endpoint secret; receivers reject timestamps older than 5 minutes, Ordered delivery per (endpoint, resource), one in flight per key, Asymmetric signatures (Ed25519) verified with a published public key, A secret token in the URL or a static auth header.

Our pick: Each request carries Webhook-Id (the event id, the same across retries and replays), Webhook-Timestamp (this attempt’s time) and Webhook-Signature: t=…,v1=hex(HMAC-SHA256(secret, t + "." + raw body)). The receiver recomputes the signature over the raw bytes rather than re-serialised JSON, compares in constant time, rejects a timestamp more than 5 minutes from its clock, inserts the event id into a table with a unique constraint, and treats the payload as a hint: it compares data.object.version with what it holds, or fetches the object from our API before doing anything that matters. Fast-changing objects can be sent as thin events, id and type only, so receivers always fetch the current state. Secrets are 32 random bytes per endpoint, encrypted under a KMS-wrapped key and decrypted only in dispatcher memory. Rotation adds a new secret and, for a window the customer picks (default 24 h), signs every request with both, as two v1 entries. Stripe’s Stripe-Signature header uses this t=…,v1=… form, and its libraries default to a 5-minute tolerance; GitHub’s X-Hub-Signature-256 signs the body alone.

1. Why sign the timestamp too, not just the body?
   Without it, anyone who captures one request, from a log or a misconfigured proxy, can replay it forever and the signature still verifies. With the timestamp inside the signed string, changing it breaks the signature and an old one is rejected by the 5-minute window. Inside the window, the receiver’s dedupe on the event id catches the replay.
2. A retry goes out two days later. Does its signature still pass the 5-minute check?
   Yes: every attempt is signed fresh with its own timestamp, so it is always current. The event id stays the same, so dedupe still works. The created field in the body tells the receiver how old the event really is, which is what it should use for staleness, not the header.
3. The receiver handles payment.succeeded, then gets an older payment.updated saying pending. What should it do?
   Compare the object’s version, or its updated time, and drop the older one. If it cannot, it should fetch the payment from the API, which is always current, and act on that. This is why the payload says “hint”: the event tells you something changed, the API tells you what is true now.
4. The outbox relay publishes the same event twice. Who stops the duplicate?
   Nobody needs to perfectly: the event id was generated in the producer’s transaction, so both copies have it. Fan-out derives delivery ids from (event, endpoint), so the copies produce identical tasks, and the receiver dedupes on the event id anyway. Being honest about at least once is cheaper than building exactly once in front of an HTTP call that cannot provide it.
5. A customer leaks their secret in a public repository. What do they do?
   Rotate with no overlap: the new secret signs every attempt from the moment the changelog reaches the dispatchers, a few seconds later, and the old one is gone. Requests already in flight finish with the old signature. The dashboard records the rotation time, so they can tell which requests to distrust in their own logs.

Ask: A customer registers a URL whose DNS later answers 10.0.3.7, your internal admin service, and reads the response in your dashboard. How do you stop it?

Good answers name: All webhook egress through a proxy fleet in an isolated subnet that resolves the name itself, refuses denied ranges, connects to the vetted IP, and tunnels TLS end to end; dispatchers have no other route out, Validate the URL at registration only, Resolve and check inside the dispatcher’s own HTTP client, Only allow domains the customer has proved they control.

Our pick: Dispatchers can reach only the egress proxies; network policy blocks everything else. Each request is an HTTP CONNECT for host:443 (HTTPS only, on a short list of ports). The proxy resolves the name with its own resolver, refuses if any returned address falls in a denied range (the IANA special-purpose registries for IPv4 and IPv6, with mapped and compatible forms normalised first, plus our own VPC ranges), connects to the first allowed address and splices bytes. TLS, with SNI and certificate checks against the hostname, runs end to end in the dispatcher, so the proxy never sees plaintext. Redirects are never followed. The proxies live in a subnet whose routes and firewall deny every internal range, so even a parsing bug lands nowhere. Registration still checks the URL for quick feedback, and every attempt records the IP we actually connected to. Stripe open-sourced an egress proxy built for exactly this job, Smokescreen, and GitHub publishes the IP ranges its webhooks come from so receivers can allowlist them.

1. The attacker writes the address as 0x7f.1, 2130706433 or [::ffff:127.0.0.1].
   Those are all tricks against string matching, and the proxy does no string matching on addresses. It parses the URL with one strict parser, resolves to binary addresses, normalises IPv4-mapped IPv6 to IPv4, and compares against CIDR ranges. Any literal IP in a URL is also refused at registration, as a courtesy.
2. The customer’s DNS has a zero TTL and alternates between a public and a private answer.
   Each connection resolves once and connects to the answer it checked, so alternating answers just mean some connections are refused. Connections are reused for keep-alive, and each new connection is checked again. We cache answers for their TTL with a 30-second floor to protect our resolvers, which changes nothing about safety because the check is on the cached answer too.
3. Could someone use your webhooks to flood a third party?
   Yes: register the victim’s URL on many accounts and generate events. Limits per destination host across all accounts cap the volume, new accounts start with low ceilings, and abuse detection looks for many new endpoints pointing at one host or floods of 4xx. Domain verification is the escalation for accounts that need high volume to one domain.
4. What about the response body you show in the dashboard?
   We read at most 64 KB and store at most 1 KB, as text, escaped, never rendered as HTML. With private destinations unreachable, it can only show what a public server chose to say to us, which is the information the customer needs to debug and nothing more.
5. How big is the proxy fleet?
   Bytes are modest, ~2.4 Gbit/s at peak; connections are the cost. With keep-alive, new tunnels are ~40 k a second at peak, each a DNS lookup, a check and a TCP connect. About a dozen hosts per region handle that with N+2 spares across zones, and dispatchers spread across proxies and fail over between them.

Ask: Customers want every attempt for 30 days, searchable per endpoint, with a replay button. What does that cost and where does it live?

Good answers name: A columnar store (ClickHouse) partitioned by day and ordered by (endpoint, time), fed in batches from an attempts topic, partitions dropped after 30 days, A wide-column store keyed by (endpoint, day), One row per delivery in the delivery store, updated on each attempt, Aggregates only, raw records to object storage.

Our pick: Dispatchers emit one record per attempt to an attempts topic keyed by endpoint, and loaders insert batches into ClickHouse every second or two: ORDER BY (endpoint_id, attempted_at), one partition per day, dropped after 30 days, about 100 GB a day compressed and ~3 TB live, with two replicas. A small table keyed by delivery id, keeping only the latest attempt, answers “delivered, pending or failed?” per event, and parked deliveries come from the delivery store. Request bodies are not stored per attempt: they are the event’s payload, kept once in the event store. Replay of one event writes a task with the original event id and a new delivery id. A backfill of a window is expanded from the event store and the endpoint’s subscriptions into tasks with lane "backfill", which always park, so they drain from the capacity live traffic leaves, oldest first, with the same ramp a recovering endpoint gets, a progress bar and a cancel button.

1. A customer backfills two million deliveries. What protects their server?
   The backfill never bypasses the endpoint’s limit: its tasks park and drain only when live traffic leaves room, starting at a concurrency of 2 and growing while latency and errors stay healthy. The customer can set a maximum rate for it and cancel it. Two million at ~1 000 a second is about half an hour, which is fast enough and slow enough.
2. Why not store the request body with every attempt, so customers see exactly what was sent?
   Because it is the same bytes for every attempt and every endpoint of an event. Storing 2 KB per attempt is ~6.6 TB a day, seven times the whole attempt log. We store the payload once by event id and the per-attempt timestamp, which is enough to reconstruct the exact signed request.
3. First-attempt successes never touch the delivery store. How does the dashboard know a delivery is done?
   From the latest-attempt table in the attempt store, which every attempt updates by key. Pending and parked deliveries come from the delivery store. The dashboard merges the two; the cost of that merge is far smaller than writing 3 B extra rows a day to the delivery store.
4. Why also have an events API if replay exists?
   Because after a long outage many customers would rather pull than be pushed four days of old events. Listing events since a time from the event store lets them reconcile at their own pace. Webhooks are the notification; the API is the record.
5. Legal wants a year of attempts, not 30 days.
   Keep 30 days hot and export each daily partition to object storage as Parquet before dropping it: ~100 GB a day compressed is ~36 TB a year, cheap at object-storage prices. A batch engine queries it when someone asks, which is rare enough that minutes of latency are fine.