Webhook delivery through endpoint outages with retries, backoff and circuit breakers: an interactive simulator

A webhook is an HTTP call your system makes to a customer’s server when something happens. The customer’s server is outside your control: it will be slow, return errors and go down for hours. A delivery system has to keep events through those outages without hammering the dead endpoint, deliver them when it comes back, and decide what to do when it never does.

The playground sends two events a minute to one endpoint for five days. Six hours in, the endpoint stops answering. Pick how long the outage lasts.

What each policy does

• Retrying a couple of times and dropping is simple and fails silently: a 12 hour outage loses 1,438 events, everything created while the endpoint was down.
• Per-event exponential backoff for days, the common pattern, delivers everything after a 12 hour outage, but every event keeps its own retry clock: 12,374 failed calls against an endpoint that is clearly down, 1,422 events delivered out of order, and some delivered 14.5 hours after creation because their next retry was hours away when the endpoint recovered.
• A queue per endpoint behind a circuit breaker stops sending after the first failure and probes with a single request at growing intervals: 28 failed calls in all. When a probe succeeds it drains the backlog in order, rate-limited so the customer is not flattened by 1,440 events at once. Nothing is lost or reordered.
• When the endpoint never comes back, retries cannot help. After 3 days per-event backoff starts dropping events, while the breaker queue disables the endpoint, emails the customer and keeps the events so they can be replayed once it is fixed.

What a delivery looks like

POST https://customer.example.com/hooks
Content-Type: application/json
Webhook-Id: evt_01JB8Z3K9Q            # stable across retries: the receiver dedupes on it
Webhook-Timestamp: 1761213600         # signed, so an old request cannot be replayed
Webhook-Signature: v1,Xg5w...=         # HMAC-SHA256(secret, id.timestamp.body)

{"type":"invoice.paid","sequence":48213,"data":{"invoice":"in_77","amount":4200}}

# the receiver: verify the signature, store the id, answer 2xx fast, process later
# anything else (timeout, 5xx, connection refused) counts as a failure and is retried

Delivery is at least once: a timeout does not prove the customer did not process the event, so retries can duplicate. Receivers dedupe on the event id, and should acknowledge quickly and process asynchronously so a slow handler does not look like a failure. Sequence numbers or timestamps let them detect and handle anything that still arrives out of order.

In a system design interview

For a webhook or notification platform, describe isolation, retries and recovery: “Events go into a durable log, and a dispatcher keeps one logical queue per endpoint, so one dead customer never delays another. On failure we open a breaker for that endpoint and probe with backoff instead of retrying every event. On recovery we drain in order with a rate limit. Each request carries an id and an HMAC signature with a timestamp. After three days of failure we disable the endpoint, notify the customer, and offer a replay from the log.” Expect the follow-ups: how to shard the dispatchers, how to handle one customer with huge volume, and whether order matters for their events.

Read the guide to webhooks or delivery semantics, see circuit breakers and dead-letter queues, or watch a strong candidate design Zapier.

Questions people ask