Exponential backoff and jitter: an interactive retry simulator

Exponential backoff is a retry strategy: after each failed attempt a client waits longer before it tries again, usually doubling the wait up to a cap. Jitter makes each wait random, so that clients that failed together do not all retry together. Backoff gives a struggling server room to recover; jitter stops the retries from arriving as a wave.

In the playground, a crowd of clients calls a server at the same moment, more than it can serve. Each strategy gets the same clients and the same server, and each lane shows the calls the server sees every 100 ms: green it served, red it turned away. Compare how many calls each strategy makes and when its last client is done.

How exponential backoff works

1. Try the call. If it succeeds, or fails with an error that will not change (a 400 or a 404), stop.
2. Wait min(cap, base × 2attempt): with a 100 ms base, 100 ms after the first failure, then 200 ms, 400 ms, 800 ms, until the cap.
3. Add jitter: pick the actual wait at random, for example anywhere between zero and that value.
4. Give up after a few attempts, or when the caller’s deadline would pass, and return the error.

The doubling means a client that keeps failing backs off fast: five failures in, it is waiting over a second and a half between tries instead of hammering the server ten times a second.

Why backoff needs jitter

Clients that fail at the same moment and back off by the same rule retry at the same moment. In the first preset, 100 clients arrive together at a server that can serve 10 every 100 ms. With plain exponential backoff the 90 it turns away all come back together 100 ms later, then at 300 ms, 700 ms and 1.5 s, each time as a single spike of which the server serves at most 10. It makes 631 calls, and because the waits keep doubling, the last client is done only after 42.7 seconds. With decorrelated jitter the same clients make 340 calls and are all done in 2.2 seconds, because their retries are spread out and arrive at a rate the server can serve.

The same thing happens at larger scale whenever many clients fail together: a deploy, a dropped connection pool, a cache that expires for everyone at once, or a service coming back after an outage. This is the thundering herd, and jitter is the cheapest fix for it.

Retry storms: when retries keep a server down

Turning a call away is cheaper than serving it, but not free: the server still accepts the connection, parses the request and sends an error. The playground assumes a rejection costs a tenth of a served call. In the Half the capacity preset, 100 clients retry every 100 ms with a fixed delay at a server that can serve 5 per 100 ms: the retries alone cost more than it can do, so after the first 5 it never serves anyone again, and after two minutes 95 clients are still retrying. Nothing is broken any more except the retries themselves.

This is a retry storm, and the state it leaves behind is called a metastable failure: the overload sustains itself after its cause is gone, and the system only recovers when something cuts the load, such as clients giving up or an operator shedding traffic. Backoff, jitter, attempt limits and retry budgets are how clients avoid causing one.

Full, equal and decorrelated jitter

With heavy contention, all three jitter strategies make far fewer calls than either strategy without jitter, and finish far sooner than plain exponential backoff. Fixed delay can finish quickly when the server keeps up, as after the 3 second outage, but only by making six or seven times the calls, and with a little less capacity it never finishes at all. Which of the three jitter strategies does best moves around as you change the clients, the capacity or the outage, so the choice between them matters much less than having jitter at all. Marc Brooker’s 2015 AWS article on backoff and jitter, which compared these same strategies, reached the same conclusion and recommended full jitter as a default.

Retrying safely

• Retry only what can succeed. Timeouts, connection errors, 429 and 503 can; 400, 401, 403 and 404 will fail the same way again.
• Make the request safe to repeat. A timed-out payment may have gone through. Retry writes only when they are idempotent or carry an idempotency key.
• Honour Retry-After. When a 429 or 503 says how long to wait, wait at least that long.
• Bound it by a deadline. Stop retrying when the caller has given up; a retry nobody is waiting for is pure load.
• Retry at one layer. If three layers each make up to four attempts, one user request can become 4 × 4 × 4 = 64 calls at the bottom.
• Use a retry budget. Cap retries at a share of normal traffic, say 10%, so that when everything is failing, retries stop adding load. Finagle has retry budgets, the AWS SDKs a retry quota and gRPC retry throttling.
• Add a circuit breaker. When a dependency keeps failing, a circuit breaker stops the calls altogether instead of retrying each one.

Exponential backoff in code

import random, time

RETRYABLE = {429, 500, 502, 503, 504}

def call_with_retries(send, attempts=4, base=0.1, cap=5.0, deadline=10.0):
    start = time.monotonic()
    for attempt in range(attempts):
        response = send()                         # send() has its own timeout
        if response.status not in RETRYABLE:
            return response                       # success, or an error retrying won't fix
        if attempt == attempts - 1:
            break
        wait = random.uniform(0, min(cap, base * 2 ** attempt))  # full jitter
        wait = max(wait, float(response.headers.get("Retry-After", 0)))
        if time.monotonic() - start + wait > deadline:
            break                                 # the caller has stopped waiting
        time.sleep(wait)
    return response

Most HTTP and cloud SDK clients have this built in; check what yours does before adding another layer of retries on top of it.

In a system design interview

Retries come up wherever one service calls another: a payment provider, a push notification service, a webhook endpoint. Say the policy in one sentence: “Workers retry timeouts and 5xx up to 5 times with exponential backoff and full jitter, 200 ms base and 30 s cap, every send carries an idempotency key, and after the last attempt the message goes to a dead letter queue.” Then expect the follow-ups: what happens when the provider is down for an hour (a circuit breaker and a queue that holds the work), how you avoid sending twice, and what stops a retry storm when it comes back.

Read the resilience guide for timeouts, retries and bulkheads, or watch a strong candidate design a notification system, where retries to push and email providers are part of the design.

Questions people ask