Retry storms and metastable failures: an interactive simulator

Retries are meant to hide brief failures, but under overload they multiply load exactly when the server can least take it. A short capacity drop makes requests time out, clients retry them, the extra load keeps the server overloaded after the original cause is gone, and the system stays down on its own. This is a metastable failure: a stable bad state that the trigger started but no longer explains.

The playground cuts a server’s capacity from 1,000 to 300 requests a second for five seconds while clients send 800 a second. Compare four retry policies, then let the server drop requests whose clients have given up.

How five seconds becomes forever

During the blip the queue grows faster than the server drains it, and requests start timing out. A server working through its queue in order still spends capacity on every one of those requests, though no client is waiting for the answer. Even with no retries, useful throughput stays near zero for nine seconds after the blip while the dead work drains. With clients retrying each timeout three times at once, they send 3,200 requests a second, four times the normal load, and the queue never drains: the server stays busy answering requests that have already been abandoned, every answer arrives too late, and every timeout breeds more retries. Backoff with jitter does not help here; it spreads retries out but sends just as many.

Breaking the loop

• Drop work nobody is waiting for. Propagate deadlines with requests and have the server reject anything already past its deadline before doing real work. In the playground this alone makes every policy recover the moment capacity returns. See deadline propagation.
• Bound retries globally. A retry budget allows retries worth at most a fixed share of normal traffic, say 10%, so retries can never multiply load. Here it turns “never recovers” into recovery at 55 s even without server help.
• Retry at one layer and stop retrying when a circuit breaker sees most calls failing.
• Shed load early with bounded queues, so overload turns into fast rejections instead of a growing backlog. See load shedding.

Backoff and jitter still matter: they stop synchronized retries from arriving in waves. But they reduce the rate of retries, not their number; only a budget or a breaker limits the total.

A retry budget and a deadline check in code

import time

class RetryBudget:
    """Allow retries worth at most 10% of the requests sent recently."""
    def __init__(self, ratio=0.1, cap=100):
        self.ratio, self.cap, self.tokens = ratio, cap, 0.0

    def on_request(self):
        self.tokens = min(self.cap, self.tokens + self.ratio)

    def try_retry(self) -> bool:
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False                 # out of budget: fail fast instead of piling on

# server side: never work on a request its caller has given up on
def handle(request):
    if time.monotonic() > request.deadline:
        return reject(request, "deadline exceeded")   # costs microseconds, not a full request
    return process(request)

In a system design interview

When you add retries, say how they are bounded: “Clients retry idempotent calls at most twice with jittered backoff, within a per-client retry budget of 10% of requests. Every request carries its deadline, and services reject expired requests at the door. Queues are bounded and overflow returns 503 at once, so a brief slowdown cannot turn into a backlog we spend minutes clearing.” Expect the follow-ups: how you would detect a metastable state, and how to bring a service back up when clients are already hammering it.

Read the guide to resilience or load shedding and backpressure, see exponential backoff, or watch a strong candidate design a payment system.

Questions people ask