Timeouts, retries and deadline propagation: an interactive simulator

A timeout is how long one caller waits for one call. A deadline is the moment the whole request stops being useful, and deadline propagation passes it down the chain of calls so every service, and the database at the end, knows how long is left. Without it, each hop guesses on its own: services retry for callers that have already gone and databases finish queries nobody will read.

The playground sends 2,000 requests from a client through an API and a service to a database, under four strategies. Make queries stall, tighten the deadline, and watch one request’s trace and the totals.

What the playground shows

• One try with 1 s timeouts fails every request whose query stalls: 95% answered when 5% of queries stall. The stalled query runs on for up to 2.5 s, long after the client gave up at 1 s.
• Retries at every layer answer nearly everything, but multiply. The API tries the service 3 times and each service call tries the database 3 times, so when every query stalls, one request causes 9 queries and 18 seconds of database time. With 5 layers retrying 3 times each, it would be 243.
• Retries at one layer answer just as well with at most 3 queries. But nothing tells the database the caller has given up, so with 5% of queries stalling, 56% of its time is spent after the client has stopped waiting.
• A propagated deadline answers the same requests and the database cancels each query when its caller stops waiting: 66 ms of database time per request instead of 155, and 900 ms instead of 6 s when everything stalls.

Propagation does not make more requests succeed in this model. What it saves is capacity, and capacity is what an overloaded system is short of: queries for abandoned requests slow down the queries for live ones, which time out and are retried, which adds more load. Cutting the abandoned work is how a slowdown stays a slowdown instead of becoming an outage.

Rules for timeouts and retries

• Set a timeout on every network call. Many client libraries default to none, or to minutes, and one hung dependency then holds threads and connections until the caller runs out.
• Pick it from the latency distribution: a little above the dependency’s p99 or p99.9, so a retry is spent on the stragglers, not on ordinary slow calls.
• Inner timeouts must fit inside outer ones. If the API waits 300 ms and the service’s first query attempt alone may take 300 ms, the service’s retries finish after nobody is listening.
• Retry at one layer, usually the one closest to the failing dependency, with backoff and jitter and a retry budget, such as at most 10% extra calls.
• Pass the time left, not a timestamp. gRPC sends the remaining time in a header and each server turns it back into a local deadline, so clock skew between machines does not matter. Subtract a little at each hop for the network.
• Act on it everywhere: don’t start work that cannot finish in time, set the database’s statement timeout from it, and stop when the request is cancelled.

Deadline propagation in code

import time

class Deadline:
    """An absolute point in time, passed down as the time left."""
    def __init__(self, seconds):
        self.at = time.monotonic() + seconds
    def left(self):
        return self.at - time.monotonic()

def get_order(order_id, deadline):
    for attempt in range(3):                        # retry here, and only here
        left = deadline.left()
        if left < 0.04:                             # cannot finish in time: don't start
            raise DeadlineExceeded()
        try:
            with db.transaction() as tx:
                # the database gives up when we do
                ms = int(min(0.3, left) * 1000)
                tx.execute(f"SET LOCAL statement_timeout = {ms}")
                return tx.query_one("SELECT * FROM orders WHERE id = %s", (order_id,))
        except QueryCanceled:
            continue
    raise DeadlineExceeded()

def handle(request):
    # the caller says how long it will wait; pass on what is left, never a fixed timeout
    deadline = Deadline(int(request.headers.get("x-deadline-ms", "1000")) / 1000)
    return get_order(request.params["id"], deadline)

In Go, the deadline travels in context.Context and every library call that takes a context respects it; in gRPC, the client’s deadline arrives as the server’s context deadline and a cancelled call cancels the server’s context too.

In a system design interview

When you draw a chain of services, give the request a budget and say where it is spent: “The client waits 1 s. The gateway passes the remaining deadline in a header. Only the order service retries, twice, with jitter and a 10% retry budget. Each database query gets a statement timeout of whatever is left, so when a request is abandoned, so is its work.” Expect the follow-ups: how timeouts relate to tail latency, where a circuit breaker fits, and what happens to a non-idempotent call that timed out but may have succeeded.

Read the guide to resilience or load shedding and backpressure, see load shedding for the server’s side, or watch a strong candidate design a payment system.

Questions people ask