Load shedding and backpressure: an interactive overload simulator

Load shedding means refusing some requests on purpose when a server has more work than it can do, so that it keeps serving the rest at full speed. Without it, an overloaded server queues everything, every request waits longer than its client is willing to, and the server ends up doing nothing but work for clients that have already given up. Backpressure is the same idea passed upstream: tell the caller to slow down instead of buffering without limit.

The playground sends 80 requests a second to a server that can answer 100, with a spike in the middle. Clients give up after one second. Compare an unbounded queue with two ways of shedding.

Why an unbounded queue fails completely

At twice capacity, the queue grows by 60 requests every second. Within moments the wait passes one second, and from then on every request times out before the server reaches it, yet the server still does the work, in order, for clients that are gone. Successes do not fall to half: they fall to almost nothing. In the playground the server spends 70 seconds of work on answers nobody receives, and the damage outlasts the spike: the backlog drains at only 20 requests a second, so 50 seconds after traffic returns to normal it is still answering requests that timed out long ago.

Retries make it worse: every timed-out request comes back as another, doubling the failed attempts to almost 14,000 in the playground. This is how a brief spike becomes an outage that does not end on its own.

Two ways to shed load

• Bound the queue: refuse new requests at once, with a 503 or 429, when more than a set amount of work is already waiting. The refusal is cheap and fast, so clients can retry elsewhere or later, and everything accepted is answered quickly: 100 a second, all within about half a second, through a spike four times capacity.
• Drop work that cannot finish in time: carry each request’s deadline with it, and skip any whose deadline will pass before it could be served. Nothing is wasted and throughput stays at capacity, but the dropped requests fail slowly, after the client has waited the whole timeout, and successful ones take up to the full second.

Production systems combine them, and add a third trick: when the queue is long, serve the newest requests first. Facebook’s adaptive LIFO and the CoDel algorithm both do this, so that during a backlog the requests served are the ones whose clients are still waiting.

What to shed first

Not all requests are equal. Shed by priority: background jobs, prefetches and analytics before checkouts and logins. Shed by tenant, so one customer’s burst does not take everyone down. And shed as early as possible, at the load balancer or the edge, where refusing costs least. Rate limits, concurrency limits and circuit breakers are all forms of the same decision made at different places.

Load shedding in code

// shed load before the queue holds more work than clients will wait for
var inFlight atomic.Int64

func handler(w http.ResponseWriter, r *http.Request) {
    if inFlight.Load() >= maxInFlight {          // sized from capacity x acceptable wait
        w.Header().Set("Retry-After", "1")
        http.Error(w, "overloaded", http.StatusServiceUnavailable)   // fast, cheap refusal
        return
    }
    inFlight.Add(1)
    defer inFlight.Add(-1)

    ctx := r.Context()                            // carries the client's deadline
    if deadline, ok := ctx.Deadline(); ok && time.Until(deadline) < expectedWork {
        http.Error(w, "too late", http.StatusServiceUnavailable)    // cannot finish in time
        return
    }
    serve(ctx, w, r)
}

In a system design interview

When asked what happens under a traffic spike, do not stop at autoscaling, which takes minutes: “Each instance caps concurrent requests at what it can finish within our 500 ms target, about 50, and answers anything beyond that with a 503 and Retry-After. Requests carry their deadline, so downstream services skip work the caller has given up on, and low-priority traffic is shed first. Autoscaling catches up behind that.” Expect the follow-ups: how the limit is chosen, how clients should retry, and how you stop retries from amplifying the overload.

Read the guide to load shedding and backpressure, see queueing and Little’s law for why queues blow up near capacity and exponential backoff for retrying without making it worse, or watch a strong candidate design a ticketing system, built for spikes.

Questions people ask