Adaptive concurrency limits vs fixed limits when a dependency degrades: an interactive simulator
A service protects itself by limiting how much work it has in flight and rejecting the rest quickly. The right limit depends on what the service and its dependencies can do right now, which changes: a slow database, a noisy neighbour, a failover. A fixed limit tuned for normal days is too high on bad ones. An adaptive limit watches latency, the way TCP congestion control watches round-trip time, and shrinks when work starts queueing.
The playground sends 400 requests a second to a service whose database can work on 50 at once, until it degrades for a minute to 20, or to 4. Clients give up after a second.
400 requests a second; each takes 100 ms while the database has room for 50 at once. In the middle minute it can only handle 4, and clients give up after 1 s. With no limit, requests pile up faster than they finish, every one waits past the timeout, and the server spends the rest of the run finishing work nobody is waiting for: goodput falls to zero and stays there after the database recovers. A fixed limit of 50, right for the healthy database, still admits 50 requests that now share room for 4: each takes about 1.3 s, past the timeout, so it rejects the rest and still answers nothing in time. The adaptive limit notices latency rising and cuts itself to what the database can do: 29 good answers a second at about 195 ms, rejecting the rest at once, and grows back when the database recovers.
What the limits do
- No limit turns a slowdown into an outage that outlives it. Requests pile up, all of them wait past the client timeout, and the server keeps finishing work for clients that have already gone. Goodput stays at zero even after the database recovers, because the backlog never drains while new requests keep joining it.
- A fixed limit of 50 rejects the excess and recovers instantly. When the database drops to 20 it still works, at 283 ms per request. When it drops to 4, the 50 admitted requests each take about 1.3 s, past the timeout: the limit sheds most of the load and still answers nothing in time.
- An adaptive limit cuts itself as latency rises and grows back when it falls. With the database at 4, it keeps about 30 good answers a second at around 200 ms, close to what the database can do; at 20, it keeps latency near 125 ms, trading a little throughput for it.
In code
// Netflix concurrency-limits style: limit in-flight work, adjust the limit from latency
const limiter = SimpleLimiter.newBuilder()
.limit(Gradient2Limit.newBuilder() // or AIMDLimit, VegasLimit
.initialLimit(20).maxConcurrency(200)
.build())
.build();
Optional<Listener> token = limiter.acquire(ctx);
if (token.isEmpty()) return reply(503); // shed now, cheaply, instead of queueing
try { handle(request); token.get().onSuccess(); }
catch (TimeoutException e) { token.get().onDropped(); } // counts as a congestion signalThe same idea appears as Envoy’s adaptive concurrency filter and in gRPC and service mesh load shedding. Pair it with deadlines, so work for clients that have given up is dropped, and with client retry budgets, so the rejected requests do not come straight back.
In a system design interview
When you discuss overload, go beyond rate limits per user: “Each service caps its in-flight requests with an adaptive limit based on latency, so when the database slows down the service rejects excess requests in a millisecond instead of queueing them until everyone times out. Requests carry deadlines and expired ones are dropped. Clients retry within a budget with backoff.” Expect the follow-ups: which requests to shed first, how the limit behaves across many instances, and how to tell an overloaded service from a broken one.
Read the guide to load shedding and backpressure or queueing theory and capacity planning, see load shedding and deadline propagation, or watch a strong candidate design a payment system.
Questions people ask
What is a concurrency limit?
A cap on how many requests a service works on at the same time. Requests beyond it are rejected immediately, typically with a 503 or 429, instead of queueing, which keeps latency bounded and protects the service and its dependencies when load exceeds capacity.
What is an adaptive concurrency limit?
A concurrency limit that adjusts itself from observed latency or errors, much like TCP congestion control: it grows while latency stays near its minimum and shrinks when latency rises, so it tracks the capacity actually available rather than a number chosen in advance.
Why is a fixed concurrency limit not enough?
Because capacity changes. A limit tuned for a healthy dependency admits too much work when that dependency slows down, so every admitted request becomes slow and may exceed client timeouts, while a limit tuned for bad days wastes capacity on good ones.
What is goodput?
The rate of requests completed successfully within their deadline, as opposed to throughput, which also counts work finished too late to be useful. Under overload a system can have high throughput and zero goodput.
Why does an overloaded service sometimes not recover after load drops?
When queues are unbounded, the backlog of requests that have already timed out keeps the service busy, so new requests also wait too long and time out. This metastable state persists until the backlog is shed, which is why bounded queues, limits and deadlines matter.
What algorithms are used for adaptive limits?
AIMD, additive increase and multiplicative decrease on a latency or error signal, and gradient or Vegas-style algorithms that compare current latency with the minimum observed. Netflix’s concurrency-limits library and Envoy’s adaptive concurrency filter implement them.
How do adaptive limits work across many instances?
Each instance runs its own limiter from its own latency, so no coordination is needed; together they shed roughly the right fraction of load. Clients should spread retries across instances and respect retry budgets.