Queueing and Little’s law: an interactive latency simulator
Little’s law says that the average number of requests in a system equals the rate they arrive times the average time each one spends there: L = λW. Queueing theory explains why that time climbs so steeply as servers get busy. When requests arrive at random, a server that is 50% busy makes each request take twice its work time on average, and one that is 90% busy makes it take ten times as long, almost all of it spent waiting in line.
The playground sends random requests to one or more servers that each do 10 ms of work per request, for ten simulated minutes. Move the utilization, add servers or change how much request sizes vary, and compare the result with the textbook curve.
One server, 80% busy, requests taking 10 ms of work on average, some more, some less: a request spends 50.0 ms in the system on average, most of it waiting in line (40.0 ms), and 226 ms at p99. At 50% busy the average would be 20.0 ms. On average 4.0 requests are in the system, which is what Little’s law predicts: 80 arrivals a second × 50.0 ms each = 4.0.
Little’s law: L = λW
L is how many requests are in the system at once, λ how many arrive per second and W how long each stays. It holds for any stable system, whatever the arrival pattern or request sizes, which makes it the most useful formula in capacity planning. A service handling 2,000 requests a second that take 50 ms each has 2,000 × 0.05 = 100 requests in flight at any moment, so it needs room for 100 concurrent requests: threads, connections or memory. Read it the other way and it gives latency: if a queue holds 500 messages and consumers take 100 a second, a new message waits about 5 seconds.
The playground checks it on every run. At 80% busy, 80 requests arrive a second and each spends 50 ms in the system; 80 × 0.05 = 4.0, and counting the requests in the system every 2 ms gives an average of 3.99.
Why latency explodes near 100% utilization
For one server with random arrivals and varied request sizes, the average time in the system is W = S / (1 − ρ), where S is the work per request and ρ the utilization. The denominator is what bends the curve:
| Utilization | Average time, as a multiple of the work | With 10 ms of work |
|---|---|---|
| 50% | 2× | 20 ms |
| 80% | 5× | 50 ms |
| 90% | 10× | 100 ms |
| 95% | 20× | 200 ms |
| 99% | 100× | 1 s |
Each step towards full costs more than the last: going from 90% to 99% busy, only 10% more traffic, makes requests ten times slower. The tail is worse than the average: at 80% busy the simulation’s p99 is 226 ms, against 50 ms on average. And near full, queues swing so widely that even ten minutes is a short sample: at 95% the simulation averages 168 ms against the theory’s 200 ms.
What makes queueing better or worse
- Variable request sizes make it worse. At the same 80% busy, requests that all take 10 ms spend 30.6 ms in the system on average; varied ones 50 ms; and when 5% of requests are big, 121 ms, because every small request stuck behind a big one waits for it. The wait grows with the variance of the request sizes.
- Pooling makes it better. Eight servers sharing one queue, each 80% busy, average 12.8 ms instead of 50 ms, because a request only waits when all eight are busy at once. This is why a few large instances queue less than many small ones at the same utilization, and why a load balancer that sends work to the least busy server behaves more like one shared queue than one that picks at random.
- Bursty arrivals make it worse. Traffic that arrives in bursts, from retries, batch jobs or a cron at the top of the hour, queues more than random arrivals at the same average rate.
What it means for capacity planning
Leave headroom. Latency-sensitive services usually run at 50 to 70% of what they can handle and autoscale before 80%, because at 90% a small spike sends latency through the roof. Look at p99 as well as the average, since the tail climbs first. Put a bound on every queue: an unbounded queue in front of an overloaded server only turns overload into latency, so reject or shed the excess early with load shedding and backpressure.
In a system design interview
Use Little’s law out loud when sizing anything concurrent: “At peak we expect 5,000 requests a second at about 40 ms each, so 200 are in flight at once. With 32 instances of 10 worker threads that is just over 60% busy, which leaves room for a spike before queueing takes over.” The same move sizes database connection pools, Kafka consumer counts and GPU batch slots. Then say what happens past capacity: requests queue, latency climbs steeply, and you shed load or scale out before it does.
Read the guide to queueing theory and capacity planning, or watch a strong candidate design an LLM inference service, where every GPU is a queue.
Questions people ask
What is Little’s law?
L = λW: the average number of items in a system equals the rate they arrive times the average time each spends there. It holds for any stable system, whatever the arrival pattern or service times. A service handling 2,000 requests a second at 50 ms each has 100 requests in flight on average.
How is Little’s law used in system design?
To size anything that holds work in progress. Multiply peak throughput by latency to get concurrency, and size thread pools, connection pools, consumers or GPU batch slots from it. Read the other way, it turns a queue length into a wait: 500 queued messages drained at 100 a second means a new one waits about 5 seconds.
Why does latency rise so sharply as utilization approaches 100%?
Because requests arrive at random, they sometimes bunch up, and the busier the server the longer each bunch takes to clear. For one server with random arrivals and varied request sizes the average time in the system is the work time divided by (1 − utilization): twice the work at 50% busy, ten times at 90% and a hundred times at 99%.
What utilization should a server run at?
Latency-sensitive services usually aim for 50 to 70% of capacity at peak and autoscale before about 80%, because past that point a small increase in traffic causes a large increase in latency, especially at p99. Batch systems that only care about throughput can run much closer to 100%.
Why do variable request sizes make queueing worse?
A small request that arrives behind a big one waits for all of it. Queueing delay grows with the variance of service times: at the same utilization, requests that all take the same time wait half as long as exponentially varied ones, and a mix of mostly small with a few very large requests waits much longer. Separating big requests into their own queue helps.
Why is one shared queue better than one queue per server?
With a shared queue a request waits only when every server is busy; with a queue per server it can wait behind a slow request while another server sits idle. At 80% utilization, eight servers sharing a queue keep the average time close to the work time, while eight separate single-server queues each behave like one server at 80% busy, five times the work time on average.
What is the difference between throughput, latency and utilization?
Throughput is how many requests are completed per second, latency is how long one request takes from arrival to completion, and utilization is the share of time the servers are busy. Utilization is throughput times work per request divided by the number of servers, and latency depends on utilization through queueing.