Load balancing algorithms: an interactive comparison

A load balancing algorithm decides which server gets each request. Round robin and random spread requests evenly without looking at the servers. Least connections sends each request to the server with the fewest in progress, and power of two choices picks two servers at random and takes the less busy one. When servers and requests are all alike, any of them works; when one server is slow or some requests are long, only the ones that look at load keep response times down.

The playground routes the same minute of traffic to four servers with each algorithm. Start with one slow server, the case where they differ most, and watch its queue.

How the load balancing algorithms work

• Round robin: send each request to the next server in the list, then start again. Weighted round robin repeats bigger servers more often.
• Random: pick any server. Even on average, but bunches up by chance.
• Least connections (or least outstanding requests): send to the server with the fewest requests in progress, which is slow or busy servers’ way of saying “not me”.
• Power of two choices: pick two servers at random and send to the one with fewer in progress.
• Hashing (IP hash, consistent hashing on a key): always send the same client or key to the same server, for sticky sessions or cache locality, whatever the load.

Round robin vs least connections

Round robin assumes every server is as fast as every other and every request takes as long as every other. When that is true it is perfect and costs nothing. When it is not, it keeps sending a fair share to a server that cannot keep up: in the playground’s slow-server case at 80% load, round robin’s p99 climbs to about 48 seconds because one server’s queue never stops growing, while least connections stays near a second by sending that server only about an eighth of the requests. Counting requests in progress is the cheapest signal of how loaded a server is, and it reacts to slow servers, long requests and garbage collection pauses without being told.

Why power of two choices works

Choosing between just two random servers instead of one makes a dramatic difference: with one random choice the busiest of n servers ends up with about log n / log log n more than average, and with two choices about log log n, an exponential improvement. It also fixes least connections’ weakness at scale: with many load balancer instances each seeing slightly stale counts, they all pick the same least-loaded server at once and overload it, while random pairs spread out. That is why large fleets use it.

Which to use

A sensible default for HTTP services is least connections behind one balancer, or power of two choices when many balancers or clients choose independently. Keep round robin for identical servers with short, similar requests, and hashing for when the same key must reach the same server.

The algorithms in code

import random

def pick_server(servers):
    # power of two choices: two servers at random, the one with fewer requests in progress
    a, b = random.sample(servers, 2)
    return a if a.in_flight <= b.in_flight else b

def least_connections(servers):
    return min(servers, key=lambda s: s.in_flight)

def round_robin(servers, counter=[0]):
    counter[0] += 1
    return servers[counter[0] % len(servers)]

Whatever the algorithm, pair it with health checks that take a failing server out of the list, and connection draining so a server being deployed finishes its requests before it leaves.

In a system design interview

Name it in a sentence: “An L7 load balancer in front of stateless API servers, least outstanding requests, health checks every few seconds and connection draining on deploys; between services, client-side power of two choices.” Expect the follow-ups: sticky sessions (avoid them by keeping servers stateless), L4 vs L7 (L7 sees requests, so it can route by path and count requests rather than connections), what happens when a server is slow but healthy (outlier detection), and balancing across regions (DNS or anycast in front).

Read the load balancing guide and the tail latency guide, or watch a strong candidate design LLM inference serving, where routing by load matters most.

Questions people ask