Updated 24 October 2026

Load balancing

Layer 4 vs layer 7 load balancers, routing algorithms, health checks, sticky sessions, global load balancing and how to talk about them in a system design interview.

Try it yourself, in a playground:Load balancing algorithms comparedGraceful shutdownRendezvous hashingHealth checksSession management

A load balancer distributes incoming requests across a pool of servers so that no one server is overloaded and a failed server is taken out of rotation. It is the component that makes horizontal scaling of stateless services work, and it is in every diagram. What interviewers want is not that you draw it but that you can say what kind it is, how it picks a server, and what happens when things fail.

Layer 4 versus layer 7

Layer 4 (transport) balancers route on IP and port. They do not read the request; they forward packets. They are very fast (millions of connections, line-rate throughput) and protocol-agnostic, so they work for databases, gRPC, WebSockets and anything else over TCP or UDP. Examples: AWS NLB, Google's Maglev, Linux IPVS, HAProxy in TCP mode.

Layer 7 (application) balancers terminate the connection, read the HTTP request, and route on path, host, headers or cookies. They can do TLS termination, compression, retries, rate limiting, and send /api/orders to one pool and /api/search to another. They cost more CPU per request. Examples: AWS ALB, NGINX, Envoy, HAProxy in HTTP mode.

The usual layout is both: an L4 balancer in front, taking the raw connection load, spreading it over a tier of L7 proxies (an API gateway), which route to services. Say this in the interview and move on.

Routing algorithms

• Round robin. Each request goes to the next server. Fine when requests and servers are uniform.
• Weighted round robin. Bigger servers get more. Also used for canary releases: send 1 % to the new version.
• Least connections. Route to the server with the fewest in-flight requests. Better than round robin when request durations vary, because slow requests pile up on a server and it stops receiving new ones.
• Least response time. Like least connections but weighted by measured latency. Handles a degraded server automatically.
• Hash-based. Hash the client IP, a session id or a URL to pick a server. Gives stickiness without state on the balancer, and with consistent hashing it keeps most mappings stable when a server is added or removed. Use it for cache locality (the same key always hits the same cache node) and for WebSocket routing.
• Random with two choices. Pick two servers at random, send to the less loaded. Nearly as good as least connections with no global state, which matters when there are many balancers.

Default answer: least connections for HTTP services, consistent hashing when locality matters.

Health checks and failure

The balancer probes each server (an HTTP GET /health every few seconds, or a TCP connect) and removes servers that fail N checks in a row. Two subtleties are worth saying:

• A shallow health check (process is up) versus a deep one (can reach the database). Deep checks detect real failures but can take the whole pool out when the database blips, turning a partial outage into a total one. Most teams run a shallow check for the balancer and deep checks for alerting.
• Connection draining. When a server is removed for a deploy, stop sending new requests but let in-flight ones finish, with a timeout. Without it every deploy drops requests.

The balancer itself must not be a single point of failure. Cloud balancers are managed redundant services. Self-hosted, you run two with a floating IP (VRRP / keepalived) or announce the IP from several machines with anycast or ECMP.

Sticky sessions

Sticky sessions route a client to the same server every time, usually via a cookie set by the balancer. They let you keep session state in server memory, which is simple but fragile: a server restart logs everyone on it out, and the pool cannot rebalance. Prefer stateless services with sessions in Redis or in a signed token, and reserve stickiness for long-lived connections (WebSockets), where it is unavoidable and consistent hashing gives it for free.

Global load balancing

Across regions, the first hop is DNS. GeoDNS returns the IP of the nearest healthy region based on the resolver's location. It is coarse (resolver location, not user location) and slow to fail over (TTLs are cached, often longer than you set). Anycast announces the same IP from every region and lets BGP route each packet to the nearest one; failover is seconds and users do not need new DNS. Cloudflare, Google and AWS Global Accelerator work this way. Inside the region the normal L4 and L7 tiers take over.

For active-active multi-region, the hard part is not the balancer but the data: which region is allowed to write which records. See replication and consistency.

Service-to-service

Inside the data center, services also load balance across each other's instances. The options are a central L7 proxy per service (simple, one extra hop), client-side balancing where each caller fetches the instance list from service discovery (Consul, Kubernetes endpoints) and picks one itself (no extra hop, more client logic), or a sidecar proxy per instance (Envoy, the service mesh pattern) that gives client-side balancing, retries, mTLS and metrics without changing application code. Say "service mesh" if the design has many services; say "the Kubernetes service abstraction" if it is smaller.

Talking points in an interview

• "L4 in front for connection load, L7 behind it for routing and TLS."
• "Least connections, because request durations vary; hash on user id for the WebSocket tier so a reconnect lands on the same server."
• "Shallow health checks with connection draining, so deploys do not drop requests and a database blip does not empty the pool."
• "The balancer is redundant; anycast across regions for fast failover."

Anatomy of a request path

Say the whole chain once, so the interviewer knows you understand where the balancing happens: DNS (or anycast) picks a region → an edge or L4 load balancer terminates the connection → an L7 proxy routes by host and path → a service-level balancer picks an instance, usually client-side with service discovery. Four layers, four different failure modes, and only the last two know anything about your application.

Queueing, in one paragraph of theory

Latency does not rise linearly with load; it rises like 1 / (1 - utilisation). At 50 % utilisation the queue is short, at 80 % it is twice as long, at 95 % it is catastrophic. That is why capacity plans target 50–70 % utilisation at peak, why a balanced fleet beats an unbalanced one so decisively, and why removing a slow instance can make the whole fleet faster. If you remember one thing about load balancing, make it that the tail is created by queueing, not by slow code.

Taming the tail

• Least-outstanding-requests beats round robin when request cost varies, because it routes away from instances that are already struggling.
• Power of two choices: pick two instances at random and send to the one with fewer in-flight requests. Nearly as good as full least-loaded, with none of the coordination.
• Hedged requests: after the p95 elapses, send a duplicate to another instance and take the first response. A few percent extra load buys a much better p99. Only safe on idempotent reads.
• Outlier ejection: remove an instance that is failing or slow relative to its peers, then probe it before returning it.

Connection draining and deploys

A deploy should never produce errors. The sequence: mark the instance unhealthy so the balancer stops sending new requests, keep serving in-flight ones until they finish (a drain timeout of 30–60 s), then shut down. The application must handle SIGTERM by failing readiness first and closing keep-alive connections gracefully. For long-lived connections (WebSocket), drain means telling clients to reconnect with jitter, a few percent at a time, or the reconnect storm becomes the outage.

Checklist

• Where each balancing decision happens, from DNS down to the instance.
• Algorithm, and why it suits the request-cost distribution.
• Health checks: shallow for liveness, deep for readiness, plus outlier ejection.
• Target utilisation at peak.
• Drain behaviour on deploy, including long-lived connections.

Questions people ask

What is a load balancer in system design?

A load balancer distributes incoming requests across a pool of servers so that no one server is overloaded and a failed server is taken out of rotation. It is the component that makes horizontal scaling of stateless services work. The usual layout has two tiers: a layer 4 balancer in front takes the raw connection load and spreads it over layer 7 proxies, such as an API gateway, which route to services.

What are the most common load balancing techniques?

The most common load balancing strategies are round robin, weighted round robin, least connections, least response time, hash-based routing and random with two choices. Round robin is fine when requests and servers are uniform, but least connections is better when request durations vary, because a server piling up slow requests stops receiving new ones. A good default is least connections for HTTP services and consistent hashing when locality matters, such as cache nodes or WebSocket routing.

Layer 4 vs layer 7 load balancer: what is the difference?

A layer 4 load balancer routes on IP and port without reading the request, while a layer 7 load balancer terminates the connection, reads the HTTP request and routes on path, host, headers or cookies. Layer 4 is very fast and protocol-agnostic, so it works for databases, gRPC and WebSockets. Layer 7 costs more CPU per request but can do TLS termination, compression, retries and rate limiting, and send /api/orders to one pool and /api/search to another.

Which kind of load balancer works for databases?

A layer 4 load balancer works for databases, because it routes on IP and port and forwards packets without reading the request, which makes it protocol-agnostic for anything over TCP or UDP. Examples include AWS NLB, Google's Maglev, Linux IPVS and HAProxy in TCP mode. A layer 7 balancer, by contrast, terminates the connection and routes on the contents of the HTTP request, such as path, host, headers or cookies.

How do load balancers handle a failed server?

The balancer probes each server, with an HTTP GET /health every few seconds or a TCP connect, and removes servers that fail N checks in a row. Most teams run a shallow check (the process is up) for the balancer and deep checks (it can reach the database) for alerting, because a deep check can take the whole pool out when the database blips. When a server is removed, connection draining lets in-flight requests finish, with a timeout, so deploys do not drop requests.

What are sticky sessions in load balancing?

Sticky sessions route a client to the same server every time, usually via a cookie set by the load balancer, so session state can stay in server memory. That is simple but fragile: a server restart logs everyone on it out, and the pool cannot rebalance. Prefer stateless services with sessions in Redis or in a signed token, and reserve stickiness for long-lived connections such as WebSockets, where it is unavoidable and consistent hashing gives it for free.