Load balancing

Layer 4 vs layer 7 load balancers, routing algorithms, health checks, sticky sessions, global load balancing and how to talk about them in a system design interview.

A load balancer distributes incoming requests across a pool of servers so that no one server is overloaded and a failed server is taken out of rotation. It is the component that makes horizontal scaling of stateless services work, and it is in every diagram. What interviewers want is not that you draw it but that you can say what kind it is, how it picks a server, and what happens when things fail.

Layer 4 versus layer 7

Layer 4 (transport) balancers route on IP and port. They do not read the request; they forward packets. They are very fast (millions of connections, line-rate throughput) and protocol-agnostic, so they work for databases, gRPC, WebSockets and anything else over TCP or UDP. Examples: AWS NLB, Google's Maglev, Linux IPVS, HAProxy in TCP mode.

Layer 7 (application) balancers terminate the connection, read the HTTP request, and route on path, host, headers or cookies. They can do TLS termination, compression, retries, rate limiting, and send /api/orders to one pool and /api/search to another. They cost more CPU per request. Examples: AWS ALB, NGINX, Envoy, HAProxy in HTTP mode.

The usual layout is both: an L4 balancer in front, taking the raw connection load, spreading it over a tier of L7 proxies (an API gateway), which route to services. Say this in the interview and move on.

Routing algorithms

• Round robin. Each request goes to the next server. Fine when requests and servers are uniform.
• Weighted round robin. Bigger servers get more. Also used for canary releases: send 1 % to the new version.
• Least connections. Route to the server with the fewest in-flight requests. Better than round robin when request durations vary, because slow requests pile up on a server and it stops receiving new ones.
• Least response time. Like least connections but weighted by measured latency. Handles a degraded server automatically.
• Hash-based. Hash the client IP, a session id or a URL to pick a server. Gives stickiness without state on the balancer, and with consistent hashing it keeps most mappings stable when a server is added or removed. Use it for cache locality (the same key always hits the same cache node) and for WebSocket routing.
• Random with two choices. Pick two servers at random, send to the less loaded. Nearly as good as least connections with no global state, which matters when there are many balancers.

Default answer: least connections for HTTP services, consistent hashing when locality matters.

Health checks and failure

The balancer probes each server (an HTTP GET /health every few seconds, or a TCP connect) and removes servers that fail N checks in a row. Two subtleties are worth saying:

• A shallow health check (process is up) versus a deep one (can reach the database). Deep checks detect real failures but can take the whole pool out when the database blips, turning a partial outage into a total one. Most teams run a shallow check for the balancer and deep checks for alerting.
• Connection draining. When a server is removed for a deploy, stop sending new requests but let in-flight ones finish, with a timeout. Without it every deploy drops requests.

The balancer itself must not be a single point of failure. Cloud balancers are managed redundant services. Self-hosted, you run two with a floating IP (VRRP / keepalived) or announce the IP from several machines with anycast or ECMP.

Sticky sessions

Sticky sessions route a client to the same server every time, usually via a cookie set by the balancer. They let you keep session state in server memory, which is simple but fragile: a server restart logs everyone on it out, and the pool cannot rebalance. Prefer stateless services with sessions in Redis or in a signed token, and reserve stickiness for long-lived connections (WebSockets), where it is unavoidable and consistent hashing gives it for free.

Global load balancing

Across regions, the first hop is DNS. GeoDNS returns the IP of the nearest healthy region based on the resolver's location. It is coarse (resolver location, not user location) and slow to fail over (TTLs are cached, often longer than you set). Anycast announces the same IP from every region and lets BGP route each packet to the nearest one; failover is seconds and users do not need new DNS. Cloudflare, Google and AWS Global Accelerator work this way. Inside the region the normal L4 and L7 tiers take over.

For active-active multi-region, the hard part is not the balancer but the data: which region is allowed to write which records. See replication and consistency.

Service-to-service

Inside the data center, services also load balance across each other's instances. The options are a central L7 proxy per service (simple, one extra hop), client-side balancing where each caller fetches the instance list from service discovery (Consul, Kubernetes endpoints) and picks one itself (no extra hop, more client logic), or a sidecar proxy per instance (Envoy, the service mesh pattern) that gives client-side balancing, retries, mTLS and metrics without changing application code. Say "service mesh" if the design has many services; say "the Kubernetes service abstraction" if it is smaller.

Talking points in an interview

• "L4 in front for connection load, L7 behind it for routing and TLS."
• "Least connections, because request durations vary; hash on user id for the WebSocket tier so a reconnect lands on the same server."
• "Shallow health checks with connection draining, so deploys do not drop requests and a database blip does not empty the pool."
• "The balancer is redundant; anycast across regions for fast failover."