Design a Load Balancer

Spread 5 M requests a second over 50 k backends, terminate TLS, route by host and path, survive losing any balancer or data centre.

Last updated 2026-10-26. Difficulty: hard. Patterns: load-balancing, networking, consistent-hashing, health-checks, control-plane. Reported at Google, Amazon, Microsoft, Meta, Cloudflare, Uber.

Functional requirements

• Spread traffic across healthy backends. Per connection at L4, per request at L7. A failing backend leaves rotation within seconds, and nothing new is sent to a backend that is draining.
• Terminate TLS for every public hostname. A few hundred hostnames, certificates renewed automatically, private keys never on disk. Re-encrypt to backends with mTLS.
• Route by host, path and headers. api.example.com/v1/orders to the orders service, weighted splits for canaries (95 / 5), header matches for internal testing.
• Survive the loss of any balancer, zone or region. Losing an L4 or L7 box may reset at most that box’s own share of connections. Losing a region moves its users to the others within a minute.
• Drain and slow-start backends for deploys. Stop sending new requests, let in-flight ones finish, then remove. Ramp a new instance up over a minute instead of handing it a full share while it is cold.
• Send users to the nearest healthy region. Anycast for the front door; DNS for deliberate moves such as draining a region or keeping an account in its home region for residency.
• Self-service config for a few hundred services. Teams own their routes. Changes are reviewed, versioned, rolled out in stages and reverted in one step.
• Internal service-to-service balancing. The same routing, health and config model, applied by a proxy on the caller’s host, so an internal call takes no extra hop through a shared tier.
• Boundary features. Per-client rate limits (the design from the Rate Limiter question), request ids, access logs and per-route metrics, all at the L7 tier.
• Out of scope. Volumetric DDoS scrubbing beyond SYN cookies and drop rules, the CDN and static caching, WAF rules, and the service registry itself, which the compute platform owns.

Non-functional requirements

• Throughput (5 M req/s · 10 M conns). Global peak over 4 regions: ~1.25 M requests a second per region, 1.7 M when a region is lost. Capacity grows by adding boxes; nothing central sits on the request path.
• Added latency (L4 µs · L7 p99 < 2 ms). L4 is a table lookup and an encapsulation. L7 parses HTTP and picks a backend. Full TLS handshakes are the expensive part, so the resumption rate matters more than proxy code speed.
• Availability (99.995 %). Everything sits behind the balancer, so its availability caps every service’s. That is about 26 minutes a year for all services combined.
• Connection stability (≤ 1/N reset per box lost). Adding or losing one of N boxes may disturb only that box’s share of connections, never reshuffle all of them. This drives the hardest trade-off in the design: L4 boxes must agree on where every flow goes without sharing any state.
• Failure detection (backend < 10 s · region < 30 s). Passive outlier detection reacts in under a second on real traffic, active probes catch idle backends, and regional failover is bounded by detection, not by DNS TTLs.
• Config propagation (endpoints p99 < 5 s). Endpoint changes from deploys and autoscaling must reach every proxy fast. Route changes are riskier and roll out in stages over ~30 minutes. If the control plane is down, the data plane keeps its last good config.
• Capacity headroom (N+1 regions). Each region is provisioned a third above its normal peak, so losing one region at peak does not overload the other three and cascade into a global outage.
• Key custody (no private keys on disk). TLS keys exist only in memory on proxies, or stay in an HSM-backed key service. A stolen disk, image or backup must expose nothing.

Back-of-envelope estimates

• Requests per region: ~1.25 M/s · 1.7 M/s after a region loss. 5 M/s at peak ÷ 4 regions = 1.25 M/s. Lose one and the other three carry 5 M ÷ 3 ≈ 1.7 M/s each, so every region is sized for 1.7 M: a third above its normal peak. That headroom is the price of N+1.
• Inbound packets at the L4 tier: ~50 M pps · ~17 M per region in a failover. About 10 inbound packets per request: request segments, ACKs for a ~10 KB response, amortised handshakes and teardown. 5 M × 10 = 50 M pps globally, and 50 M ÷ 3 ≈ 17 M pps per region with one region down. Responses never cross L4 (direct server return), which keeps 5 M × 10 KB ≈ 400 Gbps of response traffic off it.
• L4 boxes per region: ~5 needed · run 8. 17 M pps ÷ 4 M pps per box (a conservative planning number for XDP or DPDK forwarding on a 2 × 25 GbE server) ≈ 4.3, so 5. Run 8: two spare for maintenance and failure, and room for a SYN flood. The L4 tier is small and is sized by attack headroom more than by legitimate load.
• L7 proxies per region: ~34 needed · run 40. 1.7 M req/s ÷ 50 k req/s per proxy (TLS mostly resumed, keep-alive, 32 cores at ~60 % CPU) ≈ 34. Run 40 per region, 160 globally. Per box, L4 moves about 8× the requests L7 can parse (4 M pps ≈ 400 k requests), which is why L7 is the tier that scales out and changes often.
• Connections per L7 proxy: ~83 k · ~3.3 GB of buffers. 10 M connections ÷ 4 regions = 2.5 M; with a region down, 3.3 M per region ÷ 40 proxies ≈ 83 k per proxy. At ~40 KB per TLS connection (socket buffers, TLS and HTTP/2 state) that is ~3.3 GB, easy for a 128 GB box. The L4 tier tracks the same flows at ~100 B each: 3.3 M ÷ 8 boxes × 100 B ≈ 40 MB per box.
• TLS handshakes: ~170 k new conns/s · ~50 k full handshakes/s. 10 M connections lasting ~60 s on average = ~170 k new connections a second globally. With 70 % resumed from session tickets, ~50 k full handshakes a second. At ~1 ms of CPU each with RSA-2048 that is ~50 cores; with ECDSA P-256 at ~0.1 ms, ~5. Steady state is cheap: the risk is a failover moving 2.5 M connections at once, where resumption decides whether it is a blip or a CPU wall.
• Health check probes: 8 M/s naive · 75 k/s central. If all 160 edge proxies probed all 50 k backends once a second: 160 × 50 k = 8 M probes/s, 160 a second landing on every backend, before counting the ~5 k per-host proxies. Instead, 3 central checkers per backend (one per zone) every 2 s: 50 k × 3 ÷ 2 = 75 k probes/s, 1.5 per backend per second.
• Config size and push: ~6 MB snapshot · ~30 GB per full push. 50 k endpoints × ~100 B = 5 MB, plus ~1 MB of routes and clusters for 300 services. Sending the whole 6 MB to ~5 k proxies is 30 GB per change, and deploys and autoscaling change about 100 k endpoints a day, tens a second during a big deploy. So pushes must be incremental: one endpoint change is ~100 B, sent only to the proxies that route to that service.

Components

• Users and apps: Browsers, mobile apps and partner servers on the internet. They resolve one hostname and open TLS connections that live about a minute on average. A reset connection shows up to them as an error or a retried request, which is why the design counts how many connections each failure moves.
• Authoritative DNS (anycast · health-aware): Answers queries for the company’s hostnames, itself served from anycast addresses in every region. Normally it returns the global anycast VIP with a 60 s TTL; for a deliberate move (draining a region, residency) it returns per-region addresses instead. It is the steering wheel for planned changes, not the failover mechanism.
• Global traffic controller (region health · BGP policy): Decides which regions take traffic. Probes each region from ~40 outside vantage points, reads each region’s capacity, and tells the region’s edge routers to announce, prepend or withdraw the anycast prefixes, and DNS what to answer. Guardrail: it never withdraws a second region without a human, and fails back more slowly than it fails over.
• Edge routers (BGP · ECMP): Border routers in each region that peer with ISPs and announce the anycast prefixes. Inside the region they spread packets for a VIP over the L4 boxes announcing it, by ECMP on a hash of the 5-tuple. They keep no per-connection state, and when an L4 box disappears they rehash, which can move flows from one L4 box to another.
• L4 balancers (XDP · Maglev hash · DSR): About 8 boxes per region that forward packets and never terminate them. Each picks an L7 proxy from a consistent-hash lookup table built from the same versioned proxy list on every box, so any L4 box sends a given flow to the same proxy. A small connection table keeps flows in place while the proxy list changes; packets are encapsulated to the proxy, and replies bypass L4 entirely (direct server return).
• L7 proxies (Envoy-style · TLS · HTTP): ~40 proxies per region that terminate TCP and TLS, parse HTTP, match host, path and header routes, apply rate limits, and pick a backend per request with power of two choices. They keep pooled HTTP/2 connections with mTLS to backends, retry within a budget, and eject failing backends on their own. All of their behaviour is config pushed by the control plane.
• Key service (HSM · certs · ticket keys): Issues and rotates certificates (public ones through ACME from a public CA, internal ones from our own CA) and delivers private keys to proxies over mTLS, held in memory only. Also distributes the session-ticket keys that every proxy in every region shares, rotated every 12 hours. For edge sites we do not fully control it keeps the key and signs handshakes itself (keyless).
• Rate limit store (Redis · token buckets): Shared counters for per-client and per-route limits at the boundary: the design from the Rate Limiter question. Proxies check local buckets first and the shared store only for limits that must hold across proxies, with a short timeout and fail-open. A rejected request costs a 429 and never reaches a backend.
• Service backends (50 k instances · 300 services): The instances behind every route, on ~5 k hosts. Each exposes a cheap readiness endpoint that says whether it can serve, not whether every dependency is perfect, and handles SIGTERM by failing readiness, finishing in-flight work and closing keep-alive connections. Membership comes from the scheduler and health from the checkers, on purpose.
• Per-host proxies (internal calls · same xDS): One proxy on each of ~5 k hosts handles that host’s outgoing internal calls with the same routes, P2C balancing, outlier detection and mTLS as the edge, programmed by the same control plane. Internal traffic takes no extra hop and no shared tier, so a broken proxy affects only the callers on its own host.
• Control plane (xDS · versioned snapshots): Watches the registry and the health checkers, reads routes from the config store, compiles versioned snapshots and streams incremental updates to ~5 k proxies, and the proxy list to the L4 boxes. Endpoints take a fast path of seconds; route changes are validated and rolled out in stages. It is off the request path: if it dies, every proxy keeps serving its last accepted config.
• Config store (versioned · reviewed): Source of truth for listeners, routes, clusters, weights and limits, one versioned document per service, changed through reviewed commits. Every applied version is kept, so rollback means re-pushing an earlier version. Small: about 1 MB for 300 services.
• Service registry (owned by the scheduler): Which instances exist for each service, with their addresses and labels, maintained by the compute platform (Kubernetes-style endpoints). The balancer consumes it and does not own it. Membership and draining come from here, health comes from the checkers, and the control plane combines the two.
• Health checkers (3 per backend · every 2 s): A fleet in each zone that probes every backend’s readiness endpoint every 2 s, three checkers per backend, and reports a transition after 3 consecutive results agreed by 2 of 3 checkers. It replaces every proxy probing every backend, which would be 8 M probes a second. Only transitions are reported, so a healthy steady state costs the control plane nothing.
• Telemetry pipeline (Kafka → metrics, logs): Access logs and per-route, per-backend metrics from every proxy, batched and shipped asynchronously, never on the request path. Feeds dashboards, canary analysis for weighted rollouts and the controller’s view of each region’s load. Metrics are complete; access logs are sampled for successes and kept for every error.

User flows

1. An HTTPS request from a user to a backend. The main path through every layer. What matters is where each decision is made, which layers hold state, and that the response skips the L4 tier on its way back.
   1. The client resolves the hostname and gets the global anycast address. One address for every region, so the answer does not depend on where the resolver is and a regional failure needs no DNS change. TTL 60 s. DNS is kept for deliberate moves, where it hands out per-region addresses instead.
   2. The SYN reaches the nearest region over BGP, and an edge router hashes it to one of the L4 boxes. ECMP hashes the 5-tuple (source IP and port, destination IP and port, protocol) over the 8 boxes announcing the VIP, so every packet of a flow takes the same path while the set of boxes is stable. The router keeps no per-flow state, which is how it does this at line rate.
   3. The L4 box looks the flow up, picks an L7 proxy from its lookup table, and forwards the packet encapsulated. A SYN misses the connection table, so the box indexes its 65,537-entry Maglev table by hash(5-tuple) and records the choice. Every L4 box builds the same table from the same proxy list, so a later packet of this flow that lands on a different L4 box still picks the same proxy. The outer header is addressed to the proxy; the inner packet still says VIP:443.
   4. The proxy decapsulates, completes TCP and TLS, and resumes the client’s session from its ticket. The proxy has the VIP on a loopback interface, so its kernel accepts the inner packet as its own. Resumption works on any proxy in any region because ticket keys are shared, which matters: a new connection has a new source port and usually hashes to a different proxy. A full handshake signs with an ECDSA key held only in memory.
   5. The proxy matches a route on host and path, then checks the client’s rate limit. Routes are compiled into a lookup by host and then longest path prefix: microseconds per request. This route says cluster orders, split 95 / 5 between v1 and v2 for a canary. Rate limiting is a boundary feature: local token buckets first, the shared store only for limits that must hold across proxies, fail open on a timeout.
   6. The proxy picks a backend with power of two choices and sends the request on a pooled connection. Two random healthy endpoints of orders-v1; the one with fewer requests in flight from this proxy wins, adjusted by weight and slow start. The upstream connection is a long-lived HTTP/2 connection over mTLS, so there is no handshake per request. The proxy adds a request id, the client address and the remaining deadline.
   7. The response goes from the proxy straight out through the edge router to the client, skipping the L4 tier. The proxy replies with the VIP as the source address and routes it out directly. Responses are ~10× the bytes of requests, so direct server return keeps ~400 Gbps off the L4 tier globally. The cost: L4 never sees the return path, so it cannot track TCP state and expires flows by idle timeout.
   8. The proxy records the request asynchronously for logs and metrics. Counters per route and per backend live in memory and are flushed every few seconds; access logs are batched. Per-backend latency and error rates are also what canary analysis compares, so they must be broken down by version.
2. An L4 box dies at peak. The failure the L4 design exists for. One of 8 boxes disappears with ~400 k flows on it, the routers rehash, and every flow must still reach the proxy that holds its TCP state.
   1. An L4 box loses power; the edge routers notice in under a second and drop it from the ECMP group. BFD on the box’s BGP session misses three 100 ms hellos, the route through it is withdrawn, and the routers hash over 7 next hops instead of 8. Routers with resilient hashing move only the dead box’s flows; plain modulo hashing moves most flows between the surviving boxes too.
   2. Packets of established flows start arriving at L4 boxes that have never seen them. These are mid-connection packets, not SYNs, and the receiving box has no connection-table entry for them. A design that picked proxies by local round robin, or from a table each box built its own way, would now send them to a proxy with no TCP state, and the client would get a reset.
   3. The new box computes the same lookup and sends the flow to the same proxy as before. Every box built its table from the same versioned proxy list, and the lookup depends only on the 5-tuple, so l4-05 gets the answer l4-03 would have given. The proxy carries on with the connection without noticing anything, and l4-05 records the flow so later packets hit its table.
   4. Flows whose proxy changed since they started get a second chance. If a proxy was added or drained during the flow, the current table version can point at a proxy with no state for it. That proxy forwards non-SYN packets it has no socket for to the owner under the previous table version, the same idea as the second-chance step in GitHub’s GLB Director. Only flows hit by an L4 move and a proxy-list change at once are at risk, and second chance saves most of them.
   5. The surviving boxes absorb the load, and a replacement joins only after loading the current table. Even at failover peak, 17 M pps over 7 boxes is 2.4 M pps each, under the 4 M planning number, which is why the region runs 8 when 5 would do. A new box fetches the proxy list and builds its table before it announces the VIP, so it never forwards with a stale view.
3. A backend starts failing requests. One instance of a 2,000-instance service starts returning 503 in a millisecond. A host that fails fast looks lightly loaded, so balancing alone would send it more traffic; ejection has to beat that without ever emptying the pool.
   1. An instance starts answering every request with a fast 503. Its in-flight count stays near zero because it fails in a millisecond, so least-request balancing favours it: the black hole effect. Power of two choices limits the damage, since it is picked only when it is one of the two sampled, but every proxy still sends it a steady trickle of requests to fail.
   2. Each proxy ejects it after 5 consecutive failures, edge and internal alike. Passive outlier detection reacts on real traffic within a few hundred milliseconds, without waiting for a probe. Ejection lasts 30 s and doubles each time the same host is ejected again, and no proxy ejects more than 20 % of a cluster on its own. Each proxy decides independently, so a host that fails only for some callers is ejected only by them.
   3. The request that hit it is retried once on another backend, within the retry budget. A 503 returned before the request was processed is safe to retry on a different host. Retries are capped at 10 % of requests per cluster, so a whole-service failure does not double the load on whatever is left.
   4. Central health checkers confirm the failure from three zones and report the transition. 3 failed probes in a row from 2 of 3 checkers, about 6 s. Active checks matter for hosts that get little traffic, where passive detection never sees enough requests, and they decide when an ejected host may come back. Only the transition is reported, not every probe.
   5. The control plane checks the change against its fail-open guard, then pushes it. One host in 2,000 is safe. If an update would mark more than a third of a cluster unhealthy at once, the control plane holds it and pages, because a broken checker or a bad network path is likelier than a third of a fleet failing in the same second. Proxies have their own guard: below 50 % healthy they ignore health and spread over every host, which Envoy calls the panic threshold.
   6. When the instance recovers, probes pass and it returns with slow start. It needs 3 consecutive passes, then its weight ramps from 10 % to 100 % over 60 s. Returning it at full weight hands it a full share of traffic with cold caches at once, which is how a flapping host keeps flapping.
4. A canary deploy with slow start and draining. A team rolls v2 out to 2,000 instances without a single failed request. Endpoint changes take the fast path through the control plane; the traffic split is a route change and takes the slow, staged path.
   1. New v2 instances start and appear in the service registry, not yet ready. Membership comes from the scheduler: the instance exists, with its address and its labels (version v2). The control plane watches the registry, but sends an endpoint to proxies only once the health checkers say it is ready, so a slow-booting process never receives traffic.
   2. Probes pass, and the control plane pushes the new endpoints to the proxies that use them. An incremental update of ~100 B per endpoint goes only to proxies that route to orders, and reaches them in about a second, p99 under 5 s. Each proxy ramps a new endpoint’s weight over 60 s, so JIT warm-up and empty caches do not meet a full share of traffic.
   3. The deploy tool commits a route change that sends 5 % of traffic to v2. The split is a route, owned by the team and reviewed like code. The control plane validates the new version before any proxy sees it: the schema, that the referenced clusters exist and have healthy endpoints, and that no route claims another team’s host.
   4. The control plane rolls the route out in stages: canary proxies, one zone, one region, everywhere. Each stage waits a few minutes and compares 5xx and latency on proxies running version 319 against those still on 318; a regression stops the rollout and re-pushes 318. Proxies ACK a version they loaded or NACK one they rejected, so the control plane knows exactly which version every proxy runs.
   5. Canary analysis compares v2 with v1, and the split steps up to 25, 50 and 100 %. Per-backend metrics from the proxies give v2’s error rate and latency against v1 on the same traffic mix. Each step is another route version, so stepping back is exactly as fast as stepping forward.
   6. The v1 instances are marked draining, and proxies stop sending them new requests. Draining is an endpoint state, so it takes the fast path. Proxies send no new requests to a draining host, send HTTP/2 GOAWAY or Connection: close on its idle pooled connections, and let requests already in flight finish normally.
   7. After the drain window, the scheduler stops the old instances. The window is config propagation p99 plus the longest request: 5 s + 30 s = 35 s. Stopping sooner kills requests on proxies that have not yet heard about the drain; the instance’s SIGTERM handler, which fails readiness and finishes in-flight work, is the second line of defence.
5. A whole region fails at peak. One of four regions goes dark with 1.25 M requests a second and 2.5 M connections on it. Anycast moves new packets within seconds; whether users notice depends on headroom in the other three and on whether TLS resumes.
   1. The global controller’s probes to the region fail from most vantage points. Probes run every 5 s from ~40 outside locations against the region’s own unicast probe address, through L4, L7 and a canary backend. Acting needs a majority of vantage points failing for 3 rounds, about 15 s, so one ISP’s routing problem does not move a region.
   2. The region’s anycast routes disappear from the internet. If the region is dark, its BGP sessions drop and the routes vanish on their own; if it is up but broken, the controller tells its edge routers to withdraw. Either way ISPs converge on the next-nearest region in seconds to a minute, with no DNS change and no tail of resolvers holding an old answer.
   3. Clients reconnect, and their packets now reach routers and L4 boxes in the surviving regions. The 2.5 M open connections are gone, since their TCP state lived on the failed region’s proxies. Clients see a reset or a timeout and reconnect, roughly 2.5 M over 30 s: ~80 k new connections a second on top of normal, ~28 k per surviving region.
   4. Most TLS sessions resume, because ticket keys are shared across regions. A ticket issued in eu-west decrypts in us-east, so a reconnect costs a key exchange, not a full handshake with a signature. With per-region ticket keys every reconnect would be a full handshake: fine with ECDSA at ~0.1 ms each, most of a core per proxy with RSA-2048.
   5. Each surviving region now carries 1.7 M requests a second, which it was sized for. N+1 means every region already had a third of headroom at peak: proxies go from ~31 k to ~42 k requests a second each, under the 50 k planning number. If a region were short, proxies would shed the lowest-priority routes first (batch APIs, prefetch) rather than let latency climb for everything.
   6. Sticky sessions pinned to the failed region are lost; stateless sessions carry on. An affinity cookie naming a backend in eu-west now points at nothing, so the proxy picks a new backend and rewrites the cookie. Users notice only where a service kept session state in that backend’s memory, which is the argument for sessions in a shared store or a signed token. WebSocket clients reconnect with jitter.
   7. When the region recovers, the controller brings it back gradually. Re-announcing everything at once would hit a cold region with a flood. The controller first sends a small share through DNS weights to the region’s unicast address, then announces the anycast prefixes with AS-path prepending, then without it, watching errors at each step. Failover is fast on purpose; failback is slow on purpose.

Deep dives

The theory behind it

• Consistent hashing on a hash ring: an interactive playground. Add and remove servers on a consistent hashing ring with 300 keys, change the virtual nodes, and compare how many keys move with hash mod N.
• Load balancing algorithms compared: an interactive playground. Route the same traffic with round robin, random, least connections and power of two choices, then add a slow server and watch the queues and p99 latency.
• Health checks: an interactive playground. Break one of five servers so it fails fast, and see least connections funnel most traffic to it, shallow health checks miss it, and deep checks or outlier detection take it out.
• DNS failover: an interactive playground. Fail one of two regions and compare DNS failover with long and short TTLs, multiple addresses with client retries and anycast, with and without resolvers that ignore the TTL.
• Load balancing: Layer 4 vs layer 7 load balancers, routing algorithms, health checks, sticky sessions, global load balancing and how to talk about them in a system design interview.
• Global traffic management, DNS and anycast: How users reach the nearest healthy region: DNS resolution and TTLs, GeoDNS and latency-based routing, anycast, global load balancers, health checks and failover, the limits of DNS-based failover, and steering traffic during incidents and migrations.
• Consistent hashing: Why hash mod N breaks when nodes change, how a hash ring with virtual nodes fixes it, replication on the ring, rendezvous hashing and jump hash, fixed slots, and handling hot keys.

Related

• Design a Distributed Cache
• Design a Distributed Key-Value Store
• Design an API Gateway
• All 83 system design interview questions
• Run this question for a friend: the interviewer kit