Design a Load Balancer

• Spread traffic across healthy backends: Per connection at L4, per request at L7. A failing backend leaves rotation within seconds, and nothing new is sent to a backend that is draining.
• Terminate TLS for every public hostname: A few hundred hostnames, certificates renewed automatically, private keys never on disk. Re-encrypt to backends with mTLS.
• Route by host, path and headers: api.example.com/v1/orders to the orders service, weighted splits for canaries (95 / 5), header matches for internal testing.
• Survive the loss of any balancer, zone or region: Losing an L4 or L7 box may reset at most that box’s own share of connections. Losing a region moves its users to the others within a minute.
• Drain and slow-start backends for deploys: Stop sending new requests, let in-flight ones finish, then remove. Ramp a new instance up over a minute instead of handing it a full share while it is cold.
• Send users to the nearest healthy region: Anycast for the front door; DNS for deliberate moves such as draining a region or keeping an account in its home region for residency.
• Self-service config for a few hundred services: Teams own their routes. Changes are reviewed, versioned, rolled out in stages and reverted in one step.
• Internal service-to-service balancing: The same routing, health and config model, applied by a proxy on the caller’s host, so an internal call takes no extra hop through a shared tier.
• Boundary features: Per-client rate limits (the design from the Rate Limiter question), request ids, access logs and per-route metrics, all at the L7 tier.
• Out of scope: Volumetric DDoS scrubbing beyond SYN cookies and drop rules, the CDN and static caching, WAF rules, and the service registry itself, which the compute platform owns.

• Throughput (5 M req/s · 10 M conns): Global peak over 4 regions: ~1.25 M requests a second per region, 1.7 M when a region is lost. Capacity grows by adding boxes; nothing central sits on the request path.
• Added latency (L4 µs · L7 p99 < 2 ms): L4 is a table lookup and an encapsulation. L7 parses HTTP and picks a backend. Full TLS handshakes are the expensive part, so the resumption rate matters more than proxy code speed.
• Availability (99.995 %): Everything sits behind the balancer, so its availability caps every service’s. That is about 26 minutes a year for all services combined.
• Connection stability (≤ 1/N reset per box lost): Adding or losing one of N boxes may disturb only that box’s share of connections, never reshuffle all of them. This drives the hardest trade-off in the design: L4 boxes must agree on where every flow goes without sharing any state.
• Failure detection (backend < 10 s · region < 30 s): Passive outlier detection reacts in under a second on real traffic, active probes catch idle backends, and regional failover is bounded by detection, not by DNS TTLs.
• Config propagation (endpoints p99 < 5 s): Endpoint changes from deploys and autoscaling must reach every proxy fast. Route changes are riskier and roll out in stages over ~30 minutes. If the control plane is down, the data plane keeps its last good config.
• Capacity headroom (N+1 regions): Each region is provisioned a third above its normal peak, so losing one region at peak does not overload the other three and cascade into a global outage.
• Key custody (no private keys on disk): TLS keys exist only in memory on proxies, or stay in an HSM-backed key service. A stolen disk, image or backup must expose nothing.

• Requests per region: ~1.25 M/s · 1.7 M/s after a region loss. 5 M/s at peak ÷ 4 regions = 1.25 M/s. Lose one and the other three carry 5 M ÷ 3 ≈ 1.7 M/s each, so every region is sized for 1.7 M: a third above its normal peak. That headroom is the price of N+1.
• Inbound packets at the L4 tier: ~50 M pps · ~17 M per region in a failover. About 10 inbound packets per request: request segments, ACKs for a ~10 KB response, amortised handshakes and teardown. 5 M × 10 = 50 M pps globally, and 50 M ÷ 3 ≈ 17 M pps per region with one region down. Responses never cross L4 (direct server return), which keeps 5 M × 10 KB ≈ 400 Gbps of response traffic off it.
• L4 boxes per region: ~5 needed · run 8. 17 M pps ÷ 4 M pps per box (a conservative planning number for XDP or DPDK forwarding on a 2 × 25 GbE server) ≈ 4.3, so 5. Run 8: two spare for maintenance and failure, and room for a SYN flood. The L4 tier is small and is sized by attack headroom more than by legitimate load.
• L7 proxies per region: ~34 needed · run 40. 1.7 M req/s ÷ 50 k req/s per proxy (TLS mostly resumed, keep-alive, 32 cores at ~60 % CPU) ≈ 34. Run 40 per region, 160 globally. Per box, L4 moves about 8× the requests L7 can parse (4 M pps ≈ 400 k requests), which is why L7 is the tier that scales out and changes often.
• Connections per L7 proxy: ~83 k · ~3.3 GB of buffers. 10 M connections ÷ 4 regions = 2.5 M; with a region down, 3.3 M per region ÷ 40 proxies ≈ 83 k per proxy. At ~40 KB per TLS connection (socket buffers, TLS and HTTP/2 state) that is ~3.3 GB, easy for a 128 GB box. The L4 tier tracks the same flows at ~100 B each: 3.3 M ÷ 8 boxes × 100 B ≈ 40 MB per box.
• TLS handshakes: ~170 k new conns/s · ~50 k full handshakes/s. 10 M connections lasting ~60 s on average = ~170 k new connections a second globally. With 70 % resumed from session tickets, ~50 k full handshakes a second. At ~1 ms of CPU each with RSA-2048 that is ~50 cores; with ECDSA P-256 at ~0.1 ms, ~5. Steady state is cheap: the risk is a failover moving 2.5 M connections at once, where resumption decides whether it is a blip or a CPU wall.
• Health check probes: 8 M/s naive · 75 k/s central. If all 160 edge proxies probed all 50 k backends once a second: 160 × 50 k = 8 M probes/s, 160 a second landing on every backend, before counting the ~5 k per-host proxies. Instead, 3 central checkers per backend (one per zone) every 2 s: 50 k × 3 ÷ 2 = 75 k probes/s, 1.5 per backend per second.
• Config size and push: ~6 MB snapshot · ~30 GB per full push. 50 k endpoints × ~100 B = 5 MB, plus ~1 MB of routes and clusters for 300 services. Sending the whole 6 MB to ~5 k proxies is 30 GB per change, and deploys and autoscaling change about 100 k endpoints a day, tens a second during a big deploy. So pushes must be incremental: one endpoint change is ~100 B, sent only to the proxies that route to that service.

• Users and apps: Browsers, mobile apps and partner servers on the internet. They resolve one hostname and open TLS connections that live about a minute on average. A reset connection shows up to them as an error or a retried request, which is why the design counts how many connections each failure moves.
• Authoritative DNS (anycast · health-aware): Answers queries for the company’s hostnames, itself served from anycast addresses in every region. Normally it returns the global anycast VIP with a 60 s TTL; for a deliberate move (draining a region, residency) it returns per-region addresses instead. It is the steering wheel for planned changes, not the failover mechanism.
• Global traffic controller (region health · BGP policy): Decides which regions take traffic. Probes each region from ~40 outside vantage points, reads each region’s capacity, and tells the region’s edge routers to announce, prepend or withdraw the anycast prefixes, and DNS what to answer. Guardrail: it never withdraws a second region without a human, and fails back more slowly than it fails over.
• Edge routers (BGP · ECMP): Border routers in each region that peer with ISPs and announce the anycast prefixes. Inside the region they spread packets for a VIP over the L4 boxes announcing it, by ECMP on a hash of the 5-tuple. They keep no per-connection state, and when an L4 box disappears they rehash, which can move flows from one L4 box to another.
• L4 balancers (XDP · Maglev hash · DSR): About 8 boxes per region that forward packets and never terminate them. Each picks an L7 proxy from a consistent-hash lookup table built from the same versioned proxy list on every box, so any L4 box sends a given flow to the same proxy. A small connection table keeps flows in place while the proxy list changes; packets are encapsulated to the proxy, and replies bypass L4 entirely (direct server return).
• L7 proxies (Envoy-style · TLS · HTTP): ~40 proxies per region that terminate TCP and TLS, parse HTTP, match host, path and header routes, apply rate limits, and pick a backend per request with power of two choices. They keep pooled HTTP/2 connections with mTLS to backends, retry within a budget, and eject failing backends on their own. All of their behaviour is config pushed by the control plane.
• Key service (HSM · certs · ticket keys): Issues and rotates certificates (public ones through ACME from a public CA, internal ones from our own CA) and delivers private keys to proxies over mTLS, held in memory only. Also distributes the session-ticket keys that every proxy in every region shares, rotated every 12 hours. For edge sites we do not fully control it keeps the key and signs handshakes itself (keyless).
• Rate limit store (Redis · token buckets): Shared counters for per-client and per-route limits at the boundary: the design from the Rate Limiter question. Proxies check local buckets first and the shared store only for limits that must hold across proxies, with a short timeout and fail-open. A rejected request costs a 429 and never reaches a backend.
• Service backends (50 k instances · 300 services): The instances behind every route, on ~5 k hosts. Each exposes a cheap readiness endpoint that says whether it can serve, not whether every dependency is perfect, and handles SIGTERM by failing readiness, finishing in-flight work and closing keep-alive connections. Membership comes from the scheduler and health from the checkers, on purpose.
• Per-host proxies (internal calls · same xDS): One proxy on each of ~5 k hosts handles that host’s outgoing internal calls with the same routes, P2C balancing, outlier detection and mTLS as the edge, programmed by the same control plane. Internal traffic takes no extra hop and no shared tier, so a broken proxy affects only the callers on its own host.
• Control plane (xDS · versioned snapshots): Watches the registry and the health checkers, reads routes from the config store, compiles versioned snapshots and streams incremental updates to ~5 k proxies, and the proxy list to the L4 boxes. Endpoints take a fast path of seconds; route changes are validated and rolled out in stages. It is off the request path: if it dies, every proxy keeps serving its last accepted config.
• Config store (versioned · reviewed): Source of truth for listeners, routes, clusters, weights and limits, one versioned document per service, changed through reviewed commits. Every applied version is kept, so rollback means re-pushing an earlier version. Small: about 1 MB for 300 services.
• Service registry (owned by the scheduler): Which instances exist for each service, with their addresses and labels, maintained by the compute platform (Kubernetes-style endpoints). The balancer consumes it and does not own it. Membership and draining come from here, health comes from the checkers, and the control plane combines the two.
• Health checkers (3 per backend · every 2 s): A fleet in each zone that probes every backend’s readiness endpoint every 2 s, three checkers per backend, and reports a transition after 3 consecutive results agreed by 2 of 3 checkers. It replaces every proxy probing every backend, which would be 8 M probes a second. Only transitions are reported, so a healthy steady state costs the control plane nothing.
• Telemetry pipeline (Kafka → metrics, logs): Access logs and per-route, per-backend metrics from every proxy, batched and shipped asynchronously, never on the request path. Feeds dashboards, canary analysis for weighted rollouts and the controller’s view of each region’s load. Metrics are complete; access logs are sampled for successes and kept for every error.

1. An HTTPS request from a user to a backend: The main path through every layer. What matters is where each decision is made, which layers hold state, and that the response skips the L4 tier on its way back.
   1. The client resolves the hostname and gets the global anycast address.
   2. The SYN reaches the nearest region over BGP, and an edge router hashes it to one of the L4 boxes.
   3. The L4 box looks the flow up, picks an L7 proxy from its lookup table, and forwards the packet encapsulated.
   4. The proxy decapsulates, completes TCP and TLS, and resumes the client’s session from its ticket.
   5. The proxy matches a route on host and path, then checks the client’s rate limit.
   6. The proxy picks a backend with power of two choices and sends the request on a pooled connection.
   7. The response goes from the proxy straight out through the edge router to the client, skipping the L4 tier.
   8. The proxy records the request asynchronously for logs and metrics.
2. An L4 box dies at peak: The failure the L4 design exists for. One of 8 boxes disappears with ~400 k flows on it, the routers rehash, and every flow must still reach the proxy that holds its TCP state.
   1. An L4 box loses power; the edge routers notice in under a second and drop it from the ECMP group.
   2. Packets of established flows start arriving at L4 boxes that have never seen them.
   3. The new box computes the same lookup and sends the flow to the same proxy as before.
   4. Flows whose proxy changed since they started get a second chance.
   5. The surviving boxes absorb the load, and a replacement joins only after loading the current table.
3. A backend starts failing requests: One instance of a 2,000-instance service starts returning 503 in a millisecond. A host that fails fast looks lightly loaded, so balancing alone would send it more traffic; ejection has to beat that without ever emptying the pool.
   1. An instance starts answering every request with a fast 503.
   2. Each proxy ejects it after 5 consecutive failures, edge and internal alike.
   3. The request that hit it is retried once on another backend, within the retry budget.
   4. Central health checkers confirm the failure from three zones and report the transition.
   5. The control plane checks the change against its fail-open guard, then pushes it.
   6. When the instance recovers, probes pass and it returns with slow start.
4. A canary deploy with slow start and draining: A team rolls v2 out to 2,000 instances without a single failed request. Endpoint changes take the fast path through the control plane; the traffic split is a route change and takes the slow, staged path.
   1. New v2 instances start and appear in the service registry, not yet ready.
   2. Probes pass, and the control plane pushes the new endpoints to the proxies that use them.
   3. The deploy tool commits a route change that sends 5 % of traffic to v2.
   4. The control plane rolls the route out in stages: canary proxies, one zone, one region, everywhere.
   5. Canary analysis compares v2 with v1, and the split steps up to 25, 50 and 100 %.
   6. The v1 instances are marked draining, and proxies stop sending them new requests.
   7. After the drain window, the scheduler stops the old instances.
5. A whole region fails at peak: One of four regions goes dark with 1.25 M requests a second and 2.5 M connections on it. Anycast moves new packets within seconds; whether users notice depends on headroom in the other three and on whether TLS resumes.
   1. The global controller’s probes to the region fail from most vantage points.
   2. The region’s anycast routes disappear from the internet.
   3. Clients reconnect, and their packets now reach routers and L4 boxes in the surviving regions.
   4. Most TLS sessions resume, because ticket keys are shared across regions.
   5. Each surviving region now carries 1.7 M requests a second, which it was sized for.
   6. Sticky sessions pinned to the failed region are lost; stateless sessions carry on.
   7. When the region recovers, the controller brings it back gradually.

Ask: Eight L4 boxes with no shared state, and routers that reshuffle flows whenever a box changes. How does every packet of a connection still reach the same proxy?

Good answers name: Stateless L4 fleet: Maglev-style lookup table, local connection table, DSR, ECMP straight to the L7 proxies, Active-standby pair with a floating IP (VRRP), L4 boxes that replicate their connection tables to each other.

Our pick: ECMP from the edge routers to 8 L4 boxes per region, each forwarding with XDP or DPDK. For every VIP each box builds a lookup table of 65,537 entries with Maglev hashing from the same versioned list of L7 proxies, pushed by the control plane: each of the 40 proxies takes about 1/40 of the entries, and removing one hands its entries to the others while leaving everyone else’s in place. A packet first checks a local connection table (5-tuple → proxy, 5 minute idle timeout); on a miss the box indexes the lookup table by hash(5-tuple), records the result, and encapsulates the packet to the proxy, which replies directly to the client. For the rare flow hit by an L4 move and a proxy change at once, a proxy that receives a non-SYN packet it has no socket for forwards it to the owner under the previous table version. This is the design in Google’s Maglev paper (NSDI 2016); Facebook’s Katran is an open-source XDP balancer built on the same ideas, and GitHub’s GLB Director uses a similar second-chance step.

1. Why not let the routers do consistent hashing and skip the L4 tier?
   Some routers offer resilient hashing, which keeps flows on the surviving next hops when one is removed, and for a small, stable fleet that is enough. It falls short here on three counts: ECMP groups hold tens of next hops while we have 40 proxies per region that we deploy weekly, adding a next hop still remaps flows, and the router gives us no draining and no second chance. The L4 tier is where we put that logic in software we deploy ourselves.
2. Why not stop at L4 and let the backends terminate TLS?
   Because most of what the system must do needs the request. Host and path routing, weighted canaries, retries, per-key rate limits and per-request metrics all need parsed HTTP. HTTP/2 makes L4-only balancing actively bad: one connection carries thousands of requests, so balancing connections can leave one backend with ten times the load of another. And TLS keys would live on 50 k backends instead of 160 proxies. L4-only is still right for non-HTTP protocols such as databases or game traffic, which the same L4 tier can serve.
3. A proxy needs to be drained for a deploy. Walk me through it.
   The control plane publishes a new proxy list without it; its table entries go to other proxies, but existing flows keep reaching it through the connection tables, and second chance covers flows that move between L4 boxes. The proxy sends GOAWAY on HTTP/2 and Connection: close on HTTP/1.1, so clients reconnect and land elsewhere. After the drain window, a few minutes for most traffic and longer for WebSockets, it stops. New connections stop reaching it within seconds, once every L4 box has the new list.
4. HTTP/3 runs over UDP. Does any of this change?
   Hashing per 5-tuple still works, but QUIC lets a client change its address mid-connection, so a phone moving from Wi-Fi to cellular would hash to a different proxy. The fix is to route on the QUIC connection ID instead: the proxy encodes its own identity in the connection ID it chooses, and the L4 tier decodes it, which is what the IETF QUIC-LB work describes. That needs QUIC awareness at L4 but keeps the tier stateless.
5. How would you know the L4 tier is breaking connections?
   Count it at the proxies, the only place that can tell: non-SYN packets for unknown flows, split into those rescued by second chance and those answered with a reset. Graph both against L4 and proxy-list changes. Replacing an L4 box should show a bump in rescued packets and almost no resets; resets that track proxy deploys mean the drain or the table versioning is wrong.

Ask: Each of 160 proxies picks a backend for every request. Why not just send each request to the backend with the fewest connections?

Good answers name: Power of two choices on requests in flight, weighted, with slow start, Weighted round robin, Least connections over the whole cluster, Consistent hashing on a key (ring hash or Maglev at L7).

Our pick: The default for every cluster is least outstanding requests with power of two choices, which is what Envoy’s least-request balancer does. Each proxy tracks requests in flight per endpoint; for each request it samples two healthy endpoints at random, divides each count by the endpoint’s weight, and takes the lower. Weights come from config (instance size, canary share), and a new endpoint enters with slow start, its effective weight ramping from 10 % to 100 % over 60 s. Outlier detection runs beside it, so a backend that fails in a millisecond, and therefore looks idle, is ejected instead of rewarded. Routes that need locality switch to consistent hashing on a header or cookie with bounded load, route by route, never globally.

1. Why two choices, and not three or ten?
   Going from one random choice to two gives most of the benefit: the busiest backend’s excess drops from about log n / log log n to log log n. Each extra sample adds less and makes proxies more likely to agree on the same best backend, which brings a little of the herding back. Two is the standard default, and I would only change it with measurements.
2. A backend fails every request in 1 ms. What does your balancer do?
   On its own it would favour that backend, because a host that fails instantly has fewer requests in flight: the black hole effect. Power of two choices limits how often it is picked, and outlier detection ejects it after 5 consecutive 5xx, well under a second at our request rates. Counting errors as load, or scoring by the latency of successful requests only, is the belt-and-braces version.
3. Why do you need slow start if power of two choices already avoids herding?
   P2C stops proxies stampeding a new backend in the same instant, but once it is in the pool it still gets a full share, and a fresh JVM with an empty cache cannot serve a full share at normal latency. A 60 s ramp lets it warm up on a fraction of the traffic. Without it, a deploy of a few hundred instances shows a latency spike on every wave.
4. Every proxy talks to all 2,000 backends of a big service. Is that a problem?
   Yes, at this size: 160 edge proxies plus thousands of host proxies each holding connections to every backend means tens of thousands of mostly idle connections per backend, and every proxy tracking every host. Deterministic subsetting fixes it: each proxy uses a stable subset of, say, 50 backends, chosen so every backend sits in about the same number of subsets. The cost is coarser balancing per proxy, which P2C tolerates well.
5. How would you check that the balancing is actually even?
   Plot requests in flight and CPU per backend within a cluster, and track the ratio of the busiest backend to the median; with P2C it should stay close to 1. Compare per-backend p99 latency too: one host consistently slower than its peers points at a hardware or noisy-neighbour problem that the balancer is quietly hiding. Look at edge and host proxies separately, since they see different traffic.

Ask: Your health checker has a bug and reports every backend of a service as down. What happens?

Good answers name: Central active probes, passive outlier detection, fail-open guards, Every proxy actively probes every backend, Passive outlier detection only, Deep health checks that test every dependency.

Our pick: Health checkers in each zone probe every backend’s readiness endpoint every 2 s, three checkers per backend, 75 k probes a second in total; a host turns unhealthy after 3 consecutive failures seen by 2 of 3 checkers, about 6 s, and healthy again after 3 passes. Readiness means this process can serve, not that every dependency is perfect. Inside every proxy, outlier detection ejects a host after 5 consecutive 5xx or connect failures for 30 s, doubling on repeats, never more than 20 % of a cluster per proxy. Two fail-open guards: the control plane holds any update that would mark more than a third of a cluster unhealthy at once and pages a human, and each proxy ignores health entirely when fewer than 50 % of a cluster are healthy (Envoy’s panic threshold, whose default is also 50 %). Membership from the scheduler and health from the checkers stay separate signals, so a registry hiccup cannot look like a health failure.

1. The checkers in one zone lose their network to the others. What happens?
   Each backend is probed from three zones and a transition needs 2 of 3 checkers to agree, so one zone’s checkers seeing failures change nothing on their own. If the partition really cuts a zone’s backends off from the other zones, it probably cuts them off from proxies there too, and passive detection in those proxies reacts within a second. The two signals together handle partial partitions better than either alone.
2. Panic mode sends traffic to dead backends. Isn’t that worse?
   When 60 % of a cluster is really dead, spreading over everything fails 60 % of requests, while obeying health would pile all the traffic onto the remaining 40 % and likely melt them too. When the health data is wrong, which is the more common reason most of a cluster looks down, panic mode means no outage at all. Retries on a different host recover part of the real-failure case.
3. The health checking service itself goes down. What do proxies do?
   Nothing changes for traffic: proxies keep the last health state they were sent, and outlier detection keeps working on real requests. New endpoints cannot become ready until the checkers are back, so deploys and scale-ups pause, and the control plane alerts once health data is more than a minute stale. That is the right failure: the system freezes in a known-good state instead of guessing.
4. What should the readiness endpoint actually check?
   That this process can serve a typical request now: it has finished booting, its thread pools are not saturated, and it can get a connection from its own datastore pool. Not whether every downstream dependency is healthy, since one shared dependency failing would fail every host together. It must also be cheap, because it is called 1.5 times a second per host, forever.
5. How do you tune outlier detection so it does not eject healthy hosts during a real incident?
   Rely on the max-ejection cap: if every host returns errors, the problem is a dependency, not the hosts, and the cap stops proxies emptying the pool chasing it. Watch ejections per cluster, because many hosts ejected at once is an incident signal, not a balancing decision. For a new service, start outlier detection in log-only mode and compare its would-eject decisions with real failures before enforcing it.

Ask: Five thousand proxies need routes and endpoint lists that change every second, and one bad config can take every service down. How does config reach them safely?

Good answers name: xDS-style streaming: incremental, versioned, ACK/NACK, staged rollout for routes, Proxies poll a versioned snapshot from object storage, Every proxy watches the service registry directly, Config baked into the proxy image and deployed.

Our pick: A stateless control plane with replicas in every region; each proxy holds one gRPC stream to a nearby replica. It watches the registry and the health checkers, reads routes from the config store, and compiles per-proxy-type snapshots of listeners, routes, clusters, endpoints and secrets, the resource types of Envoy’s xDS APIs. Proxies subscribe only to the clusters their routes reference and receive incremental updates, each with a version; a proxy ACKs a version it loaded and NACKs one it rejected, so the control plane always knows what every proxy runs. Endpoints take the fast path, under 5 s at p99. Routes take the careful path: validated (schema, referenced clusters exist and have healthy endpoints, no host claimed by two teams), loaded by canary proxies serving synthetic traffic, then rolled out to 2 % of proxies, one zone, one region and everywhere over ~30 minutes, stopping and reverting on a rise in 5xx or latency. Proxies write the last accepted config to local disk and keep serving it if the control plane is gone, including across restarts.

1. A control plane bug computes an empty endpoint list for a big service. What stops it?
   Two guards. The control plane refuses to push any change that removes more than a third of a cluster’s endpoints at once, holds it and pages; scale-downs that large are rare enough to need a human. And the proxies apply the panic threshold, so even an empty or mostly unhealthy list that got through would leave them spreading over the hosts they had. The worst case becomes stale config, not no config.
2. The whole control plane is down for an hour. What breaks?
   Traffic keeps flowing everywhere on the last good config, and outlier detection keeps routing around failing hosts. What stops is change: new instances never receive traffic, so deploys and autoscaling stall, and draining instances stay in proxies’ lists, so the scheduler must not kill them until the control plane confirms the drain. An hour is survivable; a day becomes a capacity problem, which is why replicas run in every region.
3. A team pushes a route that is valid but wrong, sending /v1/orders to the payments cluster. How fast can you undo it?
   The staged rollout catches most of these at the 2 % stage, because 4xx and 5xx jump on those proxies, and reverts automatically. If one slips through, rollback is re-pushing the previous version on the fast path: seconds, not 30 minutes. Undo is always fast; only forward changes are slow.
4. How do 5 k streams stay balanced across control plane replicas?
   Proxies reach the replicas through ordinary load balancing, and each replica caps its streams and asks the excess to reconnect elsewhere. A replica deploy closes its streams gradually, a few percent a minute, because every reconnect asks for the full state and 5 k at once is a thundering herd. Proxies reconnect with jitter for the same reason.
5. How do you know config is propagating properly?
   Per version: the fraction of proxies that ACKed it, how long that took, and every NACK with its error. Alert when any proxy NACKs a version, when any proxy is more than a few versions behind, and when endpoint propagation p99 passes 5 s. The same numbers answer the team asking why their change is not live yet.

Ask: You terminate TLS on 160 proxies in four regions. Where do the private keys live, and what happens to TLS during a failover?

Good answers name: Keys pushed to proxies in memory, ECDSA first, shared rotating ticket keys, Keyless signing: keys stay in an HSM-backed key service, Per-proxy session caches only, Pass TLS through to the backends.

Our pick: Certificates are issued automatically, public ones through ACME from a public CA and internal ones from our own CA, and the key service delivers them to proxies over mTLS, held in memory only and rotated without restarts. Serve ECDSA P-256 certificates to clients that support them and RSA-2048 to the rest, so the common full handshake costs ~0.1 ms of CPU. Resumption uses TLS 1.3 session tickets encrypted with keys shared by every proxy in every region, rotated every 12 hours with the previous key kept for decryption, so a ticket issued anywhere resumes anywhere, including during a regional failover. 0-RTT data is accepted only for GET, because it can be replayed. For edge sites outside our own data centres, the key service keeps the private key and signs handshakes remotely, the model Cloudflare published as Keyless SSL.

1. Shared ticket keys sound dangerous. Why accept that?
   A ticket key can decrypt the resumption secrets of every session it protected, so stealing it weakens forward secrecy for sessions in its window. We bound the window by rotating every 12 hours and keeping keys only in memory, delivered over mTLS and never logged. The alternative, per-proxy caches, gives a resumption rate of a few percent behind 5-tuple hashing, which costs every user a round trip and turns failovers into handshake storms. It is a deliberate trade with a bounded window.
2. A region fails and 2.5 M clients reconnect within 30 seconds. Does TLS hold up?
   That is ~80 k extra connections a second over three regions, about 700 per proxy per second. With shared tickets most of them resume, costing a key exchange and no signature. Even if none resumed, ECDSA full handshakes at ~0.1 ms each are under a tenth of a core per proxy; with RSA-2048 only, it would be most of a core per proxy, survivable but the reason ECDSA is the default.
3. How do you rotate a certificate on 160 proxies without dropping connections?
   The key service pushes the new certificate and key alongside the old ones; proxies use the new pair for new handshakes at once, and existing connections are unaffected because they already have their session keys. Renew at two thirds of the certificate’s lifetime, so a failed renewal has weeks to be fixed, and alert on any certificate within 14 days of expiry on any proxy. Expired certificates are one of the most common self-inflicted edge outages.
4. Do you re-encrypt between the proxy and the backend?
   Yes, with mTLS on long-lived pooled connections, so the cost is one handshake per connection, not per request. A compromised network segment inside the data centre then sees nothing, and backends can authorise callers by the proxy’s identity. Internal certificates are short-lived, a day or less, issued by our CA and delivered the same way as edge certificates.

Ask: How do users reach the nearest healthy region, and what happens when a whole region fails at peak?

Good answers name: Anycast front door, DNS for deliberate moves, N+1 capacity, Latency-based DNS with health checks and 30 s TTLs, Client-side steering from a list of regional endpoints, Active-passive with a standby region.

Our pick: Every region announces the same anycast prefixes, so BGP takes users to the nearest region and a dead region’s routes vanish within seconds. A global traffic controller probes each region from ~40 outside vantage points and reads its load; it withdraws or prepends a region’s announcements when the region is unhealthy or overloaded, never withdraws a second region without a human, and never fails back faster than it failed over. DNS keeps the jobs anycast cannot do: weighted moves through per-region addresses for draining or canarying a region, and sending accounts to a home region for residency. Each region is provisioned for 1.7 M requests a second, a third above its normal peak, and proxies shed the lowest-priority routes first if that is not enough. Sticky sessions are a per-route option and are lost when their region fails, so services keep session state in a shared store or a signed token.

1. Anycast routes by BGP, not latency. Isn’t that sometimes badly wrong?
   Yes: BGP prefers the shortest AS path, which can send a user to a farther region when a peering is odd. We measure it with client-side timing from our apps and real-user monitoring, and fix the worst cases with BGP communities or prepending towards specific ISPs. With four regions on well-connected networks the share sent to the wrong region is small, and the failover speed is worth it.
2. BGP flaps and a user’s packets move to another region mid-connection. What happens?
   The other region’s L4 boxes send the packets to a proxy that has no state for the connection, which resets it; the client reconnects and resumes TLS from its ticket, costing a round trip and a retried request. WebSocket clients reconnect with jitter. Flaps between distant regions are rare, and when one ISP makes them frequent, that is a peering problem to fix, not a balancer one.
3. Why not run each region at 90 % and save the headroom?
   Because losing a region at peak would then put 120 % of capacity on each survivor. They queue, time out and fail health checks, and the controller is tempted to move traffic again, which is how one regional outage becomes four. A third of headroom is the cost of surviving any single region at peak; off peak, the same headroom absorbs growth and lets us take a region down for maintenance.
4. A user’s data may live only in eu-west for residency, and eu-west fails. Now what?
   Residency wins over availability for that data. Their requests can reach another region’s proxies, but the backends there cannot serve that data, so they get a clear error, or a read-only view if a compliant replica exists elsewhere in the EU. The controller must not move them silently. That is a product and legal decision to make before the outage; the routing layer only has to support “this account must reach that region”.
5. How do you test regional failover without waiting for an outage?
   Drain a region on purpose, regularly and at peak: move its traffic away with the controller over minutes and watch the survivors’ latency, errors and CPU against the plan. Then do it abruptly in a game day with a real withdrawal. Failover that is not exercised every quarter does not work when it is needed, and the drill finds the services that quietly lost their headroom.