DNS failover, TTLs and anycast when a region fails: an interactive simulator

When a whole region fails, traffic has to move to another one, and the slowest part is often not noticing the failure but getting clients to stop using the old address. DNS failover changes the answer, but every client and resolver that cached the old one keeps using it until the TTL expires, and some keep it longer. Anycast and global load balancers keep one address and move traffic behind it, so nothing cached goes stale.

The playground fails one of two regions at time zero and follows the next 20 minutes under four approaches. Then let a few resolvers ignore the TTL, as real ones do.

What each approach costs

• DNS failover with a 5 minute TTL is cheap in DNS queries, but after the health checks flip the answer, cached copies keep half the clients on the dead region for up to 5 more minutes: 7.5% of all requests in the 20 minutes fail.
• A 30 s TTL recovers within a minute, but 10 million clients now ask 333K times a second instead of 33K, and DNS lookups add latency to more requests.
• Returning both addresses lets clients that support it try the second address when the first refuses. Nothing fails, but every client that picks the dead region first waits out a connect timeout, 2.5% of requests in this run.
• Anycast or a global load balancer fails over in the time the health checks take, here 30 s, because the client’s address never changes. It is how CDNs and large cloud front doors work, and it needs a network that can announce one address from many places.
• Resolvers that ignore TTLs are the long tail of every DNS approach: some ISPs, corporate resolvers and runtimes such as older JVMs cache answers for much longer. With 5% of clients behind them, even the 30 s TTL leaves 1.7% of requests failing ten minutes in.

The records

; DNS failover: health-checked answer, short TTL
api.example.com.  30  IN A  203.0.113.10   ; us-east, while healthy
api.example.com.  30  IN A  198.51.100.20  ; eu-west, the failover answer

; both addresses: clients try one, then the other on connect failure
api.example.com.  60  IN A  203.0.113.10
api.example.com.  60  IN A  198.51.100.20

; anycast: one address announced over BGP from every region;
; a region that fails withdraws its route or is drained by the global load balancer
api.example.com.  3600  IN A  192.0.2.1

In a system design interview

When a design runs in more than one region, say how users get moved off a failed one: “Users reach us through a global load balancer with an anycast address, which health-checks each region and drains a failed one within tens of seconds. We do not rely on DNS TTLs for failover, since some resolvers ignore them; DNS only picks the nearest edge. Clients retry idempotent requests once on connection failure.” Expect the follow-ups: how health checks avoid flapping, what happens to sessions and in-flight writes in the failed region, and whether the surviving region has the capacity for everyone.

Read the guide to global traffic management or what happens when you type a URL, see health checks and multi-region databases, or watch a strong candidate design Netflix.

Questions people ask