Design a CDN

• Route every user to a nearby PoP: One of 300 points of presence, usually within 20 to 30 ms. Anycast by default: every PoP announces the same addresses and the internet’s routing picks the nearest. A PoP that fails or fills up stops receiving traffic within a minute.
• Serve cacheable content from cache: Images, scripts, video segments and API responses with cache headers, honouring the origin’s Cache-Control, s-maxage and Vary, with per-customer rules that override them (TTLs, which query parameters matter).
• Fetch misses with minimal origin load: A popular object should be fetched from a customer’s origin about once per expiry, not once per PoP or per server: a cache hierarchy plus request coalescing. Uncacheable requests are proxied on pooled connections.
• Purge globally within seconds: By URL, by tag (surrogate key) or everything for a domain. A hard purge deletes; a soft purge marks content stale so it is revalidated. The API says when every PoP has applied it.
• HTTPS on customers’ own domains: Terminate TLS at the edge for 30 M domains: certificates issued and renewed automatically or uploaded by the customer, chosen per handshake by SNI. TLS 1.3, HTTP/2 and HTTP/3.
• Self-serve configuration: Origins, cache rules, redirects, WAF and rate-limit settings through an API and a dashboard. A customer’s own change is live worldwide in seconds; changes that affect every customer roll out in stages.
• Absorb traffic spikes and attacks: Flash crowds and DDoS at layers 3, 4 and 7, from multi-terabit floods to HTTP floods. An attack on one customer must not slow down the others sharing the same servers.
• Keep serving when the origin fails: Serve the last good copy (stale-if-error) while the origin is down, fail over to a backup origin where one is configured, and do not knock the origin over again as it recovers.
• Out of scope: Edge compute (Workers-style functions), image and video transformation, the content of WAF rules, analytics and billing pipelines, hosting customers’ DNS zones, and the design of the private backbone between PoPs.

• Scale (50 M req/s · 40 Tbit/s at peak): Across 300 PoPs and 30 M customer domains, on 100 Tbit/s of capacity. The headroom is deliberate: it is what absorbs a lost PoP or a multi-terabit attack on top of the daily peak.
• Latency of a hit (RTT to PoP p50 < 25 ms · < 5 ms inside): Being close is the product. TLS 1.3 adds one round trip (none on resumption), so everything inside the PoP on a hit, from the config lookup to reading the object, must cost a few milliseconds at most.
• Origin offload (≥ 95 % of cacheable requests): This drives the hardest trade-off in the design. 300 PoPs put users close but split every object’s demand 300 ways, so PoPs that each miss on their own multiply origin load. Hashing inside the PoP, an upper tier and coalescing exist to win back the hit ratio that proximity costs.
• Purge propagation (p50 < 500 ms · p99 < 5 s): A stale price or a legal takedown that keeps serving is a correctness bug for the customer. A purge must never be lost: every PoP acknowledges it, and a PoP that cannot catch up stops serving.
• Availability (99.99 % per domain · no global outage): A PoP, server or disk failure is routine and must be invisible beyond a reconnect. A network-wide outage must be structurally hard, which mostly means controlling how changes are deployed.
• Change safety (blast radius: one customer or a few PoPs): The largest CDN outages on record came from a change pushed everywhere at once, not from hardware. A customer’s own change may be fast; anything that touches every customer is staged with health gates and a global stop.
• Tenant isolation (one domain cannot degrade another): 30 M customers share every server, cache and address range. Per-domain limits on requests, origin fetches and CPU, and cache keys scoped by zone, so one customer’s content can never be served for another’s.
• Security of keys and content (keys encrypted at rest · nothing private cached): Customers’ private keys sit on thousands of servers in other people’s buildings, so they are encrypted at rest and decrypted only inside the TLS process. Responses marked private, or carrying Set-Cookie, never enter a shared cache.

• Requests per PoP at peak: ~170 k/s average. 50 M req/s ÷ 300 PoPs ≈ 167 k/s. PoPs are far from equal: the largest 30, in big metros and at exchange points, carry about half the traffic at ~830 k/s each, and the smallest serve a few thousand. Capacity is planned per PoP, for its own peak plus a failed neighbour’s.
• Peak egress: ~40 Tbit/s. 50 M req/s × ~100 KB average response = 5 TB/s = 40 Tbit/s. Requests are mostly small (scripts, API responses, thumbnails); bytes are mostly large (video segments, images). Against 100 Tbit/s of capacity that is 2.5× headroom, which is what absorbs a lost PoP or a multi-terabit attack on top of peak.
• Cache servers: ~5 000 (~17 per PoP). For small objects TLS and HTTP processing, not the network card, is the limit: about 20 k req/s per server at full CPU. 50 M ÷ 20 k = 2 500 servers at 100 %; run at 50 % so a PoP can take a failed neighbour’s load or an attack: ~5 000 servers, each doing ~10 k req/s and ~8 Gbit/s at peak. The average PoP has ~17; the largest have 100 or more.
• Distinct content in one PoP: ~400 TB, not ~20 TB. A 20-server PoP with 20 TB of SSD per server. If every server caches whatever its own connections ask for, all of them hold roughly the same popular set and the PoP stores about one server’s worth, ~20 TB, while a long-tail object is fetched upstream once per server that sees it. Hashing each cache key to one owner server makes the disks add up: 20 × 20 TB = 400 TB. That 20× is the case for hashing inside the PoP.
• Cacheable requests reaching origins: ~1.1 M/s (97.5 % offload). 90 % of 50 M req/s are for cacheable objects: 45 M/s. Edges hit 90 %, so 4.5 M/s go to upper tiers, which hit 75 % of those: ~1.1 M/s reach origins. Without the tier origins would see 4.5 M/s, 4× more, and the extra misses are exactly the long tail an origin is worst at serving. The other 5 M/s (uncacheable) pass through the tier on pooled connections.
• Purge fan-out: ~600 k PoP deliveries/s. At a peak of 2 k purge requests a second (publish hooks from customers’ CMSs are most of them), each must reach all 300 PoPs: 2 k × 300 = 600 k deliveries a second. Batching every 20 ms per PoP turns that into 300 × 50 = 15 k messages a second on the relays, and inside a PoP a tag purge touches only the objects its index lists. Purge is a fan-out problem, not a storage one.
• Configuration on every server: ~60 GB. 30 M domains × ~2 KB of compiled rules (origins, cache rules, WAF and rate-limit settings) = 60 GB, kept on every server’s SSD in a memory-mapped store so a lookup is a local read. Changes run at ~1 k/s at peak. Small enough to replicate everywhere, which is what removes any remote call from the request path.
• Certificate renewals: ~250 k/day. About 15 M certificates cover the 30 M domains (wildcards and multi-name certificates share), ~75 GB with chains. On 90-day certificates renewed at day 60: 15 M ÷ 60 ≈ 250 k a day, ~3/s, spread with jitter to stay inside CA rate limits. The CA/Browser Forum has voted to cut maximum lifetimes to 47 days by 2029, which roughly doubles this, so issuance must be fully automatic.

• End users (browsers · apps · bots): Browsers and apps loading 30 M customers’ websites, plus crawlers and attackers that look the same at first. They never know a CDN is involved: they resolve the customer’s hostname and connect to whatever address comes back. Their network’s routing, not their geography, decides which PoP they reach.
• Steering DNS (anycast · latency maps): Authoritative DNS for the names customers CNAME to us. For most networks it returns the shared anycast address and lets BGP pick the PoP; for networks the mapper has flagged, it returns a regional address group chosen from measured latency, health and load. It is itself served over anycast, so it is as hard to take down as the edge.
• L4 load balancer (ECMP · XDP · Maglev hash): Runs on every server in the PoP: routers spread packets with ECMP, and an XDP program hashes the 5-tuple to the server that owns the connection, with a connection table so a routing change does not break flows in progress. It is also the first DDoS filter, dropping fingerprinted attack packets before the kernel spends anything on them. Replies go straight back to the user (direct server return).
• Edge proxy (TLS · HTTP/3 · rules): Terminates TLS and HTTP on whichever server received the connection, reads the domain’s config locally, applies WAF and rate limits, computes the cache key and fetches the object from the PoP server that owns that key. Any server can serve any customer, so the PoP is one pool of CPU. Per-zone limits keep one customer’s flood from using another’s share.
• PoP cache (RAM + SSD · owner by hash): Each object lives on one owner server per PoP, chosen by rendezvous hashing of the cache key, so the PoP’s disks add up instead of all holding the same objects; only hot keys are copied to every server. The hottest objects sit in RAM and warm ones on SSD, written only on a second request so one-hit wonders do not churn the disk. The owner coalesces concurrent misses into one upstream fetch and keeps a tag index for purges.
• Upper tier (shield) (per-origin · coalescing): About 15 large PoPs whose caches sit between the edges and customer origins. Each origin is assigned the upper-tier PoP nearest to it, plus a backup, so misses from all 300 edges converge on one cache that coalesces them into a single origin fetch. It keeps warm connections to origins, caps concurrency per origin and serves stale copies when an origin fails.
• Customer origin (their servers · buckets): The customer’s own web servers, API or object-storage bucket: the source of truth for content and for its Cache-Control and Surrogate-Key headers. Usually sized for a small fraction of the traffic the CDN serves. It should accept requests only from our address ranges or authenticated pulls, or attackers will simply go around the edge.
• Certificate store (per PoP · keys encrypted): Every customer certificate and its private key, about 15 M certificates and 75 GB, replicated to each PoP with keys encrypted at rest. A server loads a certificate on the first handshake that names it (SNI) and keeps hot ones in memory, so a cold name costs a local read, not a call across an ocean. OCSP staples travel with the certificate where the CA still offers OCSP.
• Edge config store (versioned KV · every server): A memory-mapped local copy of every domain’s compiled configuration, about 60 GB, on every server, applied from a replicated log in sequence order. Request-time lookups never leave the machine. Each server keeps the previous version of every entry and reports the version it runs, which makes instant rollback and “is it live everywhere?” possible.
• Purge bus (relays · sequence · acks): A durable, sequenced purge log fanned out by about ten regional relays over persistent streams to an agent in every PoP, edges and upper tiers alike, batched every 20 ms. PoPs acknowledge the highest sequence they have applied, so the control plane knows when a purge is complete and a reconnecting PoP replays exactly what it missed.
• Traffic mapper (RTT · health · load): Aggregates sampled TCP round-trip times per client network and PoP, plus each PoP’s health and load, into maps of which PoP serves each network best. Where anycast routes a network badly or a PoP runs hot, it changes DNS answers for steerable names and drives BGP changes such as withdrawing one prefix group at one PoP. It is how routing problems become visible at all.
• Config distributor (replicated log · two lanes): Compiles each change, tests it against the real proxy build in a sandbox, appends it to a replicated log and streams it to every server’s config store and every PoP’s certificate store. A customer’s change takes a fast lane, live worldwide in seconds and reverted automatically if that zone’s errors rise; changes every customer runs take a staged lane (canary, 1 %, 10 %, 100 %) with health gates between stages.
• Control plane API (config DB · versioned): The customer-facing API and dashboard for domains, origins, rules, certificates and purges. Backed by a strongly consistent database in which every change to a zone gets the next version number, so the state of every server can be named exactly. It is never on the request path: if it is down, the edge keeps serving with the config it has.
• Certificate manager (ACME · renewals · OCSP): Issues certificates for new domains through ACME with at least two CAs, proving control with HTTP-01 tokens served by our own edge or DNS-01 records, and renews every certificate at two thirds of its life, about 250 k a day. Fetches OCSP responses for CAs that still provide them so the edge can staple them, and alerts on any certificate within 14 days of expiry.
• Certificate authorities (ACME · two or more): Public CAs that sign certificates after validating control of the domain. They have their own rate limits and outages, so issuance is spread across more than one and retried with backoff. Lifetimes are shrinking (47-day maximums by 2029), which makes renewal volume a design input.

1. Load an image: hit or miss: The main read path. Anycast picks the PoP, any server terminates TLS, one owner server per object answers from RAM or SSD, and a miss climbs to the origin’s shield before it reaches the origin. Every hop is chosen by a different hash, and each one exists to keep the hit ratio up.
   1. The browser resolves the customer’s hostname and gets a shared anycast address.
   2. The connection lands at the nearest PoP and is spread across its servers.
   3. That server terminates TLS with the certificate for the requested name.
   4. The proxy reads the domain’s configuration and computes the cache key.
   5. The proxy hashes the key to the server in the PoP that owns it and asks that server.
   6. On a miss, the owner holds every concurrent request for the key on one fetch from the origin’s shield.
   7. The shield fetches from the origin over a warm connection, and both tiers store the response by its headers.
   8. The owner streams the bytes to every waiting request and decides whether the object earns a place on disk.
2. Add a domain and a cache rule: The control-plane write path. A customer adds a domain, gets a certificate without doing anything, and a rule change reaches every server in seconds. The same machinery that makes changes fast is what could take every customer down at once, so validation and per-zone rollback are part of the path, not an afterthought.
   1. The customer adds shop.com and a cache rule through the API.
   2. Once the customer’s DNS points at us, the certificate manager orders a certificate.
   3. The certificate is stored with the zone and replicated to every PoP’s certificate store.
   4. The distributor compiles the new config version and runs it through the real proxy in a sandbox.
   5. The version is appended to the replicated log and every server’s local store applies it.
   6. The first visitor’s handshake loads the new certificate on demand.
   7. Requests use the new rule, and the zone’s error rate decides whether it stays.
3. A 3 Tbit/s flood on one customer: The scale-breaking case for a CDN is an attack, not organic load. Anycast spreads it across the PoPs nearest the bots, every server drops junk at the cheapest layer it can, and per-zone limits keep the other customers on the same servers unaffected.
   1. A botnet floods the customer’s address with reflected UDP and SYN packets.
   2. Each server samples its packets, fingerprints the attack and installs a drop rule in its packet filter.
   3. HTTP floods that complete TLS reach the proxy, which applies the zone’s rate limits and challenges.
   4. Requests for cacheable URLs are answered from cache, and cache-busting variants are cut off at the key.
   5. The mapper sees one PoP running hot and steers what it can away from it.
   6. Other customers sharing the attacked address are moved, and everyone else keeps normal latency.
4. The origin goes down: keep serving: The failure path. An origin starts timing out while its content is expiring. With stale-if-error, coalesced revalidation and a cap on origin concurrency, users see slightly old pages instead of errors, and the origin is not knocked over again as it recovers.
   1. A cached API response expires and its owner asks the shield to revalidate it.
   2. The shield merges every edge’s revalidation into one origin request, which times out.
   3. The shield serves its stale copy downstream instead of the error.
   4. Users keep getting the last good response from their own PoP.
   5. The shield marks the origin unhealthy, caps its concurrency and fails over if a backup exists.
   6. When the origin recovers, revalidations succeed and fresh copies replace the stale ones.
5. Purge a tag worldwide in a second: The asynchronous path customers judge us on: one API call must stop 300 PoPs serving every object tagged product-123, within a second, with no message lost. Delivery is a sequenced log with acknowledgements, and applying a purge is an index lookup, never a scan.
   1. The customer’s CMS purges the product’s tag after a price change.
   2. The purge is published to regional relays, which stream it to an agent in every PoP.
   3. Every PoP, upper tiers included, marks the matching objects stale through a per-server tag index.
   4. Each PoP acknowledges the sequence it has applied, and the control plane marks the purge complete.
   5. The next request revalidates the stale copy, and a copy older than the purge is never cached again.

Ask: How does a user reach a nearby PoP? Why not just answer DNS with the closest one?

Good answers name: Anycast for everything, in several prefix groups, plus BGP tuning and a DNS overlay where measurements say anycast is wrong, DNS steering to per-PoP unicast addresses, Client-side steering from a list of hosts, One global anycast prefix, no measurement or overlay.

Our pick: Every PoP announces the same customer-facing prefixes, split into a few groups, to its transit providers, exchange-point peers and directly connected ISPs, and domains get addresses from those groups. The internet’s routing takes each user to a nearby PoP and shares each attack among the PoPs nearest its sources. The traffic mapper measures TCP round-trip times per client network and PoP on real connections; where a network lands more than ~30 ms worse than its best PoP, the network team fixes it with BGP (a direct peering, a community asking the ISP to prefer a route, or a prepend at the wrong PoP), and until then steering DNS answers that network with a regional group announced only from nearby PoPs. To drain a PoP it withdraws one prefix group at a time, so neighbours take the load in steps. Cloudflare runs its whole network on anycast; Akamai’s published architecture (Nygren, Sitaraman and Sun, 2010) steers with a DNS-based mapping system. Both work at scale, and the choice is about what you optimise for: here, attack absorption and simple failover.

1. A PoP loses power at peak. What happens to the people connected to it?
   Its routers stop answering, the BGP sessions drop (within a second or so with BFD, rather than the 90-second default hold time), and traffic for its prefixes moves to the next-nearest PoPs. Connections in progress reset, and browsers retry idempotent GETs on a new connection. Resumption still works because session ticket keys are shared across the fleet, so the new PoP resumes TLS without a full handshake. The neighbours must have the headroom, which is why PoPs run at about 50 % at peak.
2. Anycast is sending a big ISP’s users in Mumbai to Singapore. How do you notice, and what do you do?
   The mapper shows it: for that ISP’s client networks, the PoP they reach is 60 ms worse than the Mumbai PoP other networks in the city reach. The usual cause is that the ISP buys transit from a carrier that hands our routes over in Singapore. The fixes, in order: peer with the ISP directly in Mumbai, ask it via a BGP community to prefer our Mumbai route, or prepend our Singapore announcement toward that carrier. Meanwhile steering DNS can answer that ISP’s resolvers with the India-only group.
3. Why not withdraw a PoP that is under attack?
   Because the attack follows the route: withdrawing moves the flood to the next PoP, and every innocent user of the withdrawn PoP goes with it. It is better to drop the attack where it lands, since dropping a packet in XDP costs almost nothing. If one PoP near a dense cluster of bots is truly saturated, withdraw only the attacked customer’s prefix there, so the attack spreads over the neighbours while other customers stay local.
4. Does anycast break long-lived connections?
   It can, when a route changes mid-connection and packets start arriving at a PoP that has no state for them. In practice routes are stable for far longer than a typical request, and the mapper alerts on prefixes that flap. Long downloads resume with Range requests, and WebSocket clients reconnect. Some CDNs move very long sessions to a PoP-specific address with a redirect, which trades anycast’s protection for stability.
5. How would you know routing is working?
   Per client network, compare the RTT to the PoP actually reached with the best PoP measured for that network, from TCP samples on real connections and from beacons in a small share of pages that fetch a test object from several PoPs. The headline is the share of requests served within 10 ms of their best PoP. Alongside it: load per PoP against capacity, BGP route changes per prefix, and how long a withdrawal takes to move traffic.

Ask: A PoP has 40 servers. Which one handles a request, and why not just round-robin connections?

Good answers name: Round-robin or least-connections; every server caches what it serves, A dedicated L7 balancer tier that routes by URL hash, L4 balancing to every server, then rendezvous hashing of the cache key to an owner, with hot keys copied everywhere, Hash the client’s IP to a server.

Our pick: Routers spread incoming packets over the PoP’s servers with ECMP, and an XDP program on every server hashes each packet’s 5-tuple through a Maglev-style table to choose the server that terminates the connection, with a connection table so flows survive ECMP or membership changes; replies go straight to the user. The proxy on that server computes the cache key and uses rendezvous hashing over the PoP’s healthy servers to find the owner, which serves from RAM or SSD and alone goes upstream on a miss. Membership comes from a per-PoP health checker and is versioned, and a server must fail checks for about 5 seconds before its keys move; a moment of disagreement only costs a duplicate copy, never a wrong answer. A count-min sketch on each server flags keys above ~1 k requests a second in the PoP, and those are copied into RAM on every server. Google’s Maglev and Facebook’s Katran are public examples of this kind of L4 balancer, and consistent hashing itself came out of the research Akamai was founded on (Karger et al., 1997).

1. A cache server dies. What happens to its keys and its connections?
   Its connections reset and clients reconnect; the L4 table sends them to other servers. Its keys, about 1/40 of the PoP’s objects, now rank another server first, which misses and fetches them from the upper tier rather than the origin, so the cost is a short bump in tier traffic. When it comes back it rejoins with a low weight that rises over minutes, so its keys move back gradually instead of all becoming misses at once.
2. One image gets 300 k requests a second in Frankfurt. What breaks?
   At 100 KB each that is 240 Gbit/s from one owner, several times what one server’s network card carries. The count-min sketches on the servers receiving those requests see the key cross the threshold within a second, and each keeps its own RAM copy, fetched once from the owner. The load then spreads over all 40 servers at about 6 Gbit/s each. When the key cools, the copies age out of RAM on their own.
3. Why rendezvous hashing rather than a ring with virtual nodes?
   For 40 to 200 servers, scoring every server for a key is cheap and needs no virtual-node tables, it spreads keys evenly, and removing a server remaps only that server’s keys. It also gives a natural ranking, so “the top two servers for this key” is free when a hot key needs a second owner. A ring wins when there are thousands of members and scoring each becomes the cost, which a PoP never has.
4. What does the extra hop inside the PoP cost?
   Latency is a few hundred microseconds on the PoP’s switches, invisible next to a 20 ms user round trip. Bandwidth is the real cost: with 40 servers, about 39 in 40 hit bytes cross the PoP network from owner to terminating server before going out, so the fabric is sized for roughly twice the PoP’s egress. That is far cheaper than the alternative, 20 to 40 times more misses.
5. Why not have the owner send the response straight to the user?
   Because the TLS session lives on the server that terminated the connection, and handing a live TLS connection between servers mid-request is complex and fragile. Direct server return already keeps the balancer out of the response path, which is the big saving. The hop from owner to terminating server is the price of separating CPU from disk, and it is worth paying.

Ask: Each of 300 PoPs misses on its own. How do you keep a customer’s origin from melting, including when it is already down?

Good answers name: Edges fetch straight from the origin, Per-origin upper tier: edges send misses to the upper-tier PoP nearest the origin, with a backup, and both levels coalesce, Several regional tiers per origin, Push or pre-warm content to every PoP.

Our pick: Each origin gets a primary and a backup upper-tier PoP from about 15 large PoPs, chosen by the mapper as the lowest RTT to the origin, much as Cloudflare’s Smart Tiered Cache picks one upper tier near each origin (Fastly calls the same idea shielding). Edge misses go to that tier, where the same rendezvous hashing picks one owner server, so a burst of misses from 300 edges for one object makes one origin fetch, and waiters receive bytes as they arrive. Uncacheable requests also pass through the tier on pooled connections, which saves the origin a TLS handshake per request. stale-while-revalidate is honoured so users never wait on a refresh, and stale-if-error applies by default for up to a day unless the origin or the zone’s config says otherwise, so an origin outage becomes slightly old content. Each tier server caps concurrent fetches per origin (say 64), and a failing origin goes behind a circuit breaker that lets only probes through until it recovers.

1. Doesn’t the tier add latency to every miss?
   Hits are untouched. For a miss, the edge-to-tier leg replaces the edge-to-origin leg and covers most of the same distance, because the tier is near the origin, and it runs over warm pooled connections, so there is no fresh TCP and TLS handshake to the origin. Measured end to end, a tiered miss is often as fast as a direct one. What it costs is a second copy of the object and one more component on the miss path.
2. The upper-tier PoP for a big origin fails.
   Edges see failed fetches and switch to the backup tier within seconds. The backup is cold for that origin, but coalescing bounds the cost at about one origin fetch per object, and edges serve stale copies of anything that expires meanwhile. The mapper then picks a new primary. A tier failure becomes a short rise in that origin’s load, not a herd.
3. How does coalescing handle a response that turns out to be uncacheable?
   The waiters cannot share a private response, so each is released to fetch on its own, and the key is remembered as uncacheable for a few minutes (hit-for-pass) so later requests skip the queue instead of being serialised one behind another. Without that, a popular uncacheable URL would be fetched one request at a time. The same applies to responses with Set-Cookie unless the zone’s rules strip it.
4. A 2 GB file is requested and someone seeks to the middle.
   Large objects are cached in fixed slices, say 2 MB, each with its own cache key and the same validators. A Range request fetches only the slices it covers, so a seek costs a few slices rather than the whole file, and the slices of one file spread across servers by hashing. Coalescing works per slice, so viewers of the same video segment share fetches.
5. How do you know the hierarchy is working?
   Per customer: edge hit ratio, tier hit ratio, and origin requests as a share of edge requests, which is the offload the customer pays for. The coalescing ratio, requests served per upstream fetch, shows whether bursts collapse. And origin fetches per object per expiry should sit near one per tier, not near the number of PoPs; when it climbs, a cache key is usually varying on something it should not.

Ask: What exactly is the cache key, and with 20 TB of disk per server and an endless long tail, what do you keep?

Good answers name: Plain LRU, every miss written to SSD, Second-hit admission with a Bloom filter, segmented LRU, RAM over SSD, Frequency-based admission (TinyLFU), Separate pools by object size.

Our pick: The key is (zone, host, normalised path, the query parameters the zone’s rules keep, sorted), plus one normalised value per header the response varies on: Accept-Encoding becomes br, gzip or identity, an image Accept becomes avif, webp or jpeg, and Vary on User-Agent or Cookie is uncacheable unless the customer maps it to a small set such as mobile and desktop. Each server has 512 GB of RAM, about half of it holding the hottest objects, and 20 TB of SSD written in large log-structured segments, so writes are sequential and eviction drops whole segments. An index in RAM of ~100 bytes per object covers ~200 M objects in ~20 GB. On a miss the object is served and held in RAM, but written to SSD only if a rotating Bloom filter has seen its key in the last few hours; eviction is segmented LRU in RAM and CLOCK over SSD segments. Akamai reported that caching on the second request cut disk writes sharply while slightly improving hit rates, because one-hit wonders stopped evicting useful objects.

1. A customer’s origin sends Vary: User-Agent. What do you do?
   Taken literally, every browser version gets its own copy and the hit ratio falls to almost nothing. By default we treat the response as uncacheable at the edge and tell the customer why. If they map User-Agent to a device class (mobile, tablet, desktop) in their rules, the key carries that class and the origin receives a normalised header, so three copies replace thousands.
2. Someone requests /logo.png?x=<random> 100 k times a second.
   Every request is a new key and a miss, so it goes upstream and the origin pays. For static file extensions the default key ignores query strings unless the customer lists parameters that matter, which turns the flood into hits. Beyond that, a per-zone cap on origin fetches per PoP and a rate limit on clients with a high miss rate stop the rest. Coalescing does not help, because no two keys match.
3. Why not keep everything in RAM?
   RAM costs something like 50 to 100 times more per gigabyte than SSD, and the hot set is small: a few percent of bytes serve most hits. An SSD read takes about 100 µs, which disappears next to a 20 ms round trip to the user. So RAM holds the hottest objects and the index, and SSD holds the warm long tail.
4. An object’s TTL expires. Do you delete it?
   No. The TTL decides freshness, not residency. An expired object stays until eviction removes it, because revalidating it costs a 304 and a few headers instead of the whole body, and because it is exactly what stale-if-error serves when the origin is down. Eviction is driven by recency and frequency alone.
5. How do you stop one customer from filling every disk?
   Cache space is not reserved per customer, because that would waste most of it, but admission and eviction are fair by construction: an object stays only if it keeps being requested, and a customer whose objects are all one-hit wonders never gets past admission. For extreme cases there is a cap on the bytes any one zone may hold per server, and large plans can buy a reserved share.

Ask: A customer changes a price and purges. How does every PoP stop serving the old page within a second, and how do you know it did?

Good answers name: Short TTLs instead of purging, Broadcast each purge to every server and scan for matches, Sequenced pub/sub to an agent in every PoP, a tag index per server, zone generations, acknowledgements, and the origin-fetch time carried downstream, Purge rules checked on every read.

Our pick: The control plane stamps each purge with a sequence number and a timestamp, writes it to a replicated log and returns 202 with an id. About ten regional relays stream the log, batched every 20 ms, to a purge agent in every PoP, upper tiers included, and each PoP acknowledges the highest sequence it has applied. Inside a PoP, a URL purge goes to that key’s owner; a tag purge goes to every server, which looks up (zone, tag) in its in-memory index and marks those objects stale (soft) or drops them (hard); and a purge of everything bumps the zone’s generation so older objects are stale on read. Every object carries the time the upper tier fetched it from the origin, and each PoP keeps the last minute of purges, so a copy older than a recent purge of one of its tags is rejected and fetched again on condition that it be newer. A PoP that loses the feed for more than 60 seconds withdraws its routes until it has replayed the log, trading a little latency for never serving content its customer has purged. Fastly’s surrogate keys and soft purge (it says purges reach its network in about 150 ms) and Cloudflare’s Cache-Tag purge are public versions of the same product.

1. A PoP is cut off from the purge feed for three minutes, then comes back.
   After 60 seconds without the feed’s heartbeat it withdrew its anycast routes, so its users were served by neighbours in the meantime. On reconnect it asks its relay for everything after its last applied sequence; the log is kept for hours, so the replay is complete. Only when it has caught up does it announce its routes again. The trade-off is deliberate: a PoP slightly further away is better than one that might serve a purged price.
2. A customer purges everything on a site doing 50 k requests a second. What happens to the origin?
   The purge itself is one generation bump per PoP, so it applies instantly, and then every object is stale at once. With a soft purge, edges keep serving stale copies while revalidating, coalescing at both levels makes it about one conditional request per object at the origin, and most come back 304. With a hard purge it is one full fetch per object, and the per-origin concurrency cap at the tier spreads that over a minute or two instead of a second.
3. Why mark objects instead of deleting them?
   A soft purge marks rather than deletes because a marked object can still be revalidated with a 304, or served if the origin fails, which a deleted one cannot. A hard purge drops the index entry and leaves the bytes for normal eviction to reclaim, since rewriting SSD segments on every purge would be expensive. Either way no disk is ever scanned.
4. What if the purge reaches an edge before its upper tier?
   The edge marks its copy stale, the next request revalidates against the tier, and the tier, still unpurged, returns the old copy. That copy carries the time it was fetched from the origin, which is older than the purge the edge has applied, so the edge rejects it and asks again with a must-be-newer condition, which sends the tier to the origin. Ordering between tiers stops mattering, which is simpler and faster than purging tiers first and edges second.
5. How do you prove purges are fast, not just claim it?
   Two measurements. From the acknowledgements: the time until all 300 PoPs report the purge’s sequence, as p50 and p99 per region. And end to end: a synthetic zone changes an object every few seconds, purges it, and probes it from every PoP until each returns the new version, which also catches a PoP that acknowledged but applied wrongly. Both sit on the dashboard.

Ask: Customers want their changes live everywhere in seconds. How do you get 30 M configurations and their certificates to every server without one bad change taking down the network?

Good answers name: Central config service queried per request, with a cache, Replicate everything to every server and apply each change everywhere at once, Versioned replicated store on every server, with a fast customer lane and a staged platform lane, Load a domain’s config on its first request from a regional store.

Our pick: The control plane’s database gives every change to a zone the next version; the distributor compiles it to the binary form the proxy reads, checks limits, accepts only linear-time (RE2-style) regex engines, and replays synthetic requests through the production proxy build in a sandbox. Accepted changes are appended to a replicated log that relays stream to every server, which applies them in order to a memory-mapped local store and keeps the previous version of each entry; Cloudflare’s Quicksilver is a public example of a replicated store on every server. A customer’s change is live everywhere in a few seconds, and if that zone’s error rate rises the distributor reverts it automatically. Proxy releases, managed WAF rules and other global changes go first to canary PoPs carrying employee and free traffic, then to 1 %, 10 % and 100 % over hours, with automatic halts on error, CPU or crash regressions and a global stop that does not depend on the system being changed. New software is also run against a corpus of real customer configurations before it ships, the test that catches a bug a valid config can trigger.

1. How do you serve TLS for 30 M domains from any server?
   The ClientHello’s SNI names the domain, and the server looks for its certificate in an in-memory LRU, then in the PoP’s certificate store, where all ~15 M certificates are replicated with keys encrypted at rest. A cold name costs about a millisecond, once per server, and hot names stay in memory. ECDSA certificates are preferred because they are small and cheap to sign with, with RSA kept for old clients. Customers who will not hand over a key can keep it on their own key server and have the edge ask it to sign each handshake, as Cloudflare’s Keyless SSL does.
2. Issuing and renewing 15 M certificates: what goes wrong?
   CAs have rate limits and outages, so the certificate manager uses at least two and falls back on failure. Validation fails when the customer’s DNS does not point at us yet, so orders retry with backoff and the dashboard says why. Renewals start at two thirds of the lifetime with jitter, about 250 k a day, and any certificate within 14 days of expiry raises an alert. Lifetimes are shrinking toward 47 days by 2029, which roughly doubles the volume, so none of this can be manual.
3. Do you staple OCSP?
   Where the CA offers OCSP, yes: the certificate manager fetches each certificate’s OCSP response before the previous one expires and distributes it with the certificate, so the edge staples it into the handshake and the browser never has to ask the CA. That saves the user a round trip and the CA a flood of queries. Some CAs, Let’s Encrypt among them, have dropped OCSP in favour of revocation lists, and for those there is nothing to staple; short lifetimes do much of the same job.
4. A customer’s valid config makes the proxy crash on one request in a hundred. Contain it.
   Crashes are counted per zone: a worker that dies while handling a zone records it, and after a handful in a minute the zone is reverted to its last good version everywhere and frozen against further changes until someone looks. Workers restart in milliseconds, and other zones’ in-flight requests are retried on another worker. A crash caused by a valid config is a platform bug, so it also opens an incident against the release, and the config joins the test corpus.
5. How do you know what every server is running?
   Every server reports, with its heartbeat, the log sequence it has applied for each lane and its software version. The dashboard shows propagation lag per PoP at p50 and p99, and flags any server more than a few seconds behind. A server whose apply fails keeps its last good version and alerts, and if it falls minutes behind it takes itself out of service rather than serve old rules.