Caching

Where to cache (browser, CDN, application, database), cache-aside vs write-through vs write-back, eviction policies, invalidation, hot keys, thundering herds and cache stampedes.

Caching is storing the result of expensive work close to where it is needed so the next request skips the work. It is the first tool for read scaling and the first thing an interviewer expects when the read rate is high. It is also the source of the most subtle bugs in distributed systems, which is why "how do you invalidate it" is the follow-up every time.

Where to cache

From the user inward, each layer catches a share of requests so the next layer sees fewer.

An interview answer usually names two: CDN for static and public content, Redis for the hot working set of the application. Add a small in-process cache only when you are talking about a hot-key problem.

Read strategies

Cache-aside (lazy loading). The application checks the cache; on a miss it reads the database, writes the result to the cache, and returns it. Simple, only caches what is actually read, and a cache failure degrades to the database. The costs: the first read of every key is a miss, and stale data lives in the cache until the TTL expires or someone invalidates it. This is the default answer.

Read-through. The cache itself loads from the database on a miss. Same behaviour as cache-aside but the logic lives in the cache library or proxy. Cleaner code, less control.

Write strategies

Write-around. Write to the database only and let the cache entry expire or be invalidated. Combined with cache-aside, this is the common pattern: write to the DB, delete the cache key. The next read repopulates it.

Write-through. Write to the cache and the database synchronously. Reads after a write are always fresh, at the cost of write latency and caching data that may never be read.

Write-back (write-behind). Write to the cache, acknowledge, and flush to the database later in batches. Very fast writes and coalescing of hot updates (a counter incremented 10 k times becomes one database write), but a cache failure loses unflushed writes. Used for counters, view tallies, and other data where loss of a few seconds is acceptable.

Invalidation

There are only three real strategies and every system uses a mix.

• TTL. Every entry expires after N seconds. Simple and bounds staleness; choose the TTL by how stale the data may be. Add jitter (random ±10 %) so entries written together do not all expire together.
• Explicit invalidation on write. Delete or update the key when the source changes. Precise but requires every writer to know every cache key that depends on the data, which decays as the system grows.
• Event-driven. Writers publish change events (or the database's change stream does); a consumer invalidates affected keys. Decoupled, works across services, and eventually consistent by a few hundred milliseconds.

The dangerous race: reader misses, reads old value from DB; writer updates DB and deletes cache key; reader writes old value into cache. Now the cache is stale until TTL. Mitigations: short TTLs as a backstop, delete-then-write-then-delete-again ("double delete" with a small delay), or versioned keys where the cache stores (value, version) and a write with a lower version is ignored.

Eviction

When the cache is full, something goes. LRU (least recently used) is the default and right for most workloads. LFU (least frequently used) protects long-lived popular keys from being pushed out by a scan of one-time reads; Redis offers an approximate LFU. FIFO and random are used when the metadata cost of LRU matters. Size the cache from the working set: in a Zipfian workload, the top 1 % of keys receive most of the traffic, so a cache holding a few percent of the data gives a 90 %+ hit rate. See the estimation guide.

Hot keys and thundering herds

A hot key is a single key receiving a large fraction of traffic: a celebrity's profile, the home page config, a viral link. Redis handles ~100 k to 1 M ops/s on one node, and a single key lives on one node, so one key can saturate a shard while the others idle. Fixes: replicate the key across N shards with a suffix (key#1 … key#N) and read a random one; add a small in-process cache with a 1 to 5 s TTL on every application server; serve the value from the CDN.

A thundering herd or cache stampede happens when a hot key expires and thousands of concurrent requests all miss and all hit the database at once. Fixes: request coalescing (a per-key lock or singleflight so one request recomputes and the rest wait), stale-while-revalidate (serve the expired value while one request refreshes in the background), and probabilistic early expiration (each reader refreshes with a small probability before the TTL, spreading the refresh).

A cold cache after a restart or a new region is the same problem for every key at once. Warm it from a snapshot or from a list of the top keys before taking traffic, and rate limit the database while it fills.

Consistency: what to promise

A cache is by definition a copy, so the honest promise is "eventually consistent within the TTL, and immediately consistent for the writer's own session if we invalidate on write". Say which data cannot be cached at all: balances, inventory counts at checkout, anything the user is about to act on where a stale read causes a wrong action. For those, read from the primary or from a cache that is updated in the same transaction (rare and expensive).

Redis versus Memcached

Memcached is a simple multithreaded key-value cache; it is slightly faster per node for plain get/set and scales by adding nodes with client-side hashing. Redis is single-threaded per core but has data structures (lists, sorted sets, hashes, sets), pub/sub, Lua scripts, persistence and replication, and a built-in cluster mode. Choose Redis when you need any structure (leaderboards, rate limiting, queues) or persistence; choose Memcached for a pure large object cache. In an interview, Redis is the safe default and nobody will object.

Sizing example

100 M items, 1 KB each, Zipfian reads at 50 k/s. Caching the hottest 5 % (5 M items, 5 GB) yields roughly a 95 % hit rate, so the database sees 2.5 k/s. Three Redis nodes of 8 GB each with replicas cover it with room to grow. The point of saying this out loud is that it turns "add a cache" into a design with numbers.