Multi-region databases with read replicas, multi-leader, home regions and consensus: an interactive latency map

Serving users on three continents from one database means somebody pays an ocean round trip on every request. Moving data closer splits into a few designs: read replicas near users with one leader for writes, multi-leader replication where every region writes, a home region for each record, or consensus across regions as in Spanner and CockroachDB. Each moves the cost somewhere else: to write latency, stale reads, conflicts or what a region outage loses.

The playground places users 50% in the US, 30% in Europe and 20% in Asia, with real-world round trips between Virginia, Ireland and Singapore. Pick a design, switch between per-user data and a shared document, and take us-east down.

What each design costs

• One region is simple and consistent, but Europe waits 75 ms and Asia 220 ms on every request, and a region outage takes the whole product down.
• Leader plus read replicas makes reads local everywhere while writes still cross to Virginia. Replicas lag, so a user in Asia can miss their own write for up to 120 ms unless reads after a write go to the leader. Losing the leader loses the writes that had not replicated, about 48 ms of them, and stops writes for everyone until a replica is promoted.
• Multi-leader makes reads and writes local and survives a region loss without refusing writes. It is safe when each record is only written from one region; when the same record is written in several, concurrent writes conflict, here about 44 times an hour on one busy document, and need last-writer-wins, CRDTs or application merges.
• A home region per record gets local reads and writes for data owned by one user without conflicts, because each record still has a single leader. Shared records pay a cross-region write, and a region outage stops writes only to the records homed there.
• Consensus across regions never loses an acknowledged write and never conflicts, because a majority of regions commits every write. Each write waits for the nearest other region: 75 ms between the US and Europe, 160 ms from Asia.

Placing data by region

-- CockroachDB: each row lives, and is led, in its owner's region
ALTER DATABASE app SET PRIMARY REGION "us-east1";
ALTER DATABASE app ADD REGION "europe-west1";
ALTER DATABASE app ADD REGION "asia-southeast1";
ALTER TABLE users SET LOCALITY REGIONAL BY ROW;      -- local reads and writes for the owner
ALTER TABLE currencies SET LOCALITY GLOBAL;          -- rarely written, read fast everywhere

-- the same idea by hand: route each user to their home region's database
home = directory.lookup(user_id)          # cached; written once at sign-up
db = pools[home]

In a system design interview

When the requirements say global users or survive a region outage, split the data by who writes it: “User-owned data is homed in the user’s region, so their reads and writes stay local and there is one writer per record. Rarely changed reference data is replicated everywhere. Money and inventory, where a lost or conflicting write is unacceptable, use consensus across three regions and accept 75 to 160 ms writes. Reads that may be slightly stale come from local replicas.” Expect the follow-ups: what happens to in-flight writes when a region fails, how users who travel are handled, and how you route a request to the right region.

Read the guide to multi-region architecture or active-active vs active-passive, see the CAP theorem and database failover, or watch a strong candidate design a payment system.

Questions people ask