Replication and consistency

Leader-follower, multi-leader and leaderless replication; synchronous vs asynchronous; quorums; CAP and PACELC; consistency models from eventual to linearizable; read-your-writes and failover.

Replication keeps copies of the same data on several nodes so the system survives a node failure, serves reads closer to users, and scales reads. The price is that copies can disagree, and everything hard about distributed data is about how much disagreement you accept and for how long. Interviewers probe this by asking "what happens when the primary dies" and "can a user read their own write".

Replication topologies

Single leader (primary-replica). One node accepts writes and streams its log to followers, which serve reads. Simple, strongly consistent on the leader, and how Postgres, MySQL, MongoDB and Redis work by default. Writes are limited to one node's throughput, and failover is the hard part.

Multi-leader. Several nodes accept writes, usually one per region, and replicate to each other. Local write latency in every region and tolerance of a region outage, at the cost of write conflicts: two regions update the same row at once and someone must resolve it (last-writer-wins, per-field merge, CRDTs, or application logic). Use only when multi-region write latency is a requirement, and constrain it so that a given record is normally written in one region (home region per user).

Leaderless (Dynamo-style). Any node accepts writes; the client writes to W replicas and reads from R. Cassandra, DynamoDB, Riak. No failover because there is no leader; availability is excellent. Consistency is tunable per request through quorums.

Synchronous versus asynchronous

A synchronous follower must acknowledge before the leader confirms the write. Zero data loss on failover, but the write is as slow as the slowest follower and stalls if it is down. Asynchronous followers lag behind (milliseconds normally, seconds or minutes under load); failover can lose the writes not yet replicated. The usual compromise is semi-synchronous: one follower synchronous, the rest async, so there is always one up-to-date copy.

Say which you use and what it costs: "semi-sync, so failover loses nothing but writes pay one extra intra-region round trip, about 1 ms".

Quorums

With N replicas, writing to W and reading from R guarantees that a read sees the latest write if W + R > N. N = 3, W = 2, R = 2 is the common setting: tolerates one node down for both reads and writes. W = 1, R = 1 is fastest and only eventually consistent. W = N is strongly consistent for writes but unavailable if any node is down.

Quorums give overlap, not full linearizability: concurrent writes, sloppy quorums during partitions, and failed partial writes can still produce anomalies. Say "quorum reads and writes for the paths that need consistency, ONE for the rest".

Consistency models

From weakest to strongest; each stronger one costs latency or availability.

• Eventual. Replicas converge if writes stop. No promise about when. Fine for likes, view counts, feeds.
• Read-your-writes. A user always sees their own updates. Implement by routing a user's reads to the leader for a few seconds after they write, or by tracking a version and reading from a replica only if it has caught up. The consistency users actually notice.
• Monotonic reads. A user never sees time go backwards (a comment appears, then disappears because a second read hit a lagging replica). Pin a session to one replica.
• Causal. If B was written after seeing A, everyone sees A before B. Replies after their parent comment.
• Linearizable (strong). Every operation appears to happen at a single instant; once a write returns, all reads see it. Required for uniqueness checks, leader election, locks, and balances. Costs a round trip to a quorum or a leader on every operation.

The ordering used in most interviews: strong for money, inventory, locks and anything with a uniqueness constraint; read-your-writes for user-facing state; eventual for everything derived (feeds, counters, search indexes).

CAP and PACELC

CAP says that during a network partition a system must choose between consistency (refusing operations that could return stale or conflicting data) and availability (answering anyway). Partitions are not optional, so the choice is C or A under partition. PACELC adds that even without a partition there is a tradeoff between latency and consistency: replicating synchronously to a quorum costs a round trip.

The useful interview move is to apply it per operation, not per system: "payment authorisation is CP: I would rather fail the request than double-charge; the feed is AP: stale is fine."

Failover

When the leader dies, a follower must be promoted. The steps and their failure modes:

1. Detect. Heartbeat timeout, typically 10 to 30 s. Too short and a GC pause causes a spurious failover; too long and the outage is long.
2. Elect. Choose the follower with the most recent log position. Needs a consensus mechanism (Raft, ZooKeeper, etcd) so two followers cannot both decide they won: split brain produces two leaders accepting conflicting writes.
3. Reconfigure. Clients and other followers must learn the new leader: a virtual IP, DNS, a proxy, or a config store.
4. Fence the old leader. When it comes back it must not accept writes. Fencing tokens or STONITH.

Async replication means the new leader may lack the last few writes; those are lost or must be reconciled. This is why money systems use synchronous replication or idempotent retries from the client.

Managed databases (Aurora, Cloud SQL, Atlas) automate all of this with 30 to 120 s of write unavailability; say that number.

Multi-region

Reads: place replicas in each region and route reads locally; accept replication lag of tens to hundreds of milliseconds. Writes: either one home region per record (user's data lives where they signed up; cross-region writes go to the home region, ~100 ms) or multi-leader with conflict resolution. Spanner-style systems give global strong consistency by paying that round trip on every commit. Say which you choose and why: most consumer products are single-leader per record with regional read replicas.

Change data capture

The leader's replication log is also the best source of change events. CDC tools (Debezium, DynamoDB Streams, Postgres logical replication) turn every committed row change into an event on a stream, which feeds caches, search indexes, analytics and other services without dual writes. It is the standard answer to "how does the search index stay in sync".

In the interview

"Postgres primary with one synchronous and two async replicas per region, read replicas in two other regions. Reads go to replicas except for a user's own recent writes, which we route to the primary for 5 seconds. Failover is automated via Patroni with a 15-second detection window; the old primary is fenced. Cross-region, a user's data has a home region and writes go there." Four sentences, every follow-up pre-empted.