SysDesignPrep.com
Interactive concepts

Replication lag and read-your-writes: an interactive simulator

With asynchronous replication, followers apply the leader’s writes a little later: usually milliseconds, sometimes seconds. Read-your-writes consistency is the promise that after you change something, your own reads show the change, even when they are served by a follower. Without it, a user saves their profile, refreshes the page and sees the old one.

In the playground, one user changes their profile every 2 seconds and reads it back four times after each change. Pick how reads are routed, let follower 2 fall behind, and watch which reads come back stale.

Each read goes to a random follower. Follower 2 is up to 2.5 seconds behind, so 8 of 20 reads miss the user’s own latest change, and 1 goes backwards, showing an older version than a read before it. The leader serves none of the reads.

0 s2 s4 s6 s8 s10 sLeaderv0v1v2v3v4v5Follower 1v0v1v2v3v4v5Follower 2v0v1v2v3v4write v1write v2write v3write v4write v5read at 0.66 s from Follower 1: v1, user last wrote v1read at 1.00 s from Follower 2: v0, user last wrote v1read at 1.50 s from Follower 1: v1, user last wrote v1read at 2.21 s from Follower 2: v1, user last wrote v1read at 2.65 s from Follower 2: v1, user last wrote v2read at 3.00 s from Follower 2: v1, user last wrote v2read at 3.50 s from Follower 2: v1, user last wrote v2read at 4.21 s from Follower 2: v1, user last wrote v2read at 4.66 s from Follower 2: v1, user last wrote v3read at 5.00 s from Follower 1: v3, user last wrote v3read at 5.50 s from Follower 1: v3, user last wrote v3read at 6.21 s from Follower 2: v3, user last wrote v3read at 6.66 s from Follower 1: v3, user last wrote v4read at 7.00 s from Follower 1: v4, user last wrote v4read at 7.50 s from Follower 1: v4, user last wrote v4read at 8.21 s from Follower 1: v4, user last wrote v4read at 8.66 s from Follower 1: v4, user last wrote v5read at 9.01 s from Follower 1: v5, user last wrote v5read at 9.51 s from Follower 1: v5, user last wrote v5read at 10.21 s from Follower 1: v5, user last wrote v5
Stale reads8 of 20older than the user’s own write
Went backwards1older than an earlier read
Reads on the leader0load the followers were meant to take
Write latency5 msaverage, until acknowledged
read saw the user’s changestale readwent backwardsreplication from the leader

Why replication lag happens

A leader that replicates asynchronously acknowledges a write as soon as it has the write itself, and ships it to followers afterwards. Under normal load followers are milliseconds behind. They fall further behind when they are busy with heavy queries, recovering after a restart, replaying a large transaction, or reached over a slow link to another region, and lag of seconds or minutes is not unusual during an incident. Reads from followers are cheap and scale out, so most read traffic goes to them, and that is where the lag shows.

What lag looks like to a user

AnomalyWhat the user seesThe guarantee that prevents it
Stale read of your own writeSaves a change, refreshes, the change is goneRead-your-writes
Going backwardsSees the change, refreshes, it is gone againMonotonic reads
Effects before causesSees a reply before the message it answersConsistent prefix

In the first preset, with follower 2 up to 2.5 seconds behind, 8 of 20 reads miss the user’s own latest change and one goes backwards.

Ways to read your own writes

  • Read from the leader for a while after a write. Simple, but the window is a guess. In the second preset it is 1 second and follower 2 lags by more, so as soon as reads move to the followers the user’s change disappears again: 5 stale reads, all of them going backwards.
  • Read from the leader for anything the user can edit. Their own profile from the leader, everyone else’s from followers. It works when the user edits a small part of the data.
  • Remember the version you wrote. Keep the log position of the user’s last write in their session and only read from a replica that has reached it, falling back to another replica or the leader. In the third preset nothing is stale and only 3 of 20 reads reach the leader. This is the robust answer.
  • Stick each user to one replica. That gives monotonic reads, since one replica never moves backwards, but not read-your-writes: pinned to the lagging follower, 18 of 20 reads are stale.
  • Replicate synchronously. Every replica has every acknowledged write, so all reads are fresh, but every write waits for the slowest follower: about 1.9 seconds each in the last preset with follower 2 behind, against 5 ms without waiting. Semi-synchronous replication, waiting for one follower rather than all, is the usual compromise.

Reading your writes in SQL

-- PostgreSQL: after a write on the leader, remember where the log was
SELECT pg_current_wal_lsn();                 -- e.g. 0/3A2F1C8, kept in the user's session

-- on a follower, serve the read only if it has replayed that far
SELECT pg_last_wal_replay_lsn() >= '0/3A2F1C8'::pg_lsn;
-- false: try another follower, wait briefly, or read from the leader

-- MySQL: wait up to 1 second for this replica to apply the write's GTID
SELECT WAIT_FOR_EXECUTED_GTID_SET('3E11FA47-71CA-11E1-9E33-C80AA9429562:23', 1);

The version has to travel with the user. Within one browser session a cookie will do; across devices, the user’s phone does not know what their laptop just wrote, so the position has to be stored server side per user. Across regions, route each user to the region that holds their data, or read from the leader region after a write.

In a system design interview

When you add read replicas, say what happens to a user’s own changes: “Reads go to followers, which can be a second or more behind. After a user edits their profile we store the write’s log position in their session, and their reads go to a follower that has reached it, or to the leader. Everyone else can read a slightly stale profile.” Expect the follow-ups: what if every follower is behind, monotonic reads, and the same problem across devices or regions.

Read the guide to replication and consistency, try quorum reads and writes for the leaderless version of the problem, or watch a strong candidate design Instagram.

Questions people ask

What is read-your-writes consistency?

A guarantee that after a user writes something, that user’s later reads reflect the write, even when they are served by a replica. It says nothing about other users, who may still see the old value for a while, which is why it is much cheaper than making every read up to date.

What is replication lag?

The delay between a write being committed on the leader and being applied on a follower. With asynchronous replication it is usually milliseconds, but it grows when a follower is busy, restarting, replaying a large transaction or far away, and can reach seconds or minutes. Reads from a lagging follower return old data.

How do you guarantee read-your-writes with read replicas?

Record the log position of the user’s last write, such as a PostgreSQL LSN or a MySQL GTID, and serve their reads only from a replica that has applied that position, falling back to another replica or the leader. Simpler options are reading from the leader for a short time after a write, or always reading data the user can edit from the leader.

What are monotonic reads?

A guarantee that a user never sees data go back in time: once a read has returned a version, later reads return that version or a newer one. Reading from different replicas with different lag breaks it. Pinning each user to one replica, or tracking the newest version they have seen, provides it.

Why is reading from the leader for a fixed time after a write not enough?

Because the window is a guess about replication lag. If a follower falls further behind than the window, reads move back to the follower before it has the write, and the user sees their change disappear. Tracking the actual version written does not depend on how far behind the followers are.

Does synchronous replication solve stale reads?

Yes for replicas that are part of it: a write is acknowledged only once they have it, so reads from them include it. The cost is that every write waits for the slowest of those replicas, and a slow or unreachable one stalls writes. Semi-synchronous replication, waiting for at least one follower, is the common compromise.

What is the difference between read-your-writes and strong consistency?

Strong consistency, or linearizability, means every read by anyone sees the latest committed write. Read-your-writes only promises that a user sees their own writes; other users may briefly see older data. Most applications need read-your-writes for a user’s own changes and can accept slightly stale data elsewhere.