Design a Distributed Lock Service

• Acquire and release a named lock: Exclusive or shared, blocking or try-lock with a timeout. Locks are advisory: they coordinate cooperating processes, they do not stop anyone touching the data.
• Sessions kept alive by KeepAlives: Every lock and ephemeral node belongs to a client session. When the KeepAlives stop for a full lease, the session ends and everything it held is released.
• A fencing token with every grant: A number that only goes up, so the protected resource can reject a holder that lost the lock without noticing.
• Leader election and membership: Built from ephemeral sequential nodes: the lowest number leads, its node holds the leader’s address, and the set of nodes is the live membership.
• Small configuration values: Values up to 64 KB with a version, read by many and written rarely, with compare-and-set on the version.
• Watches: A one-shot notification when a node’s value changes, its children change or it is deleted, delivered on the client’s session.
• Consistent client caching: Clients cache values and the service invalidates them before a write completes, so a cached read is never stale while the session is healthy.
• Out of scope: Storing application data or large blobs, a general-purpose key-value store, locks replicated across regions beyond one small global cell, and the service-mesh side of service discovery.

• Safety (never two holders acting): The requirement that drives the hardest trade-off: when the service cannot be sure, it waits (a lock stays held up to a lease after its holder dies, a cell without a majority stops granting) rather than ever granting twice. Fencing makes the guarantee hold even against paused clients.
• Availability (99.99 % per cell): Survives two replica failures or the loss of a zone. Every leader election in the company depends on it, so clients must also ride through short outages without dropping their locks.
• Acquire latency (p99 < 50 ms): One Raft commit is ~5 ms in a region; the budget absorbs fsync tails. Tens of milliseconds is fine for a call each client makes rarely.
• Scale per region (50 k sessions · 20 k locks): Load is dominated by 12.5 k KeepAlives a second and ~50 k reads a second before caching. Real writes are about 200 a second.
• Takeover after a holder dies (< 15 s): The 12 s lease plus a margin. Shorter would expire healthy clients during GC pauses and during the cell’s own leader failovers.
• Cell failover (< 3 s, no sessions lost): A new Raft leader in 1 to 2 s plus reconnects. It must stay well inside the lease so that clients do not notice.
• Linearizable (lock ops, writes, lock-path reads): Everything on lock paths is ordered by one log. Cached config reads are kept consistent by invalidation, not by a TTL.
• Small state (< 1 GB in memory): Values ≤ 64 KB and ~100 MB in total today. Keeping it small is what lets every replica hold it all in memory and snapshot it in a second.

• KeepAlives per second: ~12.5 k/s. 50 k sessions renewing every 4 s (a third of a 12 s lease) = 12.5 k/s. This is the biggest stream in the system, which is why renewals stay in the leader’s memory and never touch the log.
• Reads reaching the cell: ~500/s. 50 k processes reading a config value about once a second = 50 k reads/s. With a client cache that the leader invalidates and a ~99 % hit rate, ~500/s reach the cell. The cache, not the replicas, carries the read load.
• Writes through consensus: ~200/s avg · ~2 k/s peak. 5 k short-lived locks taken about once a minute = ~83 acquires and ~83 releases a second, plus ~10 config writes and ~20 session creates and expiries: ~200/s. Mass restarts reach ~2 k/s. One cell sustains ~10 k/s, so writes are not the hard part.
• Commit latency of an acquire: ~5 ms. Client to proxy to leader ~1 ms, then the leader’s own fsync (1–2 ms) in parallel with a cross-zone round trip to two followers and their fsync (~3 ms), then the reply ~1 ms: ~5 ms. fsync tails push p99 toward tens of milliseconds, still inside the 50 ms target.
• State in memory: ~100 MB. ~100 k nodes (20 k locks, 50 k ephemeral membership nodes, ~30 k config values) at ~1 KB each with metadata = ~100 MB, plus 50 k sessions at ~100 B. Every replica holds all of it in memory and snapshots it in about a second.
• Proxies needed: ~25. 50 k client connections at ~2 k per proxy = 25 proxies, spread over three zones. The leader sees 25 batched KeepAlives a second instead of 12.5 k, and a change watched by every process fans out 25 ways from the cell.
• Raft log written per day: ~3.5 GB/day. 200 writes/s × 86 400 s ≈ 17 M entries × ~200 B ≈ 3.5 GB/day. A snapshot every 100 k entries (about every 8 minutes) truncates it, so only ~20 MB of log is ever on disk.
• Herd on one release: 2 000 events vs 1. With 2 000 waiters all watching the lock node, one release sends 2 000 events and triggers up to 2 000 re-lists and creates: a fifth of a second of the cell’s entire write capacity for one handoff. Watching only the predecessor makes it 1 event.

• Protected resource (rejects stale tokens): Whatever the lock protects: a database shard, a storage bucket, a partition of work. It stores the highest fencing token it has accepted for each lock and refuses any write carrying a lower one, which is the only thing that stops a paused former holder.
• Client cache (invalidated before writes): Values the library has read, held in process memory and valid only while the session lease is. The leader keeps a list of who caches each node and invalidates them before any write commits, so a cached read is never stale while the session is healthy. When the lease is in doubt, the cache is switched off.
• Session manager (leader-only lease timers): Runs on the leader and tracks every session’s lease expiry in memory, extending it on each KeepAlive without writing to the log. When a lease runs out it commits one expiry entry through Raft, which deletes the session’s ephemeral nodes and frees its locks at a log position every replica agrees on.
• Client + library (session · recipes · local lease): A service process that needs a lock, a leader or a config value. Its library does the protocol work: it holds one session, sends KeepAlives, runs the lock and election recipes, and keeps a conservative local lease clock. It tells the application to stop acting the moment that lease is in doubt.
• Proxy tier (~25 · KeepAlive fan-in): About 25 proxies, each terminating ~2 k client connections. They batch KeepAlives into one RPC a second per proxy, serve cached reads, register one watch per path on behalf of all their clients and fan events back out. Sessions belong to the cell, not the proxy, so a client whose proxy dies reconnects to another and keeps its locks.
• Leader replica (Raft leader · 1 of 5): The one replica of five that orders everything: it appends writes to the log, replicates them, commits on a majority, and serves lock-path reads after a ReadIndex check. It also owns sessions, the cacher list and watch delivery, which is why proxies and client caches exist to keep load off it.
• Follower replicas (4 of 5 · across 3 zones): The other four replicas, placed so the cell has two in each of two zones and one in a third, and any zone can be lost. They persist and acknowledge entries, apply committed ones to their own copy of the state, and stand for election if the leader’s heartbeats stop for 1 to 2 s.
• Cell directory (prefix → cell · cached): Maps each top-level prefix of the namespace (/payments, /indexer) to the cell that owns it, with each cell’s proxy addresses. A few hundred entries that change a few times a month and are cached by every library, so it is never on the lock path.
• Watch table (one-shot events · invalidations): Who is watching which node and who is caching it, kept on the leader (and, for their own clients, on each proxy). After a change commits it emits one-shot events; before a write to a cached node commits it sends invalidations and waits for the acknowledgements.
• State machine (nodes · sessions · in memory): The namespace every replica rebuilds by applying the log in order: nodes with data, versions and flags (ephemeral, sequential), the sessions that own them, and the last request id and result per session so a retried request is never applied twice. About 100 MB, all in memory.
• Raft log (WAL · fsync · group commit): Each replica’s write-ahead log of creates, deletes, value writes and session expiries, fsynced before an entry is acknowledged. Group commit puts many entries in one fsync. A p99 fsync above 10 ms is the warning sign that slow disks will start causing elections.
• Other cells (independent 5-replica groups): Other independent five-replica groups in the region, each with its own proxies and its own slice of the namespace. No operation spans two cells, so they share nothing and fail independently. A small global cell with replicas in several regions handles the rare truly global lock.
• Snapshots (every 100 k entries): A point-in-time copy of the state machine taken every 100 k entries, after which the log before it is discarded. Lagging and replacement replicas start from one, and an hourly copy in object storage in another region is the disaster-recovery backup.

1. Acquire a lock and write with its token: The core write path. One Raft commit puts the client in line, the lowest node holds the lock, and the grant comes back with a fencing token that the protected resource checks on every write.
   1. The library maps the lock’s path to the cell that owns its prefix.
   2. It opens a session through a proxy and starts sending KeepAlives.
   3. To acquire, the library creates an ephemeral sequential node under the lock’s path.
   4. The leader appends the create, replicates it, and commits once three of five replicas have it on disk.
   5. The entry is applied, and the library lists the lock’s children to see whether its node is the lowest.
   6. The library returns the grant with a fencing token: the create index of its node.
   7. The holder sends the token with every write, and the resource rejects any token lower than the highest it has seen.
   8. On release, the library deletes its node; the delete commits and the next waiter is notified.
2. Cached config reads and an invalidating write: The read path. Most reads never leave the client, and a write is held until every cache that might hold the old value has dropped it, so cached reads stay consistent without polling or TTLs.
   1. A process reads a config value; on a miss, the library fetches it through its proxy.
   2. Later reads are served from the client cache with no RPC.
   3. An operator writes a new value; the leader marks the node uncachable and sends invalidations to every cacher.
   4. Each proxy drops its copy and passes the invalidation only to its clients that hold the node.
   5. Once every cacher has acknowledged or expired, the write commits and the leader replies.
   6. Watchers get a one-shot event, re-read the value and re-arm the watch in the same call.
3. 2 000 contenders and a paused leader: The scale-breaking case and the failure that matters most. Two thousand processes contend for one leadership; when the leader stalls, exactly one waiter is woken, and the old leader’s late write is refused by its token.
   1. Each of 2 000 workers creates an ephemeral sequential node under the election path and lists the children.
   2. Every other worker watches only the node just below its own.
   3. The leader stalls in a 20-second garbage-collection pause, and its KeepAlives stop.
   4. When the 12-second lease runs out, the session manager commits the session’s expiry.
   5. The delete fires one watch, and only worker 42 is notified.
   6. Worker 42 confirms it is lowest, becomes leader, and fences the resource with its token.
   7. The old leader wakes and sends a write it prepared before the pause; the resource rejects it.
4. The cell’s leader replica crashes: The failure case for the service itself. A new leader is elected in about two seconds, sessions survive because their leases outlast the failover, and clients flush their caches because an invalidation may have been lost.
   1. The leader’s machine dies, and the followers stop receiving heartbeats.
   2. A follower times out first, asks for votes, and wins a majority with an up-to-date log.
   3. The new leader commits a no-op, treats every session as renewed now, and accepts KeepAlives again.
   4. Proxies find the new leader and resend requests that had no reply; sessions are intact.
   5. Clients receive a fail-over event and flush their caches.
   6. A client whose lease ran out locally while it waited enters jeopardy, stops acting, and resumes when its session is confirmed.
5. Snapshots, a replacement replica, a new cell: The background work that keeps a cell healthy: snapshots bound the log, a dead replica is replaced without ever allowing two majorities, and a prefix that outgrows its cell moves to a new one.
   1. Every 100 k entries, each replica snapshots its state machine and truncates its log.
   2. A replacement machine joins as a non-voting learner and receives the latest snapshot from the leader.
   3. Once it has caught up, membership changes one server at a time: remove the dead replica, then promote the learner.
   4. A team’s prefix outgrows its cell, and the directory points it at a new cell.
   5. Dashboards track commit latency, fsync p99, session expiries per minute and the oldest unacknowledged invalidation.

Ask: Why a consensus group at all, why five replicas rather than three, and how does a read stay linearizable?

Good answers name: Five replicas across three zones (2 + 2 + 1) running Raft; lock-path reads through the leader with ReadIndex, Three replicas, Primary-backup with asynchronous replication, Seven replicas.

Our pick: One Raft group of five replicas per cell, two in each of two zones and one in a third, so losing any zone leaves three. The leader fsyncs its own entry while replicating, commits when two followers have acknowledged, and replies; heartbeats go out every 100 ms and election timeouts are randomised between 1 and 2 s. Lock-path reads (the waiters under a lock, the current leader of a service) go to the leader, which confirms it still leads with a ReadIndex round, one heartbeat exchange with a majority batched across all reads in flight, before answering from memory. A leader lease, serving reads for a bounded time after a successful heartbeat, saves that round trip but relies on bounded clock drift, so it is an optimisation to add once clocks are monitored. Chubby ran five replicas per cell with Paxos and served reads from the master under a master lease (Burrows, 2006); etcd and ZooKeeper are usually run with three or five.

1. Why not four replicas?
   Four needs three for a majority, the same as five, but tolerates only one failure, the same as three. You pay for an extra machine and a slower quorum and gain nothing. Odd sizes are the only ones that make sense.
2. A zone holding two replicas goes down while a third replica is being restarted. What happens?
   That leaves two of five, which is not a majority, so the cell stops committing: no new grants, no expiries, no writes. Existing holders carry on while their libraries ride the grace period, and nobody is ever granted a lock twice. The rule that prevents getting there is operational: never take a replica down for maintenance while a zone is impaired, and alert on "cell at minimum quorum".
3. How does a read go wrong without ReadIndex?
   A leader that has been partitioned away still believes it leads until its own timeout fires. Meanwhile the majority elects a new leader that frees the lock and grants it to someone else. A list served from the old leader’s memory would show the old holder, and a client could act on it. ReadIndex closes that: the leader notes its commit index, confirms with a heartbeat that a majority still follows it, waits until it has applied that far, and only then answers.
4. Could followers serve reads to take load off the leader?
   Not from their own state for lock paths: a follower can be behind, and "who holds the lock" must never be stale. A follower can serve a linearizable read by asking the leader for its read index and waiting until it has applied that far, which spreads CPU but not the round trip. For config values the client cache already removes most reads, so it is rarely worth it; non-voting observers are the better way to add read capacity.
5. What write rate can one cell sustain, and what limits it?
   The leader’s fsync and its network. Each commit needs a local fsync (1 to 2 ms on datacenter NVMe) and a cross-zone round trip of about 1 ms to two followers, and batching many entries into each AppendEntries and each fsync lifts throughput to roughly 10 k small writes a second. We use about 200 on average and 2 k at peak, so the case for more cells is blast radius long before it is throughput.

Ask: How long should a lease be, and what must a client do when it is not sure it still holds the lock?

Good answers name: One session lease per client (12 s, KeepAlive every 4 s), a conservative local timer, jeopardy and a 45 s grace period, A TTL per lock, renewed one by one (Redis SET PX style), Short leases (2 to 3 s) for fast takeover, No lease: hold until an explicit release.

Our pick: Each client has one session with a 12 s lease, renewed every 4 s, so two renewals can be lost before anything happens. The leader tracks expiries in memory and commits an expiry entry through Raft only when a lease actually runs out, which deletes the session’s ephemeral nodes and frees its locks at one agreed log position. The library starts its local lease clock when it sends a KeepAlive, uses a monotonic clock and subtracts a margin for clock-rate drift, so it always believes the lease ends before the cell does. When the local lease runs out it enters jeopardy: it tells the application to stop acting on its locks, switches off the cache and keeps trying other proxies for a 45 s grace period, then reports safe or expired. A new cell leader gives every session a full lease from the moment it takes over, so a 2 to 3 s failover never expires anyone. These are the defaults the Chubby paper reports: a 12 s lease and a 45 s grace period.

1. Why start the local clock when the KeepAlive is sent rather than when the reply arrives?
   Because the leader extends the lease when it processes the request, which is after the client sent it. Measuring from the send makes the client’s estimate of the expiry earlier than the leader’s by the request’s travel time, which is the safe direction. Measuring from the reply would make it later by the reply’s travel time, and a slow reply could leave the client acting after its locks were freed.
2. The cell is unreachable for 30 seconds. What does a lock holder do?
   Once its local lease runs out, under 12 s in, it is in jeopardy: the application gets an event, pauses work that needs the lock and stops issuing fenced writes. The library keeps retrying through the 45 s grace period. If the cell comes back and the session survived, it gets "safe" and carries on with the same locks and tokens; if not, it gets "expired" and must acquire again, with higher tokens.
3. Why are renewals not written to the Raft log?
   At 12.5 k a second they would be by far the largest write stream in the cell, sixty times the real writes, recording something nobody needs after a failover. A new leader does not need each session’s last renewal: it assumes every session renewed at the moment it took over. That only ever makes leases longer, which is the safe direction for the lock, and costs at most one extra lease of takeover time for clients that died during the failover.
4. How would you choose the lease length for a new deployment?
   Start from the longest silence a healthy client should survive: the cell’s failover time (2 to 3 s), the p99.9 GC pause of the client fleet, and a cross-zone network blip, and make the lease three to four times the largest. Then check that the takeover time it implies is acceptable to the services using it. Afterwards, measure false expiries, meaning sessions that expire and whose client reconnects seconds later; their rate tells you the lease is too short.
5. Does a client in jeopardy still hold the lock?
   It does not know, and that is the point. It may still hold it (the cell is down, so nobody can take it) or the cell may already have freed it. The only safe behaviour is to act as if it does not: start no new work under the lock, and rely on the fencing token to stop anything already in flight.

Ask: A lock holder is paused for 30 seconds by garbage collection, and meanwhile the lock is given to someone else. How do you stop its next write?

Good answers name: Fencing token (the Raft index of the holder’s node) checked by the resource on every write, Lock-delay: after an unclean release, nobody may take the lock for a while, The resource asks the lock service whether the token is still current, Check the lease before each write.

Our pick: Every grant returns a token: the Raft index at which the holder’s node was created. It is unique, it only increases over the cell’s life, leader changes included, and because waiters are served in creation order, each new holder of a lock has a higher token than the last. The holder sends it with every write; the resource stores the highest token per lock next to the data and makes the write conditional on it in the same transaction, for example UPDATE … SET fence = :t WHERE fence <= :t. A new holder’s first act is a fenced write that raises the high-water mark before it reads any state, so nothing from the old holder can land afterwards. For resources that cannot check, a lock-delay of up to a minute after an unclean release is the fallback, and we say plainly that it lowers the odds rather than removing them. Chubby calls the token a sequencer and offers both mechanisms (Burrows, 2006); ZooKeeper’s zxid and etcd’s revision serve the same purpose.

1. The old holder’s write arrives before the new holder has written anything. Is it accepted?
   Yes, if the resource has never seen the higher token, which is why the new holder must fence first. Its first operation presents its token (a claim, or a no-op write that bumps the high-water mark), and only then does it read the state it will work from. Any old write that landed before that point is ordered before the new holder’s view, which is the same as the old holder having finished in time.
2. Why the Raft index rather than a counter per lock?
   Both work, and a per-lock generation number is what Chubby’s sequencer carries. The Raft index is free, unique across the whole cell and needs no extra state per lock. Its one caveat is that it is only comparable within one cell, so when a prefix moves to a new cell the token carries the prefix’s directory epoch in its high bits, and the new cell’s tokens compare above every old one.
3. The protected resource is an object store that cannot compare tokens. What do you do?
   Make the write conditional on something the store can check. Write to a key that includes the token and have readers take the highest, or use the store’s conditional writes on an ETag so a stale writer’s overwrite fails. If neither fits, fall back to lock-delay plus the client stopping on jeopardy, and document that it is best effort.
4. How would you know fencing is actually working in production?
   Count rejections at every resource that checks tokens, per lock. A trickle after failovers is fencing doing its job; zero over months on a busy lock usually means the check was never wired in. Prove it with fault injection in staging that pauses a holder past its lease and asserts that its write is refused.

Ask: Two thousand processes are waiting for the same lock. What happens when it is released?

Good answers name: Ephemeral sequential nodes; the lowest holds; each waiter watches only its predecessor, Server-side wait queue: Acquire blocks and the cell grants the next waiter directly, Every waiter watches the lock node, Poll with backoff.

Our pick: Acquire creates an ephemeral sequential node under the lock’s path (…/w-0000000057) and lists the children through the leader. If its node is lowest it holds the lock, and its token is the node’s create index. Otherwise it watches the node immediately below its own and waits. When that node is deleted (released, or its session expired) the waiter re-lists rather than assuming it is next, because the predecessor may have been a waiter that gave up, and then either holds the lock or watches its new predecessor. Leader election is the same recipe, with the leader writing its address into its node so others find it by reading the lowest child. Watches are one-shot: an event says only that something changed, so the client re-reads and re-arms in one call (Get with watch=true), and after reconnecting it re-registers its watches with the last index it saw so the cell fires any it missed. This is the lock recipe from the ZooKeeper paper; etcd’s lock and election helpers do the same with revisions.

1. A waiter in the middle of the line crashes. Does the lock break?
   No. Its session expires, its node is deleted, and the one waiter that was watching it gets an event. That waiter re-lists, finds it is still not lowest, and watches the next node down instead. Nobody else is woken.
2. Why must a woken waiter re-list instead of assuming it now holds the lock?
   Because the deleted predecessor might not have been the holder. If waiter 41 gave up while 40 still holds the lock, 42 is woken but is not first. Re-listing costs one read and keeps the only invariant the recipe relies on: the holder is always the lowest existing node.
3. What can a client miss between a watch firing and its re-read?
   Intermediate values, never the latest one. If a value changes three times in quick succession, the client gets one event, reads the third value and re-arms. That is fine for configuration and membership, where only the current state matters. If every change matters, model the data as a sequence of nodes, or use a watch that is a stream resuming from an index, as etcd’s watches are.
4. Does the same recipe work for shared (read) locks?
   Yes, with one change. Readers create nodes marked read and wait only for the nearest lower write node; writers wait for the nearest lower node of either kind. Many readers then hold the lock together, a writer waits for all of them, and still only the processes that can make progress are woken.
5. How do you spot a herd in production?
   Watch events per committed write. In a healthy cell it is small; a release that produces thousands of events and a burst of creates means some team is using the naive recipe. Per-prefix metrics on watch fan-out and creates per second find them quickly, and the client library should make the correct recipe the only one on offer.

Ask: Most traffic is KeepAlives and reads from 50 k processes. How do you keep it off five replicas, and what do you do when one cell is not enough?

Good answers name: Client caches invalidated by the leader, plus a proxy tier that batches KeepAlives and serves reads and watches, Shard the namespace into independent cells by top-level prefix, Non-voting observers that serve reads and watches, More voting replicas.

Our pick: Three layers. The library caches values and the leader keeps a list of who caches what; a write is held until every cacher has dropped the old value, so cached reads stay consistent and about 99 % of reads never leave the process. About 25 proxies, each holding ~2 k client connections, send one batched KeepAlive a second covering all their live sessions (25 RPCs a second at the leader instead of 12.5 k), keep a shared cache, and register one watch per path on behalf of all their clients. A proxy that dies costs its clients a reconnect to another, well inside their lease, because sessions belong to the cell. Beyond that, the namespace is split into five-replica cells by top-level prefix through a small directory, and a new cell is added for blast radius or when one approaches ~2 k sustained writes a second or ~100 k sessions. The Chubby paper describes the same levers (longer leases, client caching, proxies, and partitioning the namespace) and reports that a cell’s RPC traffic is dominated by KeepAlives.

1. A proxy crashes. What do its 2 000 clients see?
   Their connections drop, and each library reconnects to another proxy from the directory with its existing session id, usually within a second. Their sessions are still alive at the cell, because the lease is 12 s and the proxy had renewed them a moment ago. They treat it like a fail-over and flush their caches, since an invalidation sent to the dead proxy may have been lost.
2. One flag is watched by all 50 k processes and changes. How long does the write take?
   The leader sends 25 invalidations to proxies, each fans out to its ~2 k clients, acknowledgements return within tens of milliseconds, and the write commits a few milliseconds later. If one client is partitioned, the write waits until that client’s lease runs out, up to 12 s. That bound is fine for configuration; for a value that changes every second it is a sign the value belongs in a different system.
3. A team needs 20 k writes a second for its locks. Do you scale the cell?
   No: they get their own cell, or several by sub-prefix, because one leader tops out around 10 k writes a second and their load should not raise everyone else’s commit latency. It is also worth asking why a lock service needs 20 k writes a second. That usually means a lock per request, where a database row or a partitioned owner would do better.
4. How do tokens stay monotonic when a prefix moves to a new cell?
   The new cell’s Raft index has nothing to do with the old one, so its tokens could be smaller. The fix is to put the prefix’s directory epoch in the token’s high bits and have the migration increment it. Every token issued after the move then compares above every token issued before it.
5. Why not one global cell across regions?
   Every commit would wait for a cross-region round trip, 50 to 150 ms, and a regional network problem could stall locks everywhere. Most locks protect regional resources, so they belong in regional cells. A small global cell with replicas in five regions is reasonable for the handful of truly global decisions, such as which region is primary for a database, where 100 ms writes are fine.

Ask: Everyone already runs Redis. Why not SET lock NX PX 30000 and be done?

Good answers name: Consensus-backed lock service with sessions and fencing tokens, One Redis key: SET NX PX with a random value, released by compare-and-delete, Redlock across five independent Redis masters, A row lock or conditional write in the resource’s own database.

Our pick: Classify every lock as efficiency or correctness before choosing. Efficiency locks (do not rebuild the same cache entry twice) can stay on a single Redis key with SET NX PX and a random value, released by a compare-and-delete script; an occasional double holder costs some work. Correctness locks (one writer per ledger shard, one leader per partition) go to the consensus-backed service and use fencing tokens checked by the resource. When the protected data lives in one database, skip the lock and use that database’s own conditional write. In the interview, say that Redlock’s safety is publicly disputed and why, rather than declaring a winner: the disagreement is about timing assumptions, and fencing tokens remove the dependence on them either way.

1. Walk me through how a Redis failover grants the lock twice.
   Client A sets the key on the primary and gets OK. Replication is asynchronous, so before the SET reaches the replica the primary dies. The replica is promoted without the key, client B sets it and also gets OK. Both now hold the lock, and Redis will never tell either of them.
2. Why release with a compare-and-delete script rather than DEL?
   Because by the time you release, your TTL may have run out and someone else may hold the key, and a plain DEL would delete their lock. The value is a random string you chose; a Lua script deletes the key only if it still holds your value, atomically.
3. Can you add fencing tokens to Redis?
   Partly. INCR on a counter next to the lock gives a number that goes up, but it lives in the same asynchronously replicated Redis, so a failover can hand out a number that was already issued. A token is only as monotonic as the store that issues it, which is why ours come from a consensus log.
4. The team says the lock only stops duplicate emails. Which design?
   An efficiency lock: a single Redis key is fine, because an occasional duplicate email is a nuisance rather than corruption. Better still, make sending idempotent with a key per message in the email service, and then the lock is purely an optimisation.