Sloppy quorums and hinted handoff through a network partition: an interactive simulator
In a leaderless store each key lives on N home nodes, and a write succeeds when W of them reply. When home nodes are unreachable, a strict quorum rejects the write. A sloppy quorum accepts it on the next healthy nodes around the ring instead, each keeping a hint that says whose copy it is, and hinted handoff delivers the copy to the owner once it is back. Writes stay available, at the cost of the overlap guarantee that made W + R > N mean consistent reads.
The playground follows one key on a five-node ring with N = 3, W = 2 and R = 2 through a partition. Step through the events and compare the two rings.
Client 1 writes v2. Under a strict quorum only A answers, so the write is rejected: the system chose consistency. Under a sloppy quorum D and E stand in for B and C and keep hints, so the write is accepted: the system chose availability.
Client 1 writes v2: A replied, B and C unreachable. 1 reply, 2 needed: reported as failed.
- Write of v2
- rejected
- Stale reads
- 0
- Hints held
- 0
Client 1 writes v2: A, D and E replied (hint for B, hint for C), B and C unreachable. 3 replies, 2 needed: acknowledged.
- Write of v2
- accepted
- Stale reads
- 0
- Hints held
- 2
What the two approaches trade
- A strict quorum keeps the overlap. Any 2 of the 3 home nodes share at least one node with any other 2, so a read sees the newest acknowledged write. The price is availability: during the partition client 1 cannot write at all, because only one home node can hear it.
- A sloppy quorum keeps writes available. D and E stand in for B and C, so the write is acknowledged with 2 or more replies. But those replies come from nodes the reader on the other side never asks, so a read of B and C returns the old value even though R + W > N.
- A failed write is not undone. Under the strict quorum A kept v2 although the client was told the write failed. Leaderless stores do not roll back partial writes; a later read that reaches A can return it. Clients must treat a failed write as unknown and retry it idempotently.
- Hints are a shortcut, not a guarantee. If the stand-in node dies before handing off, or the owner stays down past the hint window, the hint is lost and only read repair or anti-entropy repair with Merkle trees brings the replica up to date.
A typical configuration
# Cassandra-style settings for one keyspace
replication_factor = 3 # N: each key lives on 3 home nodes
write consistency = QUORUM # W = 2 of the home replicas
read consistency = QUORUM # R = 2, so W + R > N
hinted_handoff_enabled = true # a coordinator keeps hints for down replicas
max_hint_window = 3h # after that, stop hinting; repair must fix the replica
# hints are a delivery aid, not extra replicas: they do not count toward QUORUM in Cassandra,
# only toward ANY. Dynamo and Riak count stand-ins toward W: a sloppy quorum.In a system design interview
When you choose a Dynamo-style store, say what happens when replicas are down: “N = 3 with quorum reads and writes. For a shopping cart we prefer accepting writes during a partition, so we use a sloppy quorum with hinted handoff and merge conflicting versions on read. For anything that must not go backwards, such as a balance, we use strict quorums or a store with a leader, and accept rejected writes during a partition.” Expect the follow-ups: how conflicting writes are merged, how long hints are kept, and how replicas that missed hints are repaired.
Read the guide to quorums and leaderless replication or the CAP theorem, see quorum reads and writes and Merkle trees, or watch a strong candidate design a key-value store.
Questions people ask
What is hinted handoff?
A technique in leaderless stores such as Dynamo, Cassandra and Riak: when a replica is unreachable, another node accepts the write on its behalf and stores a hint naming the intended owner. When the owner comes back, the hint is replayed to it and deleted.
What is a sloppy quorum?
A quorum that counts replies from any healthy nodes, not only the key’s home replicas. Writes stay available during failures and partitions, but because the nodes that acknowledged a write may not be the ones a later read contacts, W + R > N no longer guarantees reading the latest write.
What is the difference between a strict and a sloppy quorum?
A strict quorum requires W replies from the N home replicas of a key, so every read quorum overlaps every write quorum. A sloppy quorum accepts replies from stand-in nodes, trading that overlap, and therefore consistency, for write availability.
Does Cassandra use sloppy quorums?
Not for QUORUM consistency: in Cassandra hints do not count toward QUORUM or ONE, only toward the ANY level. Hinted handoff there only speeds up recovery of replicas that missed writes. Amazon Dynamo and Riak do count stand-in nodes toward W.
What happens if a hint is lost?
If the node holding a hint fails, or the owner stays down longer than the hint window, the owner never receives that write through handoff. Read repair, which fixes stale replicas during reads, and anti-entropy repair, which compares replicas with Merkle trees, bring it up to date.
Why can a failed quorum write still appear later?
Leaderless stores do not roll back the replicas that did apply a write when the quorum fails. The client is told the write failed, but a later read that reaches one of those replicas can return it, so clients should treat failed writes as unknown and retry them idempotently.
When should you prefer a sloppy quorum?
When accepting writes matters more than reading the very latest value and conflicts can be merged, as in Amazon’s shopping cart, where losing an add-to-cart was worse than showing an older cart. For balances, inventory or uniqueness constraints, prefer strict quorums or a leader-based store.