Design an Object Storage Service

• Buckets: create, delete, configure: A bucket is a namespace with an owner and settings: versioning, lifecycle rules, access policy, default encryption. Written a few times a year, read on every request.
• PUT, GET, HEAD and DELETE an object by key: Keys are up to 1,024 bytes of UTF-8 in a flat namespace; the slashes in photos/2026/10/cat.jpg mean nothing to the store. Objects are immutable: an overwrite writes a new object and never edits bytes in place. GET supports byte ranges.
• LIST keys by prefix: In key order, with a delimiter to imitate folders and a continuation token for pages of 1,000. This one requirement decides how the index is partitioned.
• Objects from 0 bytes to 5 TB: A single PUT up to 5 GB; above that, multipart upload: up to 10,000 parts of 5 MB to 5 GB, sent in parallel, retried one by one, and stitched together by a final manifest.
• Versioning: When enabled, every overwrite and delete keeps the previous version, addressable by version id. A delete without a version id writes a delete marker instead of removing anything.
• Strong read-after-write consistency: Once a PUT or DELETE returns success, every later GET, HEAD and LIST in the region reflects it. Conditional writes (If-Match, If-None-Match) let clients build compare-and-swap on top.
• Lifecycle rules and storage classes: Per prefix and age: move objects to infrequent-access or archive classes, expire old versions, abort multipart uploads that were never finished.
• End-to-end integrity: The client sends a checksum with every upload; it is verified on arrival, stored with the bytes, and checked again on every read and every background scrub.
• Out of scope: Cross-region replication, event notifications, static website hosting, query in place, the CDN in front, and billing. Bucket names are globally unique, but the global name registry is treated as a lookup we call.

• Durability (11 nines per object per year): The design target AWS publishes for S3: store 10 M objects and expect to lose one every 10,000 years. Across our 100 trillion objects that still allows ~1,000 losses a year, so we design far beyond it on paper and spend the margin on correlated failures. Durability against cost is the hardest trade-off in the design.
• Availability (99.99 %, survives a data centre): Reads and writes keep working with one of the region’s three data centres completely down. That single requirement forces every consensus group and every erasure-code stripe to span all three.
• Consistency (linearisable per key): A successful write is visible to the next read anywhere in the region, LIST included. S3 has offered this since December 2020; before that, overwrites and deletes were eventually consistent and clients built workarounds.
• Latency (first byte p99 < 100 ms): An uncached GET is one metadata lookup (1–5 ms) plus one disk seek (~10 ms). Large-object throughput is bounded by the client’s parallelism, not by us.
• Request rate per prefix (3,500 writes/s · 5,500 reads/s, then split): The per-prefix rates S3 documents. Beyond them the index splits the hot range automatically; a single sequential hot point is the one case splitting cannot spread.
• Cost (≤ 1.6 raw bytes per stored byte): At 10 EB, every 0.1× of overhead is another 1 EB of disk, about 60,000 drives at 80 % fill. Archive classes must cost a fraction of standard.
• Scale (100 T objects · 10 EB · 20 M req/s peak): In one region. S3 has said publicly that it holds hundreds of trillions of objects. Object count stresses the index; bytes stress disks and repair.
• Security (every request authorised, encrypted at rest): Signed requests are checked against identity and bucket policy on every call. Each object is encrypted with its own data key, wrapped by a key-management service, so deleting the key makes leftover fragments unreadable.

• Requests per second: ~10 M/s avg · ~20 M/s peak. ~860 B requests a day ÷ 86,400 ≈ 10 M/s, with a 2× daily peak. Roughly 70 % GET and HEAD, 15 % PUT (multipart parts included), 3 % LIST. Every request touches the metadata index; only GET and PUT touch disks.
• Ingest bandwidth: ~230 GB/s in · ~350 GB/s to disk. 20 PB/day of new bytes ÷ 86,400 ≈ 230 GB/s, about 1.9 Tbit/s. Coding at 1.5× makes it ~350 GB/s of fragment writes, and two-thirds of those cross data-centre links, because every stripe is spread evenly over three data centres.
• Metadata index size: ~30 PB · ~90 PB replicated. 10^14 object versions × ~300 B per row (a ~100 B key, size, ETag, version id, storage class, user metadata, a wrapped data key, a pointer to the bytes) = 30 PB. Three replicas, one per data centre, = 90 PB on NVMe. The index of names is larger than most companies’ entire data, so it is a partitioned database in its own right, not a master’s in-memory table.
• Index partitions: ~3 M · ~7 req/s each on average. 30 PB ÷ 10 GB per partition = 3 M partitions, each a three-replica consensus group: ~9 M replicas on ~3,000 metadata servers of 30 TB, ~3,000 replicas each. 20 M req/s ÷ 3 M ≈ 7 req/s per partition at peak, against a capacity of thousands. The average is irrelevant: the problem is skew, one customer’s sequential keys putting 20 k writes/s on one partition.
• Drives: ~970 k drives · ~20 k servers. 10 EB × 1.55 blended overhead (1.5× for coded data, plus recent small objects still at 3× and dead bytes awaiting compaction) = 15.5 EB. Filled to 80 %, that is ~19.4 EB of disk ÷ 20 TB = ~970 k drives, ~20 k servers of 48 drives.
• Drive failures: ~53 a day. 970 k drives × 2 % annual failure rate ≈ 19,400 a year ≈ 53/day, one every 27 minutes, plus a server or two a day lost to boards and power supplies. Failure is not an incident here; it is the background workload.
• Repair bandwidth: ~120 GB/s, all day. A dead 20 TB drive at 80 % holds ~16 TB of fragments, and rebuilding each one reads 12 others: 192 TB of reads per drive. 53 a day × 192 TB ≈ 10 PB/day ≈ 118 GB/s, ~0.9 Tbit/s, more than half of it across data centres. Spread over 20 k servers it is ~6 MB/s each, affordable only because the rebuild is spread out.
• Copies versus coding: ~900 k more drives for 3×. Three copies of 10 EB is 30 EB against 15.5 EB coded: 14.5 EB more, ÷ 0.8 fill ÷ 20 TB ≈ 900 k extra drives, nearly doubling the fleet. The choice of code is not a detail; it decides whether the service makes money.

• IAM & KMS (policy · data keys): Identity and policy evaluation, and the key-management service that wraps data keys. Operated by other teams and called on every request, so the front end caches derived signing keys and policy decisions for seconds. Object data keys are generated locally and wrapped by a cached bucket key, so KMS sees a call per bucket every few minutes, not one per PUT.
• Bucket registry (small · consensus-replicated): Bucket name → id, owner, versioning state, lifecycle rules, policy and default encryption. Hundreds of millions of rows, read on every request and written rarely, so front ends cache it for seconds. That cache makes bucket configuration the one eventually consistent thing in the system, which S3 also documents for its own bucket settings.
• Partition map (ranges → replica groups): Knows which contiguous key range belongs to which index partition, and where its three replicas and current leader are. Front ends cache it and are corrected by a replica that answers "not my range", so a stale map costs a retry, never a wrong answer. It also watches load and size per partition and orders splits, merges and moves, which is how hot prefixes are absorbed.
• Index change log (per partition, commit order): Every committed mutation, streamed from each partition’s replicated log in commit order: put, delete, transition, multipart complete. GC uses it to keep exact live-byte counts per extent; lifecycle, billing and notifications consume it too. An event exists if and only if the commit did, because it is the commit log itself rather than a second write.
• Lifecycle engine (daily scan per partition): Scans the index partitions of buckets that have rules, in key order, once a day, and applies them: transition to infrequent access or archive by age, expire noncurrent versions, remove expired delete markers, abort stale multipart uploads. Every action is a conditional metadata commit, so a client overwrite during a transition simply wins.
• Client / SDK (signed HTTPS): Applications, data pipelines and backup tools speaking the S3 REST API through an SDK. The SDK signs every request, computes a CRC of the payload before sending, switches to multipart above ~100 MB, and parallelises big downloads with byte-range GETs. Most "storage is slow" complaints are clients making one request at a time.
• Front end (auth · route · encode): A stateless fleet in every data centre behind DNS and load balancers. It authenticates and authorises each request, finds the key’s index partition, streams bytes to and from storage (erasure coding and decoding on the fly), and commits metadata last. It stores nothing, so losing a front end loses only the requests in flight.
• Metadata index (range-partitioned · Raft ×3 DCs): One row per object version, keyed (bucket, key, version descending) and kept sorted, so LIST by prefix is a range scan and the latest version is the first row at its key. About 3 M partitions of ~10 GB, each a three-replica consensus group with one replica per data centre. The leader’s commit of a row is the moment an object exists, and leader-lease reads make every GET and LIST see the latest commit.
• GC & compaction: Keeps a running count of live bytes per extent from the change log, frees extents with no live bytes for a 48-hour grace period, and compacts mostly dead small-object extents by copying live objects out and repointing their rows with compare-and-set. A weekly reconciliation against a full index scan catches drift. It is deliberately slow and conservative, because a GC bug is a data-loss bug.
• Archive tier (dense, mostly spun down): The cold storage class: high-density drives kept mostly powered down, written in large sequential batches and read back only through an explicit restore that takes minutes to hours. The same failure-domain rules as standard apply; the saving comes from cheaper media and almost no provisioned IOPS, roughly a tenth of the cost per byte.
• Fragment cache (SSD · immutable data): An SSD cache of byte ranges keyed by (extent id, offset), in front of the disks. Extents never change after they are written, so a cached range can never be stale and there is nothing to invalidate. It absorbs release-day spikes on popular objects and protects an HDD fleet that offers only ~7 IOPS per TB.
• Replicated extents (3× · one copy per DC): Append-only extents of up to 1 GB that hold small objects (under 1 MB) packed Haystack style: header, bytes, checksum. Each has three replicas, one per data centre, and a single writer at a time. An extent is sealed when full or when a replica falters, and sealed extents that stay cold for 30 days are re-encoded to 12+6 in the background, as Facebook’s f4 and Microsoft’s Azure Storage describe.
• Extent manager (extent → disks · placement): Maps each extent id to its scheme and the disks holding its 3 replicas or 18 fragments: billions of rows, a few TB, partitioned and consensus-replicated. It allocates new extents by free space under failure-domain rules, takes heartbeats from every storage server, and drives repair and rebalancing. Object rows name extents, never disks, so moving data never touches the 30 PB index.
• Coded extents (RS 12+6 · 6 fragments per DC): Large objects stream through an encoder in 1 MB cells: 12 data and 6 parity fragments per stripe, on 18 servers, six in each data centre, up to 768 MB of data per extent. That is 1.5× storage with any six fragments losable, so a whole data centre can vanish. The code is systematic, so a healthy read returns the original bytes with no decoding.
• Scrub & repair (priority by spares left): Rebuilds lost fragments from 12 survivors, ordered by redundancy remaining so a stripe at its last spare goes first, and spread across thousands of disks so a 20 TB drive is re-protected in about an hour instead of the 28 it takes to copy onto one spare. The scrubber reads every disk end to end every two weeks and checks every cell’s checksum, because disks do not report all their errors.

1. PUT a small object: bytes first, commit last: The core write. The bytes are durable in three data centres before any metadata row exists, and the object becomes visible at exactly one moment: the consensus commit of its index row. A crash anywhere before that leaves garbage, never a half-written object.
   1. Client sends a signed PUT carrying the bytes and a CRC32C of them.
   2. Front end verifies the signature and the caller’s permission, and loads the bucket’s configuration.
   3. Front end appends the bytes to an open replicated extent, one replica in each data centre.
   4. Front end finds the key’s partition and commits the new version row through its leader.
   5. Front end returns success with the ETag and version id.
   6. The committed mutation is streamed to the partition’s change log.
   7. GC adds the new bytes to the extent’s live count; bytes of any version this PUT replaced become dead.
2. GET a large object, strongly consistent: The read path for a 2 GB video, fetched with parallel ranged GETs. The metadata read is linearisable, so it always returns the latest committed version; the data read can be cached freely, because extents never change.
   1. Client issues parallel ranged GETs for 64 MB slices of the object.
   2. Front end authorises the request and reads the latest version’s row from the partition leader.
   3. Front end maps the byte range to coded extents and looks up where their fragments live.
   4. Front end checks the fragment cache for the range.
   5. On a miss, front end reads the 12 data fragments in parallel and hedges the slowest with a parity fragment.
   6. Front end verifies every cell’s checksum and streams the range to the client.
3. Upload a 2 TB object in parts: A single 2 TB PUT would hold one connection for hours and restart from zero on any error. Multipart turns it into thousands of independent, parallel, retryable part uploads, and the object appears atomically only when one manifest commit lands.
   1. Client starts a multipart upload and gets an upload id.
   2. Client uploads 8,000 parts of 250 MB, 32 at a time, each with its own checksum.
   3. Front end streams each part through an encoder into coded extents spread over three data centres.
   4. Each finished part is recorded under the upload with its size, ETag and layout.
   5. Client completes the upload with the ordered list of part numbers and ETags.
   6. Index validates the list and commits one version row that points at the parts: the object appears atomically.
   7. Uploads that are never completed are aborted by a lifecycle rule and their bytes collected.
4. A disk dies and repair races the clock: With about a million drives, ~53 fail every day. Durability is decided by how quickly the system notices, and by rebuilding the stripes with the least redundancy left first, in parallel across the fleet.
   1. A storage server reports a failed disk, or stops sending heartbeats altogether.
   2. Extent manager lists every extent with a fragment on that disk and ranks them by spare fragments left.
   3. Repair workers read 12 surviving fragments of each extent, decode the missing one, and write it to a new disk.
   4. Extent manager records the fragment’s new home; the object index is not touched.
   5. Meanwhile, the scrubber finds a fragment whose bytes no longer match their checksum.
   6. Small-object extents that lost a replica are copied from a surviving one.
5. Delete, tiering and garbage collection: Nothing is deleted in place. A delete in a versioned bucket writes a marker; lifecycle rules later expire old versions and move cold data to a cheaper tier; space comes back only when GC finds bytes that nothing references.
   1. Client deletes a key in a versioned bucket.
   2. Lifecycle engine scans the partitions of buckets with rules and evaluates each row against them.
   3. A cold object is copied into the archive tier, then its row is switched to the new layout in one conditional commit.
   4. Each commit’s change-log entry tells GC exactly which bytes in which extents just stopped being referenced.
   5. Extents with no live bytes are freed after a grace period; mostly dead small-object extents are compacted.

Ask: Why not store each object as a file on a big distributed file system, with one master keeping the namespace?

Good answers name: Separate planes: a partitioned index for names, immutable extents for bytes, Single-master namespace (GFS or HDFS style), Metadata stored with the data and located by hashing (Ceph RADOS style), One big database holding bytes and metadata.

Our pick: Two planes joined by one pointer. The metadata index stores, per object version, its size, ETag, storage class, wrapped data key and a layout: (extent id, offset, length) for a small object, or the list of parts and extents for a large one. The data plane stores immutable extents and knows nothing about buckets or keys beyond a self-describing header on each object, which is what lets the index be rebuilt from the disks in a disaster. An extent manager in between maps extent ids to disks, so data can move without the index noticing. Writes go data first and metadata last; reads go metadata first and data second. Google made the same move when Colossus, the successor to GFS, put file system metadata in Bigtable.

1. Every GET now costs a metadata lookup before the first byte. Is that a latency problem?
   It is one round trip to a partition leader serving from memory or NVMe, about 1 to 5 ms, against ~10 ms for the disk seek that follows. The leader keeps its hot rows in memory, and that cache needs no invalidation because the leader is where writes land. For a very hot key, followers serve reads after checking the leader’s commit index. The lookup is the cheap part of an uncached GET.
2. You lose part of the index. Are those objects gone?
   Not if the data plane is self-describing. Each small object’s header carries bucket, key, version and checksum, and each coded extent records the upload and part it belongs to, so a recovery job can scan extents and rebuild missing rows. It is slow and a last resort, behind three replicas across data centres and point-in-time snapshots of the index. Keeping enough in the data plane to rebuild the index turns an index bug into an outage rather than a loss.
3. Why insist on immutable data? Customers overwrite objects all the time.
   An overwrite writes new bytes and moves the pointer in one metadata commit; the old bytes become garbage. In-place updates would need locking across 18 fragments, a read-modify-write of parity, and cache invalidation on the data plane. Immutability buys torn-write-free updates, caches that cannot be stale, and repair that copies fragments at any time without coordinating with writers. The price is GC, which failed uploads require anyway.
4. How small does an object have to be before per-object overhead dominates?
   On disk, a file per object costs an inode, a directory entry and extra seeks per read once metadata no longer fits in memory, which is what Facebook measured before building Haystack. Packing objects into 1 GB extents makes a small write one append and a small read one seek. In the index the ~300 B row is fixed, so a 10 KB object pays 3 % overhead in flash and a 1 KB object 30 %. That is why S3 bills infrequent-access objects as at least 128 KB.
5. How would you know the two planes agree?
   Three continuous checks. A weekly reconciliation recomputes every extent’s live-byte count from a full index scan, and any mismatch stops GC for that extent and pages someone. Sampled audits read random index rows and verify the extent, offset and checksum they point at. Change-log sequence numbers prove that no consumer skipped a mutation. Orphans found this way are freed only after the grace period; missing data is an incident.

Ask: How do you split an index of 100 trillion keys across machines so that LIST works and one hot prefix cannot melt a partition?

Good answers name: Range partitions split by size and by load, each a Raft group across three data centres, Hash partitioning with consistent hashing, Hash prefixes added to key names by the customer, Range partitions split only by size.

Our pick: Rows are keyed (bucket id, key, version id descending), and the keyspace is cut into ranges of up to 10 GB, about 3 M partitions, each a Raft group with one replica per data centre so that a majority survives losing any one. The partition map tracks load per partition from sampled request keys; above ~3,500 writes or 5,500 reads a second sustained, or a CPU threshold, it splits the partition at the load median rather than the size median, and moves the busier half to a lightly loaded server. Split points never fall inside one key’s rows, so a key’s versions and multipart parts always commit in one partition. While a split is in flight the partition sheds excess load with 503 Slow Down, and SDKs back off with jitter. For monotonic keys, splitting at the tail leaves a small, hot range on its own server; past that server’s capacity the honest answer is the one S3 documents, spread the writes across prefixes, because one sequential point in an ordered index is one machine.

1. A customer writes 20,000 objects a second, all named logs/<timestamp>. What exactly happens?
   All of those commits go to the partition owning the tail of logs/. It saturates and returns 503 Slow Down for the excess, and the SDKs back off. The partition map sees sustained overload and splits at the load median, which for timestamps is very close to the end, so the new right-hand range holds almost no data but takes all the writes, on a server of its own. That buys a few times the rate. To sustain 20 k a second the customer spreads keys over, say, eight prefixes, logs/0/ to logs/7/, which is what S3’s documentation recommends for parallelism.
2. Why not split ahead of time, before the 503s?
   When load is predictable we do: a prefix whose load is growing steadily, or a customer who warns us before a launch, gets pre-split into empty ranges spread over servers. Splitting a small, quiet range is a metadata change, while splitting under load means moving a hot replica. The reactive path exists for everything nobody predicted.
3. LIST on a prefix with 100 million keys: how long does it take, and is it consistent across pages?
   Each page of 1,000 keys is one range scan on one partition leader, a few milliseconds, so 100 k pages in sequence take minutes; clients in a hurry list sub-prefixes in parallel. Each page is consistent as of its read and the continuation token is the last key returned, so the listing is not a snapshot: a key added behind the cursor is missed and one added ahead is seen. A point-in-time listing of a whole bucket is an inventory job, not LIST.
4. Three million Raft groups. Isn’t the heartbeat traffic alone a problem?
   It would be at one heartbeat per group every 100 ms. Most partitions see a few requests a second, so idle groups go quiet until there is work, and a server sends one combined heartbeat per peer for all the groups they share. CockroachDB and TiKV do both, for the same reason. What remains is proportional to active partitions and servers, not to the 3 M total.
5. How do you move a partition without a blip?
   Add a replica on the target server as a non-voting learner, stream it a snapshot plus the log tail, then change membership in one consensus step and remove the old replica. Leadership is transferred explicitly to a caught-up replica, so the unavailable window is one leader handover, not an election timeout. Front ends learn the new location from a "not my range" rejection and a map refresh, never from a timeout.

Ask: Why not keep three copies of everything? And where does eleven nines actually come from?

Good answers name: RS 12+6, six fragments in each of three data centres, 3× replication, one copy per data centre, RS 10+4 inside one data centre, RS 10+4 per site plus an XOR across sites, as in Facebook’s f4.

Our pick: Objects under 1 MB go into 3× replicated extents, one replica per data centre; once an extent is sealed and has been cold for 30 days it is re-encoded to 12+6 in the background, so small objects also end near 1.5×. Larger objects are cut into 1 MB cells and encoded inline into RS 12+6 stripes, six fragments per data centre on 18 different servers: 1.5×, tolerating any six losses. Durability comes from repair speed as much as from the code: fragments are declustered across the fleet, so a dead 20 TB disk is rebuilt by thousands of disks in about an hour rather than by one replacement in 28. With a 2 % annual failure rate and one-hour repair, the independent-failure model gives about 30 nines for 12+6 and about 12 for three copies; three copies with day-long rebuilds give about 10. Those numbers are not the point: real losses come from correlated failures (a rack’s power, a bad batch of drives, a software bug, an operator), so failure domains, staged deployments and delete protection are where the eleven nines are really earned. AWS publishes eleven nines as S3’s design target; this is our design for meeting it, not a description of theirs.

1. Walk me through the durability arithmetic for three copies.
   Take a 2 % annual failure rate and a one-day rebuild. Each disk is down about 0.02 × 1/365 ≈ 5.5 × 10^-5 of the time. Data is lost if, after a first failure, both other copies fail inside the window: roughly 3 × 0.02 × (5.5 × 10^-5)² ≈ 2 × 10^-10 a year, just under ten nines. Cut repair to an hour and that improves by 24², about 600×, to over twelve nines. Each extra tolerated loss multiplies by another factor of about 1/(failure rate × repair time), which is why both parity and repair speed matter so much.
2. A whole data centre loses power for six hours. What happens?
   Partitions whose leaders were there elect new leaders in the other two sites once the old leases expire, a few seconds of 503s for those ranges, and metadata writes carry on with two of three replicas. Reads of coded objects need all 12 surviving fragments and decode the missing data cells, so latency and CPU rise and one slow disk can fail a read until the site returns; nothing is lost. New writes place 9 + 9 fragments in the two live sites, still tolerating six disk losses, and are rebalanced later. Repair does not rebuild the dark site: recreating ~5 EB for a six-hour outage would be absurd, so it fixes only stripes that also lost a fragment elsewhere.
3. Rebuilding reads 12 fragments per lost one, mostly across sites. Can you cut that?
   Yes, with a local reconstruction code. Azure published LRC 12+2+2: the data fragments form two local groups, each with its own parity, plus two global parities, so the common single-fragment repair reads 6 fragments instead of 12, at 1.33×. Adapting it to three sites means local groups that sit inside one data centre and enough global parity to survive losing a site, which costs some overhead beyond 1.5× and a more complex code. With ~120 GB/s of repair reads, halving them may well be worth it.
4. Why not erasure code small objects inline too?
   A 10 KB object split twelve ways is twelve tiny writes and, worse, twelve seeks to read back, on drives that give ~150 IOPS each. Replicating it costs 3× the bytes but one append and one seek. Small objects are ~99 % of objects but only ~10 % of bytes, so paying 3× on them while they are hot is cheap. Once their extent is sealed and cold we encode the whole 1 GB extent, and a read of one object still touches only the one data fragment it lives in, because the code is systematic.
5. How would you know you are actually meeting eleven nines?
   You cannot measure it directly; losses at that rate are too rare to count. Measure what drives it: time from failure to full redundancy, the number of stripes by spare fragments left (alerting on any at its last spare), failure rates by drive model and batch, scrub findings per petabyte, and how correlated failures are across racks. Feed those into the model continuously. Then also count what the model ignores: every incident in which a bug or an operator could have deleted data.

Ask: How do you guarantee that a GET right after a successful PUT returns the new object, and that a PUT which fails halfway leaves nothing behind?

Good answers name: The metadata commit as the linearisation point, leader-lease reads, caches only with invalidation, Eventually consistent replicas behind TTL caches, Leaderless quorums (Dynamo-style N, R, W), Metadata first, marked pending, flipped once the bytes land.

Our pick: A PUT streams bytes to storage and waits until they are durable (three replicas fsynced, or all 18 fragments), then sends one commit to the partition leader, which replicates the row to a majority across data centres and applies it; only then does the client get 200. A GET reads the latest version from the leader, which holds a lease of a few seconds during which no other replica can be elected, so it answers from its own state and is still linearisable; hot keys can also be read from followers after a ReadIndex check of the leader’s commit position. Any metadata cache outside the leader is lease-based and invalidated by the leader before a write to that key is acknowledged, and if a cache node is unreachable the write waits out its lease. Conditional PUTs, If-None-Match: * for create-only and If-Match: <ETag> for compare-and-swap, are checked at the leader inside the commit; S3 added both in 2024. Uncommitted bytes are collected once their write lease and a 48-hour grace period have passed. S3 moved to strong read-after-write consistency for all GET, PUT and LIST operations in December 2020, at no extra cost; the mechanism described here is our design.

1. The front end writes all the bytes, then crashes before the commit. What does the client see, and what happens to the bytes?
   The client sees a reset or a timeout, never a 200, so it retries; the retry writes new bytes and commits a new version, and the client cannot tell. The first copy sits in an extent with no row pointing at it. GC never counted those bytes as live, so a coded extent written only by that upload is freed once its write lease and the 48-hour grace period expire, and a shared small-object extent just carries a few dead objects until compaction.
2. Two clients PUT the same key at the same moment. Who wins?
   Whichever commit the partition leader orders second, and every reader agrees on that order because it is the leader’s log, not a client timestamp. Both writes succeed and, in a versioned bucket, both versions exist. If the application needs "only one of us may create this", it sends If-None-Match: * and the loser gets 412 Precondition Failed; for read-modify-write it sends If-Match with the ETag it read. Both are cheap because the check runs where the order is decided.
3. The leader in one data centre is cut off but still thinks it is leader. Can it serve a stale read?
   Not if leases are done right. A leader serves reads only while its lease is valid, and no new leader may be elected until that lease has expired, with a margin for clock drift, so two replicas never both believe they may serve reads. The price is a few seconds of unavailability for that partition after the cut, and a dependence on clocks drifting less than the margin, which we monitor as a hard alert.
4. Why not cache metadata at the front ends with a one-second TTL? It would cut leader load a lot.
   Because one second of staleness is exactly the bug strong consistency exists to remove: a pipeline writes a manifest and the next step reads the old one. If leader load becomes a problem for a hot key, the fixes keep consistency: follower reads after a ReadIndex check, or leased cache entries that the leader revokes before acknowledging a write. Data, unlike metadata, is cached freely, because an extent never changes.
5. Is LIST strongly consistent too? It spans partitions.
   Each page is read from the leaders of the partitions it covers, so any PUT that returned 200 before the LIST started appears in it, and any delete that returned before does not. A LIST is not a snapshot across pages, which is fine for the contract "what has committed is visible". Delete markers are the practical problem: a prefix with millions of markers and few live keys makes each page scan many rows to return a few, which is why there is a lifecycle rule to remove expired delete markers.

Ask: Where do an extent’s fragments go, how do you find them again, and what happens when a disk dies?

Good answers name: An extent map with weighted, failure-domain-aware allocation, Computed placement, like Ceph’s CRUSH, Disk locations stored in each object’s index row, Copy a dead disk onto a hot spare.

Our pick: The extent manager owns placement. A new extent gets 3 replicas or 18 fragments chosen by weighted random sampling over disks with free space, constrained to one fragment per server, at most two per rack and six per data centre, which spreads every extent’s siblings across the fleet. Storage servers heartbeat every 5 s; a failed disk is declared at once, a silent server after ~15 minutes, because most silences are reboots and re-protecting 768 TB for a reboot would cost ~9 PB of reads. Repair is a priority queue ordered by spare fragments left, so a stripe at its last spare preempts everything, and it is rate-limited per disk so foreground latency does not move. Rebuilt fragments go to new disks and the extent map row is updated with compare-and-set on a generation number. The same machinery rebalances: new racks are filled gradually by moving sealed, cold extents onto them, rather than by sending them every new write, because new data is the hottest data and a fresh rack would become an IOPS hot spot.

1. A whole rack of servers loses power. How is that different from a dead disk?
   It is worse, because hundreds of disks failed together, which is the correlation the independent-failure model ignores. The placement rule already limits any stripe to two fragments per rack, so a rack loss costs each affected stripe at most two of its six spares. The rack is treated like a silent server: unavailable at once, rebuilt only if it does not return within the window or if a stripe is now short. That is why the per-rack limit is a rule and not left to chance.
2. Two repair workers both rebuild fragment 7 of the same extent. What stops a mess?
   The extent map row has a generation number. Each worker reads it, rebuilds, and commits the new location with compare-and-set on that generation; the second commit fails and that worker deletes its orphaned fragment. Without the check the map could point at one copy while the other is counted as live, or two fragments could claim the same slot and quietly break the stripe.
3. How do you add 20 PB of new capacity without hurting anything?
   Give the new disks a higher, capped weight for new extents, and move existing sealed extents onto them at a controlled rate. If every new write went to the new rack, it would serve the newest, hottest data from too few spindles at ~7 IOPS per TB. Moving cold extents onto new disks and letting new writes spread normally keeps both capacity and IOPS balanced. Retiring old drives is the same process in reverse.
4. Drives keep getting bigger. Why is that a durability problem?
   A 30 TB drive holds 50 % more fragments than a 20 TB one with about the same bandwidth, so copying it takes longer and its loss is a bigger repair job. Declustering hides most of that, because the rebuild is spread over the fleet, but repair bandwidth per failure still grows with capacity. Bigger drives push towards faster repair, more parity or codes that read less per repair, and towards a reserve of repair bandwidth that foreground traffic never uses.
5. What do you alert on?
   Any stripe at its last spare fragment; the age of the oldest item in the repair queue; time from failure to full redundancy at p99; repair throughput against the failure rate, where a backlog growing for an hour pages someone; and drive failures per model and batch per week, the early warning of a correlated failure. Scrub coverage too: every disk scrubbed within its two-week cycle, with findings per petabyte trended.

Ask: Nothing is modified in place. So how do delete, versioning and lifecycle work, and how do you get the space back safely?

Good answers name: Delete markers and row removal in the index, GC from the change log with a grace period, Delete the bytes synchronously, Periodic mark-and-sweep over the whole index only, Expire bytes by TTL inside the storage layer.

Our pick: In an unversioned bucket, DELETE removes the row; in a versioned one it commits a delete marker as the new latest version, and DELETE with a version id removes that version permanently. Each commit’s change-log entry carries the extents and byte counts it released, so GC keeps exact live-byte counts: a coded extent at zero live bytes for 48 hours is freed, and a small-object extent below ~30 % live is compacted by copying its live objects to a new extent and repointing their rows with compare-and-set. Freed extents stay unreferenced for a few more days before their space is reused, so a GC bug can be rolled back, and a weekly reconciliation recomputes every count from the index. The lifecycle engine scans the partitions of buckets with rules daily and acts through the same conditional commits: transition to infrequent access (same media, retrieval fee) or archive (dense, spun-down drives, restore in minutes to hours, about a tenth of the cost), expire noncurrent versions, remove expired delete markers, abort stale multipart uploads. For privacy, each object’s wrapped data key lives in its row, so removing the row makes leftover fragments unreadable long before GC reaches them.

1. A customer deletes 100 million objects with one lifecycle rule. What load does that create?
   One hundred million index commits across whichever partitions hold that prefix, paced per partition by the lifecycle engine so they never compete with foreground traffic, finishing over hours rather than minutes. GC then sees a wave of released bytes in the change log. Fully dead coded extents are cheap to free; partly dead small-object extents queue for compaction, which is the expensive part and is rate-limited too. The customer sees objects disappear from LIST as each commit lands.
2. Versioning is on and a script overwrites every object with garbage. Can you recover?
   Yes, that is what versioning is for: every previous version is still there, and restoring a prefix is a metadata operation per key that copies or re-points it to its last good version. The danger is a rule that expires noncurrent versions after a few days, which turns recovery into a race. For data that must survive deliberate deletion, write-once retention that even the owner cannot shorten is the stronger tool. Durability is not a backup either way.
3. Why a 48-hour grace period before freeing an extent?
   Because "no row points at it" is briefly true of extents that are about to be referenced: a slow PUT between its last fragment write and its commit, or a multipart part written but not yet recorded. Write leases fence the first case, since a commit after the lease expired is rejected, and the grace period covers clock skew, retries and bugs nobody has thought of. Two days of extra garbage is a fraction of a percent of capacity; freeing one live extent is data loss.
4. How does a transition to archive stay correct if the client overwrites the key mid-copy?
   The lifecycle engine copies the bytes into the archive tier first, then commits the new layout only if the row still points at the old one. If the client overwrote the key in the meantime, the condition fails, the copy is discarded as garbage, and the new version starts its own lifecycle clock. Nothing ever holds a lock across the copy.
5. A user asks for an object to be erased under GDPR. When are the bytes really gone?
   The name goes immediately with the delete commit, and the wrapped data key goes with the row, so from that moment the fragments are ciphertext nobody can decrypt. The fragments themselves are freed or compacted within days. Old versions and index snapshots fall under the same policy, so the documented answer is "unreadable immediately, physically removed within N days", and both halves are worth saying in the interview.