Design an Object Storage Service

An S3-like store for objects from bytes to terabytes, with eleven nines of durability and strong read-after-write consistency at exabyte scale.

Last updated 2026-10-26. Difficulty: hard. Patterns: erasure-coding, range-partitioning, durability, strong-consistency, object-storage. Reported at Amazon, Google, Microsoft, Meta, Dropbox, Databricks.

Functional requirements

• Buckets: create, delete, configure. A bucket is a namespace with an owner and settings: versioning, lifecycle rules, access policy, default encryption. Written a few times a year, read on every request.
• PUT, GET, HEAD and DELETE an object by key. Keys are up to 1,024 bytes of UTF-8 in a flat namespace; the slashes in photos/2026/10/cat.jpg mean nothing to the store. Objects are immutable: an overwrite writes a new object and never edits bytes in place. GET supports byte ranges.
• LIST keys by prefix. In key order, with a delimiter to imitate folders and a continuation token for pages of 1,000. This one requirement decides how the index is partitioned.
• Objects from 0 bytes to 5 TB. A single PUT up to 5 GB; above that, multipart upload: up to 10,000 parts of 5 MB to 5 GB, sent in parallel, retried one by one, and stitched together by a final manifest.
• Versioning. When enabled, every overwrite and delete keeps the previous version, addressable by version id. A delete without a version id writes a delete marker instead of removing anything.
• Strong read-after-write consistency. Once a PUT or DELETE returns success, every later GET, HEAD and LIST in the region reflects it. Conditional writes (If-Match, If-None-Match) let clients build compare-and-swap on top.
• Lifecycle rules and storage classes. Per prefix and age: move objects to infrequent-access or archive classes, expire old versions, abort multipart uploads that were never finished.
• End-to-end integrity. The client sends a checksum with every upload; it is verified on arrival, stored with the bytes, and checked again on every read and every background scrub.
• Out of scope. Cross-region replication, event notifications, static website hosting, query in place, the CDN in front, and billing. Bucket names are globally unique, but the global name registry is treated as a lookup we call.

Non-functional requirements

• Durability (11 nines per object per year). The design target AWS publishes for S3: store 10 M objects and expect to lose one every 10,000 years. Across our 100 trillion objects that still allows ~1,000 losses a year, so we design far beyond it on paper and spend the margin on correlated failures. Durability against cost is the hardest trade-off in the design.
• Availability (99.99 %, survives a data centre). Reads and writes keep working with one of the region’s three data centres completely down. That single requirement forces every consensus group and every erasure-code stripe to span all three.
• Consistency (linearisable per key). A successful write is visible to the next read anywhere in the region, LIST included. S3 has offered this since December 2020; before that, overwrites and deletes were eventually consistent and clients built workarounds.
• Latency (first byte p99 < 100 ms). An uncached GET is one metadata lookup (1–5 ms) plus one disk seek (~10 ms). Large-object throughput is bounded by the client’s parallelism, not by us.
• Request rate per prefix (3,500 writes/s · 5,500 reads/s, then split). The per-prefix rates S3 documents. Beyond them the index splits the hot range automatically; a single sequential hot point is the one case splitting cannot spread.
• Cost (≤ 1.6 raw bytes per stored byte). At 10 EB, every 0.1× of overhead is another 1 EB of disk, about 60,000 drives at 80 % fill. Archive classes must cost a fraction of standard.
• Scale (100 T objects · 10 EB · 20 M req/s peak). In one region. S3 has said publicly that it holds hundreds of trillions of objects. Object count stresses the index; bytes stress disks and repair.
• Security (every request authorised, encrypted at rest). Signed requests are checked against identity and bucket policy on every call. Each object is encrypted with its own data key, wrapped by a key-management service, so deleting the key makes leftover fragments unreadable.

Back-of-envelope estimates

• Requests per second: ~10 M/s avg · ~20 M/s peak. ~860 B requests a day ÷ 86,400 ≈ 10 M/s, with a 2× daily peak. Roughly 70 % GET and HEAD, 15 % PUT (multipart parts included), 3 % LIST. Every request touches the metadata index; only GET and PUT touch disks.
• Ingest bandwidth: ~230 GB/s in · ~350 GB/s to disk. 20 PB/day of new bytes ÷ 86,400 ≈ 230 GB/s, about 1.9 Tbit/s. Coding at 1.5× makes it ~350 GB/s of fragment writes, and two-thirds of those cross data-centre links, because every stripe is spread evenly over three data centres.
• Metadata index size: ~30 PB · ~90 PB replicated. 10^14 object versions × ~300 B per row (a ~100 B key, size, ETag, version id, storage class, user metadata, a wrapped data key, a pointer to the bytes) = 30 PB. Three replicas, one per data centre, = 90 PB on NVMe. The index of names is larger than most companies’ entire data, so it is a partitioned database in its own right, not a master’s in-memory table.
• Index partitions: ~3 M · ~7 req/s each on average. 30 PB ÷ 10 GB per partition = 3 M partitions, each a three-replica consensus group: ~9 M replicas on ~3,000 metadata servers of 30 TB, ~3,000 replicas each. 20 M req/s ÷ 3 M ≈ 7 req/s per partition at peak, against a capacity of thousands. The average is irrelevant: the problem is skew, one customer’s sequential keys putting 20 k writes/s on one partition.
• Drives: ~970 k drives · ~20 k servers. 10 EB × 1.55 blended overhead (1.5× for coded data, plus recent small objects still at 3× and dead bytes awaiting compaction) = 15.5 EB. Filled to 80 %, that is ~19.4 EB of disk ÷ 20 TB = ~970 k drives, ~20 k servers of 48 drives.
• Drive failures: ~53 a day. 970 k drives × 2 % annual failure rate ≈ 19,400 a year ≈ 53/day, one every 27 minutes, plus a server or two a day lost to boards and power supplies. Failure is not an incident here; it is the background workload.
• Repair bandwidth: ~120 GB/s, all day. A dead 20 TB drive at 80 % holds ~16 TB of fragments, and rebuilding each one reads 12 others: 192 TB of reads per drive. 53 a day × 192 TB ≈ 10 PB/day ≈ 118 GB/s, ~0.9 Tbit/s, more than half of it across data centres. Spread over 20 k servers it is ~6 MB/s each, affordable only because the rebuild is spread out.
• Copies versus coding: ~900 k more drives for 3×. Three copies of 10 EB is 30 EB against 15.5 EB coded: 14.5 EB more, ÷ 0.8 fill ÷ 20 TB ≈ 900 k extra drives, nearly doubling the fleet. The choice of code is not a detail; it decides whether the service makes money.

Components

• IAM & KMS (policy · data keys): Identity and policy evaluation, and the key-management service that wraps data keys. Operated by other teams and called on every request, so the front end caches derived signing keys and policy decisions for seconds. Object data keys are generated locally and wrapped by a cached bucket key, so KMS sees a call per bucket every few minutes, not one per PUT.
• Bucket registry (small · consensus-replicated): Bucket name → id, owner, versioning state, lifecycle rules, policy and default encryption. Hundreds of millions of rows, read on every request and written rarely, so front ends cache it for seconds. That cache makes bucket configuration the one eventually consistent thing in the system, which S3 also documents for its own bucket settings.
• Partition map (ranges → replica groups): Knows which contiguous key range belongs to which index partition, and where its three replicas and current leader are. Front ends cache it and are corrected by a replica that answers "not my range", so a stale map costs a retry, never a wrong answer. It also watches load and size per partition and orders splits, merges and moves, which is how hot prefixes are absorbed.
• Index change log (per partition, commit order): Every committed mutation, streamed from each partition’s replicated log in commit order: put, delete, transition, multipart complete. GC uses it to keep exact live-byte counts per extent; lifecycle, billing and notifications consume it too. An event exists if and only if the commit did, because it is the commit log itself rather than a second write.
• Lifecycle engine (daily scan per partition): Scans the index partitions of buckets that have rules, in key order, once a day, and applies them: transition to infrequent access or archive by age, expire noncurrent versions, remove expired delete markers, abort stale multipart uploads. Every action is a conditional metadata commit, so a client overwrite during a transition simply wins.
• Client / SDK (signed HTTPS): Applications, data pipelines and backup tools speaking the S3 REST API through an SDK. The SDK signs every request, computes a CRC of the payload before sending, switches to multipart above ~100 MB, and parallelises big downloads with byte-range GETs. Most "storage is slow" complaints are clients making one request at a time.
• Front end (auth · route · encode): A stateless fleet in every data centre behind DNS and load balancers. It authenticates and authorises each request, finds the key’s index partition, streams bytes to and from storage (erasure coding and decoding on the fly), and commits metadata last. It stores nothing, so losing a front end loses only the requests in flight.
• Metadata index (range-partitioned · Raft ×3 DCs): One row per object version, keyed (bucket, key, version descending) and kept sorted, so LIST by prefix is a range scan and the latest version is the first row at its key. About 3 M partitions of ~10 GB, each a three-replica consensus group with one replica per data centre. The leader’s commit of a row is the moment an object exists, and leader-lease reads make every GET and LIST see the latest commit.
• GC & compaction: Keeps a running count of live bytes per extent from the change log, frees extents with no live bytes for a 48-hour grace period, and compacts mostly dead small-object extents by copying live objects out and repointing their rows with compare-and-set. A weekly reconciliation against a full index scan catches drift. It is deliberately slow and conservative, because a GC bug is a data-loss bug.
• Archive tier (dense, mostly spun down): The cold storage class: high-density drives kept mostly powered down, written in large sequential batches and read back only through an explicit restore that takes minutes to hours. The same failure-domain rules as standard apply; the saving comes from cheaper media and almost no provisioned IOPS, roughly a tenth of the cost per byte.
• Fragment cache (SSD · immutable data): An SSD cache of byte ranges keyed by (extent id, offset), in front of the disks. Extents never change after they are written, so a cached range can never be stale and there is nothing to invalidate. It absorbs release-day spikes on popular objects and protects an HDD fleet that offers only ~7 IOPS per TB.
• Replicated extents (3× · one copy per DC): Append-only extents of up to 1 GB that hold small objects (under 1 MB) packed Haystack style: header, bytes, checksum. Each has three replicas, one per data centre, and a single writer at a time. An extent is sealed when full or when a replica falters, and sealed extents that stay cold for 30 days are re-encoded to 12+6 in the background, as Facebook’s f4 and Microsoft’s Azure Storage describe.
• Extent manager (extent → disks · placement): Maps each extent id to its scheme and the disks holding its 3 replicas or 18 fragments: billions of rows, a few TB, partitioned and consensus-replicated. It allocates new extents by free space under failure-domain rules, takes heartbeats from every storage server, and drives repair and rebalancing. Object rows name extents, never disks, so moving data never touches the 30 PB index.
• Coded extents (RS 12+6 · 6 fragments per DC): Large objects stream through an encoder in 1 MB cells: 12 data and 6 parity fragments per stripe, on 18 servers, six in each data centre, up to 768 MB of data per extent. That is 1.5× storage with any six fragments losable, so a whole data centre can vanish. The code is systematic, so a healthy read returns the original bytes with no decoding.
• Scrub & repair (priority by spares left): Rebuilds lost fragments from 12 survivors, ordered by redundancy remaining so a stripe at its last spare goes first, and spread across thousands of disks so a 20 TB drive is re-protected in about an hour instead of the 28 it takes to copy onto one spare. The scrubber reads every disk end to end every two weeks and checks every cell’s checksum, because disks do not report all their errors.

User flows

1. PUT a small object: bytes first, commit last. The core write. The bytes are durable in three data centres before any metadata row exists, and the object becomes visible at exactly one moment: the consensus commit of its index row. A crash anywhere before that leaves garbage, never a half-written object.
   1. Client sends a signed PUT carrying the bytes and a CRC32C of them. One request, one key, no session. The signature covers the method, path, headers and a hash of the payload, so nothing on the way can change them. The client-computed checksum starts an end-to-end chain: checked at the front end, stored with the bytes, checked again on every read and every scrub.
   2. Front end verifies the signature and the caller’s permission, and loads the bucket’s configuration. Bucket configuration (versioning state, default encryption, lifecycle, policy) is read on every request but changes a few times a year, so front ends cache it for seconds. That makes it the one eventually consistent part of the system; S3 documents the same for bucket settings and asks you to wait after enabling versioning before writing.
   3. Front end appends the bytes to an open replicated extent, one replica in each data centre. Objects under 1 MB are packed Haystack style into append-only extents of up to 1 GB, so a 120 KB object costs one sequential append rather than a file and an inode. The front end holds leases on a few open extents from the extent manager, which makes it the only writer of each and costs no lookup per PUT. Every replica checks the CRC, writes header, bytes and checksum, and fsyncs before acknowledging; if one is slow, the front end seals that extent and retries the append on another rather than waiting.
   4. Front end finds the key’s partition and commits the new version row through its leader. The partition map is cached; a replica that no longer owns the key rejects the call and the front end refreshes, so a stale map never produces a wrong answer. The leader replicates the row to a majority (two of three data centres) and applies it. This commit is the linearisation point: before it no reader can see the object, after it every reader does.
   5. Front end returns success with the ETag and version id. Only now does the client hear success. Had the front end died after the appends but before the commit, the client would see a timeout and retry with fresh bytes and a new version; the first copy would be bytes that no row points at, invisible and later collected.
   6. The committed mutation is streamed to the partition’s change log. Each partition exports its replicated log in commit order. Consumers are extent accounting, lifecycle hints, event notifications and billing. Because the log is the commit log itself, an event exists exactly when the commit does, with no dual write to get wrong.
   7. GC adds the new bytes to the extent’s live count; bytes of any version this PUT replaced become dead. Live-byte counts per extent decide compaction later. Bytes of an overwritten object in an unversioned bucket are not freed now; they simply stop being counted, and an extent that is mostly dead gets compacted. Nothing in the data plane is ever edited in place.
2. GET a large object, strongly consistent. The read path for a 2 GB video, fetched with parallel ranged GETs. The metadata read is linearisable, so it always returns the latest committed version; the data read can be cached freely, because extents never change.
   1. Client issues parallel ranged GETs for 64 MB slices of the object. Large downloads parallelise by byte range, typically 8 to 64 requests at once, which is how one client reaches tens of Gbit/s. Each range is an independent request, retried on its own.
   2. Front end authorises the request and reads the latest version’s row from the partition leader. The leader holds a lease of a few seconds during which no other replica can be elected, so it answers from its own state, with no consensus round, and is still linearisable. A follower could be behind, so followers serve a hot key only after checking the leader’s commit index (Raft’s ReadIndex).
   3. Front end maps the byte range to coded extents and looks up where their fragments live. Prefix sums over part sizes find the part, then the extent and offset inside it. An extent’s locations change only when a fragment is repaired or moved, so front ends cache them and treat "fragment not here" from a storage server as the cue to refresh.
   4. Front end checks the fragment cache for the range. Cache keys are (extent id, offset, length), and an extent never changes after it is written, so a cached range cannot be stale and there is no invalidation problem on the data plane. A popular object, a game patch on release day, is served almost entirely from here.
   5. On a miss, front end reads the 12 data fragments in parallel and hedges the slowest with a parity fragment. The code is systematic: the data fragments are the original bytes, so a healthy read needs no decoding. With 12 disks per read the slowest sets latency, so past roughly p95 the front end also asks for one or two parity fragments and decodes from whichever 12 arrive first. Two-thirds of the fragments are in other data centres, 1–2 ms away, which is noise next to moving 64 MB.
   6. Front end verifies every cell’s checksum and streams the range to the client. Corruption the disk did not report, from a misdirected write, firmware or bit rot, is caught here: the bad cell is decoded from other fragments, the client still gets correct bytes, and the extent is queued for repair. A read never returns bytes that fail their checksum.
3. Upload a 2 TB object in parts. A single 2 TB PUT would hold one connection for hours and restart from zero on any error. Multipart turns it into thousands of independent, parallel, retryable part uploads, and the object appears atomically only when one manifest commit lands.
   1. Client starts a multipart upload and gets an upload id. The upload is a metadata record keyed (bucket, key, upload id), which sorts right next to the key itself, so the upload, its parts and the final object all live in one index partition. Nothing is visible to GET or LIST yet.
   2. Client uploads 8,000 parts of 250 MB, 32 at a time, each with its own checksum. Each part is an ordinary write with its own connection and its own retry, so a dropped connection costs one 250 MB part, not 2 TB. At 32 parts in flight a well-connected client sustains ~25 Gbit/s and finishes in about 11 minutes. Parts can come from many machines in any order.
   3. Front end streams each part through an encoder into coded extents spread over three data centres. Bytes are cut into 1 MB cells; every 12 data cells produce 6 parity cells, and the 18 go to 18 disks on 18 servers, six per data centre, chosen by the extent manager when it allocated the extent. The write is acknowledged when all 18 are durable, and a fragment that is slow to land past ~p99 is re-sent to another server in the same data centre, so one slow disk costs milliseconds rather than the tail.
   4. Each finished part is recorded under the upload with its size, ETag and layout. Part records are committed like objects, so a recorded part is durable and appears in ListParts. A client that crashes lists what is done and resumes from there. Parts written but never recorded are just unreferenced extents for GC.
   5. Client completes the upload with the ordered list of part numbers and ETags. The list lets the client choose exactly which uploaded parts form the object, so a part that was re-sent twice is unambiguous. The resulting ETag is a hash of the part hashes plus "-8000", not an MD5 of the object, because the object was never seen whole in one place.
   6. Index validates the list and commits one version row that points at the parts: the object appears atomically. No bytes are copied or moved; completion is a single commit in one partition, because split points never fall inside one key’s rows. Parts left out of the list lose their records and their extents lose their references. Until this commit GET returns the previous version or 404; after it, every reader sees the whole new object.
   7. Uploads that are never completed are aborted by a lifecycle rule and their bytes collected. An abandoned upload is invisible but still occupies, and bills, storage. A rule that aborts uploads older than seven days deletes their records in one commit per upload, and the change log tells GC which extents just lost their references.
4. A disk dies and repair races the clock. With about a million drives, ~53 fail every day. Durability is decided by how quickly the system notices, and by rebuilding the stripes with the least redundancy left first, in parallel across the fleet.
   1. A storage server reports a failed disk, or stops sending heartbeats altogether. A disk that fails its reads is declared dead at once. A silent server is not: most silences are reboots and deploys that end within minutes, and re-protecting a 768 TB server for a reboot would cost ~9 PB of repair reads. Its fragments count as unavailable for reads immediately, but repair waits ~15 minutes unless a stripe is already short.
   2. Extent manager lists every extent with a fragment on that disk and ranks them by spare fragments left. A 20 TB disk at 80 % holds ~16 TB of 64 MB fragments, about 250 k extents, found through a reverse index from disk to extents. The repair queue is ordered by fragments remaining minus 12: an extent that has lost three (three spares left) goes before one that lost one. Stripes at one or two spares get urgent bandwidth; the rest are rate-limited.
   3. Repair workers read 12 surviving fragments of each extent, decode the missing one, and write it to a new disk. Placement is declustered: those 250 k extents have their other fragments on tens of thousands of disks, and the replacements go to tens of thousands of others, so the rebuild runs in parallel across the fleet. Copying 20 TB onto one spare at 200 MB/s takes ~28 hours; rebuilt this way, at ~50 GB/s of reads, it takes about an hour.
   4. Extent manager records the fragment’s new home; the object index is not touched. Object rows name extents, not disks, so re-protecting 250 k extents updates 250 k small extent-map rows instead of rewriting the index rows of every object stored in them. That indirection is what makes repair cheap enough to run all day.
   5. Meanwhile, the scrubber finds a fragment whose bytes no longer match their checksum. Disks do not report every error: drives specify one unrecoverable read error per 10^15 bits, about one per 125 TB read, and silent corruption also comes from firmware and misdirected writes. Every disk is read end to end every two weeks, ~13 MB/s per disk or ~6 % of its bandwidth, with every 1 MB cell’s CRC checked. A bad cell enters the same priority queue as a dead disk.
   6. Small-object extents that lost a replica are copied from a surviving one. For replicated extents repair is a plain copy, cheaper than a decode. But with only three copies, an extent down to one survivor is the most urgent item in the whole queue, and it jumps ahead of every coded stripe.
5. Delete, tiering and garbage collection. Nothing is deleted in place. A delete in a versioned bucket writes a marker; lifecycle rules later expire old versions and move cold data to a cheaper tier; space comes back only when GC finds bytes that nothing references.
   1. Client deletes a key in a versioned bucket. With versioning on, DELETE without a version id commits a delete marker: a new latest version that says "gone". GET returns 404 and LIST skips the key, but every older version remains and deleting the marker restores the object. A DELETE with a version id removes that version for good.
   2. Lifecycle engine scans the partitions of buckets with rules and evaluates each row against them. Rules are by prefix and age: infrequent access after 30 days, archive after 180, expire noncurrent versions after 90, remove expired delete markers. A daily scan in key order, partition by partition and from follower replicas, spreads the work across the fleet and needs no secondary index on age.
   3. A cold object is copied into the archive tier, then its row is switched to the new layout in one conditional commit. Copy first, then commit only if the row still points at the old layout. If a client overwrote the key meanwhile, the condition fails and the copy becomes garbage, so no lock is held across a slow copy. Archive uses dense, mostly spun-down drives, about a tenth of the cost per byte, and reading it back needs a restore that takes minutes to hours.
   4. Each commit’s change-log entry tells GC exactly which bytes in which extents just stopped being referenced. GC never guesses liveness: bytes are live only if an index row points at them, and the change log carries every change to that in commit order, so live-byte counts per extent are an exact running sum. A weekly reconciliation recomputes every count from a full index scan to catch bugs.
   5. Extents with no live bytes are freed after a grace period; mostly dead small-object extents are compacted. A coded extent at zero live bytes for 48 hours is freed. A replicated extent that is, say, 70 % dead has its live objects copied to a fresh extent and their rows repointed with compare-and-set, then the old one is freed. The grace period exists because a slow upload may still be about to commit a reference to an extent that looks dead.

Deep dives

The theory behind it

• Replication vs erasure coding: an interactive playground. Store 10 PB with copies or Reed-Solomon codes, change disk failure and repair rates, and compare raw storage, nines of durability and repair traffic.
• Multipart uploads: an interactive playground. Upload 100 MB to 5 GB over connections that drop, with a single PUT, a resumable upload and parallel multipart parts, and compare time to finish and bytes wasted.
• Storage tiering: an interactive playground. Store a petabyte of images for three years under Standard, Infrequent Access, Glacier Instant Retrieval and Deep Archive lifecycles, and see when tiering saves money and when retrieval fees eat it.
• Distributed file systems and erasure coding: How huge files are stored across thousands of disks: the GFS and HDFS design with a metadata master and chunk servers, chunk sizes and replication, write pipelines, metadata scaling, object storage internals, and erasure coding versus replication for durability and cost.
• Object storage and large files: Blob storage, pre-signed uploads, chunking and resumable transfer, multipart, erasure coding, storage tiers and lifecycle, plus how media pipelines are structured.
• Storing billions of small files (Haystack): Why file systems struggle with billions of small photos and how Facebook's Haystack solved it: metadata overhead, packing files into large volumes, in-memory indexes, needle lookups in one disk read, deletes and compaction, warm storage with erasure coding, and lessons for photo and attachment storage.

Related

• Design Dropbox
• Design Instagram
• Design Pastebin
• All 83 system design interview questions
• Run this question for a friend: the interviewer kit