Design Dropbox

• Upload and download files of any size — A 20 GB video and a 2 KB note go through the same path. Uploads resume after a dropped connection rather than starting again.
• Sync every change to every device — A file saved on a laptop appears on the phone without the user asking. The unit of sync is a change, not a folder scan.
• Only transfer what changed — Appending a row to a 2 GB CSV must not upload 2 GB. The client sends the chunks that differ and nothing else.
• Share files and folders — With another account, or as a public link. Permissions are viewer or editor, and revoking one takes effect on the next request.
• Version history and restore — Previous versions of a file for 30 days, restorable to any device, including after a delete.
• Work offline — Edits made with no network are queued locally and reconciled on reconnect, including when the same file changed on the server.
• Out of scope — Real-time collaborative editing of a document (that is a different design — see Google Docs), full-text search of file contents, and desktop-class file locking.

• Durability (11 nines) — Losing a user's only copy of a file is the one unrecoverable failure. Everything else is an inconvenience.
• Sync latency (< 5 s for a small change) — From save on one online device to visible on another. Perceived as "instant"; anything over about ten seconds is perceived as broken.
• Upload throughput (saturate the client link) — Parallel chunk uploads, so a fast connection is the limit rather than our ingest path.
• Availability (99.9 % writes, 99.99 % reads) — A failed sync is retried later and is survivable. A download that fails during a presentation is not.
• Metadata consistency (read-your-writes per namespace) — A device that just committed a change must see it in its own next listing, or the client will loop uploading it again.
• Storage efficiency (> 2× from dedupe) — The same installer, deck or photo exists in thousands of accounts. Storing it once is the difference between a viable and an unviable business.
• Scale (100 M accounts, 10 M daily active) — Metadata operations dominate request count; bytes dominate cost.

• Stored bytes: ~1 EB raw, ~400 PB after dedupe — 100 M accounts × 10 GB average stored = 1 EB logical. Global chunk dedupe plus compression typically removes 55–65 % on a consumer corpus, so ~400 PB physical.
• New bytes per day: ~300 TB — 10 M daily active × 30 MB of genuinely new content each = 300 TB/day ≈ 3.5 GB/s average, and peak is 3–4× that in the evening. Sized against that, not the average.
• Chunk size and count: 4 MB · ~100 B of index per chunk — 4 MB balances two costs: small enough that a one-line edit re-uploads little, large enough that a 10 GB file is 2 500 chunks rather than 10 million. 400 PB ÷ 4 MB = ~10^11 chunks × 100 B of index = ~10 TB of chunk metadata.
• Metadata operations per second: ~200 k/s — 10 M daily active devices, each polling or holding a connection and committing a handful of changes: assume 20 metadata calls per device per active hour over an 8-hour spread → ~200 k/s at peak. This is the number the metadata store must serve, and it is why metadata and bytes are separate systems.
• Long-lived connections: ~3 M concurrent — Peak-concurrent devices ≈ 30 % of daily active = 3 M. At ~30 k sockets per notification server that is ~100 servers; each connection is idle almost all the time, so memory per connection is what matters, not CPU.
• Egress at peak: ~40 GB/s — Downloads outweigh uploads roughly 4:1 on a sync product (every change is uploaded once and downloaded once per other device). Peak upload ~10 GB/s → ~40 GB/s egress, which is why chunk downloads go through a CDN and never through the API tier.
• Version history cost: +15 % — Keeping 30 days of versions stores only the chunks that changed, not whole files. On a corpus where a few percent of bytes change each month, history costs ~10–20 % extra rather than 30×.

• Sync client (desktop · mobile agent) — Watches the folder with an OS file-system event API, chunks changed files, keeps a local SQLite database of (path, chunk hashes, server version), and reconciles against the server. Most of this design's cleverness lives here: the server is mostly a metadata store and a bucket.
• Load balancer — Terminates TLS and spreads API calls. Separate pools for metadata calls (small, chatty) and for notification connections (long-lived, idle), because they scale on different resources.
• Metadata service — The authority on what exists: namespaces, paths, file versions, the chunk list of each version, and permissions. Every mutation is a transaction that bumps a monotonic cursor for the namespace. Never touches file bytes.
• Metadata DB (sharded SQL, by namespace) — Files, versions, chunk lists and the per-namespace change journal. Sharded by namespace id so one user's (or one shared folder's) changes are a single-shard transaction and the journal is naturally ordered.
• Chunk service — Answers "which of these hashes do you already have?" and hands out short-lived presigned URLs for the ones we do not. Records chunk → storage location and a reference count. Bytes flow past it, not through it.
• Chunk index (key-value, hash-sharded) — content_hash → (location, size, refcount). Roughly 10^11 rows, uniformly distributed by hash, read on every upload and written once per new chunk. The single biggest table in the system.
• Block storage (S3-class object store) — Encrypted, erasure-coded chunks addressed by content hash. Immutable: a chunk is written once and never edited, which is what makes dedupe, history and caching all safe.
• CDN — Serves chunk downloads from the edge against signed URLs. Chunks are immutable and content-addressed, so they are infinitely cacheable — the ideal CDN workload.
• Notification service (long-poll / WebSocket) — Holds a connection per online device and tells it "namespace 42 has changed past cursor 9971". It sends no content, only a nudge, so a missed message costs one extra poll and nothing else.
• Change bus (Kafka, partitioned by namespace) — Every committed metadata change is published here. The notification service, the sharing service and the background jobs all read it, so adding a consumer never touches the write path.
• Sharing service — Owns membership and roles for shared namespaces, and public links with their expiry and password. A shared folder is a namespace mounted into several users' trees rather than a copy per user.
• GC & lifecycle — Expires versions past the retention window, decrements chunk reference counts, and deletes chunks whose count has been zero for a grace period. Runs deliberately late: deleting a chunk that is still referenced is data loss.
• Search & previews — Consumes the change bus to build filename search and to render thumbnails and document previews asynchronously. Off the critical path; a missing preview is cosmetic.

1. A new file is saved on a laptop — The write path: chunk locally, ask what is missing, upload only that to storage directly, then commit metadata last so a file never half-exists.
   1. The client sees a file-system event, waits for the file to settle, and chunks it.
   2. Client asks the chunk service which of those hashes the server already has.
   3. Client uploads the missing chunks straight to block storage, several at a time.
   4. Client commits the new file version: path, size, ordered chunk list, and the version it believes it is replacing.
   5. The commit is published to the change bus; devices are notified and the previews job picks it up.
2. The change reaches the user's phone — The read path: a nudge, a delta by cursor, and chunk downloads from the edge. The server never pushes content.
   1. The notification service reads the change from the bus and finds the connected devices for that namespace.
   2. Each online device is nudged over its held connection with nothing but the new cursor.
   3. The device asks for everything after the cursor it last processed.
   4. The device works out which chunks it is missing and fetches them from the CDN.
   5. The device assembles the file, writes it atomically, and records the new cursor.
3. One row is appended to a 2 GB CSV — The case that decides whether the product is usable on a real connection. Nothing but the tail should move.
   1. The client re-chunks the file with a content-defined boundary, not a fixed offset.
   2. The probe returns exactly one missing chunk out of ~500.
   3. Four megabytes are uploaded instead of two gigabytes.
   4. Commit records a new version whose chunk list is 499 old hashes and one new one.
   5. Other devices download the one new chunk and splice it into their local copy.
4. Two laptops edit the same file offline — The failure case a sync product is judged on. The rule: never silently lose a byte a user typed.
   1. Both devices are offline and each saves a different edit to /notes/plan.md at version 7.
   2. The first device reconnects and commits; the file becomes version 8.
   3. The second device commits with parent_version 7 and is rejected.
   4. The client resolves by creating a conflicted copy, and commits that as a new path.
   5. Both files sync to every device and the conflict is surfaced in the UI.
5. A folder is shared with a colleague — A shared folder is one namespace mounted in two trees, not a copy. That choice decides the whole permission model.
   1. The owner shares /Work/Q4 with another account as editor.
   2. The namespace is mounted into the invitee's tree.
   3. The invitee's devices are nudged and pull the namespace from cursor zero.
   4. Either member's edits now fan out to both members' devices from the one namespace journal.
   5. Access is revoked; the mount is removed, and any chunks left unreferenced are collected later.

Ask: Fixed-size blocks, content-defined chunking, or whole-file storage?

Good answers name: Content-defined chunking, ~4 MB average, Fixed-size blocks, Whole-file storage, Byte-level delta (rsync-style) against the previous version.

Our pick: Content-defined chunking with a rolling hash, 1 MB minimum, 4 MB average, 8 MB maximum, chunks identified by SHA-256 of their content and stored immutably. A file version is an ordered list of chunk hashes. Small files under the minimum are stored as a single chunk, and files under about 64 KB are inlined into the metadata row so a folder of text notes does not cost a storage round trip each.

1. Why SHA-256 rather than something faster?
   Because the hash is the identity: if two different chunks collide, one user gets another user's data. A cryptographic hash makes that a non-issue rather than a probability argument, and a deliberate collision attack becomes infeasible. The cost is real but small next to the I/O, and modern CPUs do it in hardware. Non-cryptographic hashes are fine for the rolling boundary function, which only decides where to cut.
2. What stops a pathological file from producing millions of tiny chunks?
   Minimum and maximum bounds on the chunker: no boundary is accepted before the minimum, and one is forced at the maximum. That bounds chunk count per byte regardless of content, at the cost of slightly worse dedupe on adversarial input.
3. A user renames a 10 GB folder. What crosses the network?
   Metadata only. A rename changes the path rows; the chunk lists and the chunks themselves are untouched. Other devices receive a delta entry describing the move and rename locally. A client that naively deleted and re-downloaded would be a serious bug, so the delta protocol has to express a move as a move.
4. How does the client avoid re-hashing every file on startup?
   The local database keys by (inode, size, mtime). On startup it walks the tree and only re-chunks files whose triple changed. A full rehash is the fallback when the local database is missing or the file system does not give stable inodes.

Ask: Deduplicate chunks across all users, per user, or not at all?

Good answers name: Global dedupe with a server-side possession check, Per-user dedupe only, No dedupe, Client-side convergent encryption.

Our pick: Global chunk dedupe, with the possession check done server-side rather than by trusting the client. A client that claims to have a chunk it did not upload must prove it by returning a challenge over a server-chosen byte range, and the first upload of a chunk by a given account is always performed in full. Reference counts are per chunk, incremented inside the commit transaction and decremented by the GC job, with a grace period before any chunk is actually deleted. Enterprise tenants can be pinned to per-tenant dedupe by policy.

1. Walk me through the attack the challenge prevents.
   Without it: I hash a chunk of a document I suspect exists — a payroll spreadsheet, say — and claim to have it. If the server accepts, that chunk exists on the service, and by varying the content I can brute-force the unknown parts of a document I mostly know. With a challenge, claiming possession requires actually holding the bytes, so the only thing I learn is that I already knew the content.
2. How do you delete a chunk safely when refcounts are distributed?
   Never delete on the decrement. The GC marks a chunk as candidate when its count reaches zero, waits a grace period well beyond any in-flight commit, re-reads the count, and only then deletes. Commits increment before the version becomes visible, so a chunk being referenced by a commit in flight cannot be seen as unreferenced. Getting this wrong is silent, permanent data loss, so it is worth the conservatism.
3. What happens if the chunk index and the object store disagree?
   A background scrubber walks the store and the index in both directions. An object with no index row is orphaned and safe to delete after a grace period; an index row with no object is the dangerous one — it means a file is unreadable, so it alerts, and the affected file versions are listed for the owners rather than silently failing on download.
4. Does dedupe break per-user encryption at rest?
   Chunks shared across users cannot be encrypted with a per-user key. The usual arrangement is a per-chunk data key, wrapped per namespace, with the store encrypting at rest under its own keys. A customer who insists on holding their own key gets their own dedupe domain, and gives up cross-user savings — which is the honest trade.

Ask: How do you shard a file tree, and what does that cost you?

Good answers name: Shard by namespace, with shared folders as their own namespace, Shard by user id, shared folders copied per user, Single globally consistent store (Spanner-class), Key-value store with application-level transactions.

Our pick: Sharded relational storage keyed by namespace id: personal root, plus one namespace per shared folder. Each shard holds files, versions, chunk lists and an append-only journal with a monotonic per-namespace cursor. Commits are single-shard transactions using compare-and-set on the file version, with an outbox row published to the change bus by a relay. Cross-namespace moves are a copy plus a delete, which is honest about not being atomic and is rare enough not to matter.

1. Why is the cursor per namespace rather than global?
   A global sequence is a single point of contention at 200 k commits a second, and no client needs it: a device syncs namespaces independently. Per-namespace, the cursor is just the shard's journal offset — free to produce, totally ordered where ordering matters, and it makes the delta query an index range scan.
2. A device has been offline for six months. What happens?
   It calls delta with its old cursor. If that cursor has been compacted out of the journal the server returns "reset", and the client falls back to a full listing and reconciles against its local state by hash — expensive but rare, and it must exist or a long-offline device can never converge. Journals are kept for well beyond the version retention window so this is uncommon.
3. One shared namespace has 50 000 members. What breaks?
   Fan-out. One commit nudges 50 000 devices, and if they all call delta at once that is a thundering herd on one shard. Mitigations: jitter the nudges, coalesce changes into one nudge per few seconds per namespace, and cache the delta response for a cursor since every device asks for the same one. Above some size a "shared folder" is really a different product with a read-mostly access pattern.
4. How do you rebalance a hot shard?
   Namespaces are the unit of movement. Copy the namespace to the target shard while it still takes writes, catch up from the journal, then briefly freeze writes for that namespace, apply the tail, and flip the routing entry. Freezing one namespace for a second is invisible; the client retries.

Ask: Should uploads and downloads go through your servers or straight to storage?

Good answers name: Presigned URLs, direct client ↔ storage, CDN for reads, Proxy every byte through an upload service, Client writes to storage with long-lived credentials, Resumable upload session API (single session, byte ranges).

Our pick: Presigned, per-chunk, write-only URLs valid for fifteen minutes, with the object key derived from the content hash so a client cannot write a chunk under a name it does not own. Downloads go through the CDN against signed URLs with the same scoping. Quota is checked optimistically at probe time and enforced at commit; a user who exceeds it has the commit rejected and the orphaned chunks collected. Malware scanning consumes the change bus and quarantines the file version after the fact, which is the honest trade for this architecture.

1. A client uploads chunks and never commits. What cleans up?
   Chunks with no reference are orphans. The GC finds them by age — written more than a day ago with a zero reference count — and deletes them after a grace period. It is worth tracking the orphan rate: a spike usually means clients are crashing between upload and commit.
2. How do you stop someone using a presigned URL to read another user's chunk?
   Upload URLs are write-only and the key is the content hash, so writing under it requires already having the content. Download URLs are issued only after the metadata service has checked that the requesting user has a version referencing that chunk, and they are short-lived and single-object. Content addressing helps here: knowing a hash means you already know the bytes.
3. The CDN caches a chunk, then the file is deleted for legal reasons. Now what?
   Purge by URL at the CDN, delete the object, and rely on the short signature lifetime for anything in flight. Because chunks are content-addressed and shared, deleting one may be wrong: it might still be referenced by another account. Legal takedowns therefore operate on file versions and namespaces, with chunk deletion only when the reference count reaches zero.
4. Mobile client on a flaky connection: what does the retry policy look like?
   Per-chunk exponential backoff with jitter, a small parallelism (three or four in flight on cellular), and no restart of completed chunks. The probe is re-run on resume because the server may have acquired the chunk from another device in the meantime — often the whole remaining upload evaporates.

Ask: Polling, long polling, WebSockets, or push notifications?

Good answers name: Long-lived connection carrying contentless nudges, plus cursor delta, Periodic polling only, Push messages through the platform services (APNs, FCM), Streaming the changes themselves to devices.

Our pick: A long-lived connection per online device (WebSocket, falling back to long poll) carrying only "namespace N is at cursor C", plus a cursor-based delta API that is the single source of truth. Mobile clients in the background are woken by platform push instead. Every client also polls on a long interval — a few minutes — so that a device whose connection silently died still converges. Nudges for a busy namespace are coalesced to at most one per second.

1. Why not send the changed file in the notification?
   Because then the notification path has to be reliable and ordered, and you have built a second, worse copy of the delta API. Keeping nudges contentless means the connection can drop, duplicate or reorder freely — the client just asks the authority again, and correctness never depends on the transport.
2. How do you balance three million connections?
   Consistent hashing of namespace to notification node, so a commit finds the right node in one hop, with the connection layer in front routing by namespace. Rebalancing on deploy disconnects clients, so deploys are slow-rolled with jittered reconnect, and the client treats a reconnect as "call delta" rather than as an error.
3. A deploy drops all connections at once. What does the server see?
   Three million reconnects and three million delta calls. Mitigations: jittered reconnect backoff in the client, staged rollout so only a fraction disconnect at a time, and a delta response that is cacheable by (namespace, cursor) so the herd mostly hits cache. This is worth load-testing deliberately, because it is the one moment the system is at its worst.
4. What is the desktop client doing when nothing is happening?
   Holding one socket, sending a keepalive every minute or so, and watching the file system. The design goal is that an idle client costs no requests and almost no battery — which is why the poll fallback is on a several-minute interval rather than a several-second one.

Ask: How long do you keep versions, and what happens when a user deletes a file?

Good answers name: Immutable versions, soft delete with a tombstone, GC by reference count, Hard delete immediately, Keep every version forever, Time-machine style snapshots of the whole namespace.

Our pick: Versions are immutable and retained 30 days on consumer plans, longer on business plans. Delete writes a tombstone version: the file disappears from listings and from every device, and is restorable from trash for the retention window. A lifecycle job expires old versions, decrements chunk reference counts, and a separate GC deletes chunks whose count has been zero for a grace period. An explicit "delete permanently" request skips the trash but still goes through refcounting, because the chunk may not be ours alone to delete.

1. A user asks for their data to be erased under GDPR. What actually happens?
   Their namespaces, metadata and mounts are deleted, which removes every path by which the data can be reached. Chunks are dereferenced and collected on the normal schedule, so the bytes go within the documented window — except for chunks still referenced by other accounts, which were never solely theirs. That distinction has to be explained in the policy rather than fudged.
2. Ransomware encrypts a user's whole folder and it syncs. Can they recover?
   Yes, and this is the case that justifies history existing. Every file has a pre-encryption version within the retention window, so a bulk rollback to a timestamp restores the namespace. It is worth detecting the pattern — a mass rewrite of many files to high-entropy content in a short window — and pausing sync with a prompt, because the user's real enemy is the retention window expiring before they notice.
3. How do you restore a folder of 100 000 files quickly?
   It is a metadata operation: write new versions pointing at the old chunk lists. No bytes move on the server at all. The devices then download whatever they no longer hold locally, which is the only slow part and is bounded by what actually changed.
4. Does a version count against the user's quota?
   A defensible answer either way, and the honest one is to pick and be consistent. Charging only for the current version is simpler to understand and is what consumers expect; charging for total stored bytes reflects the real cost. Most products charge for the current version and absorb history as a cost of the retention promise.