Design a Large File Distribution System

Get a 200 GB checkpoint from one origin onto 10,000 machines in five data centres within minutes, under capped links, verifying every byte.

Last updated 2026-10-26. Difficulty: hard. Patterns: peer-to-peer, chunking, bandwidth, integrity, stragglers. Reported at Anthropic, OpenAI, Meta, Google, NVIDIA, Databricks.

Functional requirements

• Publish an artifact once. A 200 GB model checkpoint or a 5 GB container image is split into pieces, hashed and signed once, and uploaded to a single origin.
• Distribute it to a target set. Up to 10,000 machines in five data centres, chosen by pool or label, with a priority and a deadline. Several distributions can run at once.
• Verify on every machine. Every piece is checked against a signed manifest before it is written to disk or passed on; the whole artifact is checked before it is marked ready.
• Stay inside bandwidth caps. The origin allows 10 GB/s, each data centre’s WAN link 40 Gbps for this job, and each host 20 of its 25 Gbps, because hosts keep serving while they download.
• Survive failures mid-transfer. Machines reboot, peers die, a seeder or the tracker fails. Transfers resume from what is already verified, never from zero.
• Report progress. Per machine, per data centre and overall: hosts verified, rates, an ETA, and the stragglers with a likely cause.
• Cancel, retain and roll back. Stop a bad release everywhere within seconds. Keep the previous artifact on every host so a rollback moves no bytes.
• Out of scope. Building the artifact, loading weights into GPUs and shifting traffic (the deploy system does that), delta updates between versions (content-defined chunking, as in Design Dropbox), and the training fabric.

Non-functional requirements

• Completion time (99 % of hosts < 3 min). The floor is file size ÷ per-host bandwidth: 200 GB ÷ 2.5 GB/s = 80 s. A good design puts the median near 100 s and p99 within about 2× of the floor.
• Origin egress (≤ 10 GB/s, ~1 copy). The origin is an object store shared with training jobs. Reading it once per artifact (200 GB) rather than once per machine (2 PB) is the difference between 20 s and 56 hours.
• Cross-data-centre bandwidth (≤ 40 Gbps per DC link). This is the requirement that drives the hardest trade-off: speed against link caps. It forces topology-aware peer selection and a few seeders per data centre that pull disjoint slices, so each piece crosses each WAN link about once.
• Integrity (every piece verified). No machine loads or forwards a byte that does not match the signed manifest. A flipped bit from a disk, a NIC or a bad peer costs one piece, never a corrupt model in production.
• Fault tolerance (1 % of hosts fail, no restart). Hosts, seeders and the tracker can all fail during a rollout. A failure costs at most the blocks in flight from one sender.
• Do no harm (≤ 20 of 25 Gbps per host). Hosts serve production while they download, and rack uplinks are oversubscribed 2.5:1. Per-host caps and rack-first peering keep distribution from crowding out real traffic.
• Confidentiality (mTLS, target hosts only). Model weights are among the most valuable things the company owns. Only authenticated machines in the target set may join a swarm or fetch a piece.
• Progress visibility (fresh within 5 s). The deploy system and operators need per-DC progress, an ETA and a named list of stragglers with causes, not a single percentage.

Back-of-envelope estimates

• Everyone pulls from the origin: ~56 hours. 10,000 × 200 GB = 2 PB. At the origin’s 10 GB/s cap that is 200,000 s ≈ 56 h. An origin with unlimited egress does not save it: a remote DC’s 2,000 hosts need 400 TB through a 5 GB/s WAN link, ~22 h. Central fan-out is not slow, it is impossible.
• The floor with peer-to-peer: ~80 s. Each host can receive at most 2.5 GB/s (20 Gbps), so 200 GB takes at least 80 s. The swarm’s total upload, 10,000 × 2.5 GB/s = 25 TB/s, moves 2 PB in the same 80 s, and the origin needs only 20 s to inject one copy. The bound is max(F/u_origin, F/d_host, N·F/Σu) = max(20, 80, 80) = 80 s, independent of N.
• Pieces, manifest and bitfield: 3,125 pieces · 100 KB · 391 B. 200 GB ÷ 64 MB = 3,125 pieces. A SHA-256 per piece is 32 B, so the hash list is 100 KB. One bit per piece makes a host’s bitfield 391 B, and the whole fleet’s availability map 10,000 × 391 B ≈ 4 MB.
• One copy across a WAN link: ~40 s. 200 GB ÷ 5 GB/s (40 Gbps) = 40 s per remote DC, shorter than each host’s own 80 s floor, so the WAN is not the bottleneck as long as each piece crosses it once. Topology-blind peering would push 400 TB per DC through the same link.
• Rack uplink with random peers: ~160 s per rack. 40 hosts × 200 GB = 8 TB per rack. With peers chosen at random across a 2,000-host DC, 39 of 1,999 are rack-mates, so ~98 % of bytes cross the 400 Gbps (50 GB/s) uplink: 7.8 TB ÷ 50 GB/s ≈ 160 s, twice the floor. Rack-first peering is a speed requirement, not a nicety.
• Store-and-forward doubling: ~19 min. If a host forwards only after it has the whole file, holders double each round: log₂ 10,000 ≈ 14 rounds × 80 s ≈ 1,120 s. A pipelined binary tree does better, ~160 s, because each interior host splits 2.5 GB/s between two children, but half the hosts are leaves whose upload is never used.
• Seeders’ share of demand in a DC: ~2 %. 8 DC seeders × 12.5 GB/s (100 Gbps) = 100 GB/s, against 2,000 hosts × 2.5 GB/s = 5 TB/s of demand. Seeders inject; peers must supply the other 98 %. Queued on seeders alone, the DC would take 400 TB ÷ 100 GB/s = 4,000 s.
• Tracker load: ~5 k heartbeats/s. 10,000 agents every 2 s = 5,000/s fleet-wide, 1,000/s per DC tracker, each a few hundred bytes with the bitfield. About 2.5 MB/s in total. The control plane is tiny; the whole problem is bandwidth.

Components

• Signing key (KMS · key in an HSM): Signs the manifest’s Merkle root together with its name, version, size and expiry. The private key never leaves the HSM, and every agent ships with the public key pinned. A compromised tracker or peer can then slow or misdirect a transfer, but cannot get a machine to load bytes the release pipeline did not produce.
• Manifest builder (64 MB pieces · SHA-256): Runs once per artifact, next to the bytes: splits the file into fixed 64 MB pieces (3,125 for 200 GB), hashes each, builds a Merkle tree over the piece hashes and has the root signed. Fixed size rather than content-defined chunks, so a piece’s index alone gives its offset. Hashing 200 GB is about 100 core-seconds.
• Origin store (object store · 10 GB/s cap): The single source of truth for artifacts and manifests, addressed by Merkle root so a path never changes meaning. It is shared with training jobs, so distribution may use 10 GB/s of it. In this design it is read about once per artifact, by the origin seeders, and never by the 10,000 agents.
• Origin seeders (8 hosts · 100 Gbps): A small tier in the origin’s data centre that reads each piece from the origin once (each host a disjoint eighth, 1.25 GB/s each), shares pieces among themselves, and serves every other data centre’s seeders across the WAN. They also act as the DC seeders for their own data centre, and keep the last few artifacts so rollbacks and late joiners never touch the origin.
• DC seeders (8 per DC · WAN slices): The only hosts in a data centre allowed to pull across the WAN. Each owns the pieces whose index mod 8 is its slot and pulls them under a 625 MB/s token bucket, so the eight together hold the link at its 40 Gbps budget. They inject pieces into the DC; the peers multiply them, since 100 GB/s of seeder upload is 2 % of the DC’s 5 TB/s demand.
• Release pipeline (training or build job): The job that produced the checkpoint or image. It runs the manifest builder, starts a distribution for the signed root and a target set, watches progress, and hands off to the deploy system once each pool’s quorum has verified.
• Distribution API (start · status · cancel): Authenticated entry point for release pipelines and operators: create a distribution, read its progress, cancel it. It fetches the manifest and checks its signature before anything moves, so a bad artifact fails here and not on 10,000 hosts. Starts are idempotent on a client key so a retry never launches a second swarm.
• Distribution controller (global · leader-elected): Plans each distribution: hosts per DC, the WAN relay plan, each link’s budget and the seeders’ slices, and pushes it to the trackers. Merges per-DC roll-ups into the global view and decides when a pool is done, on a quorum of verified hosts rather than 100 %. Its durable state is small and lives in the state store, so failover to a new leader is quick.
• DC tracker (soft state · 1 per DC): Admits agents in its data centre by certificate identity and target set, hands out topology-aware peer lists (about 20 rack-mates, 10 from other racks, plus the DC seeders), and takes a heartbeat from each agent every 2 s, 1,000 a second per DC. It is never on the data path, and everything it holds is rebuilt from the next round of heartbeats, so a standby needs no replication.
• Peer agent (on every host · mTLS): A daemon on each of the 10,000 hosts. Checks the manifest’s signature, then downloads pieces rarest-first from about 30 neighbours, verifies each piece’s hash before writing or serving it, and uploads a piece to others the moment it verifies. Persists its bitfield so a restart resumes, and caps itself at 20 Gbps each way.
• Distribution state (Postgres / etcd): Distributions, manifests, target sets, plans, budgets and the final outcome per host (verified, evicted, failed). Kilobytes per distribution plus one row per host. It deliberately does not hold live piece availability: that is soft state in the trackers.
• Progress view (time series · per DC): Counters and histograms every few seconds: hosts verified, rate and ETA distributions per DC, bytes per link, the rarest pieces, and stragglers with causes. Histograms rather than averages so percentiles merge correctly across data centres. A view for people and the deploy system; the per-host outcome of record is in the state store.
• Local NVMe (preallocated · atomic rename): Each verified piece is written at offset index × 64 MB into a preallocated file, and the bitfield is updated only after the data is durable. When the last piece lands the agent checks the root, fsyncs and renames the file into place, so a loader sees either the old artifact or the whole new one. The previous artifact stays for rollback.
• Neighbour peers (~30 · rack first): The other agents this one is connected to: about 20 in its own rack and 10 elsewhere in the same DC, never another DC. They exchange bitfields once and a HAVE message per verified piece, which is how each agent sees rarity. No tit-for-tat choking: every peer is ours, so upload slots exist for fairness and caps, not incentives.

User flows

1. Publish a checkpoint and seed it. The write path. The artifact is chunked and signed once, read from the origin once, and lands in every data centre’s seeders within the WAN budget, pipelined so agents can start in the first second.
   1. The release pipeline hands the finished 200 GB checkpoint to the manifest builder. The builder runs next to the bytes, in the training cluster, because it must read all 200 GB once to hash them. Hashing is about 100 core-seconds, so spread over a few dozen cores it is bounded by how fast storage can be read. This is the only time anything reads the whole file before distribution.
   2. The builder cuts 3,125 pieces of 64 MB, hashes each, and builds a Merkle tree over the hashes. Fixed size, so piece i is bytes [i × 64 MB, (i + 1) × 64 MB) and its index alone locates it. The manifest also carries a file table, which shard file covers which byte range, so the loader can map pieces back to the checkpoint’s files. The hash list is 100 KB; the Merkle root is the artifact’s 32-byte identity.
   3. The root is signed, and the pieces and signed manifest are uploaded to the origin under that root. Signing the root, not the file, means one small signature covers every piece transitively. Uploads are keyed by root, so the origin is content-addressed and immutable: uploading the same bytes twice is a no-op, and a path can never point at different bytes later.
   4. The pipeline starts a distribution for that root and a target set. The API fetches the manifest and verifies its signature before anything moves, so a bad artifact fails once here instead of on 10,000 hosts. The controller resolves the target set into hosts per DC and persists the distribution before acknowledging.
   5. The controller computes the plan: link budgets, which DC pulls from which, and each seeder’s slice. Budgets come from network configuration, not guesses: 10 GB/s from the origin, 40 Gbps per WAN link, 20 Gbps per host. The plan goes to each DC’s tracker, which tells its seeders their slice. Budgets can be changed mid-rollout, for example halved when production traffic peaks.
   6. Origin seeders read each piece from the origin once, each a disjoint eighth, and share among themselves. At 10 GB/s the origin tier holds a full copy in 20 s, but nothing waits for that: each piece is served onward the moment it verifies. Origin egress for the whole rollout is one copy, about 200 GB, instead of 2 PB.
   7. Each remote DC’s seeders pull their slices across the WAN from the origin seeders, under the link budget. Disjoint slices are what make "each piece crosses the WAN once" true: without them, eight seeders would race for the same first pieces. One copy crosses in 40 s. Seeders then fetch the other seven slices from each other inside the DC, so each holds a full copy for late joiners.
2. One machine downloads and verifies. The read path, run 10,000 times at once. The agent pulls by rarity from rack-mates first, verifies every piece before using or sharing it, and is uploading within a second of starting.
   1. The agent announces itself to its DC’s tracker and gets a peer list. The tracker picks neighbours by topology: mostly rack-mates, who share a top-of-rack switch with no oversubscription between them, plus a few from other racks so new pieces can enter the rack. Random within each group, so the swarm is a well-connected random graph rather than islands.
   2. It fetches the signed manifest from a DC seeder and checks it against its pinned key. The manifest can come from anyone, because it verifies itself: a forged one fails the signature, a tampered hash list fails the root. 10,000 fetches of 100 KB is 1 GB, nothing for the seeders, and the origin is not involved.
   3. It connects to its neighbours over mTLS and exchanges bitfields. From then on each agent knows which neighbour has which piece, and how many neighbours have each piece: its local view of rarity. There is no central lookup per piece, which is what keeps the tracker at 1,000 requests a second per DC.
   4. It requests the rarest pieces it lacks, from rack-mates first, with several blocks in flight per connection. Rarest-first among neighbours, ties broken at random, and a piece once started is finished first. Four 4 MB requests in flight per connection keep the pipe full across a sub-millisecond round trip. Seeders are asked only for pieces no neighbour has, so they spend their upload on new pieces.
   5. When a piece’s last block arrives, the agent hashes it and writes it to disk only if it matches. Verifying per piece means corruption costs at most 64 MB of re-download and points at the sender. SHA-256 with CPU extensions runs at about 2 GB/s a core, so keeping up with 2.5 GB/s takes about 1.25 cores.
   6. It sends HAVE to its neighbours and starts serving that piece immediately. This is the pipelining: a piece is useful to others a fraction of a second after it lands here, not after the whole 200 GB. It is why each host uploads about as much as it downloads and the swarm gets close to the 80 s floor.
   7. Every 2 s it reports progress to the tracker, which may swap some neighbours. The heartbeat doubles as the peer-list refresh: the tracker replaces neighbours that died or have nothing this agent needs. Three missed heartbeats, 6 s, take an agent out of everyone’s lists.
   8. With every piece verified, it checks the root, fsyncs, renames the file into place and reports complete. The rename is atomic, so a model loader sees the old artifact or the whole new one, never a mix. The agent keeps seeding until the distribution closes: a finished host is the best source there is.
3. 10,000 machines start in the same second. The scale case: an empty swarm, five data centres, one capped origin and capped WAN links. It shows the swarm reaching close to the 80 s floor instead of collapsing onto the seeders, and no link going over its budget.
   1. The controller opens the distribution and all 10,000 agents announce within a few seconds. Two thousand announces per DC tracker in a second or two is trivial. The herd problem here is not the tracker but everyone wanting the same first bytes. Responses carry jitter on the heartbeat interval so the load spreads evenly from then on.
   2. With no peer holding anything yet, agents ask the DC seeders, which give each agent different pieces. If agents simply queued on the seeders, 2,000 × 200 GB ÷ 100 GB/s would take 4,000 s. The seeders’ job is to inject each piece into the DC a few times and then step back, so they serve the pieces they have sent least and refuse the rest.
   3. Within seconds each agent holds a few distinct pieces and trades with its neighbours. The first two to four pieces are chosen at random: having something to trade quickly matters more than rarity at the start. Upload capacity grows with the swarm, since every holder is a new source, and after about 10 s agents get ~98 % of their bytes from each other, most from rack-mates.
   4. Remote seeders are still filling over the WAN, so new pieces enter each remote DC at 5 GB/s. A remote DC cannot finish before its 40 s WAN copy, and it does not need to: each host’s own floor is 80 s. Because pieces are pipelined, agents start on whatever has already crossed.
   5. Rack-first selection keeps rack uplinks well below capacity. Topology-blind peering would push ~98 % of bytes through each rack’s 400 Gbps uplink and cap the rack at ~160 s. Rack-first means a piece enters a rack two or three times and then spreads at switch speed inside it.
   6. Copies stay even: when the swarm is 90 % done, no piece is held by only a few hosts. Rarest-first equalises copy counts, so the last pieces anyone needs are about as common as the first. With random choice, the coupon-collector tail would leave a few pieces with a handful of holders and thousands of agents queueing on them in the last 10 %.
   7. The median host verifies at about 100 s and 99 % by about 3 minutes. Origin egress is about one copy, each WAN link ran at its cap and never above it, and the median landed within 25 % of the floor. The last 1 % is the subject of the straggler flow.
4. Peers die and one sends bad bytes. The failure path. A peer stalls, a disk returns garbage, a host reboots, the tracker and a seeder crash. Nothing restarts from zero: each failure costs at most the blocks in flight from one sender.
   1. A neighbour’s link flaps and a block request to it stalls past its deadline. Deadlines adapt per peer: a 4 MB block at the 300 MB/s this peer was delivering should take ~13 ms, so a 1 s floor is generous and still catches a dead link fast. TCP alone can sit on a half-open connection for minutes.
   2. The agent cancels, re-requests the block from another holder, and deprioritises the slow peer. Blocks of one piece can come from different peers, so a failure costs at most the 4 MB in flight. A peer that times out twice is dropped from this agent’s set and replaced on the next heartbeat.
   3. A peer with a failing disk sends a piece that fails its hash check. This is why the check happens before writing and before serving: the bad bytes never reach this disk or any other host. When the piece’s blocks came from several peers, the agent re-fetches each block from one sender at a time to find the culprit.
   4. The agent reports the sender, and the tracker quarantines it after two independent reports. Two reporters, not one, so a single buggy agent cannot evict healthy peers. A quarantined host stops serving but keeps downloading; it re-verifies its whole store and gets a hardware ticket.
   5. A host reboots halfway through and resumes from its persisted bitfield. The bitfield is updated only after a piece’s data is durable, so it never claims a piece the disk lacks. After a crash the agent re-hashes its pieces in the background before serving them, which takes under a minute for 100 GB at 2 GB/s per core.
   6. The DC tracker crashes; agents keep trading with the neighbours they already have. The tracker is not on the data path, so losing it stops introductions, not transfers. The standby rebuilds all its state from the next round of heartbeats, 2,000 agents in 2 s, with nothing to replicate or replay.
   7. A DC seeder dies mid-slice, and its remaining pieces are reassigned to the other seven. Pieces from that slice that already crossed are held by agents, so only the missing ones are fetched again. The WAN budget is split seven ways so the link stays at 40 Gbps rather than dropping to 35.
5. Declaring done with a slow 1 %. The background path that decides the rollout. Heartbeats roll up into a live view, the last pieces are fetched in endgame mode, and each pool completes on a quorum instead of waiting for the slowest host in 10,000.
   1. Trackers roll heartbeats up per DC and send them to the controller every 5 s. Aggregating at the edge turns 5,000 heartbeats a second into one roll-up per DC every 5 s, about one message a second at the controller. Rates and ETAs travel as histograms so percentiles merge correctly across DCs.
   2. The controller writes the merged view, and the pipeline reads it through the API. The ETA comes from each host’s remaining bytes over its own recent rate, not from the average, because the average is dominated by hosts already finished.
   3. An agent whose last blocks are all requested enters endgame and asks every holder for them. Without endgame, the last pieces arrive at the pace of whichever sender happened to be slowest. Endgame starts when every missing block is already in flight, about the last 480 MB (30 connections × 4 blocks × 4 MB), and costs well under 1 GB of duplicates per host, all inside the DC.
   4. The controller flags hosts far below the DC median and asks them why. At 10,000 hosts some are always on a failing disk, a bad cable or a host pinned by its own job. A straggler is a host below a quarter of the DC’s median rate or with an ETA past the p99. The causes have different fixes: a disk-bound host needs nothing from the network; a host short of peers needs a new list.
   5. Each pool is done when 99 % of its hosts have verified; the rest go to repair rather than being waited on. Waiting for the slowest host in 10,000 can turn a 3-minute rollout into a 30-minute one. The quorum is per pool and set by the caller, because only the consumer knows whether 1 % missing is a small capacity dip or a training job that cannot start.
   6. The pipeline is notified and starts the model swap; stragglers keep downloading or are drained. Distribution and deploy are separate jobs: this system delivers verified bytes, the deploy system decides when to load them and shift traffic. Evicted hosts are kept out of serving until they verify.
   7. Later, seeders and agents garbage-collect old artifacts, keeping the current and previous one. Keeping the previous version on every host makes a rollback a local rename that moves zero bytes. Two 200 GB artifacts are a fixed, budgeted slice of each host’s NVMe; seeders keep more, so a machine added next week fetches from them and not from the origin.

Deep dives

The theory behind it

• Distributing software updates at scale: How apps, games, devices and package managers deliver updates to millions of clients: update manifests and version checks, CDN and peer-assisted delivery, delta patches, signing and integrity, staged rollouts and kill switches, rollback, bandwidth peaks on release day, and offline or constrained devices.
• Merkle trees: How hash trees let systems compare and verify large datasets cheaply: structure and root hashes, finding differences between replicas, proofs of inclusion, uses in databases' anti-entropy, file sync, Git, certificate transparency, blockchains and audit logs, and design choices for building them.
• Tail latency, percentiles and queueing: Why p99 matters more than the average, how fan-out amplifies tail latency, Little's law and queueing near full utilisation, hedged requests, timeouts and deadlines, and how to set and measure latency budgets.

Related

• Design Dropbox
• All 83 system design interview questions
• Run this question for a friend: the interviewer kit