Design a Large File Distribution System

• Publish an artifact once: A 200 GB model checkpoint or a 5 GB container image is split into pieces, hashed and signed once, and uploaded to a single origin.
• Distribute it to a target set: Up to 10,000 machines in five data centres, chosen by pool or label, with a priority and a deadline. Several distributions can run at once.
• Verify on every machine: Every piece is checked against a signed manifest before it is written to disk or passed on; the whole artifact is checked before it is marked ready.
• Stay inside bandwidth caps: The origin allows 10 GB/s, each data centre’s WAN link 40 Gbps for this job, and each host 20 of its 25 Gbps, because hosts keep serving while they download.
• Survive failures mid-transfer: Machines reboot, peers die, a seeder or the tracker fails. Transfers resume from what is already verified, never from zero.
• Report progress: Per machine, per data centre and overall: hosts verified, rates, an ETA, and the stragglers with a likely cause.
• Cancel, retain and roll back: Stop a bad release everywhere within seconds. Keep the previous artifact on every host so a rollback moves no bytes.
• Out of scope: Building the artifact, loading weights into GPUs and shifting traffic (the deploy system does that), delta updates between versions (content-defined chunking, as in Design Dropbox), and the training fabric.

• Completion time (99 % of hosts < 3 min): The floor is file size ÷ per-host bandwidth: 200 GB ÷ 2.5 GB/s = 80 s. A good design puts the median near 100 s and p99 within about 2× of the floor.
• Origin egress (≤ 10 GB/s, ~1 copy): The origin is an object store shared with training jobs. Reading it once per artifact (200 GB) rather than once per machine (2 PB) is the difference between 20 s and 56 hours.
• Cross-data-centre bandwidth (≤ 40 Gbps per DC link): This is the requirement that drives the hardest trade-off: speed against link caps. It forces topology-aware peer selection and a few seeders per data centre that pull disjoint slices, so each piece crosses each WAN link about once.
• Integrity (every piece verified): No machine loads or forwards a byte that does not match the signed manifest. A flipped bit from a disk, a NIC or a bad peer costs one piece, never a corrupt model in production.
• Fault tolerance (1 % of hosts fail, no restart): Hosts, seeders and the tracker can all fail during a rollout. A failure costs at most the blocks in flight from one sender.
• Do no harm (≤ 20 of 25 Gbps per host): Hosts serve production while they download, and rack uplinks are oversubscribed 2.5:1. Per-host caps and rack-first peering keep distribution from crowding out real traffic.
• Confidentiality (mTLS, target hosts only): Model weights are among the most valuable things the company owns. Only authenticated machines in the target set may join a swarm or fetch a piece.
• Progress visibility (fresh within 5 s): The deploy system and operators need per-DC progress, an ETA and a named list of stragglers with causes, not a single percentage.

• Everyone pulls from the origin: ~56 hours. 10,000 × 200 GB = 2 PB. At the origin’s 10 GB/s cap that is 200,000 s ≈ 56 h. An origin with unlimited egress does not save it: a remote DC’s 2,000 hosts need 400 TB through a 5 GB/s WAN link, ~22 h. Central fan-out is not slow, it is impossible.
• The floor with peer-to-peer: ~80 s. Each host can receive at most 2.5 GB/s (20 Gbps), so 200 GB takes at least 80 s. The swarm’s total upload, 10,000 × 2.5 GB/s = 25 TB/s, moves 2 PB in the same 80 s, and the origin needs only 20 s to inject one copy. The bound is max(F/u_origin, F/d_host, N·F/Σu) = max(20, 80, 80) = 80 s, independent of N.
• Pieces, manifest and bitfield: 3,125 pieces · 100 KB · 391 B. 200 GB ÷ 64 MB = 3,125 pieces. A SHA-256 per piece is 32 B, so the hash list is 100 KB. One bit per piece makes a host’s bitfield 391 B, and the whole fleet’s availability map 10,000 × 391 B ≈ 4 MB.
• One copy across a WAN link: ~40 s. 200 GB ÷ 5 GB/s (40 Gbps) = 40 s per remote DC, shorter than each host’s own 80 s floor, so the WAN is not the bottleneck as long as each piece crosses it once. Topology-blind peering would push 400 TB per DC through the same link.
• Rack uplink with random peers: ~160 s per rack. 40 hosts × 200 GB = 8 TB per rack. With peers chosen at random across a 2,000-host DC, 39 of 1,999 are rack-mates, so ~98 % of bytes cross the 400 Gbps (50 GB/s) uplink: 7.8 TB ÷ 50 GB/s ≈ 160 s, twice the floor. Rack-first peering is a speed requirement, not a nicety.
• Store-and-forward doubling: ~19 min. If a host forwards only after it has the whole file, holders double each round: log₂ 10,000 ≈ 14 rounds × 80 s ≈ 1,120 s. A pipelined binary tree does better, ~160 s, because each interior host splits 2.5 GB/s between two children, but half the hosts are leaves whose upload is never used.
• Seeders’ share of demand in a DC: ~2 %. 8 DC seeders × 12.5 GB/s (100 Gbps) = 100 GB/s, against 2,000 hosts × 2.5 GB/s = 5 TB/s of demand. Seeders inject; peers must supply the other 98 %. Queued on seeders alone, the DC would take 400 TB ÷ 100 GB/s = 4,000 s.
• Tracker load: ~5 k heartbeats/s. 10,000 agents every 2 s = 5,000/s fleet-wide, 1,000/s per DC tracker, each a few hundred bytes with the bitfield. About 2.5 MB/s in total. The control plane is tiny; the whole problem is bandwidth.

• Signing key (KMS · key in an HSM): Signs the manifest’s Merkle root together with its name, version, size and expiry. The private key never leaves the HSM, and every agent ships with the public key pinned. A compromised tracker or peer can then slow or misdirect a transfer, but cannot get a machine to load bytes the release pipeline did not produce.
• Manifest builder (64 MB pieces · SHA-256): Runs once per artifact, next to the bytes: splits the file into fixed 64 MB pieces (3,125 for 200 GB), hashes each, builds a Merkle tree over the piece hashes and has the root signed. Fixed size rather than content-defined chunks, so a piece’s index alone gives its offset. Hashing 200 GB is about 100 core-seconds.
• Origin store (object store · 10 GB/s cap): The single source of truth for artifacts and manifests, addressed by Merkle root so a path never changes meaning. It is shared with training jobs, so distribution may use 10 GB/s of it. In this design it is read about once per artifact, by the origin seeders, and never by the 10,000 agents.
• Origin seeders (8 hosts · 100 Gbps): A small tier in the origin’s data centre that reads each piece from the origin once (each host a disjoint eighth, 1.25 GB/s each), shares pieces among themselves, and serves every other data centre’s seeders across the WAN. They also act as the DC seeders for their own data centre, and keep the last few artifacts so rollbacks and late joiners never touch the origin.
• DC seeders (8 per DC · WAN slices): The only hosts in a data centre allowed to pull across the WAN. Each owns the pieces whose index mod 8 is its slot and pulls them under a 625 MB/s token bucket, so the eight together hold the link at its 40 Gbps budget. They inject pieces into the DC; the peers multiply them, since 100 GB/s of seeder upload is 2 % of the DC’s 5 TB/s demand.
• Release pipeline (training or build job): The job that produced the checkpoint or image. It runs the manifest builder, starts a distribution for the signed root and a target set, watches progress, and hands off to the deploy system once each pool’s quorum has verified.
• Distribution API (start · status · cancel): Authenticated entry point for release pipelines and operators: create a distribution, read its progress, cancel it. It fetches the manifest and checks its signature before anything moves, so a bad artifact fails here and not on 10,000 hosts. Starts are idempotent on a client key so a retry never launches a second swarm.
• Distribution controller (global · leader-elected): Plans each distribution: hosts per DC, the WAN relay plan, each link’s budget and the seeders’ slices, and pushes it to the trackers. Merges per-DC roll-ups into the global view and decides when a pool is done, on a quorum of verified hosts rather than 100 %. Its durable state is small and lives in the state store, so failover to a new leader is quick.
• DC tracker (soft state · 1 per DC): Admits agents in its data centre by certificate identity and target set, hands out topology-aware peer lists (about 20 rack-mates, 10 from other racks, plus the DC seeders), and takes a heartbeat from each agent every 2 s, 1,000 a second per DC. It is never on the data path, and everything it holds is rebuilt from the next round of heartbeats, so a standby needs no replication.
• Peer agent (on every host · mTLS): A daemon on each of the 10,000 hosts. Checks the manifest’s signature, then downloads pieces rarest-first from about 30 neighbours, verifies each piece’s hash before writing or serving it, and uploads a piece to others the moment it verifies. Persists its bitfield so a restart resumes, and caps itself at 20 Gbps each way.
• Distribution state (Postgres / etcd): Distributions, manifests, target sets, plans, budgets and the final outcome per host (verified, evicted, failed). Kilobytes per distribution plus one row per host. It deliberately does not hold live piece availability: that is soft state in the trackers.
• Progress view (time series · per DC): Counters and histograms every few seconds: hosts verified, rate and ETA distributions per DC, bytes per link, the rarest pieces, and stragglers with causes. Histograms rather than averages so percentiles merge correctly across data centres. A view for people and the deploy system; the per-host outcome of record is in the state store.
• Local NVMe (preallocated · atomic rename): Each verified piece is written at offset index × 64 MB into a preallocated file, and the bitfield is updated only after the data is durable. When the last piece lands the agent checks the root, fsyncs and renames the file into place, so a loader sees either the old artifact or the whole new one. The previous artifact stays for rollback.
• Neighbour peers (~30 · rack first): The other agents this one is connected to: about 20 in its own rack and 10 elsewhere in the same DC, never another DC. They exchange bitfields once and a HAVE message per verified piece, which is how each agent sees rarity. No tit-for-tat choking: every peer is ours, so upload slots exist for fairness and caps, not incentives.

1. Publish a checkpoint and seed it: The write path. The artifact is chunked and signed once, read from the origin once, and lands in every data centre’s seeders within the WAN budget, pipelined so agents can start in the first second.
   1. The release pipeline hands the finished 200 GB checkpoint to the manifest builder.
   2. The builder cuts 3,125 pieces of 64 MB, hashes each, and builds a Merkle tree over the hashes.
   3. The root is signed, and the pieces and signed manifest are uploaded to the origin under that root.
   4. The pipeline starts a distribution for that root and a target set.
   5. The controller computes the plan: link budgets, which DC pulls from which, and each seeder’s slice.
   6. Origin seeders read each piece from the origin once, each a disjoint eighth, and share among themselves.
   7. Each remote DC’s seeders pull their slices across the WAN from the origin seeders, under the link budget.
2. One machine downloads and verifies: The read path, run 10,000 times at once. The agent pulls by rarity from rack-mates first, verifies every piece before using or sharing it, and is uploading within a second of starting.
   1. The agent announces itself to its DC’s tracker and gets a peer list.
   2. It fetches the signed manifest from a DC seeder and checks it against its pinned key.
   3. It connects to its neighbours over mTLS and exchanges bitfields.
   4. It requests the rarest pieces it lacks, from rack-mates first, with several blocks in flight per connection.
   5. When a piece’s last block arrives, the agent hashes it and writes it to disk only if it matches.
   6. It sends HAVE to its neighbours and starts serving that piece immediately.
   7. Every 2 s it reports progress to the tracker, which may swap some neighbours.
   8. With every piece verified, it checks the root, fsyncs, renames the file into place and reports complete.
3. 10,000 machines start in the same second: The scale case: an empty swarm, five data centres, one capped origin and capped WAN links. It shows the swarm reaching close to the 80 s floor instead of collapsing onto the seeders, and no link going over its budget.
   1. The controller opens the distribution and all 10,000 agents announce within a few seconds.
   2. With no peer holding anything yet, agents ask the DC seeders, which give each agent different pieces.
   3. Within seconds each agent holds a few distinct pieces and trades with its neighbours.
   4. Remote seeders are still filling over the WAN, so new pieces enter each remote DC at 5 GB/s.
   5. Rack-first selection keeps rack uplinks well below capacity.
   6. Copies stay even: when the swarm is 90 % done, no piece is held by only a few hosts.
   7. The median host verifies at about 100 s and 99 % by about 3 minutes.
4. Peers die and one sends bad bytes: The failure path. A peer stalls, a disk returns garbage, a host reboots, the tracker and a seeder crash. Nothing restarts from zero: each failure costs at most the blocks in flight from one sender.
   1. A neighbour’s link flaps and a block request to it stalls past its deadline.
   2. The agent cancels, re-requests the block from another holder, and deprioritises the slow peer.
   3. A peer with a failing disk sends a piece that fails its hash check.
   4. The agent reports the sender, and the tracker quarantines it after two independent reports.
   5. A host reboots halfway through and resumes from its persisted bitfield.
   6. The DC tracker crashes; agents keep trading with the neighbours they already have.
   7. A DC seeder dies mid-slice, and its remaining pieces are reassigned to the other seven.
5. Declaring done with a slow 1 %: The background path that decides the rollout. Heartbeats roll up into a live view, the last pieces are fetched in endgame mode, and each pool completes on a quorum instead of waiting for the slowest host in 10,000.
   1. Trackers roll heartbeats up per DC and send them to the controller every 5 s.
   2. The controller writes the merged view, and the pipeline reads it through the API.
   3. An agent whose last blocks are all requested enters endgame and asks every holder for them.
   4. The controller flags hosts far below the DC median and asks them why.
   5. Each pool is done when 99 % of its hosts have verified; the rest go to repair rather than being waited on.
   6. The pipeline is notified and starts the model swap; stragglers keep downloading or are drained.
   7. Later, seeders and agents garbage-collect old artifacts, keeping the current and previous one.

Ask: Why not have every machine download from the origin, or build a broadcast tree?

Good answers name: Every machine pulls from the origin or a CDN, Store-and-forward tree (forward only after the whole file arrives), Pipelined fixed broadcast tree (forward each piece as it arrives), Swarm: pieces, rarest-first, everyone uploads as soon as a piece verifies.

Our pick: A swarm with 64 MB pieces: each agent pulls the rarest pieces it lacks from about 30 neighbours and serves each piece to others the moment its hash verifies. Pipelining is the whole trick: a piece reaches its 10,000th host a few hops and a second or two after its first, so the rollout costs about one file time, 80 s at 2.5 GB/s, plus startup and a tail. A seeder tier reads the origin once and each WAN link carries about one copy, so the capped resources stay well under their caps. Trees are not wrong: interleaved pipelined trees reach the same throughput, but every slow interior host becomes the problem of everyone below it, and repairing a tree mid-transfer is the hard part. Facebook has described using BitTorrent to push its large web server binary to its fleet during deploys, and Uber’s Kraken and Alibaba’s Dragonfly are open-source peer-to-peer distributors for container images built on the same idea.

1. Derive the 80 seconds. What makes the real number worse?
   Each host must receive 200 GB and can take at most 2.5 GB/s, so no design beats 80 s for that host, and the swarm’s 25 TB/s of total upload is exactly enough to feed all 10,000 at that rate. Startup adds a few seconds while the first pieces spread, pipelining adds a few hops of 0.1 to 0.2 s, and the tail adds whatever the slowest few percent cost. I would expect the median near 100 s and p99 at 2 to 3 minutes. If the median were far above that, I would look at rack uplinks and seeders first, because that is where a topology mistake shows up.
2. It is a 5 GB container image instead. Does anything change?
   The shape stays and the numbers shrink: 5 GB at 2.5 GB/s is a 2 s floor, so fixed costs dominate, such as announces, TLS handshakes and the first pieces spreading. I would use 16 MB pieces so there are still ~300 to trade. Below about 1 GB, or for a few hundred machines, a per-DC pull-through registry cache is simpler and fast enough; the swarm earns its complexity when size × machine count is large.
3. Half the hosts have 100 Gbps NICs and half 25 Gbps. Does the swarm cope?
   Better than a tree would. Fast hosts download faster and, more importantly, upload more, so they become the main sources without any configuration. Each host finishes near its own floor, 20 s or 80 s, as long as total upload covers the demand, which it does here. The one change is a per-host upload cap proportional to its NIC, so a slow host is never asked to serve more than it can.
4. Why not IP multicast? The network can copy packets for free.
   Most data-centre networks do not enable multicast across racks, and it has no reliability of its own, so you add a repair protocol for lost packets. One slow receiver then forces the group to slow down or needs a separate repair path, and those repair paths end up looking like a peer-to-peer swarm anyway. Where a fabric supports it well, multicast for the bulk plus unicast repair is a fair optimisation, not the base design.
5. How would you know the swarm is actually close to optimal?
   Put the floor on the dashboard: per-DC p50 and p99 against size ÷ host bandwidth, origin egress against one copy, and each WAN link’s bytes against one copy per DC. I would also track how much of the fleet’s upload budget is in use during the transfer. If it sits well below 80 %, peers are idle for lack of pieces to trade, which points at piece selection or peer lists.

Ask: How big is a piece, and how does a machine trust a piece it got from another machine?

Good answers name: One hash for the whole file, Fixed 64 MB pieces, a signed manifest of SHA-256 piece hashes with a Merkle root, Merkle tree down to small blocks, with a proof per block, Content-defined chunks.

Our pick: Fixed 64 MB pieces, 3,125 for 200 GB. The manifest lists name, version, size, piece size, the SHA-256 of every piece, a file table mapping pieces to the checkpoint’s shard files, and an expiry; the Merkle root over the piece hashes is the artifact’s identity, and the KMS signs the root with the metadata. An agent checks the signature with its pinned public key, recomputes the root from the list, and then accepts any piece from anyone whose hash matches. Pieces are requested in 4 MB blocks, possibly from several peers; a piece that fails is dropped and its blocks re-fetched one sender at a time to find the culprit. Peers talk over mTLS with workload certificates, which controls who is in the swarm and keeps the weights encrypted on the wire, but the hashes, not TLS, are what guarantee the bytes. Hashing costs about 1.25 cores at 2.5 GB/s with SHA-256 CPU extensions; BLAKE3 is several times faster if that matters.

1. Why both a signature and mTLS? Isn’t one enough?
   They protect different things. The signature proves the bytes are what the release pipeline produced, end to end, however many peers relayed them; mTLS cannot, because an authenticated but buggy or compromised peer can still send wrong bytes. mTLS proves who is on the other end and encrypts the transfer, which matters because the weights are valuable. Drop the signature and one compromised peer can poison the fleet; drop mTLS and anyone on the network can join the swarm and download the weights.
2. Someone replays an old, correctly signed manifest as if it were the new release. What stops it?
   Three checks. The manifest carries an expiry, so a stale one fails after its window. The distribution names the exact root to fetch, and the agent gets that root from the authenticated tracker, not from a peer. And the agent refuses a lower version than the one it holds unless the distribution is marked as a rollback. That is the rollback-attack protection The Update Framework formalises for software updates.
3. Why 64 MB and not 4 MB like a sync client?
   Because the costs that matter here are per piece: hashes in the manifest, bits in every bitfield and heartbeat, scheduling decisions, and HAVE messages to 30 neighbours. At 4 MB, 200 GB is 50,000 pieces and each agent sends 1.5 M HAVE messages per rollout. At 64 MB a pipeline hop is still only 0.1 to 0.2 s, and 3,125 pieces give 30 neighbours plenty to trade. For a 5 GB image I would drop to 16 MB so there are still about 300.
4. Every piece verified, yet the final root check fails on one host out of 10,000. What happened?
   Each piece was right when it was hashed, so something changed it afterwards, or it was written at the wrong offset. Both point at that host’s disk or a bug in its agent. The agent re-hashes every piece on disk, re-fetches the failing ones and reports; if it fails twice, the host is quarantined and gets a hardware ticket. One bad machine costs one machine, which is the point of per-piece hashes.
5. How do you keep the weights confidential if a host is compromised?
   First, limit what any host can get: the tracker admits a host only if its certificate identity is in the distribution’s target set. For stronger protection, encrypt the artifact with a per-release data key and let the swarm move only ciphertext; each host fetches the key from KMS with its own identity at load time. Then seeders, relaying peers and the origin never see plaintext, and revoking a host is revoking its access to the key.

Ask: Who knows which machine has which piece? Why not a DHT, like public BitTorrent?

Good answers name: A tracker per DC for peer lists and progress; peers choose pieces themselves, A central scheduler that assigns every piece transfer, A DHT (Kademlia) for discovery, Gossip and peer exchange only, seeded from a static list.

Our pick: One tracker per data centre with a hot standby, and the global controller above them. The tracker admits agents by certificate identity and target set, hands out peer lists of about 20 rack-mates, 10 hosts from other racks and the DC seeders, refreshes them on each 2 s heartbeat, and keeps every agent’s bitfield as soft state. Peers choose pieces themselves from neighbours’ bitfields and HAVE messages, so the tracker is never on the data path; if it dies, transfers carry on and the standby rebuilds from the next heartbeats. The one thing the coordinator does assign is the seeders’ WAN slices, because that is where global coordination saves real bandwidth. A DHT solves a trust and takedown problem this system does not have, and it gives up exactly the topology control this system needs. Uber’s Kraken is shaped the same way at a high level: a tracker introduces peers, and dedicated origin hosts seed.

1. DC3’s tracker is down for two minutes in the middle of a rollout. What breaks?
   Agents already in the swarm keep their connections and keep trading, so most of DC3 finishes on time. New or restarted agents cannot get a peer list; they fall back to the DC seeders in their config and to peer exchange with whoever they reach. DC3’s progress goes stale on the dashboard. When the standby takes over, it rebuilds from the next heartbeats in about 2 s, with nothing to replay.
2. At 100,000 machines, does one tracker per DC still work?
   Per DC that is 20,000 agents and 10,000 heartbeats a second of a few hundred bytes each, still one process with headroom. Beyond that I would shard the tracker by groups of racks, since peer lists are mostly rack-local anyway, and send bitfield deltas instead of whole bitfields. The controller does not notice, because it only sees per-DC roll-ups.
3. Why not let the tracker tell each peer exactly which piece to fetch next?
   Because a local choice is nearly as good and far more robust. A peer with 30 random neighbours sees rarity well enough, and rarest-first on a random graph keeps copy counts within a small factor of each other. Central assignment would put 400,000 decisions a second on the critical path and turn every scheduler pause into a fleet-wide stall. I keep central assignment for the one place it pays: which seeder fetches which slice across the WAN.
4. How do you pick neighbours so the swarm does not split into islands?
   Random within topology groups rather than nearest-only: about 20 random rack-mates and about 10 random hosts from other racks. The cross-rack links keep the graph connected and are how new pieces enter each rack, and random choice gives a graph only a handful of hops across. Each heartbeat can swap a few neighbours, which heals the graph as hosts die.
5. A bad release is caught halfway through. How do you stop it everywhere?
   Cancel through the API: the controller marks the distribution cancelled and pushes that to every tracker, and each agent learns it on its next heartbeat, within about 2 s. Agents stop requesting, refuse to serve that root to anyone, and delete partial data; seeders stop their WAN pulls. Because hosts that already finished have not loaded anything (loading is the deploy system’s job), nothing needs to be rolled back, only deleted.

Ask: Each machine sees 30 neighbours holding different pieces. Which piece should it ask for next?

Good answers name: Sequential, in file order, Random, Rarest-first among neighbours, finishing started pieces first, Endgame mode: ask every holder for the last blocks.

Our pick: Random for the first few pieces, rarest-first after that with ties broken at random, a rule that a started piece is finished before new ones are begun (so half-done, unverifiable pieces do not pile up), and endgame mode at the end. Rarity comes from neighbours’ bitfields and HAVE messages, and the seeders serve the pieces they have sent least, so new pieces enter each DC already spread out. Endgame starts when every missing block is already requested, about the last 480 MB with 30 connections × 4 blocks × 4 MB in flight: the agent asks every neighbour that has those blocks and cancels the others when one arrives. All of this is in Bram Cohen’s 2003 BitTorrent paper, minus tit-for-tat: inside a company every peer cooperates, so there is no choking for incentives, only upload slots for fairness.

1. The loader wants to stream layers into GPUs in order as they arrive. Doesn’t that argue for sequential?
   It argues for a mix. Give the next few pieces in file order a deadline-based priority and fetch everything else rarest-first, the way streaming BitTorrent clients do. In a swarm this size the head of the file spreads fast anyway, since everyone wants it. But loading usually cannot finish until the slowest piece lands, so I would measure whether streaming saves anything over waiting the extra seconds for the whole file.
2. Seeders hold every piece. Why does rarity matter at all?
   Because seeders can serve only 2 % of a DC’s demand. If a piece is held only by seeders and 500 peers want it, those 500 queue on 100 GB/s that is shared with everything else. Rarest-first makes peers grab such a piece early, so within seconds it has dozens of peer copies and the seeders are no longer needed for it. It also means a seeder dying costs nothing, because every piece already has many copies.
3. Endgame duplicates traffic. How much, and could it break a budget?
   It is bounded by what is in flight when it starts: about 480 MB per agent, asked of perhaps three holders each, with most duplicates cancelled within one block time, so well under 1 GB per host, under 0.5 % of 200 GB. It is all inside the DC and mostly inside the rack, so the WAN budget is untouched; seeders never duplicate across the WAN. When many agents hit endgame together, at the very end, the extra load falls on peers that are done and idle.
4. Peers keep choosing the same rare piece and hammer its one holder. What do you do?
   Break ties at random so peers spread over the rarest few pieces instead of one. Cap each peer at about 8 concurrent uploads, so a hot holder serves at its capacity and turns the rest away, which sends them to other rare pieces. And as soon as anyone completes that piece it sends HAVE and its rarity drops, so a hot spot lasts a fraction of a second.

Ask: How do you stop this from saturating the WAN and the rack uplinks?

Good answers name: Topology-blind random peers, Strict hierarchy: one leader per DC, then per rack, each fetching only from its parent, Rack-first, same-DC peer lists; DC seeders own disjoint WAN slices; token buckets on capped links, A DC-to-DC relay plan over the WAN.

Our pick: The tracker builds each agent’s list from its own data centre only: about 20 rack-mates, 10 random hosts from other racks, and the DC’s 8 seeders. An agent fetches a piece from a rack-mate if one has it, else from another rack, else from a seeder, and never crosses the WAN; with that rule a piece typically enters a rack two or three times and uplinks stay well below capacity. Only seeders cross the WAN, each owning the pieces whose index mod 8 is its slot, under a 625 MB/s token bucket so the eight together hold the link at 40 Gbps; the origin seeders read the origin under a 10 GB/s bucket. Each agent caps itself at 20 Gbps each way so the host’s own traffic keeps 5 Gbps. Budgets live in the controller’s plan and can change mid-rollout. It is the same principle as Netflix filling ISP caches off-peak, compressed into seconds: move the expensive copy once, then serve it locally.

1. DC5’s WAN budget is cut from 40 to 10 Gbps mid-rollout. What happens?
   The controller pushes a new plan and DC5’s seeders lower their buckets to about 156 MB/s each within a heartbeat. A full copy now takes 160 s to cross instead of 40, so DC5’s hosts finish near 160 s rather than 80; nothing else changes, because inside the DC the swarm keeps pace with whatever arrives. The dashboard shows DC5’s ETA move, and the deploy system can wait for it or go ahead in the other four.
2. Where does the topology data come from, and what if it is wrong?
   From the inventory system: host → rack → DC, which the network team already keeps for placement. Each agent reports its rack in the announce, and the tracker can cross-check against measured throughput, since rack-mates should see close to NIC speed. A wrong label costs efficiency, not correctness: that host just pulls more over its uplink. A rack whose uplink runs far hotter than its peers’ during a rollout is the signal to fix the inventory.
3. Why dedicated seeders? Ordinary peers could do the WAN fetch.
   They could, and in a small system I would let them. Dedicated seeders get 100 Gbps NICs, so one copy crosses at the full budget with few hosts; they are the only hosts that need WAN egress rules; and they keep the last few artifacts, so late joiners and rollbacks never go back to the origin. They also make the budget simple: eight known hosts per DC with a bucket each, not a budget spread over whichever peers happen to be fast.
4. The service on these hosts says distribution slows it down. What can you do?
   Lower the per-host cap, for example from 20 to 10 Gbps, which moves the floor to 160 s. Mark distribution traffic with a lower DSCP class so switches favour production packets. And schedule large rollouts off-peak, the Netflix idea. The swarm’s speed is simply the per-host budget, so this is a policy setting, not a redesign.

Ask: 99 % of machines finished in three minutes. Why is the rollout still running at minute twenty, and what do you do about it?

Good answers name: Wait for 100 % of targets, Per-block deadlines and re-requests, endgame, and a per-pool quorum for done, with stragglers sent to repair, Send every block request to two peers from the start.

Our pick: Inside the swarm, each block request has a deadline of max(1 s, 4× the expected time at that peer’s recent rate); on expiry it is cancelled and re-requested from another holder, and a peer that times out twice is dropped and replaced at the next heartbeat. Agents persist their bitfield after each durable piece so a reboot resumes, and a seeder that dies has its remaining slice reassigned. At the rollout level, agents heartbeat every 2 s, trackers roll up per DC every 5 s, and the controller flags a host as a straggler when its rate is below a quarter of the DC median or its ETA is past the p99, then asks the agent for a diagnosis: disk write rate, NIC errors, peer count, hashing rate. A pool is done when its quorum, 99 % by default, has verified; stragglers keep downloading and the deploy system keeps them out of serving until they verify. A host that fails verification twice, or cannot reach a tenth of the median rate, gets a repair ticket instead of more retries.

1. A training job needs all 1,024 of its hosts to have the weights. A quorum does not help. What then?
   Then done is 100 % of that job’s hosts, and the lever is replacing stragglers rather than waiting for them. Give the job a deadline, say two minutes past its p99; at the deadline the scheduler swaps the slow hosts for spares that already hold the artifact and starts. Distributing to the spare pool before the job starts makes the swap free. Waiting indefinitely on one bad disk is the expensive option.
2. How do you tell a host starved by the network from one limited by its own disk?
   From the agent’s own counters. If blocks arrive slowly from several peers that are serving others quickly, it is this host’s network path; if blocks arrive fast but disk write throughput or fsync latency is poor, it is the disk; if hashing is the slow stage, it is CPU contention. The cause goes into the roll-up, so the dashboard says "88 stragglers: 51 disk, 30 network, 7 unreachable" rather than one number. Each cause has its own fix, and only the network one is ours.
3. A whole rack goes dark for 30 seconds in the middle of a rollout.
   Its 40 agents miss heartbeats and drop out of other agents’ lists after 6 s; their neighbours re-request whatever they were fetching from them elsewhere, losing at most the blocks in flight. When the rack comes back, its agents resume from their persisted bitfields and re-announce. Because rarest-first kept every piece well replicated, the swarm loses no data; only those 40 hosts lose 30 seconds.
4. How does the progress view stay accurate without 10,000 machines flooding it?
   Aggregation at the edge: agents report to their DC tracker every 2 s, the tracker merges counts and histograms and sends one roll-up per DC every 5 s, so the controller sees about one message a second. Histograms of rate and ETA merge correctly across DCs, where averages and percentiles would not. Per-host detail stays in the tracker and is fetched only when someone drills into a straggler.
5. What would you alert on?
   Origin egress above two copies per artifact, which means the seeders are being bypassed; any WAN link above its budget; rack uplink utilisation above 70 % during a rollout; p99 completion more than twice the floor; hash-mismatch reports above a small baseline, which signal failing hardware or a bad agent release; and any distribution below its quorum past its deadline. The first two cost money or hurt other teams, so they page.