Design a Code Deployment System

Build each merge once into a signed artifact, roll it out in stages to 50,000 hosts in 10 regions, and roll back automatically on regression.

Last updated 2026-10-26. Difficulty: hard. Patterns: deployment, canary, reconciliation, rollback, blast-radius. Reported at Google, Meta, Amazon, Microsoft, Netflix, Uber.

Functional requirements

• Build each merge once. A hermetic build of the merged commit into an immutable container image named by its content digest, with a signed provenance record (commit, builder, tests). Staging, the canary, every region and any later rollback run those exact bytes; nothing is rebuilt.
• Deploy a release to a service’s fleet. “checkout runs release r1842 in these cells” is recorded as desired state, and the system converges 1,000 services on 50,000 hosts (about 500,000 instances) to it. Engineers say what should run, never how to get each host there.
• Staged rollout plans. Per service, chosen by its tier: staging, a canary of a few instances in one cell, the rest of that cell, the first region, then waves of regions. Each stage has a bake time and a health gate before the next one starts.
• Automated canary analysis. Compare the canary with a control group of the current release on errors, latency and saturation, and promote, hold or roll back with no human in the loop. Most of the 3,000 deploys a day should never need a person.
• Automatic and manual rollback. Back to the last known-good release within minutes, from any stage, including halfway through a wave. Rollback is a new rollout of the old digest, not an undo.
• Human controls. Pause, resume, promote (skip the rest of a bake, with a reason), abort and roll back from a CLI, UI or chat. Every action is authorised against the service’s owners and recorded.
• Concurrency and change policy. One rollout per service at a time, with newer releases queued behind it. Freezes and change windows by scope (global, org, service, region), and a global halt that stops every rollout within a minute.
• Audit and change history. Who deployed what, where, when and why, plus every automatic decision with its evidence, answerable as “what changed in this cell in the last hour?” during an incident.
• Out of scope. Code review and the merge queue, the test frameworks, moving the bytes to hosts (the peer-to-peer distribution system does that: we give it a digest and a target set), the cluster scheduler that places instances on hosts, and the feature flag service (we use it, we do not build it).

Non-functional requirements

• Bounded blast radius (canary ≤ 1 % · first cell 2 % · first region 10 %). The requirement that drives the hardest trade-off: speed against safety. Each stage’s bake buys confidence and costs time, so the plan, not the code, decides how many users a bad release can reach before it is caught.
• Time to detect and restore (detect < 10 min · restore < 5 min). A regression is caught within one or two 5-minute analysis intervals, and a cell is back on the previous release in about 2 minutes because that release is still on every host’s disk. Restore is restarts, never a build or a download.
• Deploy throughput and lead time (3,000 deploys/day · ~2 h merge to everywhere). Slow pipelines make engineers batch changes, and big batches are riskier and harder to bisect. A standard plan reaches every region in about 2 hours; tier-0 services take a day or more on purpose.
• Off the serving path (fleet serves and heals with the control plane down). The deploy system must never be a dependency of serving traffic. If the global control plane is down, nothing new ships, but every cell keeps restarting crashed instances and placing replacements at their current versions.
• One writer per service (no interleaved or stale writes). Two rollouts of one service must never interleave, and an orchestrator worker that paused past its lease must not be able to write an old decision over a newer one, such as a rollback.
• Integrity of what runs (only signed CI builds from main). A host starts an artifact only if its digest matches and its signature checks against a pinned key held by CI. A stolen engineer credential must not be enough to run code on 50,000 hosts.
• Capacity during a rollout (≥ 90 % per cell · ≥ 75 % in a rollback). Forward rollouts change at most 10 % of a cell’s instances at a time (or surge new ones first). A rollback accepts 25 % at a time, because the version it replaces is the problem.
• Auditability (append-only · kept for years). Change-management audits and incident reviews both need a complete record: no change without an entry, no entry ever edited.

Back-of-envelope estimates

• Deploys per working hour: ~300/h · 3,000/day. 1,000 services × ~3 deploys per working day = 3,000 deploys/day. Nearly all start inside a ~10-hour working window: 300/h, about 5 a minute. As a request rate this is nothing; throughput is not the hard part.
• Rollouts in flight at peak: ~600. Little’s law: 300/h × ~2 h on the standard plan = 600 rollouts in progress at once. With one rollout per service at a time, a service on that plan ships at most ~5 times a day, so each rollout carries every merge since the last one (a release train): ~10,000 service changes a day ÷ 3,000 deploys ≈ 3 merges per release.
• Artifact storage growth: ~150 TB/year. ~10,000 images built a day (7,500 merges touching ~1.3 services each) × ~400 MB = 4 TB/day logically, but layers are content-addressed and most are unchanged: ~40 MB of new layers per build ≈ 400 GB/day ≈ 146 TB/yr. Keep everything that ever reached production; expire the rest after 30 days.
• Bytes shipped to hosts: ~60 TB/day · ~1.7 GB/s. 3,000 deploys × ~500 instances each (500,000 instances over 1,000 services) × ~40 MB of changed layers = 60 TB/day, about 1.7 GB/s over a 10-hour day. The fleet’s NICs total 50,000 × 25 Gbps ≈ 156 TB/s. Distribution only matters for one huge service at once, which the P2P layer handles and prestaging moves off the critical path.
• Control-plane load: ~5 k status reports/s. 50,000 host agents report to their cell controller every 10 s = 5,000/s, 100/s per cell (50 cells of 1,000 hosts). Orchestrators read per-cell summaries, not host reports: 600 rollouts × a few cells every 10 s is well under 1 k/s. The design is about safety and failure, not load.
• Requests to judge a canary: ~24 k per group. To see the error rate double from 0.1 % to 0.2 % (80 % power, 5 % significance): n ≈ 16·p(1−p)/δ² = 16 × 0.0015 ÷ 0.001² ≈ 24,000 requests each for canary and control. 3 canary instances at 200 req/s get there in 40 s; a quiet service at 5 req/s per instance needs ~27 min. So a stage ends on a request count with a minimum time, not on the clock alone.
• Error budget a bad release spends: ~0.7 % in one cell · ~35 % everywhere. A release that fails half its requests, caught and reverted 15 minutes after reaching one cell carrying 2 % of traffic: 0.02 × 0.5 × 15 = 0.15 full-outage minutes, 0.7 % of a 99.95 % monthly budget (21.6 min). Pushed everywhere at once: 0.5 × 15 = 7.5 minutes, 35 %. Cells turn a bad deploy from an incident into a footnote.
• Time to roll a cell back: ~2 min. The previous release is still on every host’s disk, so rollback is restarts only: drain 10 s + start 15 s + readiness 5 s ≈ 30 s per batch. At 25 % of instances per batch that is 4 × 30 s = 2 min, with all affected cells in parallel. A forward roll at 10 % per batch takes 10 × 30 s = 5 min per cell.

Components

• Repo + merge queue (review · always green): Where every release starts. The merge queue tests each change rebased on the latest main before landing it, so main is always green and any commit on it is a valid release candidate. It tells CI about the commit; it never triggers a deploy itself.
• Build & test (hermetic · signs provenance): Builds each affected service once per merged commit, in a hermetic sandbox with pinned toolchains, hash-locked dependencies and no network, then runs its tests. Produces an image named by its content digest and a provenance record (commit, builder, inputs, test results) signed with a key only CI builders can use. Staging, production and rollbacks all promote that digest; nothing is built twice.
• Artifact registry (content-addressed · immutable): Image layers keyed by SHA-256, plus their signed attestations. A digest can never be overwritten, so “which bits are running” has one answer forever, and tags are never used for deploys. About 400 GB of new layers a day after deduplication; everything that reached production is kept for rollback and audit, the rest expires after 30 days.
• Artifact distribution (P2P swarm · prestaged): Moves layers from the registry to the hosts that will run them, as the large file distribution question designs it: a peer-to-peer swarm that reads the registry about once per release and stays inside per-host and per-link caps. Here it is a boundary. The orchestrator hands it a digest and a target set when the rollout starts, well before any stage needs the bytes.
• Load balancers (weights · slow start · drain): Spread requests over a service’s instances in each cell. They give canary and control equal weight so both see the same traffic mix, ramp new instances in on slow start, and drain old ones (no new requests, in-flight ones finish, 30 s cap) before they stop. A rollout is invisible to users only because of this step.
• Engineer / on-call (CLI · UI · chat): Merges code, watches rollouts and is the human in the loop: pause, resume, promote, abort, roll back, halt. Every action goes through the deploy API under the person’s own identity, never straight to hosts, so it is authorised and recorded. Most rollouts need no human; the controls exist for the few that do.
• Deploy API (authz · policy · halt): The one door for changing production: register releases, start, pause, promote, abort and roll back rollouts, set freezes, raise the halt. It checks the caller against the service’s owners, the change calendar and the release’s signature before writing anything, and every write carries an outbox row so no change exists without its audit record. The halt it also broadcasts straight to every cell controller.
• Rollout orchestrator (durable workflows): Runs each rollout as a durable workflow over its plan: take the service’s lock, prestage, write desired state stage by stage, ask for a canary verdict, wait out bakes, promote or roll back. Each decision is recorded in the release DB before it acts, so a worker that dies is replaced by one that resumes the same step (the Step Functions pattern, used here, not rebuilt). About 600 rollouts run at once on a small pool of stateless workers.
• Cell controller (one per cell · reconciler): Holds the desired state of every service in its cell (“checkout at r1842, generation 59, 10 % at a time”) in a cell-local replicated store and reconciles toward it: picks which instances change next, drains, restarts and waits for readiness, batch by batch. Rejects any write whose generation is lower than the one it holds, which fences out stale orchestrators. Keeps the cell running and healing when the global control plane is down.
• Host agent (50,000 hosts · level-triggered): Runs on every host, takes its targets from the cell controller and makes reality match: fetch, verify digest and signature, start, check readiness, register, drain and stop the old instance. It compares actual with desired on every loop instead of reacting to events, so a missed message or a reboot only delays convergence. Keeps the last two releases on disk, so a rollback moves no bytes.
• Audit log (append-only · change feed): Every change and decision: who or what deployed which digest where, approvals and overrides, and every automatic verdict with its evidence. Fed from the release DB’s outbox, so a change cannot commit without its record. Indexed by cell, service and time to answer the first question of every incident, “what changed here in the last hour?”, and kept for years.
• Release DB (consensus · multi-region): Releases, plans, rollouts and their step history, freezes, the halt flag, and one row per service holding the lock (which rollout owns it), a generation counter and the last known-good release. Strongly consistent because two orchestrator workers must never both believe they own a service. A few writes a second, replicated across regions so losing one does not stop deploys elsewhere.
• Canary analysis (canary vs control · stats): Given a canary group, a control group and a window, pulls the same metrics for both and tests each for a significant difference (a two-proportion test on errors, Mann-Whitney U on latency distributions), then combines them into a score: pass, fail, or not enough data yet. Also runs the guards in later stages, such as SLO burn rate on updated cells. Netflix and Google’s open-source Kayenta works this way.
• Metrics store (tagged by release · group): Per-instance time series (requests, errors, latency histograms, CPU, memory and each service’s business metrics) tagged with release, cell and canary or control group. The tags are what make the comparison possible: without them a canary’s errors vanish into the service’s totals. Histograms rather than precomputed percentiles, so groups and intervals merge correctly.

User flows

1. Merge to a signed release. Build once, deploy many. One merged commit becomes one immutable, signed artifact and one release record; every environment and every later rollback refers to that digest, so what was tested is exactly what runs.
   1. The merge queue lands a reviewed change on main and tells CI which commit to build. The queue tests each change rebased on the latest main before it lands, so main never breaks and every commit on it is a release candidate. About 7,500 merges a day touch ~10,000 service builds. CI builds the affected services at exactly that commit, never at “the tip of main” a few minutes later.
   2. CI builds each affected service hermetically from that commit and runs its tests. Hermetic means pinned toolchains, dependencies locked by hash and no network during the build, so the output depends only on the commit. Google’s SRE book puts hermetic builds at the base of its release engineering for this reason. Builds and tests take 10 to 15 minutes.
   3. CI pushes the image to the registry, where its name is the digest of its content. Layers already in the registry are skipped, so a 400 MB image usually adds about 40 MB. Tags such as checkout:latest are mutable and the deploy system never reads them: it speaks only digests, so “what is running” has exactly one answer.
   4. CI signs a provenance record for the image: commit, builder, inputs and test results. The signing key lives in a KMS that only CI builders can call. Host agents verify the signature against a pinned public key before starting anything, so an artifact built on a laptop, or by a stolen credential, is refused by every host in the fleet.
   5. CI registers a release: one immutable record that points at the digest. A release is (service, digest, attestation, commits since the previous release) and is never edited. Promotion from staging to production changes which stage a release has reached, never its bits. The outbox row written in the same transaction becomes the audit entry.
   6. When the service’s lock is free, the orchestrator starts a rollout of the newest waiting release. If r1840 is still rolling out, r1841 and r1842 wait. When the lock frees, only r1842 deploys, because it contains r1841’s commits too; queues never grow and each rollout carries about 3 merges. Taking the lock is a compare-and-set on the service’s row.
2. Roll a release out in stages. The main path: staging, a canary judged against a control group, one cell, one region, then waves, with a bake after each. Every stage is a desired-state write that the cells reconcile, and every decision is durable before it acts.
   1. The orchestrator asks the distribution layer to prestage the image on every host that will run it. Transfer is separated from activation. Bytes move at background priority, under per-host caps, long before a stage needs them, so switching a cell later is restarts only. The swarm reads the registry about once, however many hosts there are.
   2. Staging: the pre-production cell runs the release and its integration and smoke tests. Same digest as production; only the config differs, and config is its own versioned artifact. About 10 minutes. The write carries the service’s generation, which the cell stores and checks on every later write.
   3. Canary: the first production cell starts 3 instances of the new release and 3 fresh instances of the current one as a control. Starting the control at the same moment means both groups have cold caches and new processes; compared with week-old instances, every release would look slow. Both groups are extra (surge) instances with equal load-balancer weight, so the cell loses no capacity and both see the same traffic.
   4. Agents fetch the pieces, check digest and signature, start the instance and register it once it is ready. Verification happens before exec, not after. Readiness (serving, dependencies connected) gates registration with the load balancer; liveness is a separate check. Metrics from these instances carry release and group tags from the first request.
   5. Every 5 minutes canary analysis compares canary and control on errors, latency and saturation. Each metric is tested (a two-proportion test for errors, Mann-Whitney U for latency) and classed pass, high or low; the score is the weighted share that pass. ≥ 95 promotes, < 75 rolls back, in between keeps collecting. The stage needs at least 20 minutes and 24 k requests per group.
   6. On a pass, the rest of the cell moves to the new release 10 % at a time, then bakes for 15 minutes. The cell controller sequences instances: drain, stop, start, ready, next, so the cell keeps at least 90 % of its capacity. A whole cell exercises the release at full load on its shared dependencies while confining any damage to about 2 % of traffic. During the bake, guards watch the cell’s SLO burn rate.
   7. The orchestrator widens stage by stage: the region’s other cells, then 3 regions, then the last 6, baking after each. Each stage is recorded in the release DB before any cell is written, with the generation bumped under the lock, so a worker that crashes is replaced by one that rewrites the same stage. 4 cells (bake 30 min), 15 cells (bake 20 min), 30 cells: about 10 %, 40 % and then 100 % of traffic.
   8. After the last wave and a 15-minute watch, the release becomes the service’s known-good version and the lock is released. The last-good pointer is what every automatic rollback targets, so only releases that finished a whole plan qualify. About 2 hours from staging to everywhere on the standard plan. The next queued release can now start.
3. Canary fails and rolls itself back. The case the system exists for: a regression far too small to move any fleet-wide alert, caught on 3 instances by comparing them with their control, and reverted with no human involved.
   1. Five minutes in, analysis finds the canary’s error rate at 0.4 % against the control’s 0.1 %. With 30 k requests per group the difference is about 7 standard errors, and p99 latency is up 35 %. The service’s fleet-wide error rate moved only in the fourth decimal place, so no threshold alert would ever have fired. Errors are a critical metric and can fail the canary on their own.
   2. The orchestrator records the failed verdict and its evidence before changing anything. Decision first, action second. If the worker dies now, the next one finds the rollout in rolling_back and finishes the rollback instead of judging the canary again. The evidence goes to the audit log with the decision.
   3. It writes the cell’s desired state back to the current release alone, without canary or control. Only 3 instances ever ran r1842; the rest of the cell never changed. The write names r1841 explicitly, with a higher generation, rather than saying “undo”: rollback is always a roll forward to a known-good release.
   4. Agents drain and stop the canary and control instances; all traffic is back on the old version within a minute. Deregister first, let in-flight requests finish (up to 30 s), then stop. The control goes too: it was surge capacity, and leaving it would only skew the next canary.
   5. The release is marked rejected, the lock is released and the owner is paged with the analysis report. A rejected release can never be deployed again, so the queue waits for a newer build. The page links the failing metrics by group and the commits in the release, which is usually enough to find the culprit among about 3 merges.
   6. The owner fixes forward: a new build, r1843, starts the whole plan again from staging. The fix gets no shortcut through the stages: it is as untested as the bug was. Only an incident fix may use the expedited plan, with a second approver.
4. The biggest service: 20,000 instances. What changes at the top of the size distribution: a 2 GB image on 20,000 hosts, a tier-0 plan, surge instead of shrink, slow start and draining, and restarts paced so they do not stampede the service’s own dependencies.
   1. A web frontend release, 20,000 instances with a 2 GB image, starts by prestaging the image everywhere. 20,000 × 2 GB = 40 TB. Pulled straight from the registry at 10 GB/s that is over an hour; through the swarm the registry is read about once and hosts finish in minutes. Prestaging ends when 99 % of hosts have the image; stragglers fetch it when their turn comes.
   2. The tier-0 plan applies: a stricter canary, then one region per wave with 60-minute bakes, inside each region’s business hours. Plans are data chosen by tier. This one takes one to two days, because the web frontend is every user’s first request and an outage there is a company-wide incident. A regression found at 3 a.m. by a tired on-call costs more than a day’s delay.
   3. Canary analysis uses the extra traffic for a stricter test and adds business metrics. 4 canary instances at 500 req/s reach 24 k requests in 12 seconds, so the minimum time, not the sample size, sets the stage: 60 minutes, long enough to see memory and connection counts trend. Page load success and sign-in rate are in the config next to errors and latency.
   4. Within each cell the controller surges instead of shrinking: start 40 new instances, wait until they are ready, then drain 40 old ones. The frontend runs hot at peak, so it keeps 100 % of its capacity during the roll at the cost of 10 % extra hosts for the duration. 400 instances per cell in batches of 40 is 10 batches, about 10 minutes a cell.
   5. New instances join the load balancer on slow start, and old ones drain their connections before they stop. Slow start ramps a new instance’s weight over 60 s, so cold caches and an unwarmed JIT do not take a full share at once. Draining stops new requests, sends GOAWAY on long-lived HTTP/2 connections and waits up to 30 s; clients reconnect with jitter.
   6. The cell controller paces batches by its agents’ readiness reports, so restarts do not stampede shared dependencies. 40 instances starting together each open database and cache connections and load config. Starts are jittered over 20 s, and the next batch waits until this one is ready and its error rate is flat. A batch that fails readiness stops the cell and reports to the orchestrator.
   7. The orchestrator reads one summary per cell, never 20,000 host reports. A cell is done when it reports the stage’s generation observed and at least 99 % of target instances ready on the new release. Hosts that never converge are named in the summary and replaced by the scheduler rather than waited on.
5. Stop everything during an incident. The operational path: find what changed, stop every rollout within a minute without depending on the orchestrators, roll back the suspect, let the fix through, and resume. Every step lands in the audit log.
   1. On-call asks what changed in the failing cells in the last hour. The first question of every incident. The change feed lists rollouts, config pushes and flag changes by cell and time, so “search r2211 reached use1 23 minutes before errors rose” is one query, not an hour of asking in chat.
   2. The incident commander raises the global halt. Changes during an incident muddle diagnosis and can make it worse. Anyone on call for a tier-0 service can set the halt; lifting it takes two people, because forgetting to lift it is cheap and lifting it by mistake is not.
   3. The API broadcasts the halt to every cell controller directly, not through the orchestrators. The brake must not depend on the engine: if orchestrators are what is broken, the halt still lands. 50 cells acknowledge within seconds. Each finishes the batch it is restarting (at most 30 s of work) and starts no new one.
   4. Orchestrators see the halt before their next step and pause all 612 rollouts where they stand. Paused, not rolled back. Each service sits in a mixed but stable state, which is safe because every release must run alongside the one before it. Bake clocks stop, so no stage is promoted on a bake that ran during the incident.
   5. The suspect release is rolled back to its last known-good version; rollbacks are allowed through the halt. Rolling back moves toward a version that ran everywhere recently, so it is the one change the halt must allow. It takes search’s lock from the paused rollout and bumps the generation, so that rollout can never write again. All 5 affected cells roll back in parallel, 25 % at a time: about 2 minutes.
   6. The incident’s own fix ships as a break-glass rollout on the expedited plan, with a second approver. Still a signed CI build of a reviewed commit; only the plan changes: a 5-minute canary, then regions in parallel. The approver and the incident id are on the rollout and in the audit log.
   7. Two people lift the halt; paused rollouts resume, each re-baking its current stage. Resuming does not trust the bake time that ran during the incident: metrics from that hour say nothing about the releases. Rollouts that were in their canary restart the analysis window.

Deep dives

The theory behind it

• Canary deployments: an interactive playground. Roll out a buggy release all at once, rolling, blue-green and as a canary, and compare how fast each catches the bug, how many requests fail and how long a healthy rollout takes.
• Burn rate alerting: an interactive playground. Run three days of errors with an outage, a slow leak and a blip past fixed thresholds and multi-window burn-rate alerts, and see which page, when, and for what.
• Online schema migration: an interactive playground. Rename a column while four app servers roll through deploys: in one step, with expand and contract done in the wrong order, and in the right order, and count failures, stale reads and lost changes.
• Feature flag rollouts: an interactive playground. Roll a feature out to a percentage of 400 users by random choice, by a hash of the user id and by a hash of flag and user, and see flicker, lost access and flags that overlap.
• Deployment strategies and safe releases: Shipping changes without outages: rolling deployments, blue-green, canary releases with automated analysis, feature flags versus deploys, backward-compatible changes across services and databases, progressive delivery across regions, rollbacks and deploy freezes.
• Cell-based architecture: Limiting the blast radius of failures and bad deploys by splitting a system into independent cells: what a cell contains, routing customers to cells, sizing and placement, cell migration, shared global services, deployment by waves, and the costs of running many copies of a system.
• SLIs, SLOs and error budgets: Defining reliability with numbers: choosing SLIs, setting SLO targets, availability nines and what they allow, error budgets and how teams spend them, burn-rate alerting, and SLAs versus SLOs.

Related

• Design a Container Orchestrator
• All 83 system design interview questions
• Run this question for a friend: the interviewer kit