Design a Code Deployment System

• Build each merge once: A hermetic build of the merged commit into an immutable container image named by its content digest, with a signed provenance record (commit, builder, tests). Staging, the canary, every region and any later rollback run those exact bytes; nothing is rebuilt.
• Deploy a release to a service’s fleet: “checkout runs release r1842 in these cells” is recorded as desired state, and the system converges 1,000 services on 50,000 hosts (about 500,000 instances) to it. Engineers say what should run, never how to get each host there.
• Staged rollout plans: Per service, chosen by its tier: staging, a canary of a few instances in one cell, the rest of that cell, the first region, then waves of regions. Each stage has a bake time and a health gate before the next one starts.
• Automated canary analysis: Compare the canary with a control group of the current release on errors, latency and saturation, and promote, hold or roll back with no human in the loop. Most of the 3,000 deploys a day should never need a person.
• Automatic and manual rollback: Back to the last known-good release within minutes, from any stage, including halfway through a wave. Rollback is a new rollout of the old digest, not an undo.
• Human controls: Pause, resume, promote (skip the rest of a bake, with a reason), abort and roll back from a CLI, UI or chat. Every action is authorised against the service’s owners and recorded.
• Concurrency and change policy: One rollout per service at a time, with newer releases queued behind it. Freezes and change windows by scope (global, org, service, region), and a global halt that stops every rollout within a minute.
• Audit and change history: Who deployed what, where, when and why, plus every automatic decision with its evidence, answerable as “what changed in this cell in the last hour?” during an incident.
• Out of scope: Code review and the merge queue, the test frameworks, moving the bytes to hosts (the peer-to-peer distribution system does that: we give it a digest and a target set), the cluster scheduler that places instances on hosts, and the feature flag service (we use it, we do not build it).

• Bounded blast radius (canary ≤ 1 % · first cell 2 % · first region 10 %): The requirement that drives the hardest trade-off: speed against safety. Each stage’s bake buys confidence and costs time, so the plan, not the code, decides how many users a bad release can reach before it is caught.
• Time to detect and restore (detect < 10 min · restore < 5 min): A regression is caught within one or two 5-minute analysis intervals, and a cell is back on the previous release in about 2 minutes because that release is still on every host’s disk. Restore is restarts, never a build or a download.
• Deploy throughput and lead time (3,000 deploys/day · ~2 h merge to everywhere): Slow pipelines make engineers batch changes, and big batches are riskier and harder to bisect. A standard plan reaches every region in about 2 hours; tier-0 services take a day or more on purpose.
• Off the serving path (fleet serves and heals with the control plane down): The deploy system must never be a dependency of serving traffic. If the global control plane is down, nothing new ships, but every cell keeps restarting crashed instances and placing replacements at their current versions.
• One writer per service (no interleaved or stale writes): Two rollouts of one service must never interleave, and an orchestrator worker that paused past its lease must not be able to write an old decision over a newer one, such as a rollback.
• Integrity of what runs (only signed CI builds from main): A host starts an artifact only if its digest matches and its signature checks against a pinned key held by CI. A stolen engineer credential must not be enough to run code on 50,000 hosts.
• Capacity during a rollout (≥ 90 % per cell · ≥ 75 % in a rollback): Forward rollouts change at most 10 % of a cell’s instances at a time (or surge new ones first). A rollback accepts 25 % at a time, because the version it replaces is the problem.
• Auditability (append-only · kept for years): Change-management audits and incident reviews both need a complete record: no change without an entry, no entry ever edited.

• Deploys per working hour: ~300/h · 3,000/day. 1,000 services × ~3 deploys per working day = 3,000 deploys/day. Nearly all start inside a ~10-hour working window: 300/h, about 5 a minute. As a request rate this is nothing; throughput is not the hard part.
• Rollouts in flight at peak: ~600. Little’s law: 300/h × ~2 h on the standard plan = 600 rollouts in progress at once. With one rollout per service at a time, a service on that plan ships at most ~5 times a day, so each rollout carries every merge since the last one (a release train): ~10,000 service changes a day ÷ 3,000 deploys ≈ 3 merges per release.
• Artifact storage growth: ~150 TB/year. ~10,000 images built a day (7,500 merges touching ~1.3 services each) × ~400 MB = 4 TB/day logically, but layers are content-addressed and most are unchanged: ~40 MB of new layers per build ≈ 400 GB/day ≈ 146 TB/yr. Keep everything that ever reached production; expire the rest after 30 days.
• Bytes shipped to hosts: ~60 TB/day · ~1.7 GB/s. 3,000 deploys × ~500 instances each (500,000 instances over 1,000 services) × ~40 MB of changed layers = 60 TB/day, about 1.7 GB/s over a 10-hour day. The fleet’s NICs total 50,000 × 25 Gbps ≈ 156 TB/s. Distribution only matters for one huge service at once, which the P2P layer handles and prestaging moves off the critical path.
• Control-plane load: ~5 k status reports/s. 50,000 host agents report to their cell controller every 10 s = 5,000/s, 100/s per cell (50 cells of 1,000 hosts). Orchestrators read per-cell summaries, not host reports: 600 rollouts × a few cells every 10 s is well under 1 k/s. The design is about safety and failure, not load.
• Requests to judge a canary: ~24 k per group. To see the error rate double from 0.1 % to 0.2 % (80 % power, 5 % significance): n ≈ 16·p(1−p)/δ² = 16 × 0.0015 ÷ 0.001² ≈ 24,000 requests each for canary and control. 3 canary instances at 200 req/s get there in 40 s; a quiet service at 5 req/s per instance needs ~27 min. So a stage ends on a request count with a minimum time, not on the clock alone.
• Error budget a bad release spends: ~0.7 % in one cell · ~35 % everywhere. A release that fails half its requests, caught and reverted 15 minutes after reaching one cell carrying 2 % of traffic: 0.02 × 0.5 × 15 = 0.15 full-outage minutes, 0.7 % of a 99.95 % monthly budget (21.6 min). Pushed everywhere at once: 0.5 × 15 = 7.5 minutes, 35 %. Cells turn a bad deploy from an incident into a footnote.
• Time to roll a cell back: ~2 min. The previous release is still on every host’s disk, so rollback is restarts only: drain 10 s + start 15 s + readiness 5 s ≈ 30 s per batch. At 25 % of instances per batch that is 4 × 30 s = 2 min, with all affected cells in parallel. A forward roll at 10 % per batch takes 10 × 30 s = 5 min per cell.

• Repo + merge queue (review · always green): Where every release starts. The merge queue tests each change rebased on the latest main before landing it, so main is always green and any commit on it is a valid release candidate. It tells CI about the commit; it never triggers a deploy itself.
• Build & test (hermetic · signs provenance): Builds each affected service once per merged commit, in a hermetic sandbox with pinned toolchains, hash-locked dependencies and no network, then runs its tests. Produces an image named by its content digest and a provenance record (commit, builder, inputs, test results) signed with a key only CI builders can use. Staging, production and rollbacks all promote that digest; nothing is built twice.
• Artifact registry (content-addressed · immutable): Image layers keyed by SHA-256, plus their signed attestations. A digest can never be overwritten, so “which bits are running” has one answer forever, and tags are never used for deploys. About 400 GB of new layers a day after deduplication; everything that reached production is kept for rollback and audit, the rest expires after 30 days.
• Artifact distribution (P2P swarm · prestaged): Moves layers from the registry to the hosts that will run them, as the large file distribution question designs it: a peer-to-peer swarm that reads the registry about once per release and stays inside per-host and per-link caps. Here it is a boundary. The orchestrator hands it a digest and a target set when the rollout starts, well before any stage needs the bytes.
• Load balancers (weights · slow start · drain): Spread requests over a service’s instances in each cell. They give canary and control equal weight so both see the same traffic mix, ramp new instances in on slow start, and drain old ones (no new requests, in-flight ones finish, 30 s cap) before they stop. A rollout is invisible to users only because of this step.
• Engineer / on-call (CLI · UI · chat): Merges code, watches rollouts and is the human in the loop: pause, resume, promote, abort, roll back, halt. Every action goes through the deploy API under the person’s own identity, never straight to hosts, so it is authorised and recorded. Most rollouts need no human; the controls exist for the few that do.
• Deploy API (authz · policy · halt): The one door for changing production: register releases, start, pause, promote, abort and roll back rollouts, set freezes, raise the halt. It checks the caller against the service’s owners, the change calendar and the release’s signature before writing anything, and every write carries an outbox row so no change exists without its audit record. The halt it also broadcasts straight to every cell controller.
• Rollout orchestrator (durable workflows): Runs each rollout as a durable workflow over its plan: take the service’s lock, prestage, write desired state stage by stage, ask for a canary verdict, wait out bakes, promote or roll back. Each decision is recorded in the release DB before it acts, so a worker that dies is replaced by one that resumes the same step (the Step Functions pattern, used here, not rebuilt). About 600 rollouts run at once on a small pool of stateless workers.
• Cell controller (one per cell · reconciler): Holds the desired state of every service in its cell (“checkout at r1842, generation 59, 10 % at a time”) in a cell-local replicated store and reconciles toward it: picks which instances change next, drains, restarts and waits for readiness, batch by batch. Rejects any write whose generation is lower than the one it holds, which fences out stale orchestrators. Keeps the cell running and healing when the global control plane is down.
• Host agent (50,000 hosts · level-triggered): Runs on every host, takes its targets from the cell controller and makes reality match: fetch, verify digest and signature, start, check readiness, register, drain and stop the old instance. It compares actual with desired on every loop instead of reacting to events, so a missed message or a reboot only delays convergence. Keeps the last two releases on disk, so a rollback moves no bytes.
• Audit log (append-only · change feed): Every change and decision: who or what deployed which digest where, approvals and overrides, and every automatic verdict with its evidence. Fed from the release DB’s outbox, so a change cannot commit without its record. Indexed by cell, service and time to answer the first question of every incident, “what changed here in the last hour?”, and kept for years.
• Release DB (consensus · multi-region): Releases, plans, rollouts and their step history, freezes, the halt flag, and one row per service holding the lock (which rollout owns it), a generation counter and the last known-good release. Strongly consistent because two orchestrator workers must never both believe they own a service. A few writes a second, replicated across regions so losing one does not stop deploys elsewhere.
• Canary analysis (canary vs control · stats): Given a canary group, a control group and a window, pulls the same metrics for both and tests each for a significant difference (a two-proportion test on errors, Mann-Whitney U on latency distributions), then combines them into a score: pass, fail, or not enough data yet. Also runs the guards in later stages, such as SLO burn rate on updated cells. Netflix and Google’s open-source Kayenta works this way.
• Metrics store (tagged by release · group): Per-instance time series (requests, errors, latency histograms, CPU, memory and each service’s business metrics) tagged with release, cell and canary or control group. The tags are what make the comparison possible: without them a canary’s errors vanish into the service’s totals. Histograms rather than precomputed percentiles, so groups and intervals merge correctly.

1. Merge to a signed release: Build once, deploy many. One merged commit becomes one immutable, signed artifact and one release record; every environment and every later rollback refers to that digest, so what was tested is exactly what runs.
   1. The merge queue lands a reviewed change on main and tells CI which commit to build.
   2. CI builds each affected service hermetically from that commit and runs its tests.
   3. CI pushes the image to the registry, where its name is the digest of its content.
   4. CI signs a provenance record for the image: commit, builder, inputs and test results.
   5. CI registers a release: one immutable record that points at the digest.
   6. When the service’s lock is free, the orchestrator starts a rollout of the newest waiting release.
2. Roll a release out in stages: The main path: staging, a canary judged against a control group, one cell, one region, then waves, with a bake after each. Every stage is a desired-state write that the cells reconcile, and every decision is durable before it acts.
   1. The orchestrator asks the distribution layer to prestage the image on every host that will run it.
   2. Staging: the pre-production cell runs the release and its integration and smoke tests.
   3. Canary: the first production cell starts 3 instances of the new release and 3 fresh instances of the current one as a control.
   4. Agents fetch the pieces, check digest and signature, start the instance and register it once it is ready.
   5. Every 5 minutes canary analysis compares canary and control on errors, latency and saturation.
   6. On a pass, the rest of the cell moves to the new release 10 % at a time, then bakes for 15 minutes.
   7. The orchestrator widens stage by stage: the region’s other cells, then 3 regions, then the last 6, baking after each.
   8. After the last wave and a 15-minute watch, the release becomes the service’s known-good version and the lock is released.
3. Canary fails and rolls itself back: The case the system exists for: a regression far too small to move any fleet-wide alert, caught on 3 instances by comparing them with their control, and reverted with no human involved.
   1. Five minutes in, analysis finds the canary’s error rate at 0.4 % against the control’s 0.1 %.
   2. The orchestrator records the failed verdict and its evidence before changing anything.
   3. It writes the cell’s desired state back to the current release alone, without canary or control.
   4. Agents drain and stop the canary and control instances; all traffic is back on the old version within a minute.
   5. The release is marked rejected, the lock is released and the owner is paged with the analysis report.
   6. The owner fixes forward: a new build, r1843, starts the whole plan again from staging.
4. The biggest service: 20,000 instances: What changes at the top of the size distribution: a 2 GB image on 20,000 hosts, a tier-0 plan, surge instead of shrink, slow start and draining, and restarts paced so they do not stampede the service’s own dependencies.
   1. A web frontend release, 20,000 instances with a 2 GB image, starts by prestaging the image everywhere.
   2. The tier-0 plan applies: a stricter canary, then one region per wave with 60-minute bakes, inside each region’s business hours.
   3. Canary analysis uses the extra traffic for a stricter test and adds business metrics.
   4. Within each cell the controller surges instead of shrinking: start 40 new instances, wait until they are ready, then drain 40 old ones.
   5. New instances join the load balancer on slow start, and old ones drain their connections before they stop.
   6. The cell controller paces batches by its agents’ readiness reports, so restarts do not stampede shared dependencies.
   7. The orchestrator reads one summary per cell, never 20,000 host reports.
5. Stop everything during an incident: The operational path: find what changed, stop every rollout within a minute without depending on the orchestrators, roll back the suspect, let the fix through, and resume. Every step lands in the audit log.
   1. On-call asks what changed in the failing cells in the last hour.
   2. The incident commander raises the global halt.
   3. The API broadcasts the halt to every cell controller directly, not through the orchestrators.
   4. Orchestrators see the halt before their next step and pause all 612 rollouts where they stand.
   5. The suspect release is rolled back to its last known-good version; rollbacks are allowed through the halt.
   6. The incident’s own fix ships as a break-glass rollout on the expedited plan, with a second approver.
   7. Two people lift the halt; paused rollouts resume, each re-baking its current stage.

Ask: Why not just build from the branch in each environment, or let each region build its own copy?

Good answers name: Hermetic build once, content-addressed image, signed provenance, promote by digest, Build per environment from the same commit, Deploy by mutable tag (service:prod, service:latest), Build on the hosts (pull the source and install).

Our pick: CI builds each affected service once per merged commit in a hermetic sandbox: pinned toolchain, dependencies locked by hash, no network. The output is an image named by its SHA-256 digest plus a provenance record (commit, builder identity, inputs, tests passed) signed through a KMS by a key only CI builders can use. A release is an immutable record pointing at that digest; promotion from staging to canary to each region changes which stage the release has reached, never the bits. Configuration that differs by environment is its own versioned artifact, rolled out by the same system. Host agents check digest and signature before starting anything and refuse unsigned builds or builds not from main. Google’s SRE book describes hermetic builds as the foundation of its release engineering, and supply-chain frameworks such as SLSA standardise the provenance part.

1. A critical CVE lands in a base image. Do you rebuild every service?
   Yes, but as new releases, never by changing existing ones. Every image containing the vulnerable base layer is rebuilt as a new digest with new provenance and rolled out through its normal plan, or the expedited plan with approval if the CVE is being exploited. The registry’s index of which images contain which layer digest answers “who is affected” in seconds. Old digests are marked vulnerable so they cannot become rollback targets without an override.
2. Can an engineer deploy a build from their laptop in an emergency?
   No. Agents run only artifacts signed by the CI key with provenance from main, so a laptop build is refused by every host. The emergency path is the expedited plan: the fix goes through the merge queue and CI in about 15 minutes, then a break-glass rollout with a second approver skips the long bakes. Allowing laptop builds would mean one stolen credential can run arbitrary code on 50,000 hosts.
3. How long do you keep artifacts?
   Everything that ever reached production, for at least the audit retention period, because rollbacks and incident reviews need the exact bits. Builds that never deployed expire after 30 days. Layers are reference-counted by digest, so collection deletes a layer only when no kept image uses it. At about 400 GB of new layers a day that is roughly 150 TB a year before expiry, which is cheap object storage.
4. How do you know your builds are really hermetic?
   Rebuild a sample of releases on a different builder and compare digests; a mismatch is a hermeticity bug and pages the build team. This design never rebuilds to deploy, so perfect reproducibility is not required for safety, but it is what makes provenance checkable: anyone can rebuild the commit and confirm the digest.
5. Config changes: same pipeline or a separate one?
   The same pipeline and the same staged plans, as a separate artifact. Several of the largest public cloud outages came from a global config push that skipped the safety a code deploy would have had. A config release is a content-addressed bundle with its own digest, rolled out cell by cell with canary analysis, and a cell’s desired state names both the code digest and the config digest so either can be rolled back alone.

Ask: Why not have the deploy server SSH into hosts in batches and run the install? And what happens to the fleet when the deploy system is down?

Good answers name: Hierarchical reconciliation: orchestrator → cell controller → host agent, each persisting its desired state, Imperative push over SSH in batches, Every host agent watches one global desired-state store, GitOps: desired state committed to a repository and synced by controllers.

Our pick: Three levels, all declarative. The orchestrator writes per-cell desired state (“checkout at r1842, generation 59, at most 10 % unavailable”) to that cell’s controller, which persists it in a cell-local replicated store before acknowledging. The cell controller turns it into per-host targets and sequences instances in batches, honouring max unavailable, readiness and the cell’s halt flag. Host agents are level-triggered: every loop they compare running instances with their targets, fix any difference and keep the last targets on local disk. Nothing below the orchestrator needs the orchestrator to keep working, so a dead global control plane means no new deploys while every cell keeps restarting crashed instances and placing replacements at their current versions. AWS calls this property static stability; Kubernetes controllers are the best-known public example of the reconcile loop.

1. An orchestrator worker crashes in the middle of a stage. What happens?
   The rollout’s record in the release DB says which stage it is in and which generation it wrote. Workers hold a short lease on each rollout, so another worker takes it within about 10 seconds and replays the stage: it writes the same desired state with the same generation to every cell in the stage. Cells that already have it treat the write as a no-op. Nothing is lost, because the decision was durable before it was acted on and the action is idempotent.
2. If everything is eventual, how do you know a stage is done?
   Read it back. Each cell reports its observed generation and its ready instances by release. A stage is done when every cell in it has observed the stage’s generation with at least 99 % of targets ready on the new release, and the bake has passed. Hosts that never converge are named in the summary and replaced by the scheduler instead of being waited on.
3. The global control plane is down for an hour during an incident. Can you still ship the fix?
   Yes, through a break-glass path that skips it: a CLI that writes signed desired state straight to the affected cell controllers, one cell at a time, with two approvers, logged locally and replayed into the release DB afterwards. It is deliberately minimal and practised in game days. Cells accept it because they check the signature, not where the request came from.
4. How do you deploy the deploy system?
   With the previous version of itself, on the same staged plans, under a rule that the cell controller and agent protocols stay compatible one version back in both directions. Agents and cell controllers roll cell by cell like any service; the global API and orchestrators run in several regions and roll one region at a time. If a deploy-system release is broken, the break-glass CLI can roll the cell controllers back without it.
5. Why not have agents talk straight to the orchestrator and drop the cell controller?
   Fan-out and isolation. One global service sequencing 50,000 hosts is one large shared failure domain: a bug in it touches every cell at once. A controller per cell sequences about 1,000 hosts, keeps its state inside the cell and fails alone. It is the same reason the serving stack itself is split into cells.

Ask: Why not roll a release out by percentage across the whole fleet, 1 % then 10 % then 100 %? Why bother with cells and regions?

Good answers name: Staged by isolation boundary: canary, one cell, one region, waves of regions, with bakes and gates, Percentage of the whole fleet (1 %, 10 %, 50 %, 100 %), Blue-green per cell, Everything at once.

Our pick: Plans are data, chosen by the service’s tier. The standard plan: staging (about 10 minutes of integration tests); a canary of 3 instances with 3 fresh controls in one cell, for at least 20 minutes and 24 k requests per group; the rest of that cell at 10 % per batch with a 15-minute bake; the region’s other 4 cells with a 30-minute bake; 3 more regions with a 20-minute bake; then the last 6 regions and a 15-minute watch before the release becomes known-good. That is about 2 hours, exposing roughly 1 %, 2 %, 10 % and 40 % of traffic before 100 %. Bakes end on evidence (request counts and verdicts) with a minimum wall-clock time for slow-burning problems. Tier 0 goes one region per wave with 60-minute bakes inside each region’s business hours; tier 2 skips the region waves. This is the shape AWS describes publicly: one box first, then one zone or cell, then region by region, with bake time and automatic rollback on alarms between them.

1. Which region goes first?
   One with enough traffic for the analysis to mean something, but not the largest, and never the region the others fail over to. Keeping it the same makes its on-call aware they are the canary region; rotating it spreads the risk across users. Either is defensible; what is not is starting in the region you would need if the release broke.
2. A regression appears only after six hours, like a slow memory leak. Your bakes are 30 minutes.
   Bakes are sized for most regressions, not all; slow ones are caught by guards that keep running after the rollout. Memory by release is a tracked metric, and a leak trending toward the limit triggers a rollback to last-good even after the release was marked known-good, while the previous artifact is still on disk. Tier-0 plans use hour-long bakes partly for this reason.
3. Engineers complain deploys take two hours. What do you change?
   First check that the time goes into evidence, not waiting: bakes end when the analysis has enough data, so busy services clear the canary at its minimum. Then match plans to risk: internal tier-2 services get a 45-minute plan. And move risk out of deploys: changes behind a feature flag can take the fast plan because the risky part is released separately. What I would not shorten is the first cell’s bake, which is where most regressions are caught.
4. Two dependent services deploy at the same time and a cell breaks. Which release caused it?
   At the canary, each service is compared with its own control in the same cell, so a dependency’s change affects both groups equally and is not blamed on the release. At the cell stage, the change feed shows both rollouts touching the cell and the guard pauses both. A rule that at most one tier-0 service is in its first-cell stage in a given cell at a time keeps this rare.
5. How would you know the stages are working?
   Track where regressions are caught: the share of rolled-back releases stopped at staging, canary, first cell, first region and later. Most should stop at the canary or the first cell; each one found after wave 2 gets a review, because some earlier stage should have seen it. Also track error budget spent per bad release, which should fall as detection moves earlier.

Ask: How does the system decide a canary is healthy? Why not just alert if the error rate goes above 1 %?

Good answers name: Canary vs a fresh control, per-metric statistical tests, combined score, Fixed thresholds on fleet-wide alerts, Before and after on the same instances, Canary vs the rest of the running fleet.

Our pick: Each canary gets a control: the same number of instances of the current release, started at the same time in the same cell with equal load-balancer weight, both as surge capacity, and every metric is tagged by release and group. Every 5 minutes the analysis pulls each configured metric for both groups: error rate (a two-proportion test), the latency distribution (Mann-Whitney U rather than one percentile), CPU and memory, and the service’s business metrics. Each metric is classed pass, high or low, with an effect-size floor so a significant but harmless 0.5 % latency change does not fail a release; the score is the weighted share that pass, and a few critical metrics such as error rate can fail the canary alone. A score of 95 or more promotes, under 75 rolls back, anything between keeps collecting until the stage’s maximum and then asks the owner. The stage needs at least 20 minutes and 24 k requests per group. This is how Kayenta, the open-source canary judge from Netflix and Google, works, and Google’s SRE Workbook chapter on canarying makes the same case for canary versus control over before and after.

1. You test 20 metrics every 5 minutes. Won’t one fail by chance?
   Yes: at a 5 % level, about two in three runs would see at least one false “high”. Three defences: require an effect size as well as significance; score across metrics instead of failing on any one, with only a few critical metrics able to fail alone; and require a failure to persist for two consecutive intervals unless it is extreme. Track the false rollback rate per service; above about 1 in 50 rollouts, its config is too sensitive.
2. What if there is not enough traffic to decide?
   The stage runs to its maximum time and the verdict is “inconclusive”, which is not a pass. For tier 0 and 1 that pauses the rollout and asks the owner; for tier 2 it proceeds to the first cell with the SLO burn guard active. The better fix is in the plan: quiet services get a larger canary or a longer window, sized from the same sample-size formula.
3. The metrics pipeline is down. What happens to rollouts?
   No data is never a pass, so rollouts waiting on analysis pause and nothing promotes until metrics return. Rollbacks do not need analysis and keep working. This is also why the analysis reads a metrics store in each region rather than one global pipeline.
4. The release breaks only for one app version, 1 % of users. Will the canary see it?
   Only if those users reach the canary and the metric is sliced finely enough; the overall error rate barely moves. For known risky segments the config adds sliced metrics, by client version or endpoint, at the cost of more tests and more false alarms. Some segment-specific bugs will get past any canary and be caught at the first cell or by the owner’s own alerts, which is why the later stages exist.
5. Why start fresh control instances instead of comparing with the ones already running?
   Because new processes differ from old ones for reasons unrelated to the code: empty caches, an unwarmed JIT, a smaller heap, fresh connections. Against week-old instances, every release looks slower for its first minutes. Starting the control at the same moment makes warm-up symmetric, and the extra cost is 3 instances for 20 minutes.

Ask: Rollback is just deploying the old version again, right? When does that not work?

Good answers name: Roll forward to last known-good, with every release rollback-safe: expand and contract, read new before writing new, flags, Down migrations: reverse the schema change as part of the rollback, Fix forward only, Restore the database from a snapshot.

Our pick: Rollback is a rollout: a new rollout, with a higher generation, of the service’s last known-good digest on a fast plan (every affected cell in parallel, 25 % at a time, no bakes), allowed even during a freeze or a halt, and it preempts the bad rollout’s lock. Making that safe is each release’s job, checked by the pipeline. Schema changes are their own releases following expand and contract: add the nullable column, deploy code that writes both, backfill, switch reads, stop writing the old one, drop it weeks later. A new data format takes two releases: first one that can read it, then, once that is known-good everywhere, a flag that starts writing it. CI runs the previous release against the new schema and against data written by the new release, so a release the old one cannot survive fails before it ships. Risky behaviour ships dark behind a feature flag, so turning it off takes seconds and no deploy at all.

1. The bad release already wrote a million rows in a new format. Now what?
   If the previous release was built to read that format, which is what the read-new-before-write-new rule guarantees, roll back and the rows are fine. If not, rolling back breaks those reads, so it becomes a fix-forward decision: ship a release that reads both formats, or rewrite the rows with a backfill, and keep the writing flag off meanwhile. This case is why the pipeline tests old code against new data.
2. Should flag flips go through the deploy system?
   Flags are evaluated by their own service and change in seconds, which is their whole value, so they do not take a 2-hour plan. But they are production changes: they go into the same change feed and audit log, the halt freezes risky ones, and big ramps follow a percentage schedule with the same guards. On-call should see deploys and flag changes on one timeline.
3. How do you stop someone dropping a column the previous release still reads?
   The migration tool refuses destructive steps unless no running or known-good release has used the column for a set period, which it checks against each release’s recorded schema usage. Migrations ship as their own releases and get their own review. And the rollback-compatibility test in CI fails if the previous release’s queries break against the new schema.
4. Isn’t a rollback during a halt also a change?
   It is, and it is the one change the halt must allow, because it moves toward a version that recently ran everywhere without trouble. Blocking rollbacks during an incident would remove the main mitigation exactly when it is needed. It is still recorded with a reason and still runs in batches, so it cannot take out a cell’s capacity.

Ask: Two engineers start deploys of the same service at once, and a third starts one during the holiday freeze. What stops them, and how do you stop everything at once?

Good answers name: Per-service lock row with a generation that fences every desired-state write; freezes and halt as data read at every step; halt broadcast to cells, Lock held in orchestrator memory, or a lease without fencing, Allow concurrent rollouts of one service and merge them, Freezes by convention: an announcement in chat.

Our pick: Each service has one row in the release DB: the rollout that holds it, a generation counter and its last known-good release. Starting a rollout is a compare-and-set on that row; each stage increments the generation with a write conditional on still holding it, and every desired-state write carries the generation, which cell controllers check: anything lower than what they hold is rejected. A rollback takes the row from the bad rollout and bumps the generation, so the old rollout’s next write fails and it stops. New releases queue, and when the lock frees only the newest deploys, since it contains the rest. Freezes are rows with a scope (global, org, service, region) and a time window, checked by the API at start and by the orchestrator before every stage, so an in-flight rollout pauses at its next stage boundary. The global halt is one row plus a signed broadcast from the API straight to all 50 cell controllers, which finish the batch in hand and start no new one. Rollbacks and approved break-glass rollouts pass through both, and every override is in the audit log with its approver.

1. The release DB loses a region. Can anyone deploy?
   The release DB is consensus-replicated across regions (Spanner-style, or a Raft group across five regions), so losing one loses no committed state and the remaining quorum keeps accepting writes. Orchestrator workers run in several regions and take over leases. If the whole control plane were down, cells keep serving and the break-glass CLI can still write desired state to cells directly.
2. A paused rollout holds checkout’s lock for a day. Is checkout blocked?
   Yes, deliberately: a paused rollout means a human is looking at something. To stop pauses being forgotten, they page the owner after an hour and expire after a day, after which the rollout rolls back to last-good unless someone acts. The owner can also abort and start the newer release, which takes the lock with a higher generation.
3. How do you enforce “no deploys when the error budget is spent”?
   As a freeze generated from data: the SLO system publishes each service’s remaining budget, and the API refuses non-urgent rollouts for a service below zero while allowing rollbacks and fixes tagged as reliability work. It is applied by the same check as calendar freezes, so there is one place to look when a deploy is refused.
4. How fast does the halt take effect?
   Setting it is one API call. The broadcast reaches all 50 cells within seconds, and orchestrators read it before their next step, so within about a minute nothing new starts anywhere. Each cell finishes the batch it is restarting, at most 30 seconds of work, rather than leaving instances half-drained.
5. What exactly goes into the audit log?
   Every request with the caller’s identity, the service, the digests from and to, the cells, approvers and reason; every automatic decision with its evidence, such as analysis scores and the metrics that failed; every override of a freeze or halt; and every state change of a rollout. It is written through the release DB’s outbox so a change cannot commit without it, stored append-only, and indexed by cell and time.