Design a Container Orchestrator

Place 300,000 containers on 10,000 machines from declared desired state, keep them running through failures, and give every service a stable address.

Last updated 2026-10-27. Difficulty: hard. Patterns: scheduling, reconciliation, consensus, bin-packing, failure-detection. Reported at Google, Microsoft, Amazon, Databricks, Uber, Nvidia.

Functional requirements

• Declare a workload. A service (keep N replicas running) or a batch job (run N tasks to completion): image digest, replicas, CPU and memory requests and limits, placement constraints (zone, machine class, GPUs, spread), health checks and a priority. Teams say what should run; the system works out where and keeps it true.
• Place pods on machines. A pod is one or more containers placed together, usually one. Each lands on a machine that meets its constraints and has room for its requests. Production pods are placed within seconds; a batch job of 100,000 tasks may queue, but it must never delay production.
• Keep the declared count running. Restart a crashed container in place, and replace the pods of a failed or drained machine elsewhere. Never run two copies of a pod that declares itself at-most-one, such as a shard leader.
• Roll out a new version. A new template replaces pods gradually inside a surge and an unavailability budget, gated on readiness, and can be paused or reversed. Staged rollouts across clusters and canary analysis belong to the deployment system, which drives this API.
• Stable service addresses. A service is a name and a virtual IP backed by its ready pods. Endpoint changes reach every per-machine proxy and load balancer within seconds, and a pod that is unready or draining stops getting new traffic before it goes away.
• Priorities, quotas and preemption. Production outranks batch: when the cluster is full, a production pod evicts batch pods to fit. Quotas per team per priority stop everyone from declaring themselves production.
• Resource isolation and overcommit. Limits are enforced on the machine with cgroups, so a noisy pod hurts only itself. Batch may run on capacity that production reserved but is not using, and gives it back the moment production needs it.
• Planned maintenance. Drain a machine for a kernel upgrade or repair by evicting its pods only as fast as each service’s disruption budget allows, so 500 machines can be upgraded in a night without anyone noticing.
• Status and events. Desired against actual for every workload, and why a pod is pending (“no machine in use1b has 8 free GPUs”). An orchestrator nobody can debug is one nobody trusts.
• Out of scope. Building images and staged deploys across clusters (the deployment system), the container runtime itself (containerd, runc), the packet data plane between pods, persistent volume provisioning, and buying or retiring machines (the capacity planner).

Non-functional requirements

• Cluster scale (10,000 machines · 300,000 pods). A fixed, tested maximum per cluster. A region runs many clusters of this size rather than one bigger one, and every number below is sized against these two.
• Scheduling throughput and latency (2,000 placements/s · prod p99 < 5 s). Batch bursts set the throughput; production pods must never wait behind them. Measured from pod created to pod bound, when capacity exists.
• Recovery from machine failure (detect ≤ 40 s · replaced ≤ 3 min). About 5 machines fail a day, each taking ~30 pods. Faster detection means more false alarms from a stalled agent or a rebooting switch, and the wait before rescheduling is set by what a duplicate would cost.
• Static stability (control plane down → 0 pods stop). Running pods, restarts on the same machine and the traffic between pods must not depend on the control plane. An outage of the store or the API servers stops changes, never serving.
• Consistent desired state (linearizable · every write conditional). Two writers (two scheduler workers, a scheduler and a controller) must never both succeed from stale reads. Every object carries a version and every update is conditional on it.
• Utilisation against safety (≥ 60 % of CPU used). The requirement that drives the hardest trade-off. Packing tightly saves hundreds of machines (1 % of the fleet is 100 machines), while spreading replicas across failure domains and keeping room to reschedule costs capacity. Batch on unused production reservations closes most of the gap.
• Endpoint freshness (readiness change → every proxy < 5 s). Traffic sent to a pod that is draining or dead is user-visible errors. The change must reach 10,000 proxies in seconds, which at rollout peaks is a million updates a second.
• Fairness in the control plane (per-team shares of API and scheduler). A runaway controller or a 100,000-task submission must not slow other teams’ writes, watches or placements. API servers queue requests per client with fair shares; the scheduling queue orders by priority, then by team.

Back-of-envelope estimates

• State in the store: ~1.5 GB. 300,000 pods × ~4 KB (spec and status) = 1.2 GB; 10,000 machines × ~10 KB = 100 MB; ~20,000 services and jobs × ~5 KB = 100 MB; endpoint slices, quotas and config about 100 MB. Roughly 1.5 GB: it fits in memory on every replica and in a Raft store’s comfortable few-GB range only because it holds state. One day of these containers’ logs would be tens of terabytes.
• Placements per second: ~170/s avg · 2,000/s peak. ~100,000 batch pods run at any moment with a typical life of ~10 minutes: 100,000 ÷ 600 s ≈ 170 starts/s. 200,000 service pods replaced about once a day by deploys add only ~2/s. Peaks come from submissions: a 100,000-task job placed at 2,000/s takes 50 s, about 12× the average.
• Scheduler CPU at peak: ~20 cores full scan · ~1 core sampled. Checking and scoring one machine costs ~1 µs, so a pass over 10,000 machines is 10 ms a pod; at 2,000 pods/s that is 20 cores kept busy and a 10 ms floor on every decision. Stopping after ~500 feasible machines and scoring only those is ~0.5 ms, about 1 core. Sampling, not hardware, is what makes 2,000/s cheap.
• Store writes at peak: ~8 k/s. Each pod start is ~4 writes: created by its controller, bound by the scheduler, running and then ready from the node agent. 2,000 starts/s × 4 = 8,000 writes/s of ~4 KB objects. One five-member Raft group on NVMe with batched commits can take this, but it is the busiest the store gets: heartbeats, events and metrics must go elsewhere, and this is a reason to cap the cluster.
• Heartbeat traffic: 1,000/s · ~0.3 MB/s. 10,000 node agents renew a lease every 10 s = 1,000 writes/s of ~300 B, about 0.3 MB/s. Trivial in bytes, but as Raft writes it is an eighth of the main store’s peak load forever, so leases live in their own small Raft group, and full machine status is written only when it changes or every 5 minutes.
• Machine failure to replacements serving: ~135 s. Suspect after 40 s without a renewal (four missed), then wait 60 s more in case it is a partition rather than a death. Placing ~30 pods takes under 5 s; pulling and starting takes ~30 s with most layers cached. 135 s. Traffic is protected sooner: the machine’s endpoints are removed at 40 s, and proxies eject failing endpoints on their own within seconds.
• Endpoint updates fanned out: ~1 M/s at rollout peaks. 200,000 service pods in ~5,000 services; rollouts and failures change up to ~100 endpoints a second at the busiest. A proxy on each of 10,000 machines needs every service: 100 × 10,000 = 1 M updates/s. As ~200 B deltas that is 200 MB/s; resending a 5,000-endpoint service’s whole list (~500 KB) per change would be hundreds of GB a second. This is the biggest stream in the system.
• Capacity reclaimed by overcommit: ~1,500 machines. Production requests reserve ~70 % of the fleet’s CPU but use ~40 % on average, because teams size for peak. Letting batch use half of that 30-point gap, at a priority that can be throttled or evicted, gives back 15 % of 10,000 machines = 1,500 machines, more than any packing tweak.

Components

• Teams / deploy system (CLI · CI · deploy plans): Teams declare workloads, and the deployment system writes new versions into each cluster as one stage of its plan. Both only ever write desired state through the API and read status back; neither talks to a machine. Every change is conditional on the version it read, so two writers cannot silently overwrite each other.
• API servers (stateless · admission · fair queues): The only path to the store, run as ~10 stateless replicas. Authenticates, validates, applies defaults and enforces quotas before anything is written, and turns every update into a write conditional on the object’s version. Queues requests per client with fair shares, so a runaway controller or a 100,000-task submission cannot starve everyone else.
• Watch cache (in API servers · recent history): Each API server holds every object in memory, fed by one watch on the store, plus a few minutes of recent changes. Lists and watches are served from here, so 10,000 agents, the discovery tier and the controllers cost the store ten streams, not thousands. A client that reconnects resumes from its resource version instead of relisting 1.2 GB of pods.
• State store (Raft × 5 · MVCC · ~1.5 GB): An etcd-style key-value store: a Raft log across five replicas, every write stamped with a cluster-wide increasing revision, conditional transactions, and watches that start from a given revision. It holds only state (desired and actual for every pod, machine, workload and endpoint), about 1.5 GB, compacted to a few minutes of history. Its write rate, 8 k/s at peak, is the ceiling for the whole cluster.
• Lease store (separate Raft group · 1 k/s): A small, separate consensus group for node leases (and leader-election leases), so 1,000 heartbeat writes a second never compete with binds in the main store. Each lease is ~300 B: holder, renew time, a 40-second duration and an epoch. Losing it pauses failure detection, not serving.
• Scheduler (filter · score · optimistic bind): Places unbound pods. Keeps an in-memory snapshot of all 10,000 machines; four workers each walk machines from a random offset, filter on constraints and free requests, stop after 500 feasible machines and score those (spread first for production, packing for batch). Binds are writes conditional on the machine’s allocation version, so a conflict requeues the pod instead of overcommitting a machine.
• Scheduling queue (priority · fair share · backoff): Pending pods ordered by priority, then the team’s fair share, then age, so a production pod created mid-burst jumps ahead of 100,000 batch tasks. Pods that fit nowhere move to an unschedulable set and are retried only when the cluster changes in a way that could help, not in a hot loop.
• Node agent (one per machine · cgroups): Runs on every machine and watches only the ~30 pods bound to it. Pulls images by digest, starts containers with cgroup CPU weights and memory limits, runs liveness and readiness probes, restarts crashed containers in place and reports status. Keeps running its pods when the control plane is unreachable, and kills any pod the store no longer lists for it.
• Image registry (by digest · layer cache): Where images come from, run by another team. Agents pull by digest, and most layers are already cached on a busy machine, so a start usually fetches tens of MB. Large rollouts can be prestaged by the deployment system so pods start in seconds.
• Usage metrics (time series · not in the store): Per-pod and per-machine CPU and memory usage every 10 s, from every agent, in a time-series store built for volume. Kept out of the state store on purpose: it is millions of points a second. Feeds autoscaling and the reclaim controller that works out how much reserved but unused capacity batch may borrow.
• Node lifecycle (leases · taints · eviction): Decides when a silent machine is gone. Measures lease silence on its own monotonic clock: at 40 s it marks the machine unreachable and its pods not ready, 60 s later it evicts ordinary pods. If more than 10 % of a zone goes silent at once it assumes a partition and slows eviction to a trickle, because mass rescheduling would turn a network fault into an outage.
• Controllers (level-triggered · sharded): One reconcile loop per kind: services and their rollouts, jobs, endpoint slices, disruption budgets, reclaimed capacity. Each lists, watches from that revision, and repeatedly compares desired with actual for the objects it owns, acting only through the API. Leader-elected, and the heavy ones are sharded by hash; none keeps state the store does not have.
• Discovery service (versioned deltas · ~20 replicas): A stateless fan-out tier between endpoint slices and their 10,000 consumers. Watches the slices once and streams each subscriber versioned deltas of only what changed (~200 B per endpoint), so a busy rollout costs 200 MB/s instead of hundreds of GB. A reconnecting proxy sends its last version and gets only what it missed.
• Node proxy (per machine · VIP → endpoints): Runs on every machine and programs local forwarding (eBPF or IPVS) from each service’s virtual IP to its ready endpoints, preferring the caller’s zone. Holds every service’s endpoints, about 20 MB, and keeps forwarding on its last copy if discovery or the control plane is down. When every endpoint of a service is unready it fails open and uses them anyway.
• Load balancers (L4 / L7 · north-south): The edge balancers from the load balancer design, owned by the networking team. For the few hundred services exposed outside the cluster they subscribe to the same endpoint deltas and still health-check endpoints themselves, with fail-open thresholds, so a readiness bug here cannot empty a pool there.

User flows

1. Declare a service, get running pods. The main write path. A team writes desired state once, and four independent loops (controller, scheduler, node agent, endpoints) each watch for their part and write their result back. None of them calls another, so any of them can crash and resume.
   1. A team applies a service spec: 30 replicas of payments-api, 2 CPUs and 4 GB each, spread over three zones, at production priority. Admission runs before anything is stored: authentication, schema validation, defaults, and the team’s production quota (30 × 2 = 60 cores must fit in what it has left). The write is a create conditional on the name not existing. The only thing that has happened when it returns is that a record exists; nothing is placed by this call.
   2. The service controller sees the new workload on its watch and creates 30 pod objects, none of them bound to a machine. The controller compares desired (30) with actual (pods owned by this workload: 0) and creates the difference; it never asks what event just happened. Its watch is served from the API servers’ watch cache, so twenty controllers and 10,000 agents cost the store one stream per API server. Each pod records its owner, so a restarted controller recounts instead of creating 30 more.
   3. The scheduler takes the pods from its queue, filters the machines and scores the ones that pass. Filter: enough unrequested CPU and memory, the right machine class, no taint the pod does not tolerate, and the zone spread still satisfiable. Score: spread across zones and racks first, then prefer fuller machines so whole machines stay free for large pods. It all runs against the scheduler’s in-memory snapshot, about 0.5 ms a pod when it stops after 500 feasible machines.
   4. The scheduler binds each pod with a write conditional on the machine’s allocation version it decided from. Binding is optimistic: the scheduler assumes its snapshot is right, and the transaction checks that the pod is still unbound and that nothing else has changed the machine’s allocations since. If another worker took that capacity first, the bind fails and the pod goes back to the queue, a few milliseconds lost. Workers start their walks at different machines, so collisions are rare.
   5. The node agent on that machine sees a pod bound to it, pulls the image and starts the container inside its limits. Each agent watches only the ~30 pods bound to its machine, so 10,000 agents are 10,000 small filtered watches, not 10,000 copies of the cluster. It re-checks fit as a last line of defence, pulls by digest (most layers are cached), and starts the container with a CPU weight from its 2-core request and a hard 4 GiB memory limit, so a leak kills only this container.
   6. The agent runs the readiness probe and reports the pod running, then ready. Running means the process started; ready means it passed its readiness probe and may get traffic, and everything downstream keys on ready. The agent writes only the status part of the object and coalesces updates, so a pod start costs two status writes, not one per probe.
   7. The endpoints controller adds the ready pod to the service’s endpoint slice, and discovery pushes the change to every proxy and the load balancers. A service’s endpoints are split into slices of at most 100, so adding one pod rewrites one ~10 KB slice rather than a 5,000-entry list. Discovery watches slices once and sends each subscriber a one-endpoint delta. About 2 s after ready the pod takes traffic; apply to serving is ~30 s, mostly image pull and start.
2. A call, and a pod that stops being ready. The read path: a stable name resolves to healthy endpoints on the caller’s own machine, with no control-plane call per request. And when a pod fails its readiness probe, it leaves every proxy’s rotation in seconds without being restarted.
   1. A pod calls payments-api by name, and the name resolves to the service’s virtual IP. The name and the virtual IP are fixed for the life of the service; only the endpoints behind them change. Because the IP never moves, clients that cache DNS far past its TTL do no harm, which is the classic DNS failure this avoids. The node proxy answers these lookups locally from the data discovery gave it.
   2. The node proxy on the caller’s machine picks a ready endpoint, preferring its own zone, and forwards the connection. Client-side balancing: no hop through a central balancer, and zone preference keeps traffic off cross-zone links. The proxy holds every service’s endpoints, about 200,000 entries at ~100 B, ~20 MB per machine, and keeps forwarding on that copy if discovery or the whole control plane is unreachable.
   3. Traffic from outside reaches the same pods through the load balancers, which subscribe to the endpoints of exposed services. Only the few hundred services exposed outside the cluster are sent to the load balancers, not all 5,000. They also health-check endpoints themselves, with fail-open thresholds, as a second opinion: a bug in readiness reporting must not empty a public pool.
   4. One payments pod starts failing its readiness probe because its database connection pool is exhausted. Three failed probes at 2 s intervals, about 6 s. The agent flips ready to false and does not restart the container, because its liveness probe still passes. Restarting a pod that is merely overloaded would turn a slow dependency into a restart storm.
   5. The endpoints controller marks that endpoint not ready in its slice. Marked not ready rather than removed, so it can come back without churning the slice, and so a proxy that sees every endpoint of a service unready can fall back to using them all (fail open) instead of failing every request. The controller writes each slice at most once a second, batching changes during a busy rollout.
   6. Discovery streams a one-endpoint delta to all 10,000 node proxies and to the load balancers. One ~200 B delta × 10,000 proxies is 2 MB, sent from a horizontally scaled tier rather than from 10,000 watches on the API servers. Subscribers acknowledge each version, so a proxy that has fallen behind is visible. Within ~2 s of the slice write the pod gets no new connections, and existing ones finish.
   7. When the probe passes again the same path puts the pod back; if liveness fails instead, the agent restarts the container in place. A restart in place needs neither the scheduler nor the rest of the control plane. The agent backs off (10 s, 20 s, 40 s, up to 5 minutes) so a crash-looping container does not burn its machine. A pod moves to another machine only if its machine fails or it is evicted.
3. A 100,000-task batch job meets production. The scale case: a batch job submits 100,000 tasks at once while production keeps deploying. The scheduler must place 2,000 pods a second, keep production at the front, let batch borrow idle reservations, and when the cluster is full, evict batch for production and never the reverse.
   1. A team submits a job of 100,000 tasks at batch priority, 1 CPU and 2 GB each. Admission checks the team’s batch quota, which is larger and cheaper than production quota because batch can be evicted. One job object is written; its tasks do not exist yet.
   2. The job controller creates task pods in batches, paced to what the scheduler can place. 100,000 pod objects written in one second would be 400 MB landing on a store that holds 1.5 GB. Paced at ~2,000 a second, creation matches placement, the pending set stays at a few thousand, and the API servers’ per-client fair share keeps this controller from crowding out everyone else’s writes.
   3. The scheduling queue orders by priority, so production pods created during the burst jump ahead of the batch tasks. A heap keyed by (priority, team share, age). A production pod created mid-burst is placed in milliseconds instead of waiting 50 s behind 100,000 tasks. Tasks that fit nowhere move to an unschedulable set and are retried when a machine is added or a pod finishes, not every few milliseconds.
   4. Four scheduler workers decide in parallel against the same snapshot, each sampling machines from a different starting point. Each worker walks machines from a random offset, stops at 500 feasible ones and scores those: ~0.5 ms a pod, so four workers have 8,000 a second of capacity for a 2,000 a second burst. The tasks of one job share an equivalence class, so the filter result for task 51,234 is reused for 51,235. Batch scoring favours packing: the fullest machine that fits.
   5. Batch tasks may land on capacity that production reserved but is not using, as estimated from recent usage. Agents report usage every 10 s. A reclaim controller writes each machine a batch allocatable figure: unreserved capacity plus production’s requests minus its p95 usage over the last hour, minus a 10 % margin. Batch is filtered against that figure and production against hard requests, so borrowing never weakens a production reservation.
   6. Binds commit optimistically; the few that lose a race for a machine’s last free slot are requeued. Each bind is a transaction conditional on the machine’s allocation version. With 2,000 binds a second spread over 10,000 machines, two workers rarely pick the same machine within the same few milliseconds, and losing costs one requeue. The Omega paper argued for this shared-state, optimistic model; a lock per machine would serialise exactly the machines everyone wants.
   7. The cluster is full; a production pod preempts two batch tasks on one machine to fit. When nothing fits, the scheduler finds the machine where evicting the fewest, lowest-priority, most recently started pods frees enough room, and never preempts inside the production band, which would cascade. It deletes the victims with a 10 s grace and nominates the machine, so the freed room is held for the production pod rather than refilled by the next batch task.
   8. The job controller sees the evicted tasks and recreates them; they wait at batch priority for capacity. Batch is built to be evicted: tasks are short or checkpoint, and an eviction does not count against the job’s retry limit. The Borg paper reports eviction rates for non-production work far above production’s, which is the deal batch accepts for cheap capacity.
4. A machine stops heartbeating. The failure path, and the dangerous one: a machine that stops answering might be dead, or cut off and still running its pods. The control plane must replace capacity quickly without ever running two copies of a pod that must be unique, and without mistaking a partitioned zone for 3,000 dead machines.
   1. Every 10 s each node agent renews its machine’s lease; the renewals from m-use1b-04417 stop. A lease is a ~300 B object (holder, renew time, 40 s duration, epoch) in its own Raft group, so 1,000 renewals a second never compete with binds. Full machine status is written only when it changes or every 5 minutes. Silence alone does not say whether the machine is dead or just unreachable.
   2. After 40 s without a renewal, the node lifecycle controller marks the machine unreachable and its pods not ready. It measures silence on its own monotonic clock from the last renewal it saw, so a skewed clock on the machine cannot fake liveness. Marking the pods not ready takes them out of every proxy within seconds: traffic stops at 40 s although nothing has been rescheduled. A NoSchedule taint stops new pods landing there.
   3. Before evicting, it checks how many machines in the zone went silent; a zone-sized silence pauses eviction. One machine failing is normal, about 5 a day. A thousand machines in one zone going silent in the same minute is almost certainly a network fault between them and the control plane, and evicting 30,000 pods would turn a partition into an outage and a rescheduling storm. Above 10 % of a zone silent, eviction slows to a few machines a minute and a human is paged.
   4. After a further 60 s it evicts the machine’s ordinary pods; at-most-one pods are released only once their lease has surely expired on the machine too. The 60 s wait trades capacity for safety, since most shorter silences are reboots or blips that come back. A pod that declares itself at-most-one (a shard leader, a singleton consumer) is safe to replace only because its agent kills it on its own after 30 s without a successful renewal; its replacement also gets a higher epoch to present to whatever it writes to.
   5. Controllers see fewer pods than declared and create replacements, which the scheduler places elsewhere. Nothing here is special to failures: the controller sees 29 of 30 and creates one, exactly as on the first day, which is the payoff of level-triggered loops. Thirty pods are placed in well under a second and are serving about 135 s after the machine died. Spread scoring already kept replicas on different racks, so one machine rarely holds two replicas of one service.
   6. The partition heals; the returning agent finds its pods gone from the store and stops them. During the partition its pods kept running (static stability), so ordinary replicas briefly existed twice, which is harmless for stateless pods behind a balancer. On reconnect the agent relists its own pods, sees that the evicted ones are gone, and stops them. The Borg paper describes the same rule: a returning machine is told to kill tasks that were rescheduled.
   7. The machine renews its lease, its taint is lifted and it rejoins empty. It comes back empty rather than reclaiming its old pods: they were already replaced, and moving them back is churn for nothing. The lease epoch increments, fencing anything that still holds 18. A machine that goes silent three times in an hour stays tainted and goes to repair, because it costs more in reschedules than it adds in capacity.
5. Roll a service to a new version. A rollout is just another controller: a new template is new desired state, and the service controller replaces pods inside a surge and an unavailability budget, gated on readiness. The deployment system drives this cluster by cluster and judges the result; it does not re-implement it.
   1. The deployment system writes payments-api’s new image digest into this cluster as one stage of its plan. The write is conditional on the resourceVersion it read, so a concurrent edit by the owning team makes it fail with 409 and re-read rather than silently win. The generation goes to 2: comparing it with the controller’s observed generation is how anyone tells whether the new spec has been acted on yet.
   2. The service controller starts 3 surge pods from the new template. maxSurge 3 and maxUnavailable 0 mean at least 30 ready pods and at most 33 at any time: 10 % extra capacity for the duration in exchange for never dropping below full strength. Pods carry their template hash, so the controller can count old and new from the store at any moment and remembers nothing.
   3. The scheduler places them and agents start them with the new image. The same path as any new pod. Spread scoring counts old and new replicas together, so the new version does not pile into one zone. If the deployment system prestaged the image, a start takes seconds.
   4. Each new pod that stays ready lets the controller retire one old pod, which leaves rotation and drains before it stops. A new pod counts only after 10 s of continuous readiness, so one that crashes right after its first probe does not advance the rollout. The old pod is marked terminating, removed from endpoints, and only then sent SIGTERM with up to 30 s to finish in-flight requests; stopping it first would cut requests mid-flight.
   5. The controller repeats until all 30 pods run the new template, and reports the generation it has observed. Three at a time at ~30 s per batch, 30 pods take about 5 minutes. The status reports observed generation 2 with 30 updated and 30 ready, which is what the deployment system waits for before judging this cluster and moving to the next one.
   6. If new pods never become ready, the rollout stops after its surge and reports itself stalled; the old pods keep serving. With maxUnavailable 0 a broken release costs 3 surge pods and nothing else. After a 10-minute progress deadline the controller sets progressing to false. Rolling back is writing the old template again, which the same controller converges to; whether to do that automatically is the deployment system’s decision.
   7. Draining a machine for a kernel upgrade uses budgets too: each pod is evicted only if its service can spare it. The eviction API refuses an eviction that would take payments-api past its disruption budget (at most 1 unavailable), so a drain that meets a service already one short waits instead of making it two. Draining 500 machines a night is just many evictions that obey the same rule, which is why planned maintenance stays invisible.

Deep dives

The theory behind it

• Kubernetes for system design interviews: The Kubernetes concepts that come up in system design: pods, deployments, services and ingress, autoscaling of pods and nodes, health probes, resource requests and limits, stateful sets, jobs, rolling updates, multi-zone scheduling, and what Kubernetes does not solve for you.
• Consensus, leases and coordination: Leader election, Raft and Paxos at interview depth, ZooKeeper and etcd, distributed locks and why they are fencing tokens, plus how to avoid needing coordination at all.
• Gossip protocols and failure detection: How large clusters know who is alive without a central coordinator: heartbeats and timeouts, phi accrual failure detectors, gossip dissemination and SWIM, membership changes, split brain, and how these show up in databases, caches and service discovery.

Related

• Design a Matchmaking System
• Design an LLM Inference Service
• Design a Web Crawler
• Design a Distributed Job Scheduler
• All 83 system design interview questions
• Run this question for a friend: the interviewer kit