Canary deployments vs blue-green vs rolling: an interactive simulator
A canary deployment sends a small share of real traffic to a new version, compares how it behaves with the old version, and widens the share step by step only while it looks healthy. When a release is broken, the canary turns an outage for everyone into a few failed requests for a few users. Rolling deployments replace servers in batches, and blue-green deployments switch all traffic to a second fleet at once and keep the old one ready to switch back.
The playground rolls out one release four ways and charts how much traffic is on the new version over an hour. Make the release buggy, more or less badly, and see who catches it and what it costs.
The new version fails an extra 1% of the requests it serves, on top of the usual 0.5%. Every strategy has a fleet-wide alarm that rolls back when more than 2% of all requests fail for two minutes. That is too small to trip the alarm, so all at once, rolling and blue-green serve the bug to everyone for the whole hour: 36,000 extra failures and counting, until someone notices. The canary compares its own error rate with the old version's and rolls back after 3 minutes, with 1% of traffic exposed: 18 failed requests.
- Caught
- not within the hour
- Failed requests caused
- 36,000 and counting
- Most traffic on the bug
- 100%
- Caught
- not within the hour
- Failed requests caused
- 31,500 and counting
- Most traffic on the bug
- 100%
- Caught
- not within the hour
- Failed requests caused
- 36,000 and counting
- Most traffic on the bug
- 100%
- Caught
- after 3 minutes, by the canary check
- Failed requests caused
- 18
- Most traffic on the bug
- 1%
Why the canary catches what alarms miss
A release that fails an extra 1% of requests moves the fleet’s error rate from 0.5% to 1.5%. That stays under a 2% alarm, so all at once, rolling and blue-green serve it to everyone for the whole hour: 36,000 extra failed requests and counting. The canary compares its own error rate with the old version’s, on the same traffic at the same time, and the difference is clear after 3 minutes at 1% of traffic: 18 failed requests. For a bad bug, +10%, the alarm does fire within 2 minutes, but everyone is already on the new version: 24,000 failures while all at once redeploys, 12,000 for blue-green, which only has to switch traffic back, and 60 for the canary.
The cost is time. A healthy release takes the canary 30 minutes to reach everyone with 10 minute steps, against 15 for rolling and none for blue-green. Small steps also need enough traffic to tell signal from noise: a bug adding 0.3% failures is not caught at 1%, and the canary only rolls back after 11 minutes, once it is serving 10% of traffic.
Choosing a strategy
- Rolling is the default in Kubernetes and most platforms: cheap, no extra capacity, but rollback is another deploy and the health checks are per server, not a comparison.
- Blue-green needs a second full fleet, or at least the capacity for one, and gives an instant, complete rollback. It suits releases you want to switch all at once, such as a new database schema read path, after testing the idle fleet.
- Canary needs a router that can split traffic by percentage, such as a service mesh or load balancer, and metrics per version. It is the safest for services with enough traffic to compare.
- Feature flags do the same at the level of a feature rather than a binary: ship the code dark, then turn it on for 1% of users. They pair well with canaries.
No traffic-based strategy catches everything. Memory leaks show up only after hours, and a migration that rewrites data cannot be rolled back by moving traffic. Keep releases backward compatible, so the old and new versions can run side by side, and separate schema changes from code changes.
A canary rollout in Kubernetes
# Argo Rollouts: move traffic in steps; a background analysis can fail the rollout,
# which sends all traffic back to the stable version (selector and pod template omitted)
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: checkout
spec:
strategy:
canary:
steps:
- setWeight: 1
- pause: { duration: 10m }
- setWeight: 10
- pause: { duration: 10m }
- setWeight: 50
- pause: { duration: 10m }
analysis:
templates:
- templateName: error-rate-vs-stableIn a system design interview
When asked how you ship safely, describe the pipeline: “Each release goes to one canary cell at 1% of traffic for 10 minutes, then 10% and 50%. An automated analysis compares error rate and p99 latency against the old version on the same traffic and rolls back on its own; humans are paged only if the rollback fails. Risky features also sit behind a flag.” Expect the follow-ups: what the canary compares against, how you handle database migrations, and how the system behaves while two versions run at once.
Read the guide to deployment strategies or SLOs and error budgets, see the circuit breaker for failing fast at runtime, or watch a strong candidate design Netflix.
Questions people ask
What is a canary deployment?
A release strategy that sends a small share of real traffic, often 1%, to the new version, compares its metrics with the old version, and increases the share in steps only while the new version looks healthy. If it looks worse, traffic moves back to the old version, so a bad release affects few users.
What is the difference between canary and blue-green deployments?
Blue-green runs the new version on a separate fleet and switches all traffic to it at once, keeping the old fleet ready to switch back instantly. A canary shifts traffic gradually and compares the two versions as it goes, so problems are caught while only a small share of users is exposed.
What is a rolling deployment?
A strategy that replaces servers with the new version in batches, such as a quarter at a time, checking that each batch starts and passes health checks before continuing. It needs no extra capacity, but health checks rarely catch higher error rates, and rolling back means deploying the old version again.
How does automated canary analysis work?
It collects the same metrics, such as error rate and latency percentiles, from the canary and from the old version serving the same traffic at the same time, and runs a statistical comparison. If the canary is significantly worse, the rollout stops and traffic moves back. Netflix’s Kayenta and Argo Rollouts analyses work this way.
Why compare the canary with the old version instead of using an alarm threshold?
Because a bug confined to a small share of traffic barely moves fleet-wide metrics, and a fixed threshold must sit above normal variation. Comparing the canary with the old version at the same moment cancels out time of day and traffic mix, so small differences become detectable quickly.
How long should each canary step last?
Long enough to collect the traffic needed to detect the smallest regression you care about. A busy service can decide in minutes at 1%; a quiet one needs longer steps or a larger share. Some problems, such as memory leaks, need hours, so long-running checks follow the rollout.
What can canary deployments not catch?
Problems that appear only at full load or after a long time, such as memory leaks or capacity limits, and changes that cannot be undone by moving traffic, such as data migrations. Keep releases backward compatible, ship schema changes separately, and use feature flags for risky features.