Autoscaling explained: an interactive capacity simulator
Autoscaling adds servers when load rises and removes them when it falls, so you pay for roughly what you use. It works by reacting: it measures load, decides, and launches instances that take minutes to be ready. That makes it good at following the slow curve of a day and bad at sudden spikes, which arrive faster than any instance can boot. The settings that matter are the target utilization, how fast new capacity arrives, and whether you scale ahead of load you can predict.
The playground runs a day of traffic, with a push notification at 9:00 that sends traffic up two and a half times, against an autoscaled fleet. Change the target, the boot time and the schedule, add an unexpected flash crowd, and count the minutes when demand exceeded capacity.
Each instance serves 100 requests a second; the autoscaler aims for 70% utilization and new instances take 4 minutes to boot. At 9:00 a push notification sends traffic up two and a half times within two minutes. The autoscaler only sees it after it has happened, and the instances it launches arrive 4 minutes later: 6 minutes overloaded. Over the day: 255 instance-hours at 66% average utilization.
How autoscaling decides
- Measure a metric across the fleet, usually CPU, requests per instance or queue length, averaged over a short window.
- Compare with the target: the utilization you want instances to run at, such as 70%.
- Compute the desired count: current load divided by what one instance handles at the target.
- Scale out by launching the difference; they help only once booted, warmed up and passing health checks.
- Scale in slowly, after a cooldown, so a brief dip does not remove capacity that is needed again a minute later.
The target is your headroom
Running at 70% leaves 30% spare for load that arrives before new instances do. Raise it to 90% and the fleet costs less, 200 instance-hours instead of 255 in the playground, but the 9:00 surge overloads it for longer and turns away more requests. Lower it to 50% and the surge costs 4 minutes instead of 6, for 353 instance-hours. Headroom is cheap insurance only up to a point: no reasonable target absorbs a surge of two and a half times.
Boot time decides how long you suffer
From the moment a surge begins, the autoscaler needs a few minutes to notice and the new instances need their boot time before they help. With a 1 minute boot the 9:00 surge overloads the fleet for 3 minutes; with 4 minutes, 6; with 10 minutes, 12. Faster capacity comes from smaller images, pre-baked machine images, containers on already-running nodes, warm pools of stopped instances, or serverless functions that start in milliseconds.
Scale ahead of load you can predict
Much traffic is not a surprise: the daily curve, a scheduled push notification, a sale at noon, the start of a match. Scheduled or predictive scaling adds capacity before it is needed, from a calendar or from yesterday’s traffic. In the playground it takes the 9:00 surge from 6 overloaded minutes to none, for 6 more instance-hours. AWS predictive scaling and Kubernetes tools such as KEDA’s cron scaler do this.
What autoscaling cannot do
A flash crowd with no warning triples traffic within two minutes, and reactive scaling is always behind: 7 minutes overloaded at a 70% target, 5 even at 50%. Until capacity catches up, the service must protect itself by shedding load or serving from cache. Scaling also only helps the tier you scale: if the database behind the web servers is the bottleneck, more web servers send it more load. And scaling on CPU misses work that waits on I/O; scaling on queue length or concurrency often tracks real load better.
An autoscaler in Kubernetes
# Kubernetes: scale on CPU, keep headroom, scale out fast and in slowly
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: api }
spec:
scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: api }
minReplicas: 6 # never below what a surprise needs while scaling
maxReplicas: 200
metrics:
- type: Resource
resource: { name: cpu, target: { type: Utilization, averageUtilization: 60 } }
behavior:
scaleUp: { stabilizationWindowSeconds: 0, policies: [{ type: Percent, value: 100, periodSeconds: 30 }] }
scaleDown: { stabilizationWindowSeconds: 600, policies: [{ type: Percent, value: 10, periodSeconds: 60 }] }In a system design interview
Say what you scale on and how you handle the gap: “The API tier scales on requests per instance with a 60% target, which leaves room for a doubling while new pods start in about a minute. Known peaks, like the 8 pm match start, are scheduled an hour ahead. For anything faster, the gateway sheds low-priority traffic, and the database is sized for peak rather than autoscaled.” Expect the follow-ups: what you scale on, scale-in safety for stateful work, and what bottleneck appears when the stateless tier scales.
Read the guide to autoscaling, see queueing and Little’s law for why the target cannot be 100%, or watch a strong candidate design a ticketing system, where the spike is the whole problem.
Questions people ask
How does autoscaling work?
An autoscaler measures a metric across a group of instances, such as average CPU or requests per instance, compares it with a target, and adds or removes instances so the metric moves back toward the target. New instances only help once they have booted and passed health checks, and removals usually wait for a cooldown to avoid flapping.
What target utilization should autoscaling use?
Often 50 to 70% for latency-sensitive services. The gap between the target and 100% is headroom that absorbs load arriving before new instances are ready; a higher target saves money but overloads sooner on a spike, and queueing makes latency climb steeply as utilization approaches 100%.
Why can autoscaling not handle sudden spikes?
Because it is reactive: it notices load after it has arrived, then launches instances that take minutes to boot. A spike that doubles traffic in a minute overloads the existing fleet for that whole time. Headroom, faster-starting capacity, scheduled scaling for predictable peaks and load shedding cover the gap.
What is predictive or scheduled scaling?
Adding capacity before load arrives, based on a schedule or a forecast from past traffic, rather than reacting to current load. It suits predictable peaks such as the daily curve, a marketing email, a sale or the start of a live event, and is usually combined with reactive scaling for everything else.
What metric should you autoscale on?
The one that best tracks how close instances are to their limit. CPU is common and works for compute-bound services; requests per instance, concurrent requests or queue length are better for services that wait on I/O, and queue length or lag is natural for workers consuming from a queue.
What is a cooldown in autoscaling?
A period after a scaling action during which further actions, usually scale-ins, are held back, so the metric can settle and the fleet does not oscillate between too many and too few instances. Scale-out should generally react quickly and scale-in slowly, because too little capacity costs more than a few idle minutes.
Does autoscaling fix database bottlenecks?
No. Scaling a stateless tier sends more load to whatever it depends on, and databases cannot usually add capacity in minutes. Size stateful tiers for the peak, add read replicas or caches ahead of time, and make sure the autoscaled tier cannot overwhelm them, for example with connection limits.