Serverless cold starts, keep-warm pings and provisioned concurrency: an interactive simulator
A serverless platform runs your function on instances it creates on demand and removes when idle. A request that finds no idle instance waits while a new one starts: a cold start, from a couple of hundred milliseconds for a lean runtime to seconds for a heavy one. How often that happens depends on the shape of the traffic, and the fixes, keep-warm pings and provisioned concurrency, help some shapes and not others.
The playground sends an hour of traffic to a function that runs 100 ms per request on instances that each serve one request at a time and shut down after 5 idle minutes.
When traffic jumps from 5 to 150 requests a second, the warm pool is far too small: 99 requests start new instances at once and wait 900 ms, and because each waits through its cold start, the spike needs 97 instances instead of about 15. A ping keeps one instance warm, which is no help here. Provisioning 10 instances absorbs part of it: 54 cold starts remain, for $124/mo instead of $33/mo. Shorter cold starts help twice: each waits less, and fewer instances are needed.
- Cold starts
- 99 0.37%
- Slowest requests
- 900 ms
- Peak instances
- 97
- Cost
- $33/mo
- Cold starts
- 98 0.37%
- Slowest requests
- 900 ms
- Peak instances
- 97
- Cost
- $33/mo
- Cold starts
- 54 0.2%
- Slowest requests
- 900 ms
- Peak instances
- 64
- Cost
- $124/mo
Which fix fits which traffic
- Quiet traffic is where cold starts hurt most often: with a request every few minutes, instances shut down in between, and 5 of 8 requests in the hour start cold. A scheduled ping keeps one instance warm for almost nothing; provisioned concurrency also works, at about $110 a month.
- Steady traffic keeps its own instances warm. After the first few, every request is warm, and paying $117 a month for 10 provisioned instances instead of $22 buys nothing.
- Spikes are the hard case. At 150 requests a second the warm pool is far too small, 99 requests wait for new instances at once, and since every one of them holds an instance through its cold start, the spike needs 97 instances instead of 15. A ping cannot help. Provisioned concurrency helps only as far as it is sized for the peak, which is why it is often scaled up on a schedule before known events.
- Shorter cold starts help twice. With a 200 ms start the spike causes 49 cold starts; with a 2 s JVM start, 224, and 217 instances. Small packages, fewer dependencies loaded at start-up and snapshotting runtimes cut both the wait and the concurrency it causes.
Configuring it
# AWS Lambda: keep 10 instances initialised all the time
aws lambda put-provisioned-concurrency-config \
--function-name checkout-api --qualifier live \
--provisioned-concurrent-executions 10
# scale provisioned concurrency up before a known peak, down after
aws application-autoscaling put-scheduled-action \
--service-namespace lambda --scheduled-action-name before-sale \
--resource-id function:checkout-api:live \
--scalable-dimension lambda:function:ProvisionedConcurrency \
--schedule "cron(45 11 * * ? *)" \
--scalable-target-action MinCapacity=200,MaxCapacity=200
# make cold starts cheaper: small bundles, lazy imports, SnapStart for JavaIn a system design interview
If you put a latency-sensitive path on functions, say how cold starts are handled: “The API runs on functions with provisioned concurrency sized for the normal daytime peak, scaled up on a schedule before the evening rush. Bursty background work runs on plain on-demand functions, where a cold start costs nothing users can see. If the traffic were steady and high, containers on a service with autoscaling would be cheaper.” Expect the follow-ups: what concurrency limits apply, how connections to the database are pooled when hundreds of instances start at once, and where serverless costs more than servers.
Read the guide to serverless or autoscaling, see autoscaling and connection pooling, or watch a strong candidate design a workflow engine.
Questions people ask
What is a cold start in serverless?
The delay when a request arrives and no initialised instance of the function is free, so the platform must start one: provision an environment, load the runtime and the code, and run initialisation. It ranges from about 100 ms for small functions to several seconds for large JVM ones.
How do you reduce cold starts?
Keep packages small and initialisation light, choose faster-starting runtimes, use snapshotting features such as Lambda SnapStart, and keep instances warm with provisioned concurrency where latency matters. Scheduled pings help only for low, steady traffic.
What is provisioned concurrency?
A setting that keeps a number of function instances initialised and ready at all times, so requests up to that concurrency never cold-start. You pay for the provisioned instances whether or not they are used, so it suits predictable baseline or scheduled peak load.
Do keep-warm pings work?
They keep roughly one instance warm, which removes cold starts for very low traffic. They do nothing for bursts that need many instances at once, where most of the extra instances still cold-start.
Why do cold starts increase concurrency?
Each request that cold-starts occupies its new instance for the whole start-up time plus the request. During a burst, longer cold starts mean more requests are in flight at once, so the platform has to create even more instances, each of them cold.
When is serverless more expensive than containers?
At high, steady utilisation: per-request pricing is efficient for spiky or low traffic, but a function busy all the time costs more than a container or virtual machine sized for the same load, and provisioned concurrency adds a constant charge.
How do serverless functions connect to databases at scale?
Hundreds of instances starting at once can exhaust database connections, so functions connect through a pooler or proxy such as RDS Proxy or PgBouncer, or use HTTP-based data APIs that do not hold connections per instance.