Updated 24 October 2026

Scalability fundamentals

Vertical vs horizontal scaling, stateless services, the standard web architecture, and how to reason about which tier breaks first.

Scalability is the property that a system can handle more load by adding resources, ideally in proportion. Almost every system design interview is, underneath, the question "what breaks first as load grows, and what do you do about it?" This guide covers the vocabulary and the standard architecture that the rest of the guides build on.

Vertical versus horizontal

Vertical scaling (scale up) means a bigger machine: more cores, more memory, faster disks. It is the right first move more often than interview folklore suggests: a single modern server has 100+ cores, terabytes of RAM and NVMe disks doing a million IOPS, and there is no distributed-systems complexity. The limits are a hard ceiling, a single point of failure, and cost that grows faster than linearly at the top end.

Horizontal scaling (scale out) means more machines. It has no ceiling and gives redundancy, but it forces you to answer: how do requests find a machine, where does state live, and what happens when one machine fails or is slower than the rest? Every other topic in system design is an answer to one of those questions.

The practical rule: scale stateless tiers horizontally from day one because it is cheap, and scale stateful tiers vertically until you have a specific reason not to (size, throughput, or availability), then partition.

Stateless services

A service is stateless when any instance can handle any request, because the request carries or looks up everything it needs. That is what makes horizontal scaling and load balancing trivial: add an instance, register it with the load balancer, done. Failure is equally simple: the load balancer stops sending to it.

The state has to go somewhere. The options, in increasing order of cost:

• The client. Session tokens (a signed JWT) instead of server-side sessions. The server verifies the signature and needs no lookup.
• A shared store. Redis for sessions, carts, and rate-limit counters. Every instance reads the same thing.
• The database. The durable state that the system exists to keep.

The one place stateful services are unavoidable is long-lived connections (WebSockets). There the "state" is which server holds a user's socket, and you need a registry so other servers can find it. See real-time systems.

The standard architecture

Nearly every web-scale system is a variation of this shape, and drawing it in the first minute of the high-level design is expected.

clients → DNS → CDN → load balancer → API gateway → stateless services
                                                        ├── cache
                                                        ├── database (primary + replicas)
                                                        ├── object storage
                                                        └── message queue → workers

• DNS maps the name to one or more IPs and, with GeoDNS, sends users to the nearest region.
• CDN serves static assets and cacheable responses from edge locations near the user. See CDNs and edge.
• Load balancer spreads requests over healthy instances and terminates TLS. See load balancing.
• API gateway authenticates, rate limits, routes to services, and hides internal topology.
• Services hold the business logic and are stateless.
• Cache absorbs repeated reads so the database sees only misses. See caching.
• Database is the source of truth. Replicated for availability and read scaling; partitioned when one node cannot hold or serve it.
• Object storage (S3, GCS) for large blobs: images, video, backups. Cheap, durable, and separate from the database so the database stays small and fast.
• Queue decouples work that does not have to happen before the response: sending an email, resizing an image, updating a search index. See queues and streams.

What breaks first

Load grows and something saturates. Knowing the usual order lets you say "at this scale, the bottleneck is X" without measuring.

1. The database, almost always. It is the one thing that cannot be trivially cloned. The fixes, in order: add indexes and fix slow queries; add a cache for reads; add read replicas; partition (shard) writes; move data that does not need transactions to a specialised store.
2. Hot spots. One user, one key, one partition receiving disproportionate load: a celebrity, a viral link, a global counter. The fix is to spread the hot key (see sharding) or absorb it in a cache or in-memory aggregation.
3. Synchronous fan-out. A request that calls five services in sequence has the latency of their sum and the availability of their product. Parallelise, cache, or make it asynchronous.
4. Connections. Databases handle hundreds to low thousands of connections; a thousand stateless instances each holding a pool exhaust that. Connection poolers (PgBouncer, ProxySQL) sit between.
5. The network. Serving video or large files from your own servers saturates NICs long before CPUs. Push bytes to a CDN or object storage and serve URLs.

Latency versus throughput

They are different and both matter. Throughput is how many requests per second the system completes; latency is how long one takes. Adding machines raises throughput; it does not lower latency, and it can raise it if it adds a hop. Lowering latency means fewer sequential hops, less work per request, closer data (cache, edge), and avoiding tail amplification: with 100 parallel calls, a 1 % chance each is slow becomes a 63 % chance the request is slow. Hedged requests, timeouts, and tight per-call budgets are the answer.

Saying it in the interview

"I will scale the API tier horizontally behind a load balancer since it is stateless. The database is the bottleneck: at 5 k writes/s a single Postgres primary is fine, and I will add read replicas and a cache for the 50 k reads/s. If writes reached 50 k/s I would shard by user id." That is two sentences and covers the whole tier list with a threshold for each transition.

Scale cube, in one sentence each

A useful way to name the three independent axes of scaling:

• X: clone. Run more identical instances behind a load balancer. Cheapest, and the first move for stateless services.
• Y: split by function. Separate services for separate concerns, so the expensive part scales on its own and a slow feature cannot exhaust the shared pool.
• Z: split by data. Shard, so each instance owns a slice of the data. Necessary once the dataset, not the traffic, is the constraint.

Most designs need X first, then Z for the one store that is too big, and Y only where the team or the load profile genuinely differs.

Asynchrony as a scaling tool

The most effective scaling move is usually not another machine; it is moving work off the request path. Send email, generate thumbnails, update search indexes, recompute counters and fan out to followers after the response.

Two consequences to state, because they are what interviewers probe: the user now sees an intermediate state ("processing"), so the UI needs to represent it; and the work is now retryable and duplicable, so every consumer must be idempotent.

Isolation and blast radius

Scaling is also about making sure one bad thing stays small:

• Bulkheads: separate thread pools, connection pools and queues per dependency, so one slow downstream cannot consume every worker.
• Cells / shuffle sharding: partition customers across independent stacks so an outage or a poison-pill tenant takes out a fraction of users, not all of them.
• Priority separation: critical traffic (checkout, login) gets its own capacity and its own queue, separate from batch and analytics.

A capacity plan in five lines

1. Peak requests per second, from the estimates.
2. Requests per second per instance, from a load test.
3. Instances = peak ÷ per-instance, then divide by target utilisation (0.6) and add one zone's worth of headroom.
4. The stateful tier: database write capacity, cache memory for the working set, connection limits.
5. The growth rate, and which of the above hits its ceiling first.

That last line is the answer to "what breaks first", and it is the one interviewers remember.

Checklist

• What is stateless, and where the state actually lives.
• The scaling axis for each tier.
• What is moved off the request path.
• Isolation boundaries and blast radius.
• The capacity numbers and the first ceiling you will hit.

Questions people ask

What is scalability in system design?

Scalability is the property that a system can handle more load by adding resources, ideally in proportion. Underneath, almost every system design interview asks what breaks first as load grows and what you do about it. The usual starting point is a standard architecture: DNS, a CDN, a load balancer, an API gateway and stateless services, backed by a cache, a replicated database, object storage, and a message queue with workers.

What is the difference between vertical and horizontal scaling?

Vertical scaling (scale up) means a bigger machine with more cores, memory and faster disks, while horizontal scaling (scale out) means more machines. Vertical scaling adds no distributed-systems complexity, but it has a hard ceiling, a single point of failure and cost that grows faster than linearly at the top end. Horizontal scaling has no ceiling and gives redundancy, but forces you to decide how requests find a machine, where state lives, and what happens when one machine fails or is slow.

Should you scale vertically or horizontally first?

Scale stateless tiers horizontally from day one, because it is cheap, and scale stateful tiers vertically until there is a specific reason not to (size, throughput or availability), then partition. Vertical scaling is the right first move more often than interview folklore suggests: a single modern server has 100+ cores, terabytes of RAM and NVMe disks doing a million IOPS. For example, a single Postgres primary is fine at 5 k writes/s, and sharding by user id becomes the answer if writes reach 50 k/s.

What is a stateless service?

A stateless service is one where any instance can handle any request, because the request carries or looks up everything it needs. That makes horizontal scaling and load balancing trivial: add an instance and register it with the load balancer, and if one fails, the load balancer stops sending to it. The state moves to the client (a signed JWT instead of a server-side session), a shared store such as Redis, or the database.

What usually breaks first when a system scales?

The database breaks first, almost always, because it is the one component that cannot be trivially cloned. The fixes, in order, are to add indexes and fix slow queries, cache reads, add read replicas, shard writes, and move data that does not need transactions to a specialised store. After the database, the usual bottlenecks are hot spots, synchronous fan-out, database connection limits, and the network when serving video or large files.

What is the difference between latency and throughput?

Throughput is how many requests per second a system completes, while latency is how long one request takes. Adding machines raises throughput but does not lower latency, and can raise it if it adds a hop. Lowering latency means fewer sequential hops, less work per request, closer data and avoiding tail amplification: with 100 parallel calls, a 1 % chance that each is slow becomes a 63 % chance that the request is slow.