Scalability fundamentals

Vertical vs horizontal scaling, stateless services, the standard web architecture, and how to reason about which tier breaks first.

Scalability is the property that a system can handle more load by adding resources, ideally in proportion. Almost every system design interview is, underneath, the question "what breaks first as load grows, and what do you do about it?" This guide covers the vocabulary and the standard architecture that the rest of the guides build on.

Vertical versus horizontal

Vertical scaling (scale up) means a bigger machine: more cores, more memory, faster disks. It is the right first move more often than interview folklore suggests: a single modern server has 100+ cores, terabytes of RAM and NVMe disks doing a million IOPS, and there is no distributed-systems complexity. The limits are a hard ceiling, a single point of failure, and cost that grows faster than linearly at the top end.

Horizontal scaling (scale out) means more machines. It has no ceiling and gives redundancy, but it forces you to answer: how do requests find a machine, where does state live, and what happens when one machine fails or is slower than the rest? Every other topic in system design is an answer to one of those questions.

The practical rule: scale stateless tiers horizontally from day one because it is cheap, and scale stateful tiers vertically until you have a specific reason not to (size, throughput, or availability), then partition.

Stateless services

A service is stateless when any instance can handle any request, because the request carries or looks up everything it needs. That is what makes horizontal scaling and load balancing trivial: add an instance, register it with the load balancer, done. Failure is equally simple: the load balancer stops sending to it.

The state has to go somewhere. The options, in increasing order of cost:

• The client. Session tokens (a signed JWT) instead of server-side sessions. The server verifies the signature and needs no lookup.
• A shared store. Redis for sessions, carts, and rate-limit counters. Every instance reads the same thing.
• The database. The durable state that the system exists to keep.

The one place stateful services are unavoidable is long-lived connections (WebSockets). There the "state" is which server holds a user's socket, and you need a registry so other servers can find it. See real-time systems.

The standard architecture

Nearly every web-scale system is a variation of this shape, and drawing it in the first minute of the high-level design is expected.

clients → DNS → CDN → load balancer → API gateway → stateless services
                                                        ├── cache
                                                        ├── database (primary + replicas)
                                                        ├── object storage
                                                        └── message queue → workers

• DNS maps the name to one or more IPs and, with GeoDNS, sends users to the nearest region.
• CDN serves static assets and cacheable responses from edge locations near the user. See CDNs and edge.
• Load balancer spreads requests over healthy instances and terminates TLS. See load balancing.
• API gateway authenticates, rate limits, routes to services, and hides internal topology.
• Services hold the business logic and are stateless.
• Cache absorbs repeated reads so the database sees only misses. See caching.
• Database is the source of truth. Replicated for availability and read scaling; partitioned when one node cannot hold or serve it.
• Object storage (S3, GCS) for large blobs: images, video, backups. Cheap, durable, and separate from the database so the database stays small and fast.
• Queue decouples work that does not have to happen before the response: sending an email, resizing an image, updating a search index. See queues and streams.

What breaks first

Load grows and something saturates. Knowing the usual order lets you say "at this scale, the bottleneck is X" without measuring.

1. The database, almost always. It is the one thing that cannot be trivially cloned. The fixes, in order: add indexes and fix slow queries; add a cache for reads; add read replicas; partition (shard) writes; move data that does not need transactions to a specialised store.
2. Hot spots. One user, one key, one partition receiving disproportionate load: a celebrity, a viral link, a global counter. The fix is to spread the hot key (see sharding) or absorb it in a cache or in-memory aggregation.
3. Synchronous fan-out. A request that calls five services in sequence has the latency of their sum and the availability of their product. Parallelise, cache, or make it asynchronous.
4. Connections. Databases handle hundreds to low thousands of connections; a thousand stateless instances each holding a pool exhaust that. Connection poolers (PgBouncer, ProxySQL) sit between.
5. The network. Serving video or large files from your own servers saturates NICs long before CPUs. Push bytes to a CDN or object storage and serve URLs.

Latency versus throughput

They are different and both matter. Throughput is how many requests per second the system completes; latency is how long one takes. Adding machines raises throughput; it does not lower latency, and it can raise it if it adds a hop. Lowering latency means fewer sequential hops, less work per request, closer data (cache, edge), and avoiding tail amplification: with 100 parallel calls, a 1 % chance each is slow becomes a 63 % chance the request is slow. Hedged requests, timeouts, and tight per-call budgets are the answer.

Saying it in the interview

"I will scale the API tier horizontally behind a load balancer since it is stateless. The database is the bottleneck: at 5 k writes/s a single Postgres primary is fine, and I will add read replicas and a cache for the 50 k reads/s. If writes reached 50 k/s I would shard by user id." That is two sentences and covers the whole tier list with a threshold for each transition.