Shuffle sharding: an interactive blast radius simulator

In a multi-tenant service, one customer can take down everyone: a request that crashes servers, a traffic flood, a query that never ends. Shuffle sharding limits that blast radius by giving each customer its own random combination of a few workers. A bad customer can only break its own workers, and almost no other customer depends on exactly that combination, so the rest lose at most part of their capacity.

The playground has 24 customers on 8 workers. Click any customer to make it the poison one and compare a shared fleet, cells and shuffle sharding.

From a shared fleet to shuffle shards

On a shared fleet, a customer whose requests crash workers reaches every worker through retries, and all 23 other customers go down with it. Splitting the workers into cells of 2 confines the damage to one cell, but the 5 customers sharing that cell go down too. With shuffle sharding each customer gets 2 of the 8 workers, one of 28 combinations. The poison customer still kills its 2, but no other customer has that exact pair: 10 lose one of their two workers and keep running on the other, and 13 are untouched.

The numbers grow quickly. Choosing 5 of 100 workers gives over 75 million combinations, so the chance that another customer shares all of a poison customer’s workers is negligible. Amazon Route 53 gives each domain 4 of 2,048 virtual name servers, about 730 billion combinations.

What it needs, and what it costs

• Clients that fail over. A degraded customer stays up only if its requests retry on another worker in its shard. With bigger shards, here 4 of 8, nobody else goes down, but almost everyone loses some capacity.
• Stable assignments. Derive the shard from the customer id, or store it, so a customer always lands on the same workers and stays isolated.
• Spare capacity. Each worker carries several customers’ traffic, and a customer whose shard is degraded loads its remaining workers harder.
• Shared dependencies. Workers sharded this way still share their database, cache and network; isolation is only as good as the weakest shared layer, which is why designs combine shuffle sharding with cells, bulkheads and per-tenant rate limits.

Assigning a shuffle shard in code

import hashlib, random

def shuffle_shard(customer_id: str, workers: list[str], k: int = 4) -> list[str]:
    """The same customer always gets the same k workers; different customers rarely share all k."""
    seed = int.from_bytes(hashlib.sha256(customer_id.encode()).digest()[:8], "big")
    return random.Random(seed).sample(workers, k)

# the router sends a customer's request to one of its workers, and its retries to another
def route(customer_id, workers, healthy):
    shard = shuffle_shard(customer_id, workers)
    return [w for w in shard if w in healthy] or shard   # all down: fail, but only this customer

In a system design interview

Bring it up for multi-tenant systems: “Tenants share a fleet of 100 stateless workers, but each tenant is pinned to 5 of them, chosen by hashing the tenant id. If one tenant sends poison requests or a flood, it can only exhaust its own 5; any other tenant shares at most a few of those workers and keeps serving from the rest, because the client retries on another worker in its shard. Per-tenant rate limits sit in front.” Expect the follow-ups: how to detect the poison tenant, what happens to a tenant whose whole shard is unhealthy, and how the database tier is isolated.

Read the guide to cell-based architecture or multi-tenancy, see load shedding, or watch a strong candidate design a job scheduler.

Questions people ask