Distributed rate limiting: local limits vs a central Redis counter, simulated

A rate limit is easy on one server: keep a counter. With a fleet of API gateways behind a load balancer, each sees only part of a tenant’s traffic, and the limit belongs to the tenant as a whole. Distributed rate limiting is the choice of where the count lives: split across the gateways, in one shared store such as Redis, or kept locally and shared now and then.

The playground limits one tenant to 100 requests a second across several gateways and runs the same traffic through all three designs. Try uneven load balancing, bursts and different sync intervals.

Three designs, three failure modes

• Local limits give each of 4 gateways a quarter of the limit. With traffic spread evenly that is exact and free. But load balancers are rarely even, especially with long-lived connections: when 55% of a tenant’s 81 requests a second land on one gateway, it runs out while the others sit idle, and 189 requests are rejected although the tenant never exceeded its limit. Every time the fleet scales, every share changes.
• A central counter in Redis is exact whatever the spread: every request increments the tenant’s counter for the current second and is allowed if the count is within the limit. It costs a round trip on every request, about a millisecond inside a data center, and a Redis operation for every request, 313 a second here, and Redis becomes a dependency of every call.
• Local counts with periodic sync cut Redis traffic: each gateway counts on its own and adds its count to Redis every 100 ms, 40 calls a second instead of 313. Between syncs each gateway knows only its own count, so the fleet lets 123 requests a second through a limit of 100, and during a burst 220. Syncing once a second lets nearly 300 through; syncing every 10 ms is almost exact but makes 400 Redis calls a second, more than the central counter.

In practice the central counter is the default: a Redis call per request is cheap next to the request itself, and one Redis shard handles tens of thousands of operations a second. The sync design suits very high volumes where a few percent of overshoot is acceptable. Either way, gateways can remember locally that a tenant is blocked until the current window ends and skip Redis until then.

Details that matter

• Atomicity. Read-then-write races let two gateways both see 99 and both allow. Use INCR, which returns the new count, or a Lua script for token buckets and sliding windows.
• When Redis is down. Decide in advance whether to fail open, allowing traffic and protecting availability, or fail closed, protecting the backend. Many fall back to local limits.
• Scale. Key counters by tenant, so Redis Cluster spreads tenants across shards; one very hot tenant can still need its own handling.
• Where to enforce. At the edge for abuse, per tenant at the API gateway, and with load shedding inside services for overload that no per-tenant limit predicts.

A fixed-window limiter on Redis

import time
import redis

r = redis.Redis()

def allow(tenant: str, limit: int = 100) -> bool:
    window = int(time.time())                 # one counter per tenant per second
    key = f"rl:{tenant}:{window}"
    pipe = r.pipeline()                       # MULTI/EXEC: both or neither
    pipe.incr(key)
    pipe.expire(key, 2)                       # old windows delete themselves
    count, _ = pipe.execute()
    return count <= limit

In a system design interview

Say where the count lives and why: “Limits are per API key, enforced in the gateway. Each request does an atomic INCR on a Redis key for the key and the current second, about half a millisecond. Redis is sharded by API key and replicated; if it is unreachable, gateways fall back to a local limit of the global limit divided by the number of gateways, so we degrade to approximate rather than unlimited.” Expect the follow-ups: the boundary burst of fixed windows, hot tenants, and limits across regions.

Read the guide to rate limiting algorithms or API gateways, or watch a strong candidate design a rate limiter.

Questions people ask