Design an API Gateway

One front door for public and partner APIs: authenticate, limit and route 1 M requests a second to 300 services, adding under 10 ms.

Last updated 2026-10-26. Difficulty: hard. Patterns: api-gateway, authentication, control-plane, quotas, multi-tenancy. Reported at Amazon, Google, Microsoft, Stripe, Netflix, Uber.

Functional requirements

• One front door for every public and partner API. Route by host, method, path and version to ~300 services. /v1 and /v2 live side by side, and a team can send 5 % of a route to a new backend version, assigned by key so each partner sees one version.
• Authenticate every request. API keys for partner servers, OAuth 2.0 access tokens (JWTs) for apps acting for our users. Bad credentials are rejected before any backend sees the request, and backends get a verified identity, never the credential itself.
• Authorise by scope and plan. Each route declares the scopes it needs and the API product it belongs to, which the caller’s plan must include. Object-level checks (may this tenant read order 123?) stay in the service.
• Rate limits and quotas per key. Per-second limits with bursts to protect capacity, and monthly quotas that enforce plans and feed billing. They look alike and are built as two different systems.
• Validate and transform requests cheaply. Size limits, schema validation from each team’s OpenAPI spec, header and path rewrites, REST to gRPC transcoding. Bodies stream by default; only small ones are buffered.
• Protect backends and each other. Per-route timeouts, retries only where safe, circuit breakers, and load shedding by priority, so a slow service or a public traffic spike does not reach partner traffic.
• Revoke credentials in seconds. A leaked key or a compromised app stops working on every proxy within seconds, without a lookup on every request.
• Observe everything per key. Calls, errors and latency per key and route, for us and for the developer; sampled logs and traces by request id. All of it off the request path.
• Self-service config for 300 teams. Teams change their own routes and policies without a ticket to the gateway team. Changes are validated, rolled out in stages and reverted in one step.
• Out of scope. The identity service itself (login, consent, token issuance), the portal UI, invoicing, DDoS scrubbing and WAF rules, GraphQL federation and backends for frontends, and service-to-service traffic, which the mesh handles.

Non-functional requirements

• Throughput (1 M req/s peak · 3 regions). Average ~400 k/s, ~35 B requests a day. Each region is sized for 500 k/s so that two can carry the peak when one is lost.
• Added latency (p99 < 10 ms · p50 ~1 ms). This drives the hardest trade-off in the design. A remote call costs 5–10 ms at p99, so nothing on the hot path may call the network except the backend, and revocation speed and quota exactness have to be won some other way.
• Availability of the data plane (99.99 %). Every API is behind the gateway, so its availability caps theirs. The control plane, the auth server and the key service may all be down without stopping traffic.
• Revocation delay (p99 < 5 s · worst case 15 min). Pushed to every proxy in seconds; if every push path failed, a revoked token still dies when it expires.
• Counting accuracy (billing exact · limits ±10 %). Billed usage must have no lost or double-counted request after reconciliation. Quota enforcement may lag ~20 s; rate limits are protective and may be approximate.
• Tenant isolation (partner p99 unchanged). One key, one app or one slow backend must not move the latency or errors of anyone else, above all partners during a public spike.
• Config safety (one team’s blast radius). A bad change can touch only its own team’s routes, reaches a few proxies for a few minutes before it is caught, and reverts in seconds.
• Credential hygiene (hashes only · never logged). API keys are stored as hashes, credentials are never logged or forwarded, and identity headers sent by clients are stripped.

Back-of-envelope estimates

• Requests per day: ~35 B · 1 M/s peak. Peak 1 M/s, average about 400 k/s (peaks run ~2.5× the daily mean). 400 k × 86,400 ≈ 35 B requests a day. Every one is authenticated, limited, counted and logged, so the cost of each check per request is what the design is about.
• Gateway proxies: ~25 per region · run 30 × 3. Three regions, each sized so two can carry the peak: 1 M ÷ 2 = 500 k/s per region. At 20 k req/s per proxy (32 cores at ~60 %, TLS mostly resumed, with auth, limits, validation and logging) that is 25. Run 30 per region, 90 in all: 12 in the partner pool and 18 in the public pool.
• CPU per request for the checks: ~0.1 ms. Route match ~10 µs, an ES256 JWT signature check ~50 µs (an API key hash and lookup is ~2 µs), token buckets ~1 µs, validating a 2 KB body ~50 µs: about 0.1 ms. At 1 M/s the JWT checks alone are ~50 cores, under 2 % of the fleet’s 2,880. A network call costs ~0.5 ms at p50 and 5–10 ms at p99, so one remote auth or quota call per request would spend the whole p99 budget.
• API key table: 100 k keys × ~200 B ≈ 20 MB. A key record is the SHA-256 hash (32 B) plus key id, app, tenant, plan, scopes, priority and status: ~200 B. 100 k × 200 B = 20 MB, small enough to push the whole table to every proxy and never look a key up over the network. 1 M keys would still be only 200 MB; around 100 M, a cache in front of a lookup service becomes the better design.
• Revocation deny list: ~100 live entries. Access tokens live 15 min, so a revoked token only needs denying until it would have expired anyway. At ~10 k revocations a day: 10 k ÷ 96 fifteen-minute windows ≈ 100 live entries. Revoking a whole app or user is one entry, (client_id or sub, issued before T). A few KB, pushed to every proxy.
• Token refresh load: ~11 k/s at 15 min. 10 M active user sessions, each refreshing once per token lifetime: 10 M ÷ 900 s ≈ 11 k/s at the authorization server. 5-minute tokens triple it to 33 k/s; 1-hour tokens cut it to 2.8 k/s but leave a revoked token working for up to an hour if a push is missed. Token lifetime is where revocation delay is traded against auth server load.
• Usage data: ~10 TB/day raw · ~250 GB/day aggregated. A 300 B access record × 400 k/s ≈ 120 MB/s ≈ 10 TB/day. Billing and quotas need counts, not records, so each proxy sums per (key, route, status class) over 10 s windows: ~3 k rows per window per proxy, ~27 k rows/s across 90 proxies, ~250 GB a day. Full records are kept only for errors and a 1 % sample of successes.
• Gateway config: ~5 k routes · ~10 MB · ~200 changes/day. 300 services × ~17 operations ≈ 5 k routes. ~1 KB of match, auth, limits, timeouts and retries per route plus ~1 KB of compiled schema ≈ 10 MB, plus the 20 MB key table. One team’s change is ~30 KB, so a delta to 90 proxies is ~3 MB, and a booting proxy loads all 30 MB in about a second. Pushing config is not the hard part; deciding what is safe to push is.

Components

• API clients (partner servers · apps): Partner servers calling with API keys (about 40 % of traffic), third-party apps calling on behalf of our users with OAuth access tokens, and our own mobile apps. Partners use their own hostname, which lands them on a separate proxy pool. Every response carries one error format, rate-limit and quota headers, and a request id, which is what clients quote when something goes wrong.
• Developer portal (keys · plans · usage): Where developers register apps, create and rotate API keys, pick a plan and watch their usage, errors and quota. A new key is shown exactly once, and only after every proxy has acknowledged it, so the first call with it never fails. Usage pages read the usage store, not the gateway.
• L4 edge (anycast · one VIP per pool): The L4 tier from the load balancer design: anycast addresses, ECMP and consistent hashing of connections onto gateway proxies. Each pool has its own VIP, so partner and public traffic are separated before anything is decrypted. It is also the cheapest place to drop a flood, and the control plane can push temporary drop rules for one abusive source to it.
• Gateway proxies (Envoy-style · stateless): 90 proxies, 30 per region in two pools (12 partner, 18 public), each handling ~20 k requests a second. They terminate TLS, verify JWTs and API keys, check scopes and plans, take tokens from local buckets, validate small bodies, route by host, method, path and version, and forward with identity headers over mTLS. Every decision uses data held in memory and pushed by the control plane, so the only network hop they add is the one to the backend.
• Backend services (300 services · mTLS): About 300 services owned by their own teams, each publishing an OpenAPI spec that the gateway compiles into routes and validators. They receive the verified identity in x-auth-* headers, accepted only from the gateway’s mTLS identity, and never the raw credential. Object-level authorisation, such as whether this tenant may read order 123, stays here.
• Rate limit counters (Redis · synced every 500 ms): A Redis cluster per region holding each key’s global token-bucket state: the component from the rate limiter design, used in its local mode. Proxies never call it per request; every 500 ms each proxy makes one pipelined call that reports what it consumed and returns its share of every limit, weighted by its share of that key’s traffic. If it is unreachable, proxies keep their last shares.
• Authorization server (OAuth 2.0 · ES256 JWTs): The identity team’s OAuth 2.0 and OpenID Connect server; the gateway consumes it but does not own it. It issues 15-minute access tokens signed with ES256, publishes each new signing key a day before using it, and sends revocations (one token, a user or a whole app) into the control plane’s fast lane. Short tokens are the revocation backstop, and they cost it ~11 k refreshes a second.
• API key service (issue · rotate · revoke): Generates keys of the form ak_live_ plus 32 random base62 characters and a checksum, stores only their SHA-256 hash, and lets a key have a successor so partners rotate without downtime. Every change (a new key, a revocation, a plan or scope change) is committed, then published to the control plane’s fast lane. It also takes leaked-key reports from secret scanners and revokes those keys automatically.
• Key and plan store (Postgres · hashes only): Source of truth for apps, tenants, plans, scopes and API keys: about 100 k keys at ~200 B each, 20 MB. The raw key is never stored, so a leaked backup is useless to an attacker. Proxies hold a pushed copy of the key table, so this store is off the request path entirely.
• Gateway control plane (compile · stage · fast lane): Compiles every team’s routes, policies and schemas, plus the platform’s plans, into one versioned snapshot, rejects anything that fails validation, and rolls changes out in stages, reverting on regressions. A second, unstaged fast lane carries signing keys, key changes, deny entries and over-quota flags to every proxy in seconds. It is never on the request path: if it disappears, proxies serve their last accepted snapshot.
• Route and policy repo (one directory per team): Versioned config with one directory per service: its routes, limits, timeouts, retry rules and OpenAPI schemas, changed through that team’s own reviewed commits. The platform owns plans, scopes, global policy and the registry of which team owns which host and path prefix. About 5 k routes and ~200 changes a day.
• Usage stream (Kafka · 10 s windows): Kafka topics keyed by key id, carrying each proxy’s 10 s usage windows tagged with (proxy, sequence), per-route metrics, sampled access logs (every error, 1 % of successes) and sampled spans. Proxies spool to local disk first, so a slow Kafka delays usage but never loses it. Nothing on the request path waits for it.
• Quota service (durable period totals): Consumes usage windows, drops duplicate batches by (proxy, sequence), and keeps per-key, per-product totals for the billing period in a durable store. It warns developers at 80 % and 100 % and, for hard-capped plans, publishes an over-quota flag through the fast lane. Its totals trail reality by ~15 s, which is why enforcement is eventual and billing is reconciled.
• Usage store (columnar · per key per minute): A columnar store (ClickHouse or Druid style) holding calls, status classes and latency per key, route and minute for 13 months, plus the raw per-proxy windows for reconciliation. It serves the portal’s analytics, support’s investigations and the billing export: ~250 GB a day of aggregated rows instead of 10 TB of request logs.
• Usage reconciler (hourly + at period close): Recounts each period from the per-proxy windows, checks that every proxy’s sequence numbers are contiguous, asks proxies to re-send missing windows from their spools, and compares totals with each proxy’s own request counters. Only then is a period’s usage final and exported to billing. Differences go back to the quota service as corrections.

User flows

1. A call is authenticated, limited and routed. The hot path. Every check happens in the proxy’s memory, so the only network hop the gateway adds is the one to the backend, and the backend receives a verified identity instead of a credential.
   1. A third-party app holds a 15-minute access token for one of our users, issued by the authorization server. The user consented through the authorization code flow with PKCE, which the identity service owns, and the app refreshes the token every 15 minutes. The token is a JWT signed with ES256 that carries the user, the app’s client_id and the scopes granted. Fifteen minutes is the revocation backstop if every push fails.
   2. The request reaches a gateway proxy in the public pool through the L4 edge. The hostname picks the pool: api.example.com is served by the public pool and partners.example.com by the partner pool, each behind its own VIP, so the L4 tier keeps them apart without looking inside TLS. The proxy terminates TLS, as the L7 tier for these hostnames, and deletes any x-auth-* headers the client sent, so nobody can claim an identity the gateway did not verify.
   3. The proxy verifies the token locally and checks the deny list. Algorithm pinned to ES256, the kid looked up in the signing keys the control plane pushed, signature checked in ~50 µs, then issuer, audience and expiry with 30 s of clock leeway, then the deny list by jti, sub and client_id. An API key instead is checked for format and checksum, hashed with SHA-256 and looked up in the pushed key table in ~2 µs. No network call either way.
   4. The proxy matches the route and checks the scope and the plan. Routes are compiled into a lookup by host, then method and longest path prefix, then headers such as a version header: ~10 µs. This route needs orders:write and belongs to the orders API product, which the app’s plan must include. A missing scope is a 403 insufficient_scope; whether this user may order for this account is the orders service’s decision, not the gateway’s.
   5. The proxy takes tokens from its local buckets and counts the call. Three buckets, each a microsecond in memory: per (user, app), per app across all its users, and the plan’s write class. None is checked over the network; the proxy’s share of each limit is refreshed in the background. The call is also added to the current 10 s usage window for (app, orders-api).
   6. Every 500 ms the proxy reports what it consumed and gets back its share of each limit. One pipelined call carries this proxy’s consumption for every key it saw since the last sync and returns each key’s global remaining tokens and this proxy’s share, weighted by its recent fraction of that key’s traffic. That is the rate limiter design in its local mode, chosen because the gateway cannot afford a network call per request. If Redis is unreachable, proxies keep their last shares.
   7. The proxy validates the body, rewrites the path and forwards with identity headers and a deadline. The 1.8 KB body is under the route’s 64 KB limit, so it is buffered and checked against the compiled CreateOrder schema in ~50 µs. The request goes on a pooled HTTP/2 connection with mTLS, and backends accept x-auth-* headers only from the gateway’s mTLS identity. The deadline header lets the service give up when the 2 s route timeout is nearly spent.
   8. The proxy returns the response with limit headers and ships the record off the hot path. Headers report the most restrictive bucket. The call is in the 10 s usage window, which is spooled to disk and shipped when it closes; errors are always logged in full and successes at 1 %. Spans are sampled at 1 % unless this key is flagged for debugging, when every request is traced.
2. A team canaries a new backend version. The self-service path. A team changes only its own routes, the control plane compiles and validates the change with everyone else’s, and the rollout is staged so a mistake reaches a few proxies for a few minutes. Partners are assigned to the canary by key, so each sees one version consistently.
   1. The orders team merges a change that sends 5 % of /v2/orders to a new backend version. The change lives in the orders directory and is reviewed by the orders team alone; nobody on the gateway team is in the loop. The split is by hash of the key or app, not per request, so one partner does not see responses alternate between versions. Partners on an exclusion list never join a canary.
   2. The control plane compiles every team’s config into a new snapshot and runs the validators. Checks: the schema; the orders team owns api.example.com/v2/orders; no other route overlaps; auth is required unless the route is marked public with a security review; the scopes exist; the timeout is within 30 s; regexes are RE2, so matching stays linear; the new cluster exists and has healthy endpoints. A failure stops here with a message to the team, and nothing reaches a proxy.
   3. Canary proxies load the new snapshot first and run synthetic checks. Two proxies per region, serving synthetic traffic and a little real traffic, get the delta first. A proxy that cannot load it NACKs and keeps the previous version, which stops the rollout. The push mechanics are the load balancer’s control plane; what is new here is that each stage watches only the routes that changed.
   4. The rollout widens to 10 % of proxies, one region, then all, comparing the orders routes on the new and old versions. At each stage the control plane compares the 5xx rate and p99 of the orders routes on proxies running 18233 against those still on 18232, from the per-route metrics in the usage stream. A regression re-pushes 18232 automatically. The whole rollout takes about 15 minutes.
   5. Each proxy sends a caller to the canary when the hash of its key or app falls in the first 5 of 100 buckets. Every proxy computes the same hash, so the assignment holds whichever proxy a request lands on. Metrics are tagged with the backend version, so the team compares v2-next against v2 on the same mix of partners.
   6. The team steps the split to 25, 50 and 100 %, each a new version that takes the same staged path. Going forward is staged; going back is not. A rollback re-pushes a version that already ran everywhere, so it goes to every proxy at once and takes seconds.
   7. If the control plane fails mid-rollout, proxies keep serving the version they last accepted. Proxies keep their snapshot in memory and on local disk, so a restart comes back on the same version. The rollout pauses with some proxies on the old version and some on the new, both of which passed validation. Traffic is unaffected; only further changes wait.
3. Search slows down at evening peak. The scale-breaking case. One backend slows under peak load, clients start retrying, and the public pool runs hot. Timeouts, per-cluster caps, retry budgets, a breaker and priority shedding each take part of the blow, and partner traffic never notices.
   1. At peak, a bad search deploy pushes search latency from 80 ms to 3 s. Each public-pool proxy sends search ~2 k requests a second. At 80 ms that is ~160 in flight; at 3 s it would be 6 k per proxy, each holding buffers and an upstream stream on a proxy that also serves orders, catalogue and payments. Without bounds, one slow service would take every route on the proxy down with it.
   2. The route’s timeout cuts each search off at 1 s and tells search to stop working on it. The proxy sends the remaining deadline with the request, so search can abandon work for callers that are already gone instead of finishing it. The client gets a 504 with the request id, and since a search is a GET it may retry with backoff.
   3. Search’s in-flight cap fills, and extra searches fail in microseconds. Each proxy allows a cluster about 3× its normal concurrency, ~500 for search. Past that, new searches get an immediate 503 with Retry-After instead of queueing, so search can hold at most 500 requests’ worth of memory and streams on any proxy, and orders on the same proxy keep theirs.
   4. The proxy adds no retries: search’s retry budget is spent. Searches are GETs, so a retry on another host would be safe, but retries are capped at 10 % of a cluster’s requests and that budget went in the first seconds. Retrying every failure would multiply the load on search exactly when it can least take it.
   5. Badly written clients retry at once, and per-key and per-app buckets reject the worst of them locally. Public traffic climbs ~30 % as apps retry without backoff. Each key and app is held to its plan’s rate and the excess gets 429s from local buckets, but a rejection still costs TLS and header parsing, about a fifth of a full request, so the public pool runs hotter.
   6. The search circuit breaker opens, and searches fail fast for 10 s at a time. More than half of at least 100 requests in 10 s failed, so each proxy stops calling search and answers 503 at once. After 10 s it lets 5 % through: if they succeed it closes, if not it waits another 10 s. Search gets room to recover instead of meeting the full load the moment it improves.
   7. Public-pool CPU passes 75 %, and proxies shed by priority: sheddable traffic first, then free. Bulk exports, analytics and requests marked as client retries go first, then free-tier traffic, each shed fraction ramping up in steps rather than all at once. Paid traffic in the public pool is shed only if that is not enough. The order was decided with the product team in advance.
   8. The partner pool sees none of it: different hostname, different proxies. Partner traffic arrives on its own VIP at its own 12 proxies per region, so neither the retries nor the shedding reach it. Partners calling search see the open breaker like everyone else, but orders, payments and catalogue calls keep their normal latency.
4. A leaked API key is revoked in seconds. The security failure path. A live key turns up in a public repository, and it must stop working on all 90 proxies within seconds, even though no proxy looks anything up per request. The same lane revokes OAuth tokens.
   1. A secret scanner finds a live key in a public repository and reports it. The ak_live_ prefix and checksum make our keys easy to match with few false positives, which is why GitHub’s secret scanning partner programme can find them and post them to us within minutes of a push. Reports are signed by the scanner, and the endpoint accepts nothing else.
   2. The key service hashes the key, finds its record and marks it revoked. The store only knows hashes, so the reported key is hashed with SHA-256 to find it. The revocation records who reported it and why, and the key’s owner is emailed with where it was found.
   3. The change goes into the control plane’s fast lane. The fast lane is unstaged on purpose. A revocation can only make one credential stop working, so its worst case is one wrongly blocked key, while staging it like a route change would leave a leaked key working for 15 minutes.
   4. Every proxy applies the change and acknowledges it. Typically about a second to all 90 proxies, under 5 s at p99. The key service also writes a signed snapshot of the key table that proxies poll every 60 s, so revocation still lands within a minute if the control plane’s streams are down.
   5. The next request with the key is refused with a 401 by whichever proxy it reaches. The proxy hashes the key, finds it revoked in memory and answers without calling anything. Whoever copied the key from the repository gets nothing, and so does the partner’s own integration, which is why the portal makes rotation painless.
   6. The developer creates a replacement, and the portal shows it only after every proxy has it. The key service commits the new key, pushes it through the fast lane and returns once the control plane reports every proxy has acknowledged that version, typically in 1–2 s. The first call with the new key therefore never fails. For planned rotation a key can have a successor, and both work for 24 hours.
   7. OAuth tokens are revoked through the same lane, as deny entries that expire with the tokens. One token is denied by its jti, a user or a whole app by (sub or client_id, issued before T), and each entry is dropped 15 minutes later, when every token it covers has expired anyway. That is why the list stays around 100 entries. If every push failed, the 15-minute lifetime is the backstop.
5. A key reaches its monthly quota. The asynchronous path. Usage is counted in the proxies, shipped in windows, deduplicated and summed, and an over-quota flag comes back through the fast lane about 20 s after the line is crossed. A reconciler makes the count exact for billing.
   1. Each proxy counts calls per (key, product) in 10 s windows and spools every closed window to disk before shipping it. Counting is an in-memory increment on the hot path; everything else happens after the window closes. The spool means a slow or unavailable Kafka delays usage but never loses it, and each batch carries the proxy id and a sequence number, so duplicates and gaps can both be found later.
   2. The quota service drops duplicate batches and adds each window to the key’s total for the period. Proxies and Kafka retry, so the same batch can arrive twice; the (proxy, seq) pair makes adding it idempotent. Totals live in a durable store keyed by (key, product, period). They trail reality by ~15 s: the window, plus shipping, plus processing.
   3. The same stream is rolled up per key, route and minute into the usage store. This is what developers see in the portal and what support queries: calls, error classes and latency per key and route per minute, kept for 13 months. The per-proxy windows are kept too, for reconciliation.
   4. At 80 % the developer is warned; at 100 % on a hard-capped plan the quota service publishes an over-quota flag. Plans with billed overage are never blocked; they only accrue. For a hard cap the flag goes through the fast lane like a revocation, because it only restricts one key. It carries the period end, so proxies lift it without another message.
   5. Proxies refuse that key’s calls to that product with a 429 quota_exceeded until the period resets. The same status as a rate limit, but a different body and headers, because the client must react differently: wait a second, or stop until the period resets. From crossing 100 % to the first refusal is ~20 s, so a key at 1 k calls a second overshoots by ~20 k calls, 0.2 % of its quota; those are counted but not billed.
   6. The reconciler recounts the period from per-proxy windows and chases any gaps in the sequence numbers. Every proxy’s sequence numbers must be contiguous. A gap means a batch never arrived, so the reconciler asks that proxy to re-send it from its spool, and each proxy’s totals are compared with its own request counters. Only then is the period’s usage final and exported to billing.
   7. Any remaining difference goes back to the quota service, so live totals match the final count. Re-sent batches flow through the stream like any other and are counted once by the (proxy, seq) check. The reconciler’s corrections cover the rest, such as a window a consumer bug dropped, so a key near its cap is judged on the number it will be billed on.

Deep dives

The theory behind it

• API gateways and backends for frontends: What an API gateway does at the edge of a system: routing, authentication, rate limiting, request transformation and aggregation, the backend-for-frontend pattern, GraphQL gateways, and how to keep the gateway from becoming a bottleneck or a monolith.
• JWT vs session cookies: Two ways to keep users logged in: server-side sessions with cookies versus self-contained JSON Web Tokens. How each works, revocation, size, security pitfalls, refresh tokens, storage in browsers and mobile apps, and the hybrid most large systems use.
• Rate limiting and resilience: Token bucket, leaky bucket, fixed and sliding windows, distributed rate limiters, plus the resilience toolkit: timeouts, retries with backoff and jitter, circuit breakers, bulkheads, load shedding and graceful degradation.

Related

• Design Zapier
• Design a Distributed Logging System
• Design a Load Balancer
• Design a RAG System
• All 83 system design interview questions
• Run this question for a friend: the interviewer kit