Design an API Gateway

• One front door for every public and partner API: Route by host, method, path and version to ~300 services. /v1 and /v2 live side by side, and a team can send 5 % of a route to a new backend version, assigned by key so each partner sees one version.
• Authenticate every request: API keys for partner servers, OAuth 2.0 access tokens (JWTs) for apps acting for our users. Bad credentials are rejected before any backend sees the request, and backends get a verified identity, never the credential itself.
• Authorise by scope and plan: Each route declares the scopes it needs and the API product it belongs to, which the caller’s plan must include. Object-level checks (may this tenant read order 123?) stay in the service.
• Rate limits and quotas per key: Per-second limits with bursts to protect capacity, and monthly quotas that enforce plans and feed billing. They look alike and are built as two different systems.
• Validate and transform requests cheaply: Size limits, schema validation from each team’s OpenAPI spec, header and path rewrites, REST to gRPC transcoding. Bodies stream by default; only small ones are buffered.
• Protect backends and each other: Per-route timeouts, retries only where safe, circuit breakers, and load shedding by priority, so a slow service or a public traffic spike does not reach partner traffic.
• Revoke credentials in seconds: A leaked key or a compromised app stops working on every proxy within seconds, without a lookup on every request.
• Observe everything per key: Calls, errors and latency per key and route, for us and for the developer; sampled logs and traces by request id. All of it off the request path.
• Self-service config for 300 teams: Teams change their own routes and policies without a ticket to the gateway team. Changes are validated, rolled out in stages and reverted in one step.
• Out of scope: The identity service itself (login, consent, token issuance), the portal UI, invoicing, DDoS scrubbing and WAF rules, GraphQL federation and backends for frontends, and service-to-service traffic, which the mesh handles.

• Throughput (1 M req/s peak · 3 regions): Average ~400 k/s, ~35 B requests a day. Each region is sized for 500 k/s so that two can carry the peak when one is lost.
• Added latency (p99 < 10 ms · p50 ~1 ms): This drives the hardest trade-off in the design. A remote call costs 5–10 ms at p99, so nothing on the hot path may call the network except the backend, and revocation speed and quota exactness have to be won some other way.
• Availability of the data plane (99.99 %): Every API is behind the gateway, so its availability caps theirs. The control plane, the auth server and the key service may all be down without stopping traffic.
• Revocation delay (p99 < 5 s · worst case 15 min): Pushed to every proxy in seconds; if every push path failed, a revoked token still dies when it expires.
• Counting accuracy (billing exact · limits ±10 %): Billed usage must have no lost or double-counted request after reconciliation. Quota enforcement may lag ~20 s; rate limits are protective and may be approximate.
• Tenant isolation (partner p99 unchanged): One key, one app or one slow backend must not move the latency or errors of anyone else, above all partners during a public spike.
• Config safety (one team’s blast radius): A bad change can touch only its own team’s routes, reaches a few proxies for a few minutes before it is caught, and reverts in seconds.
• Credential hygiene (hashes only · never logged): API keys are stored as hashes, credentials are never logged or forwarded, and identity headers sent by clients are stripped.

• Requests per day: ~35 B · 1 M/s peak. Peak 1 M/s, average about 400 k/s (peaks run ~2.5× the daily mean). 400 k × 86,400 ≈ 35 B requests a day. Every one is authenticated, limited, counted and logged, so the cost of each check per request is what the design is about.
• Gateway proxies: ~25 per region · run 30 × 3. Three regions, each sized so two can carry the peak: 1 M ÷ 2 = 500 k/s per region. At 20 k req/s per proxy (32 cores at ~60 %, TLS mostly resumed, with auth, limits, validation and logging) that is 25. Run 30 per region, 90 in all: 12 in the partner pool and 18 in the public pool.
• CPU per request for the checks: ~0.1 ms. Route match ~10 µs, an ES256 JWT signature check ~50 µs (an API key hash and lookup is ~2 µs), token buckets ~1 µs, validating a 2 KB body ~50 µs: about 0.1 ms. At 1 M/s the JWT checks alone are ~50 cores, under 2 % of the fleet’s 2,880. A network call costs ~0.5 ms at p50 and 5–10 ms at p99, so one remote auth or quota call per request would spend the whole p99 budget.
• API key table: 100 k keys × ~200 B ≈ 20 MB. A key record is the SHA-256 hash (32 B) plus key id, app, tenant, plan, scopes, priority and status: ~200 B. 100 k × 200 B = 20 MB, small enough to push the whole table to every proxy and never look a key up over the network. 1 M keys would still be only 200 MB; around 100 M, a cache in front of a lookup service becomes the better design.
• Revocation deny list: ~100 live entries. Access tokens live 15 min, so a revoked token only needs denying until it would have expired anyway. At ~10 k revocations a day: 10 k ÷ 96 fifteen-minute windows ≈ 100 live entries. Revoking a whole app or user is one entry, (client_id or sub, issued before T). A few KB, pushed to every proxy.
• Token refresh load: ~11 k/s at 15 min. 10 M active user sessions, each refreshing once per token lifetime: 10 M ÷ 900 s ≈ 11 k/s at the authorization server. 5-minute tokens triple it to 33 k/s; 1-hour tokens cut it to 2.8 k/s but leave a revoked token working for up to an hour if a push is missed. Token lifetime is where revocation delay is traded against auth server load.
• Usage data: ~10 TB/day raw · ~250 GB/day aggregated. A 300 B access record × 400 k/s ≈ 120 MB/s ≈ 10 TB/day. Billing and quotas need counts, not records, so each proxy sums per (key, route, status class) over 10 s windows: ~3 k rows per window per proxy, ~27 k rows/s across 90 proxies, ~250 GB a day. Full records are kept only for errors and a 1 % sample of successes.
• Gateway config: ~5 k routes · ~10 MB · ~200 changes/day. 300 services × ~17 operations ≈ 5 k routes. ~1 KB of match, auth, limits, timeouts and retries per route plus ~1 KB of compiled schema ≈ 10 MB, plus the 20 MB key table. One team’s change is ~30 KB, so a delta to 90 proxies is ~3 MB, and a booting proxy loads all 30 MB in about a second. Pushing config is not the hard part; deciding what is safe to push is.

• API clients (partner servers · apps): Partner servers calling with API keys (about 40 % of traffic), third-party apps calling on behalf of our users with OAuth access tokens, and our own mobile apps. Partners use their own hostname, which lands them on a separate proxy pool. Every response carries one error format, rate-limit and quota headers, and a request id, which is what clients quote when something goes wrong.
• Developer portal (keys · plans · usage): Where developers register apps, create and rotate API keys, pick a plan and watch their usage, errors and quota. A new key is shown exactly once, and only after every proxy has acknowledged it, so the first call with it never fails. Usage pages read the usage store, not the gateway.
• L4 edge (anycast · one VIP per pool): The L4 tier from the load balancer design: anycast addresses, ECMP and consistent hashing of connections onto gateway proxies. Each pool has its own VIP, so partner and public traffic are separated before anything is decrypted. It is also the cheapest place to drop a flood, and the control plane can push temporary drop rules for one abusive source to it.
• Gateway proxies (Envoy-style · stateless): 90 proxies, 30 per region in two pools (12 partner, 18 public), each handling ~20 k requests a second. They terminate TLS, verify JWTs and API keys, check scopes and plans, take tokens from local buckets, validate small bodies, route by host, method, path and version, and forward with identity headers over mTLS. Every decision uses data held in memory and pushed by the control plane, so the only network hop they add is the one to the backend.
• Backend services (300 services · mTLS): About 300 services owned by their own teams, each publishing an OpenAPI spec that the gateway compiles into routes and validators. They receive the verified identity in x-auth-* headers, accepted only from the gateway’s mTLS identity, and never the raw credential. Object-level authorisation, such as whether this tenant may read order 123, stays here.
• Rate limit counters (Redis · synced every 500 ms): A Redis cluster per region holding each key’s global token-bucket state: the component from the rate limiter design, used in its local mode. Proxies never call it per request; every 500 ms each proxy makes one pipelined call that reports what it consumed and returns its share of every limit, weighted by its share of that key’s traffic. If it is unreachable, proxies keep their last shares.
• Authorization server (OAuth 2.0 · ES256 JWTs): The identity team’s OAuth 2.0 and OpenID Connect server; the gateway consumes it but does not own it. It issues 15-minute access tokens signed with ES256, publishes each new signing key a day before using it, and sends revocations (one token, a user or a whole app) into the control plane’s fast lane. Short tokens are the revocation backstop, and they cost it ~11 k refreshes a second.
• API key service (issue · rotate · revoke): Generates keys of the form ak_live_ plus 32 random base62 characters and a checksum, stores only their SHA-256 hash, and lets a key have a successor so partners rotate without downtime. Every change (a new key, a revocation, a plan or scope change) is committed, then published to the control plane’s fast lane. It also takes leaked-key reports from secret scanners and revokes those keys automatically.
• Key and plan store (Postgres · hashes only): Source of truth for apps, tenants, plans, scopes and API keys: about 100 k keys at ~200 B each, 20 MB. The raw key is never stored, so a leaked backup is useless to an attacker. Proxies hold a pushed copy of the key table, so this store is off the request path entirely.
• Gateway control plane (compile · stage · fast lane): Compiles every team’s routes, policies and schemas, plus the platform’s plans, into one versioned snapshot, rejects anything that fails validation, and rolls changes out in stages, reverting on regressions. A second, unstaged fast lane carries signing keys, key changes, deny entries and over-quota flags to every proxy in seconds. It is never on the request path: if it disappears, proxies serve their last accepted snapshot.
• Route and policy repo (one directory per team): Versioned config with one directory per service: its routes, limits, timeouts, retry rules and OpenAPI schemas, changed through that team’s own reviewed commits. The platform owns plans, scopes, global policy and the registry of which team owns which host and path prefix. About 5 k routes and ~200 changes a day.
• Usage stream (Kafka · 10 s windows): Kafka topics keyed by key id, carrying each proxy’s 10 s usage windows tagged with (proxy, sequence), per-route metrics, sampled access logs (every error, 1 % of successes) and sampled spans. Proxies spool to local disk first, so a slow Kafka delays usage but never loses it. Nothing on the request path waits for it.
• Quota service (durable period totals): Consumes usage windows, drops duplicate batches by (proxy, sequence), and keeps per-key, per-product totals for the billing period in a durable store. It warns developers at 80 % and 100 % and, for hard-capped plans, publishes an over-quota flag through the fast lane. Its totals trail reality by ~15 s, which is why enforcement is eventual and billing is reconciled.
• Usage store (columnar · per key per minute): A columnar store (ClickHouse or Druid style) holding calls, status classes and latency per key, route and minute for 13 months, plus the raw per-proxy windows for reconciliation. It serves the portal’s analytics, support’s investigations and the billing export: ~250 GB a day of aggregated rows instead of 10 TB of request logs.
• Usage reconciler (hourly + at period close): Recounts each period from the per-proxy windows, checks that every proxy’s sequence numbers are contiguous, asks proxies to re-send missing windows from their spools, and compares totals with each proxy’s own request counters. Only then is a period’s usage final and exported to billing. Differences go back to the quota service as corrections.

1. A call is authenticated, limited and routed: The hot path. Every check happens in the proxy’s memory, so the only network hop the gateway adds is the one to the backend, and the backend receives a verified identity instead of a credential.
   1. A third-party app holds a 15-minute access token for one of our users, issued by the authorization server.
   2. The request reaches a gateway proxy in the public pool through the L4 edge.
   3. The proxy verifies the token locally and checks the deny list.
   4. The proxy matches the route and checks the scope and the plan.
   5. The proxy takes tokens from its local buckets and counts the call.
   6. Every 500 ms the proxy reports what it consumed and gets back its share of each limit.
   7. The proxy validates the body, rewrites the path and forwards with identity headers and a deadline.
   8. The proxy returns the response with limit headers and ships the record off the hot path.
2. A team canaries a new backend version: The self-service path. A team changes only its own routes, the control plane compiles and validates the change with everyone else’s, and the rollout is staged so a mistake reaches a few proxies for a few minutes. Partners are assigned to the canary by key, so each sees one version consistently.
   1. The orders team merges a change that sends 5 % of /v2/orders to a new backend version.
   2. The control plane compiles every team’s config into a new snapshot and runs the validators.
   3. Canary proxies load the new snapshot first and run synthetic checks.
   4. The rollout widens to 10 % of proxies, one region, then all, comparing the orders routes on the new and old versions.
   5. Each proxy sends a caller to the canary when the hash of its key or app falls in the first 5 of 100 buckets.
   6. The team steps the split to 25, 50 and 100 %, each a new version that takes the same staged path.
   7. If the control plane fails mid-rollout, proxies keep serving the version they last accepted.
3. Search slows down at evening peak: The scale-breaking case. One backend slows under peak load, clients start retrying, and the public pool runs hot. Timeouts, per-cluster caps, retry budgets, a breaker and priority shedding each take part of the blow, and partner traffic never notices.
   1. At peak, a bad search deploy pushes search latency from 80 ms to 3 s.
   2. The route’s timeout cuts each search off at 1 s and tells search to stop working on it.
   3. Search’s in-flight cap fills, and extra searches fail in microseconds.
   4. The proxy adds no retries: search’s retry budget is spent.
   5. Badly written clients retry at once, and per-key and per-app buckets reject the worst of them locally.
   6. The search circuit breaker opens, and searches fail fast for 10 s at a time.
   7. Public-pool CPU passes 75 %, and proxies shed by priority: sheddable traffic first, then free.
   8. The partner pool sees none of it: different hostname, different proxies.
4. A leaked API key is revoked in seconds: The security failure path. A live key turns up in a public repository, and it must stop working on all 90 proxies within seconds, even though no proxy looks anything up per request. The same lane revokes OAuth tokens.
   1. A secret scanner finds a live key in a public repository and reports it.
   2. The key service hashes the key, finds its record and marks it revoked.
   3. The change goes into the control plane’s fast lane.
   4. Every proxy applies the change and acknowledges it.
   5. The next request with the key is refused with a 401 by whichever proxy it reaches.
   6. The developer creates a replacement, and the portal shows it only after every proxy has it.
   7. OAuth tokens are revoked through the same lane, as deny entries that expire with the tokens.
5. A key reaches its monthly quota: The asynchronous path. Usage is counted in the proxies, shipped in windows, deduplicated and summed, and an over-quota flag comes back through the fast lane about 20 s after the line is crossed. A reconciler makes the count exact for billing.
   1. Each proxy counts calls per (key, product) in 10 s windows and spools every closed window to disk before shipping it.
   2. The quota service drops duplicate batches and adds each window to the key’s total for the period.
   3. The same stream is rolled up per key, route and minute into the usage store.
   4. At 80 % the developer is warned; at 100 % on a hard-capped plan the quota service publishes an over-quota flag.
   5. Proxies refuse that key’s calls to that product with a 429 quota_exceeded until the period resets.
   6. The reconciler recounts the period from per-proxy windows and chases any gaps in the sequence numbers.
   7. Any remaining difference goes back to the quota service, so live totals match the final count.

Ask: Three hundred teams change their routes and policies every week. How do they do it without filing a ticket with the gateway team, and without one team’s mistake breaking everyone’s API?

Good answers name: Team-owned config, compiled and validated centrally, rolled out in stages, A central gateway team reviews and applies every change, Each team runs its own gateway, Custom plugin code per team in the shared proxy.

Our pick: Each service has a directory in the config repo with its routes, policies and OpenAPI schemas, owned by its team; the platform owns plans, scopes, global policy and the registry of which team owns which host and path prefix. The control plane compiles everything into one versioned snapshot and rejects a change that fails any check: schema, ownership of every host and prefix, no overlapping routes, auth on every route not explicitly marked public, scopes that exist, timeouts within the 30 s platform maximum, RE2 regexes so matching stays linear, and the compiled size. A change that compiles goes to two canary proxies per region running synthetic checks, then 10 % of proxies, one region and everywhere over ~15 minutes, comparing 5xx and latency on the changed routes between proxies on the new and old versions; a regression stops it and re-pushes the previous version. Signing keys, key changes, deny entries and quota flags take a separate fast lane with no staging, because each can only ever affect one credential. Proxies keep the last snapshot they accepted on local disk and serve it if the control plane is gone. The push mechanics (versioned deltas, ACK and NACK) are the load balancer’s control plane; Kong’s hybrid mode splits its gateway the same way, with data plane nodes that keep serving their cached config when the control plane is unreachable.

1. Two teams both claim /v2/orders. What happens?
   The compiler refuses whichever change arrives second and names the owner of the prefix. Ownership is registered once per host and path prefix in a platform-owned file, so a new claim needs the current owner’s approval. Without this check the outcome would depend on route order, and a team could silently capture another team’s traffic, which is a security problem as well as an outage.
2. The control plane is down for two hours. What still works?
   All traffic, on the last accepted snapshot: routes, the key table, signing keys and deny lists stay in memory and on disk. Route changes stop, which is an inconvenience. Revocation matters more, so it does not depend on the control plane alone: the key service and the authorization server each write a signed snapshot of their state to object storage that proxies poll every 60 s, so a leaked key still dies within about a minute.
3. A change passes validation but is wrong: the team points /v2/orders at its staging cluster. How far does it get?
   To the canary proxies and the 10 % stage, where synthetic checks and the 5xx comparison on that team’s routes catch it within minutes, and the rollout reverts itself. Even if it reached every proxy, only the orders routes would be affected, because the change could only touch the orders namespace. Rollback re-pushes the previous version to every proxy at once: seconds, not another 15-minute rollout.
4. Why not let teams write plugins? Kong and Envoy both support them.
   Because a shared proxy runs every team’s code in one process: one slow regex or memory leak stalls traffic for all 300 services. Declarative config covers almost everything teams ask for: routes, auth, limits, timeouts, header and path rewrites, schema validation. For the rest the platform offers a few reviewed plugins, or Wasm filters with CPU and memory limits on routes that opt in, never arbitrary code merged 200 times a day.
5. How do you know a proxy is running the config it should?
   Every proxy reports the snapshot it runs, ACKs each version it loaded and NACKs any it rejected. Dashboards show versions per proxy and the time from merge to every proxy acknowledging; alert on any NACK and on any proxy several versions behind its peers. A proxy whose own stream broke while its peers are current takes itself out of rotation after a minute, but when every proxy is behind because the control plane is down, they all keep serving.

Ask: Why not just ask the auth service whether each token or key is valid? It is the only way to know about a revocation immediately.

Good answers name: Local checks: JWTs against pushed signing keys, API keys against a pushed table of hashes, revocations pushed, Introspect every token and look up every key, Introspect, then cache the answer for 60 s, Long-lived JWTs with no revocation list.

Our pick: Access tokens are JWTs signed with ES256 by the authorization server, with a 15-minute lifetime and iss, aud, sub, client_id, scope, exp and jti. Proxies pin the algorithm, find the kid among signing keys pushed by the control plane, verify the signature and check issuer, audience and expiry with 30 s of clock leeway; an unknown kid is a 401, never a fetch, so a forged kid cannot make proxies call anyone, and the authorization server publishes each new key a day before signing with it. API keys look like ak_live_ plus 32 random base62 characters and a checksum: a malformed key is rejected from its format alone, and the rest is hashed with SHA-256 and looked up in the pushed 20 MB key table. A fast hash is safe because the key has ~190 random bits; bcrypt would cost ~100 ms per request for nothing. Revocations go through the control plane’s fast lane, by jti for one token, by (sub or client_id, issued before T) for a user or an app, and as status changes for keys, reaching every proxy at p99 under 5 s, with a polled snapshot in object storage every 60 s as the second path. If both fail, tokens still die within 15 minutes, which is why the lifetime is short; the price is ~11 k refreshes a second at the authorization server.

1. Why hash the API keys at all, if the table lives in our own memory?
   Because the table also lives in the key store, its backups and every proxy, and any of them can leak. With only SHA-256 hashes stored, a leaked table gives an attacker nothing they can send. The raw key exists once, when the portal shows it to the developer, and after that only its first few characters are shown so people can tell keys apart. For the same reason proxies never log the Authorization or X-Api-Key header.
2. A developer creates a key and calls the API one second later. Does it work?
   Yes, because the portal does not hand the key over until it is live: the key service commits it, the fast lane pushes it, and the create call returns once every proxy has acknowledged that version, typically in one to two seconds. An unreachable proxy does not block it forever: after 5 s the key is returned and that proxy is marked stale, and a proxy that stays behind its peers leaves rotation. Two seconds at creation is far cheaper than a lookup path on every request.
3. How do you rotate the authorization server’s signing key without failing requests?
   Publish the new public key a day before signing with it, so every proxy has it long before the first token carrying its kid. Switch signing to the new key, keep the old one for at least the 15-minute token lifetime, then remove it. Because unknown kids are rejected rather than fetched, the order matters: announce, wait, sign, wait, retire.
4. A partner’s plan is downgraded and loses the orders API. How fast does that take effect?
   Entitlements tied to an API key or an app’s plan are in the pushed key table, so a plan change reaches every proxy in seconds. Scopes inside an OAuth token stay as issued until the token expires, at most 15 minutes, unless we also push an (client_id, issued before T) entry, which forces a refresh that returns the new scopes. For a downgrade 15 minutes is usually fine; for an incident we push the entry.
5. At 10 M or 100 M keys, does pushing the table still work?
   1 M keys is 200 MB per proxy, still fine to push with no lookups. Around 10 M to 100 M, typically when every end user gets a personal token, memory and boot time push towards a cache: proxies keep recently used keys for 10 minutes, look up misses at a regional key service with a negative cache for unknown hashes, and keep the pushed change events to evict revoked keys. The checksum in the key format matters more then, because it filters random keys before they cost a lookup.

Ask: A plan says 100 requests a second and 10 million a month. Why not enforce both with the same counters?

Good answers name: Rate limits as synced local buckets; quotas counted from the usage stream, enforced by pushed flags, reconciled, One shared Redis counter per key for both, A durable decrement of the quota on every request, Quota leases: each proxy takes a block of the remaining quota and spends it locally.

Our pick: Rate limits reuse the rate limiter design in its local mode: each proxy keeps token buckets per (key, route class), plus a ceiling per OAuth app, and every 500 ms makes one pipelined call to a regional Redis that reports what it consumed and returns its share of each limit, weighted by its recent share of that key’s traffic. A key new to a proxy starts with an even share; if Redis is unreachable, proxies keep their last shares, which fails open within a bound. Quotas are counted from usage, never from limiter state: each proxy sums calls per (key, product) over 10 s windows, appends each closed window to a local disk spool and ships it to Kafka with (proxy id, sequence number). The quota service drops duplicate batches by that pair, keeps per-key totals for the period in a durable store, warns at 80 % and 100 %, and for hard-capped plans publishes an over-quota flag that proxies enforce with a 429 quota_exceeded until the period resets. A reconciler recounts each period from the windows, chases sequence gaps from the spools, compares against each proxy’s own counters and finalises billable usage; the ~0.2 % overshoot on a hard cap is counted but not billed.

1. A proxy crashes with 8 seconds of uncounted usage in memory. Is that money lost?
   Up to 10 s of one proxy’s counts, at most ~200 k calls at peak against 35 B a day: real but small, and bounded by the window. Windows that had already closed are on the spool and are shipped when the proxy comes back, or the reconciler sees the gap and pulls them. If the bound mattered more, 1 s windows would cut it tenfold at ten times the rows. We do not bill what we cannot count, so the error favours the customer.
2. Why ±10 % on rate limits? Can you make it tighter?
   The error comes from shares being up to 500 ms stale: when a key’s traffic moves between proxies, some hold share they are not using while others run out. Syncing every 100 ms cuts the error at five times the Redis traffic, and the rate limiter’s exact mode, one Redis call per request, removes it at the cost of a network call on the hot path. For a protective limit, 10 % is the right trade.
3. The billing period ends at midnight UTC. What happens at 00:00:00?
   The quota service rolls each key to a new period and clears over-quota flags, and the flags already carry their own expiry, so proxies lift them even if that message is late. A window that straddles midnight is split by request timestamp, so each call lands in exactly one period. Plans anchored to a customer’s billing date work the same way with a per-key period start; the dangerous version is computing periods in local time, which creates 23- and 25-hour days.
4. A customer disputes their invoice: “we never made 12 million calls”. How do you answer?
   From the usage store: calls per day, per route and per key, with status classes, plus the sampled access logs for spot checks of individual requests. Because counts come from deduplicated, sequence-checked windows that the reconciler compared with each proxy’s own counters, we can show the figure is complete and not double counted. Usually the answer is a forgotten integration or a retry loop on 5xx, both visible per route.
5. A partner says 5 % of their calls fail. How do they, and you, see why?
   The portal reads the usage store: calls, status classes and latency per key and route per minute, so the partner can see whether the failures are their 4xx or our 5xx and on which route. Individual requests are found by the request id on every response, in the sampled logs that keep every error. For a hard case, support switches on full tracing for that one key for an hour through the fast lane; otherwise spans are sampled at 1 %, which at 400 k requests a second is still 4 k traces a second.

Ask: Search latency goes from 80 ms to 3 seconds. Why doesn’t that take down the orders API too, on the same proxies?

Good answers name: Per-route timeouts and deadlines, per-cluster in-flight caps, idempotent-only retries within a 10 % budget, circuit breakers, Retry every failure up to 3 times, Queue requests at the gateway until the backend recovers, Adaptive concurrency limits per cluster.

Our pick: Every route has a timeout set by its team within a 30 s platform maximum, 1 s for search and 2 s for creating an order, and the proxy sends the remaining deadline in a header so the backend can abandon work the caller has stopped waiting for. Each cluster has a per-proxy in-flight cap of about three times its normal concurrency (requests a second × typical latency), ~500 for search, and requests beyond it get an immediate 503 with Retry-After, so search can never hold the memory and connections that orders needs. Retries happen once, on a different host, only for connect failures, resets before any response and 503s that say the request was not processed, and only for GET, HEAD, PUT and DELETE, or a POST with an Idempotency-Key on a route that declares its service honours one; retries are capped at 10 % of a cluster’s requests. A circuit breaker per cluster on each proxy opens when more than half of at least 100 requests in 10 s fail or time out, answers 503 at once for 10 s, then lets 5 % through to probe. Ejecting single bad hosts inside a cluster is the load balancer’s outlier detection, reused as is.

1. Why does the in-flight cap matter if you already have a timeout?
   A timeout bounds how long each request is held, not how many are held. At 2 k searches a second and a 1 s timeout, a dead search cluster still occupies 2 k requests per proxy at every moment, most of them doomed. The cap of ~500 turns the rest into instant 503s. Together they bound a slow cluster’s footprint on a proxy whatever happens downstream.
2. A partner’s POST /v2/orders times out at the gateway. Did the order get created?
   Maybe, and the gateway cannot know, which is exactly why it did not retry. It returns a 504 with the request id. The partner retries with the same Idempotency-Key, and the orders service either returns the original result or creates the order once. Only routes whose service implements idempotency keys let the gateway retry a POST itself, and even then only for failures before any response.
3. The breaker on search opens. What does a client see, and how does it close?
   A 503 with Retry-After and an error code saying the service is unavailable, in under a millisecond, from every proxy whose breaker opened. Each proxy decides on its own, so proxies that still see search succeeding keep sending to it. After 10 s a breaker lets 5 % of requests through; if they succeed it closes, if not it stays open another 10 s, so search does not meet the full load the instant it recovers.
4. Client, gateway and service all retry. How do you stop retries multiplying?
   Retry at one layer per hop, and budget it. The gateway retries at most once and within 10 % of traffic, services do the same through their mesh, and our SDKs back off with jitter and honour Retry-After. The gateway marks a retried request with a header so the backend does not retry it again, and it counts client retries by Idempotency-Key, which is how we spot a client library with a bad retry loop.
5. How do you choose a route’s timeout?
   From that route’s own latency, not a default: a few times its p99.9, within the platform maximum. Search at a p99.9 of 300 ms gets 1 s; an export that takes minutes is not a synchronous route at all and returns 202 with an operation id. The control plane warns when a route’s timeout is far above its observed latency, because a 30 s timeout on an 80 ms route is a 30 s pile-up waiting to happen.

Ask: A free-tier developer’s batch job loops and sends 200,000 requests a second. How do your paying partners not notice?

Good answers name: Per-key and per-app limits, separate partner and public pools, priority classes shed in order, edge drops for floods, Per-key rate limits only, A dedicated gateway deployment per large tenant, Autoscale the proxies.

Our pick: Limits per key, plus a ceiling per OAuth app (client_id), because a popular app’s users each have their own token but share one bug. Partners and enterprise plans use their own hostname on a separate pool, 12 proxies per region against 18 for the public pool, so public traffic never reaches partner proxies. Every request gets a priority from its plan and route: critical (partner and enterprise writes), standard (paid), free, and sheddable (bulk exports, analytics, requests marked as retries). When a pool’s CPU passes 75 % or a backend cluster signals overload, proxies shed sheddable traffic first, then free, then standard, with a 503 and Retry-After, ramping the shed fraction rather than flipping it. A key rejected more than 90 % of the time for a minute at over 10 k requests a second is reported to the control plane, which pushes a 15-minute drop rule for its source addresses to the L4 edge, where dropping a packet costs almost nothing; that is for floods from a few servers, never for an app whose traffic comes from millions of phones behind shared carrier addresses. Expensive routes such as search take several tokens per call, so a heavy endpoint cannot hide inside a request count.

1. Why separate pools by hostname rather than by API key?
   Because the L4 edge in front of the proxies sees addresses and ports, not keys, and the key is inside TLS. A hostname with its own VIP is the earliest point where partner traffic can be told apart. Partners get the partner hostname at onboarding, and the public hostname refuses partner keys after a migration period, so nobody ends up on the wrong pool.
2. Isn’t shedding free-tier traffic just breaking the product for free users?
   Only during overload, and the alternative is worse: without priorities every request slows down together until everything times out, free and paid alike. Shedding part of the free traffic for a few minutes keeps the product working for everyone else and keeps free users’ retries from deepening the hole. The order is a product decision written down in advance, not improvised in an incident.
3. How does the gateway know a backend is overloaded before it starts timing out?
   Three signals: the cluster’s in-flight count approaching its cap, latency rising against its normal p99, and the backend saying so with a 503 and an overload header when its own concurrency limiter fires. Any of them starts priority shedding for that cluster’s routes only, so an overloaded search sheds free search traffic without touching orders.
4. A critical-class partner spikes within its limits. Can it hurt other partners?
   Within the partner pool, yes, if the spike is big enough, which is why partner limits are sized against pool capacity: the largest few partners’ limits together must fit in the pool’s headroom. Beyond that, per-cluster caps protect each backend, and the very largest partners can get their own priority within the critical class or, by contract, their own deployment.
5. How would you test that partner traffic survives a public spike?
   Game days. In a staging copy, replay recorded public traffic at several times its peak; in production, move a region’s public traffic onto fewer proxies until shedding starts. Partner p99 and errors should not move, and the shed counters should climb class by class in the right order. If partner latency moves, something is shared that should not be.

Ask: Teams want the gateway to validate every request against their schema and to reshape payloads for old API versions. How much body work should a gateway do?

Good answers name: Stream by default; validate small bodies on routes that opt in; rewrite headers, paths and protocols only, Full transformation templates in the gateway, No body inspection at the gateway, Buffer and validate every body.

Our pick: Bodies stream by default under a size limit per route, 64 KB for most JSON routes and more only where declared, enforced from Content-Length and again by counting bytes as they arrive, so an oversized upload is cut off with a 413 without being buffered. Routes that opt in get JSON Schema validation compiled from the team’s OpenAPI spec when the snapshot is built, run only when the body fits under the route’s 1 MB buffer limit; a 2 KB body validates in ~50 µs and a failure returns a 400 naming the bad fields. Transformations are limited to what needs no understanding of the payload: header and path rewrites (/v2/orders to /orders), identity and deadline headers, compression, and JSON to gRPC transcoding for gRPC backends, which Envoy has a filter for. Responses always stream. Mapping old API versions onto new ones stays in the service: Stripe, for example, pins each account to a dated API version and keeps the per-version changes in its API code, not at its edge.

1. Why validate at the gateway at all if the service validates too?
   To reject garbage before it costs a backend hop, and to give every API the same 400 format with field-level errors, generated from the same OpenAPI spec as the docs. It is a filter, not a guarantee: services still validate, because they are also called from inside. Routes opt in, so a team with a hot, simple endpoint can skip it.
2. A client sends Content-Length: 1000 and then streams 10 GB.
   The proxy counts bytes as they arrive and resets the stream at the route’s limit, whatever the header said; with chunked encoding there may be no header at all. The client gets a 413 if it is still listening, and the connection is closed. Header checks are a fast path, the byte count is the real limit.
3. How does the gateway serve /v1 and /v2 of the same API?
   As separate routes, usually to separate clusters, or to one service with the version passed through, owned by the same team. The gateway routes by path prefix, or by a version header for APIs with dated versions, and adds a Deprecation header with a sunset date to v1 responses. Usage per key per version from the usage store tells the team which partners still call v1 before they turn it off.
4. A response is 50 MB. Does the proxy buffer it?
   No. Responses stream to the client as the backend sends them, with flow control, so memory per request stays at a few buffers. Anything that would need the whole response, such as rewriting its fields, is not done at the gateway, and compression streams too.