Design an LLM Inference Service

Serve a large language model behind an API: queue the requests, batch them onto scarce GPUs, stream tokens back, and keep every accelerator busy without blowing the latency budget.

Last updated 2026-09-22. Difficulty: hard. Patterns: gpu-batching, streaming, scheduling, kv-cache. Reported at Anthropic and 5 more with Pro.

Functional requirements

• Completion API with streaming. A caller sends a prompt and receives tokens as they are produced, not one response at the end. Streaming is the product: time to first token is what users perceive as speed.
• Synchronous request, batched execution. Callers wait on their own request while the server groups many requests onto one GPU pass. The mapping from a batch slot back to the right caller is the core plumbing of this design.
• Multiple models and versions. Several models, each pinned to its own replicas, plus versioned rollouts. A request names a model; routing must never send it to a replica holding different weights.
• Per-tenant quotas and priorities. Tokens per minute and concurrent requests per tenant, with at least two priority classes (interactive and batch) so a bulk job cannot starve a chat session.
• Cancellation. A caller that disconnects must stop being generated for. Decoding a cancelled request is pure waste of the scarcest resource in the system, so cancellation has to reach the engine within a step or two.
• Deterministic request limits. Maximum prompt length, maximum output tokens, and a hard deadline per request, all enforced before the request reaches a GPU.
• Safety filtering on input and output. Prompts are screened before admission and streamed output is screened as it is produced, which means the filter runs on partial text and can stop a stream mid-flight.
• Usage metering. Prompt and output tokens counted per request for billing and quota enforcement, emitted even when the request fails midway.
• Batch (offline) mode. A cheaper asynchronous tier with a long deadline, used to fill the GPUs during troughs. It shares the fleet with interactive traffic and yields to it.
• Out of scope. Training and fine-tuning, the model architecture itself, embeddings and vector search, and the tool-use or agent loop layered on top.

Non-functional requirements

• Time to first token (p95 < 500 ms). This is prefill plus queueing. It is the number users feel, and it is the one that queueing destroys first under load.
• Inter-token latency (p95 < 40 ms). Roughly 25 tokens a second, faster than reading speed. It is set by how many sequences share the decode step, which is exactly the knob that also sets throughput.
• Throughput (80 k output tokens/s). The real capacity unit is tokens per second, not requests per second. Every capacity decision in the design is denominated in tokens.
• GPU utilisation (> 70 % of peak achievable). Accelerators dominate the cost, so idle GPU time is the main efficiency metric. This requirement is what forces continuous batching rather than simple request-per-GPU serving.
• Availability (99.9 % for the API). Degradation is a queued or slower response, or a smaller model, rather than an error. Hard failures are reserved for overload that shedding cannot absorb.
• Fairness and isolation (no tenant above its share under contention). One customer submitting ten thousand long generations must not add a second to everyone else's first token.
• Cost per million tokens (tracked and budgeted). Unusual for a system design, but here the design decisions (batch size, quantisation, prefix caching, spot capacity) are directly cost decisions, so the number belongs in the requirements.
• Correctness under retry (no duplicate billing, no duplicate side effects). A retried request must not be metered twice, and a partially streamed response must be resumable or cleanly restarted.

Back-of-envelope estimates

• Peak requests per second: ~280/s. 1 M requests/hour at peak ÷ 3 600 s = ~280/s. Modest as request rates go: the load is in the tokens each request generates, not in the request count.
• Output tokens per second: ~84 k/s. 280 req/s × 300 output tokens per request = 84 k tokens/s. This is the number that sizes the decode fleet; requests per second is almost irrelevant by comparison.
• Prompt tokens per second: ~280 k/s. 280 req/s × 1 000 prompt tokens = 280 k tokens/s to prefill. Prefill processes a whole prompt in one pass, so it is compute-bound and cheap per token; decode is one token at a time and memory-bandwidth-bound.
• GPUs for decode: ~42. A replica of a 70 B model sustains roughly 2 000 output tokens/s aggregated across its batch. 84 k ÷ 2 000 = 42 replicas worth of decode capacity. Batch size is what makes that number possible: one sequence alone would get a few dozen tokens a second from the same hardware.
• GPUs for prefill: ~28. Prefill runs at roughly 10 k prompt tokens/s per replica because it processes the prompt in parallel. 280 k ÷ 10 k = 28 replicas. Total fleet ~70, round to 80 for headroom and failure domains.
• KV cache per token: ~320 KB. 2 (keys and values) × 80 layers × 8 grouped KV heads × 128 head dimension × 2 bytes (fp16) = 327 680 B ≈ 320 KB per token for the whole model. Split across 4 tensor-parallel GPUs that is 80 KB per token per GPU.
• Concurrent sequences per replica: ~500. An 80 GB GPU holding 17.5 GB of weights (70 B parameters in fp8 across 4 GPUs) leaves ~55 GB for KV. At 80 KB per token per GPU that is ~700 k tokens, and at 1 300 tokens per sequence, ~500 concurrent sequences. Memory, not compute, sets the concurrency limit.
• Cold start for a replica: ~2–4 min. Loading 70 GB of weights from object storage at 2 GB/s is 35 s, plus shard initialisation, CUDA graph capture and warm-up passes, so 2–4 minutes before the replica can take traffic. Far too slow to react to a traffic spike, which is why a warm pool exists.
• Cost per million output tokens: ~$6. A GPU hour is roughly $4; a replica of 4 GPUs is $16/h and produces 2 000 tok/s = 7.2 M tokens/h. That is ~$2.2 per M tokens at full utilisation, but real utilisation is nearer 40 % once queues, traffic troughs and prefill are included, so ~$6 per M. The gap between those two numbers is the entire economic argument for batching.

Components

• Client app / SDK (streams tokens): Holds one long-lived HTTP response per request and renders tokens as they arrive. Carries an idempotency key so a retry is recognisable, and closes the connection on cancellation, which is the signal the engine uses to stop generating.
• API gateway (auth · streaming · quotas): Terminates TLS and the streaming response, authenticates the API key, and resolves the tenant. Holds the connection open for the whole generation, which makes connection count rather than request rate the thing that sizes it. Propagates client disconnects inward as cancellations.
• Admission control (quotas · deadlines · shedding): The gate in front of the scarce resource. Checks the tenant token bucket, the request size limits and the current queue depth, then admits, queues or rejects with a retry hint. Rejecting early is cheap; discovering the overload after a prompt has been prefilled is expensive.
• Safety filters (input + streaming output): Screens the prompt before admission and the generated text as it streams. Output screening works on a sliding window of partial text and can terminate a stream mid-generation, which is why the streaming path has to support a terminal error event after tokens have already been sent.
• Request queue (per model · priority tiers): One queue per model and priority class, holding admitted requests waiting for a slot. Depth is the load signal for both shedding and autoscaling. Interactive requests jump batch-tier ones, and anything past its deadline is dropped before it reaches a GPU rather than after.
• Scheduler / router (capacity + prefix affinity): Picks the replica for each request using live KV-memory headroom and current batch occupancy rather than round robin, and prefers a replica that already holds the prompt prefix in cache. Tracks per-replica capacity in tokens, not in requests, because that is the unit that actually runs out.
• Prefill pool (compute-bound · whole prompt): Runs the prompt through the model in one parallel pass and produces the KV cache for the sequence plus the first token. Compute-bound, so it batches by total token count rather than by sequence count, and a long prompt here is what delays everyone else's first token if it is not chunked.
• Decode pool (continuous batching loop): The steady-state loop: every step produces one token for every sequence in the batch. Memory-bandwidth-bound, so a larger batch is nearly free in time per step and is the reason throughput and batch size move together. Sequences join and leave the batch between steps rather than at batch boundaries.
• KV cache (paged blocks · prefix reuse): GPU memory holding the attention state of every live sequence, allocated in fixed-size blocks so memory is not reserved for the maximum possible length. Shared prefixes (system prompts, few-shot examples, conversation history) point at the same blocks, which is the single biggest saving available in a chat workload.
• Weight store (object storage · versioned): Model weights by version, pulled once per replica at start-up and cached on local NVMe so a restart does not re-download 70 GB. Distribution to a large fleet is itself a bandwidth problem, solved with peer-to-peer fan-out or a regional mirror rather than every worker reading from the same bucket.
• Model registry (versions · routing rules): Which model versions exist, which replicas serve them, and how traffic is split during a rollout. Small, strongly consistent and read on the routing path through a cached view, because sending a request to a replica with the wrong weights is a silent correctness bug.
• Autoscaler (warm pool · queue depth): Scales replicas from queue depth and time to first token rather than from GPU utilisation, which is misleading under batching. Because a cold start is minutes, it keeps a warm pool of loaded replicas and treats scaling down as the risky direction.
• Metering & billing (tokens per request): Counts prompt and output tokens per request and emits a usage event even when the request fails or is cancelled midway, since the compute was spent either way. Feeds the quota buckets that admission control reads, so a tenant's spend and its rate limit come from the same numbers.
• Telemetry stream (per-request records): One record per request with queue time, prefill time, token count, inter-token latency and the replica that served it. This is what makes the system debuggable: without per-stage timings, "the API is slow" cannot be attributed to queueing, prefill or decode.
• Dashboards & alerts (SLO views): Time to first token and inter-token latency by model and tenant, queue depth, KV utilisation, preemption rate and cost per million tokens. The alerting metric is client-observed latency; GPU utilisation is a diagnostic, never the SLO.

User flows

1. One streaming completion, end to end. The path every request takes. The interesting part is that the response begins before the work is finished, so the connection, the batch slot and the caller stay bound together for the whole generation.
   1. Client posts a prompt and asks for a streamed response. One HTTP request that will stay open for the length of the generation, typically several seconds. The gateway is therefore sized by concurrent streams, not by requests per second.
   2. Gateway authenticates the key, resolves the tenant, and screens the prompt. Input screening happens before anything expensive: a rejected prompt should never consume a GPU second. The check is a small model or classifier with a budget of a few milliseconds.
   3. Admission control checks the tenant quota, the size limits and the current queue depth. Three decisions in one place: is this tenant within its tokens-per-minute bucket, is the request within the size and deadline limits, and is the queue short enough that this request can finish in time. Anything else is rejected now with a retry hint rather than queued to die later.
   4. The request is queued for its model and priority class, then picked up by the scheduler. The scheduler resolves the model name to the set of replicas serving that exact version, then picks one by free KV memory and batch occupancy, preferring a replica that already holds this prompt's prefix.
   5. The chosen replica prefills the prompt, writing the attention state into the KV cache. One parallel pass over 1 000 prompt tokens, a few hundred milliseconds, producing the KV blocks and the first output token. This is the bulk of time to first token; everything before it is queueing.
   6. The sequence joins the running decode batch, which emits one token per step for every member. It does not wait for a new batch to form: continuous batching admits it at the next step boundary. Each step reads the whole KV cache for the batch and appends one token per sequence, which is why the step time is set by memory bandwidth rather than by how many sequences are in it.
   7. Tokens stream back through the gateway as they are produced, screened on the way out. Each token, or small group of tokens, is written to the open response. Output screening runs on a sliding window and can end the stream with a terminal event after partial text has already been delivered, which the client contract has to allow for.
   8. On the last token the sequence leaves the batch, its KV blocks are freed, and usage is metered. Freeing the blocks immediately is what lets the next queued request in; a slot held after completion is capacity lost. The usage event carries both token counts and is emitted even if the stream ended early.
2. How a batch forms, changes and drains. The mechanism the whole design exists to support. The batch is not a fixed group of requests; it is a set that changes at every step, which is what keeps GPUs busy without making anyone wait for a batch window.
   1. The engine runs a step loop: every iteration produces exactly one token for every sequence in the batch. Step time is dominated by streaming the model weights and the KV cache through memory, so a step with 8 sequences and a step with 200 take a similar amount of time. That single fact is why batching multiplies throughput almost for free.
   2. Between steps, finished sequences are evicted and their KV blocks returned to the pool. Sequences finish at wildly different times: one stops after 20 tokens, another runs to 2 000. Static batching would hold the whole batch until the longest finished, wasting most of the slots; continuous batching reclaims each slot the moment it is free.
   3. The scheduler admits waiting requests into the free slots, subject to KV memory rather than a slot count. Admission is a memory calculation: will this sequence's projected KV growth fit alongside the current batch until it finishes. A count-based limit either wastes memory on short prompts or over-commits on long ones.
   4. Prefill for newly admitted sequences is interleaved with decode steps, in chunks. A 30 k-token prompt prefilled in one pass would stall decoding for hundreds of milliseconds and show up as a latency spike for every other user in the batch. Chunked prefill splits it across several steps so ongoing generations keep flowing.
   5. When KV memory runs short, the engine preempts a sequence rather than failing it. The victim's blocks are freed and it is either swapped to host memory or recomputed later from its prompt. Preemption is a latency event for that one request instead of an error for everyone, and its rate is a metric worth alerting on.
   6. The batch size adapts continuously to the latency target. Bigger batches raise throughput and raise inter-token latency for everyone in them. The engine holds a target inter-token latency and trims the batch when it is breached, which is the concrete expression of the throughput-versus-latency tradeoff.
3. Traffic doubles and the GPUs are already full. GPU capacity cannot be conjured in seconds, so overload is handled by deciding who waits, who is refused and what gets degraded, before the queue turns into a graveyard of timed-out requests.
   1. Queue depth and time to first token rise together; the queue is the first signal. GPU utilisation looks the same at healthy load and at overload, because the engine keeps the GPUs busy either way. Queue depth and time to first token are the honest signals, and they are what the alerts and the autoscaler use.
   2. Admission control tightens: batch-tier requests are queued longer or refused first. The cheapest capacity in the system is work that does not have to happen now. Offline and low-priority traffic yields entirely before any interactive request is touched, which is the point of having a priority class at all.
   3. Tenants above their fair share are throttled before tenants below it. Weighted fair queueing by tenant, measured in tokens rather than requests, so one customer sending very long generations cannot consume the fleet while staying inside a request-count limit.
   4. Requests whose deadline cannot be met are rejected at the head of the queue, not served late. A request that has waited past its deadline is worthless to the caller but still costs a full generation to serve. Dropping it frees capacity for requests that can still be useful, and the caller has usually retried already.
   5. The autoscaler promotes warm-pool replicas immediately and starts cold ones behind them. Warm replicas take traffic in seconds; cold ones are minutes away, so the warm pool is sized from how fast traffic can plausibly double, not from average load.
   6. If the spike persists, the service degrades rather than fails: shorter outputs, a smaller model, or queued batch mode. Capping max_tokens, routing overflow to a smaller and cheaper model, or accepting the request into the asynchronous tier all keep the product working. Naming the degradation explicitly is what separates a design from a wish.
4. A GPU dies with thirty sequences in flight. Everything in the batch shares the failure, and the KV state that took a second of compute to build is gone. The system has to decide what is retryable and what the caller sees mid-stream.
   1. A worker in the tensor-parallel group stops responding and the replica fails its health check. Because the model is sharded across several GPUs, losing one loses the whole replica. The failure domain is the replica, not the GPU, which is why capacity planning counts replicas.
   2. The scheduler marks the replica unhealthy and stops placing new requests on it. Fast to do and cheap to get wrong in the safe direction: a replica wrongly marked unhealthy costs capacity, while one wrongly kept in rotation costs every request placed on it.
   3. Sequences that had not produced a token are re-queued and retried on another replica. Nothing has been delivered, so a retry is invisible to the caller apart from added latency. The idempotency key prevents the retry from being metered as a second request.
   4. Sequences already streaming cannot be silently retried, because partial text has been delivered. Restarting would produce different text after the same prefix. The honest options are an error event on the stream, or resuming generation by prefilling the prompt plus the tokens already sent, which costs a full prefill and only works if the client can tolerate a pause mid-stream.
   5. Usage is metered for the tokens actually produced, and the failure is attributed in telemetry. The compute was spent, but charging in full for a broken response is a support ticket. The usual policy is to meter but not bill interrupted generations, and to track the interrupted rate as an SLO of its own.
   6. The autoscaler replaces the replica from the warm pool while the dead one reloads weights. Weights come from local NVMe if the host survived, which turns a four-minute cold start into under a minute. Only a lost host pays the full download.
5. Ship a new model version. Two models, one fleet, no capacity to run both at full size. The rollout is a scheduling problem as much as a deployment one.
   1. The new version is published to the registry and its weights staged to the weight store. Versions are immutable and addressed by content hash, so a replica can verify that what it loaded is what routing thinks it is serving.
   2. A small number of replicas load the new weights and warm up out of rotation. Distributing 70 GB to many workers at once saturates the network, so the fan-out is peer-to-peer or through a regional mirror rather than every worker pulling from the same bucket. Warm-up includes a few synthetic generations so the first real request does not pay for graph capture.
   3. Routing sends a small share of traffic to the new version, keyed so a conversation stays on one version. Version pinning per conversation matters: switching versions halfway through a session changes the voice of the responses, which users notice more than latency.
   4. Quality and latency are compared between versions on live traffic. Not only latency and error rate: output length distribution, refusal rate, truncation rate and cost per million tokens all shift with a model change, and a version that is 10 % more verbose is 10 % more expensive to serve.
   5. The share is increased in steps, with drain rather than kill on the old replicas. A replica taken out of rotation keeps generating for the sequences it already holds, up to a drain timeout. Killing it mid-batch would break every stream on it for no reason.
   6. If a regression shows up, routing flips back in one registry write. The rollback is a traffic split change, not a redeploy, because the old replicas are still loaded and warm. That is why the old version is drained slowly rather than reclaimed immediately.

Deep dives

Related

• Design a Matchmaking System
• Design Uber
• Design a Web Crawler
• Design a Distributed Job Scheduler
• Run this question for a friend: the interviewer kit