Requests arrive one at a time and the GPU wants many at once. How do you group them?

Continuous batching with chunked prefill, admission by projected KV memory rather than slot count, and an adaptive batch size bounded by a target inter-token latency.

GPU memory holds the model and the attention state of every live sequence. How is it managed?

Paged KV cache in fixed blocks with copy-on-write sharing, a tenant-scoped prefix cache in front of it, and host-memory offload used only for preempted or idle sequences.

The two phases of generation have opposite performance characteristics. Should they share hardware?

Colocated prefill and decode with chunked prefill by default, plus dedicated replica pools for long-prompt traffic, keeping true disaggregation as the next step if the interconnect makes KV transfer cheap.

How do partial results get back to the client, and what happens when the connection breaks?

Server-sent events for the interactive API, with client disconnect propagated to the engine as a cancellation, and an asynchronous job API for the batch tier.

Capacity is fixed and demand is not. Who waits, who is refused, and how is that decided?

Token-denominated quotas per tenant at the edge, priority classes with deadline dropping in the queue, and weighted fair queueing between tenants inside each class, with queue depth bounded so that refusal beats unbounded waiting.

GPUs are expensive and slow to start. How do you match capacity to demand without wasting either?

Autoscale on queue depth and time to first token, keep a warm pool sized to the plausible doubling rate, fill troughs with batch-tier work on spot capacity, and treat scale-down as the risky direction with a long cooldown.

System design interview question

Design an LLM Inference Service

Serve a large language model behind an API: queue the requests, batch them onto scarce GPUs, stream tokens back, and keep every accelerator busy without blowing the latency budget.

Last updated 2026-09-22. Difficulty: hard. Patterns: gpu-batching, streaming, scheduling, kv-cache. Reported at Anthropic, OpenAI, Google, Meta, NVIDIA, Amazon.

Sit this as an AI interview and be asked it one question at a time; Study shows every answer, and Practice hides them until you have produced your own.

Functional requirements

Completion API with streaming. A caller sends a prompt and receives tokens as they are produced, not one response at the end. Streaming is the product: time to first token is what users perceive as speed.
Synchronous request, batched execution. Callers wait on their own request while the server groups many requests onto one GPU pass. The mapping from a batch slot back to the right caller is the core plumbing of this design.
Multiple models and versions. Several models, each pinned to its own replicas, plus versioned rollouts. A request names a model; routing must never send it to a replica holding different weights.
Per-tenant quotas and priorities. Tokens per minute and concurrent requests per tenant, with at least two priority classes (interactive and batch) so a bulk job cannot starve a chat session.
Cancellation. A caller that disconnects must stop being generated for. Decoding a cancelled request is pure waste of the scarcest resource in the system, so cancellation has to reach the engine within a step or two.
Deterministic request limits. Maximum prompt length, maximum output tokens, and a hard deadline per request, all enforced before the request reaches a GPU.
Safety filtering on input and output. Prompts are screened before admission and streamed output is screened as it is produced, which means the filter runs on partial text and can stop a stream mid-flight.
Usage metering. Prompt and output tokens counted per request for billing and quota enforcement, emitted even when the request fails midway.
Batch (offline) mode. A cheaper asynchronous tier with a long deadline, used to fill the GPUs during troughs. It shares the fleet with interactive traffic and yields to it.
Out of scope. Training and fine-tuning, the model architecture itself, embeddings and vector search, and the tool-use or agent loop layered on top.

Non-functional requirements

Time to first token (p95 < 500 ms). This is prefill plus queueing. It is the number users feel, and it is the one that queueing destroys first under load.
Inter-token latency (p95 < 40 ms). Roughly 25 tokens a second, faster than reading speed. It is set by how many sequences share the decode step, which is exactly the knob that also sets throughput.
Throughput (80 k output tokens/s). The real capacity unit is tokens per second, not requests per second. Every capacity decision in the design is denominated in tokens.
GPU utilisation (> 70 % of peak achievable). Accelerators dominate the cost, so idle GPU time is the main efficiency metric. This requirement is what forces continuous batching rather than simple request-per-GPU serving.
Availability (99.9 % for the API). Degradation is a queued or slower response, or a smaller model, rather than an error. Hard failures are reserved for overload that shedding cannot absorb.
Fairness and isolation (no tenant above its share under contention). One customer submitting ten thousand long generations must not add a second to everyone else's first token.
Cost per million tokens (tracked and budgeted). Unusual for a system design, but here the design decisions (batch size, quantisation, prefix caching, spot capacity) are directly cost decisions, so the number belongs in the requirements.
Correctness under retry (no duplicate billing, no duplicate side effects). A retried request must not be metered twice, and a partially streamed response must be resumable or cleanly restarted.

Back-of-envelope estimates

Peak requests per second: ~280/s. 1 M requests/hour at peak ÷ 3 600 s = ~280/s. Modest as request rates go: the load is in the tokens each request generates, not in the request count.
Output tokens per second: ~84 k/s. 280 req/s × 300 output tokens per request = 84 k tokens/s. This is the number that sizes the decode fleet; requests per second is almost irrelevant by comparison.
Prompt tokens per second: ~280 k/s. 280 req/s × 1 000 prompt tokens = 280 k tokens/s to prefill. Prefill processes a whole prompt in one pass, so it is compute-bound and cheap per token; decode is one token at a time and memory-bandwidth-bound.
GPUs for decode: ~42. A replica of a 70 B model sustains roughly 2 000 output tokens/s aggregated across its batch. 84 k ÷ 2 000 = 42 replicas worth of decode capacity. Batch size is what makes that number possible: one sequence alone would get a few dozen tokens a second from the same hardware.
GPUs for prefill: ~28. Prefill runs at roughly 10 k prompt tokens/s per replica because it processes the prompt in parallel. 280 k ÷ 10 k = 28 replicas. Total fleet ~70, round to 80 for headroom and failure domains.
KV cache per token: ~320 KB. 2 (keys and values) × 80 layers × 8 grouped KV heads × 128 head dimension × 2 bytes (fp16) = 327 680 B ≈ 320 KB per token for the whole model. Split across 4 tensor-parallel GPUs that is 80 KB per token per GPU.
Concurrent sequences per replica: ~500. An 80 GB GPU holding 17.5 GB of weights (70 B parameters in fp8 across 4 GPUs) leaves ~55 GB for KV. At 80 KB per token per GPU that is ~700 k tokens, and at 1 300 tokens per sequence, ~500 concurrent sequences. Memory, not compute, sets the concurrency limit.
Cold start for a replica: ~2–4 min. Loading 70 GB of weights from object storage at 2 GB/s is 35 s, plus shard initialisation, CUDA graph capture and warm-up passes, so 2–4 minutes before the replica can take traffic. Far too slow to react to a traffic spike, which is why a warm pool exists.
Cost per million output tokens: ~$6. A GPU hour is roughly $4; a replica of 4 GPUs is $16/h and produces 2 000 tok/s = 7.2 M tokens/h. That is ~$2.2 per M tokens at full utilisation, but real utilisation is nearer 40 % once queues, traffic troughs and prefill are included, so ~$6 per M. The gap between those two numbers is the entire economic argument for batching.

Components

Client app / SDK (streams tokens): Holds one long-lived HTTP response per request and renders tokens as they arrive. Carries an idempotency key so a retry is recognisable, and closes the connection on cancellation, which is the signal the engine uses to stop generating.
API gateway (auth · streaming · quotas): Terminates TLS and the streaming response, authenticates the API key, and resolves the tenant. Holds the connection open for the whole generation, which makes connection count rather than request rate the thing that sizes it. Propagates client disconnects inward as cancellations.
Admission control (quotas · deadlines · shedding): The gate in front of the scarce resource. Checks the tenant token bucket, the request size limits and the current queue depth, then admits, queues or rejects with a retry hint. Rejecting early is cheap; discovering the overload after a prompt has been prefilled is expensive.
Safety filters (input + streaming output): Screens the prompt before admission and the generated text as it streams. Output screening works on a sliding window of partial text and can terminate a stream mid-generation, which is why the streaming path has to support a terminal error event after tokens have already been sent.
Request queue (per model · priority tiers): One queue per model and priority class, holding admitted requests waiting for a slot. Depth is the load signal for both shedding and autoscaling. Interactive requests jump batch-tier ones, and anything past its deadline is dropped before it reaches a GPU rather than after.
Scheduler / router (capacity + prefix affinity): Picks the replica for each request using live KV-memory headroom and current batch occupancy rather than round robin, and prefers a replica that already holds the prompt prefix in cache. Tracks per-replica capacity in tokens, not in requests, because that is the unit that actually runs out.
Prefill pool (compute-bound · whole prompt): Runs the prompt through the model in one parallel pass and produces the KV cache for the sequence plus the first token. Compute-bound, so it batches by total token count rather than by sequence count, and a long prompt here is what delays everyone else's first token if it is not chunked.
Decode pool (continuous batching loop): The steady-state loop: every step produces one token for every sequence in the batch. Memory-bandwidth-bound, so a larger batch is nearly free in time per step and is the reason throughput and batch size move together. Sequences join and leave the batch between steps rather than at batch boundaries.
KV cache (paged blocks · prefix reuse): GPU memory holding the attention state of every live sequence, allocated in fixed-size blocks so memory is not reserved for the maximum possible length. Shared prefixes (system prompts, few-shot examples, conversation history) point at the same blocks, which is the single biggest saving available in a chat workload.
Weight store (object storage · versioned): Model weights by version, pulled once per replica at start-up and cached on local NVMe so a restart does not re-download 70 GB. Distribution to a large fleet is itself a bandwidth problem, solved with peer-to-peer fan-out or a regional mirror rather than every worker reading from the same bucket.
Model registry (versions · routing rules): Which model versions exist, which replicas serve them, and how traffic is split during a rollout. Small, strongly consistent and read on the routing path through a cached view, because sending a request to a replica with the wrong weights is a silent correctness bug.
Autoscaler (warm pool · queue depth): Scales replicas from queue depth and time to first token rather than from GPU utilisation, which is misleading under batching. Because a cold start is minutes, it keeps a warm pool of loaded replicas and treats scaling down as the risky direction.
Metering & billing (tokens per request): Counts prompt and output tokens per request and emits a usage event even when the request fails or is cancelled midway, since the compute was spent either way. Feeds the quota buckets that admission control reads, so a tenant's spend and its rate limit come from the same numbers.
Telemetry stream (per-request records): One record per request with queue time, prefill time, token count, inter-token latency and the replica that served it. This is what makes the system debuggable: without per-stage timings, "the API is slow" cannot be attributed to queueing, prefill or decode.
Dashboards & alerts (SLO views): Time to first token and inter-token latency by model and tenant, queue depth, KV utilisation, preemption rate and cost per million tokens. The alerting metric is client-observed latency; GPU utilisation is a diagnostic, never the SLO.

User flows

One streaming completion, end to end. The path every request takes. The interesting part is that the response begins before the work is finished, so the connection, the batch slot and the caller stay bound together for the whole generation.
1. Client posts a prompt and asks for a streamed response.
2. Gateway authenticates the key, resolves the tenant, and screens the prompt.
3. Admission control checks the tenant quota, the size limits and the current queue depth.
4. The request is queued for its model and priority class, then picked up by the scheduler.
5. The chosen replica prefills the prompt, writing the attention state into the KV cache.
6. The sequence joins the running decode batch, which emits one token per step for every member.
7. Tokens stream back through the gateway as they are produced, screened on the way out.
8. On the last token the sequence leaves the batch, its KV blocks are freed, and usage is metered.
How a batch forms, changes and drains. The mechanism the whole design exists to support. The batch is not a fixed group of requests; it is a set that changes at every step, which is what keeps GPUs busy without making anyone wait for a batch window.
1. The engine runs a step loop: every iteration produces exactly one token for every sequence in the batch.
2. Between steps, finished sequences are evicted and their KV blocks returned to the pool.
3. The scheduler admits waiting requests into the free slots, subject to KV memory rather than a slot count.
4. Prefill for newly admitted sequences is interleaved with decode steps, in chunks.
5. When KV memory runs short, the engine preempts a sequence rather than failing it.
6. The batch size adapts continuously to the latency target.
Traffic doubles and the GPUs are already full. GPU capacity cannot be conjured in seconds, so overload is handled by deciding who waits, who is refused and what gets degraded — before the queue turns into a graveyard of timed-out requests.
1. Queue depth and time to first token rise together; the queue is the first signal.
2. Admission control tightens: batch-tier requests are queued longer or refused first.
3. Tenants above their fair share are throttled before tenants below it.
4. Requests whose deadline cannot be met are rejected at the head of the queue, not served late.
5. The autoscaler promotes warm-pool replicas immediately and starts cold ones behind them.
6. If the spike persists, the service degrades rather than fails: shorter outputs, a smaller model, or queued batch mode.
A GPU dies with thirty sequences in flight. Everything in the batch shares the failure, and the KV state that took a second of compute to build is gone. The system has to decide what is retryable and what the caller sees mid-stream.
1. A worker in the tensor-parallel group stops responding and the replica fails its health check.
2. The scheduler marks the replica unhealthy and stops placing new requests on it.
3. Sequences that had not produced a token are re-queued and retried on another replica.
4. Sequences already streaming cannot be silently retried, because partial text has been delivered.
5. Usage is metered for the tokens actually produced, and the failure is attributed in telemetry.
6. The autoscaler replaces the replica from the warm pool while the dead one reloads weights.
Ship a new model version. Two models, one fleet, no capacity to run both at full size. The rollout is a scheduling problem as much as a deployment one.
1. The new version is published to the registry and its weights staged to the weight store.
2. A small number of replicas load the new weights and warm up out of rotation.
3. Routing sends a small share of traffic to the new version, keyed so a conversation stays on one version.
4. Quality and latency are compared between versions on live traffic.
5. The share is increased in steps, with drain rather than kill on the old replicas.
6. If a regression shows up, routing flips back in one registry write.

Deep dives

How requests are batched onto a GPU. Requests arrive one at a time and the GPU wants many at once. How do you group them? Continuous batching with chunked prefill, admission by projected KV memory rather than slot count, and an adaptive batch size bounded by a target inter-token latency.
KV cache: the resource that actually runs out. GPU memory holds the model and the attention state of every live sequence. How is it managed? Paged KV cache in fixed blocks with copy-on-write sharing, a tenant-scoped prefix cache in front of it, and host-memory offload used only for preempted or idle sequences.
Prefill and decode: same fleet or separate ones. The two phases of generation have opposite performance characteristics. Should they share hardware? Colocated prefill and decode with chunked prefill by default, plus dedicated replica pools for long-prompt traffic, keeping true disaggregation as the next step if the interconnect makes KV transfer cheap.
Streaming tokens to the caller. How do partial results get back to the client, and what happens when the connection breaks? Server-sent events for the interactive API, with client disconnect propagated to the engine as a cancellation, and an asynchronous job API for the batch tier.
Admission control, queueing and fairness. Capacity is fixed and demand is not. Who waits, who is refused, and how is that decided? Token-denominated quotas per tenant at the edge, priority classes with deadline dropping in the queue, and weighted fair queueing between tenants inside each class, with queue depth bounded so that refusal beats unbounded waiting.
Autoscaling, warm pools and cost. GPUs are expensive and slow to start. How do you match capacity to demand without wasting either? Autoscale on queue depth and time to first token, keep a warm pool sized to the plausible doubling rate, fill troughs with batch-tier work on spot capacity, and treat scale-down as the risky direction with a long cooldown.