Design an LLM Inference Service

• Completion API with streaming — A caller sends a prompt and receives tokens as they are produced, not one response at the end. Streaming is the product: time to first token is what users perceive as speed.
• Synchronous request, batched execution — Callers wait on their own request while the server groups many requests onto one GPU pass. The mapping from a batch slot back to the right caller is the core plumbing of this design.
• Multiple models and versions — Several models, each pinned to its own replicas, plus versioned rollouts. A request names a model; routing must never send it to a replica holding different weights.
• Per-tenant quotas and priorities — Tokens per minute and concurrent requests per tenant, with at least two priority classes (interactive and batch) so a bulk job cannot starve a chat session.
• Cancellation — A caller that disconnects must stop being generated for. Decoding a cancelled request is pure waste of the scarcest resource in the system, so cancellation has to reach the engine within a step or two.
• Deterministic request limits — Maximum prompt length, maximum output tokens, and a hard deadline per request, all enforced before the request reaches a GPU.
• Safety filtering on input and output — Prompts are screened before admission and streamed output is screened as it is produced, which means the filter runs on partial text and can stop a stream mid-flight.
• Usage metering — Prompt and output tokens counted per request for billing and quota enforcement, emitted even when the request fails midway.
• Batch (offline) mode — A cheaper asynchronous tier with a long deadline, used to fill the GPUs during troughs. It shares the fleet with interactive traffic and yields to it.
• Out of scope — Training and fine-tuning, the model architecture itself, embeddings and vector search, and the tool-use or agent loop layered on top.

• Time to first token (p95 < 500 ms) — This is prefill plus queueing. It is the number users feel, and it is the one that queueing destroys first under load.
• Inter-token latency (p95 < 40 ms) — Roughly 25 tokens a second, faster than reading speed. It is set by how many sequences share the decode step, which is exactly the knob that also sets throughput.
• Throughput (80 k output tokens/s) — The real capacity unit is tokens per second, not requests per second. Every capacity decision in the design is denominated in tokens.
• GPU utilisation (> 70 % of peak achievable) — Accelerators dominate the cost, so idle GPU time is the main efficiency metric. This requirement is what forces continuous batching rather than simple request-per-GPU serving.
• Availability (99.9 % for the API) — Degradation is a queued or slower response, or a smaller model, rather than an error. Hard failures are reserved for overload that shedding cannot absorb.
• Fairness and isolation (no tenant above its share under contention) — One customer submitting ten thousand long generations must not add a second to everyone else's first token.
• Cost per million tokens (tracked and budgeted) — Unusual for a system design, but here the design decisions (batch size, quantisation, prefix caching, spot capacity) are directly cost decisions, so the number belongs in the requirements.
• Correctness under retry (no duplicate billing, no duplicate side effects) — A retried request must not be metered twice, and a partially streamed response must be resumable or cleanly restarted.

• Peak requests per second: ~280/s — 1 M requests/hour at peak ÷ 3 600 s = ~280/s. Modest as request rates go: the load is in the tokens each request generates, not in the request count.
• Output tokens per second: ~84 k/s — 280 req/s × 300 output tokens per request = 84 k tokens/s. This is the number that sizes the decode fleet; requests per second is almost irrelevant by comparison.
• Prompt tokens per second: ~280 k/s — 280 req/s × 1 000 prompt tokens = 280 k tokens/s to prefill. Prefill processes a whole prompt in one pass, so it is compute-bound and cheap per token; decode is one token at a time and memory-bandwidth-bound.
• GPUs for decode: ~42 — A replica of a 70 B model sustains roughly 2 000 output tokens/s aggregated across its batch. 84 k ÷ 2 000 = 42 replicas worth of decode capacity. Batch size is what makes that number possible: one sequence alone would get a few dozen tokens a second from the same hardware.
• GPUs for prefill: ~28 — Prefill runs at roughly 10 k prompt tokens/s per replica because it processes the prompt in parallel. 280 k ÷ 10 k = 28 replicas. Total fleet ~70, round to 80 for headroom and failure domains.
• KV cache per token: ~320 KB — 2 (keys and values) × 80 layers × 8 grouped KV heads × 128 head dimension × 2 bytes (fp16) = 327 680 B ≈ 320 KB per token for the whole model. Split across 4 tensor-parallel GPUs that is 80 KB per token per GPU.
• Concurrent sequences per replica: ~500 — An 80 GB GPU holding 17.5 GB of weights (70 B parameters in fp8 across 4 GPUs) leaves ~55 GB for KV. At 80 KB per token per GPU that is ~700 k tokens, and at 1 300 tokens per sequence, ~500 concurrent sequences. Memory, not compute, sets the concurrency limit.
• Cold start for a replica: ~2–4 min — Loading 70 GB of weights from object storage at 2 GB/s is 35 s, plus shard initialisation, CUDA graph capture and warm-up passes, so 2–4 minutes before the replica can take traffic. Far too slow to react to a traffic spike, which is why a warm pool exists.
• Cost per million output tokens: ~$6 — A GPU hour is roughly $4; a replica of 4 GPUs is $16/h and produces 2 000 tok/s = 7.2 M tokens/h. That is ~$2.2 per M tokens at full utilisation, but real utilisation is nearer 40 % once queues, traffic troughs and prefill are included, so ~$6 per M. The gap between those two numbers is the entire economic argument for batching.

• Client app / SDK (streams tokens) — Holds one long-lived HTTP response per request and renders tokens as they arrive. Carries an idempotency key so a retry is recognisable, and closes the connection on cancellation, which is the signal the engine uses to stop generating.
• API gateway (auth · streaming · quotas) — Terminates TLS and the streaming response, authenticates the API key, and resolves the tenant. Holds the connection open for the whole generation, which makes connection count rather than request rate the thing that sizes it. Propagates client disconnects inward as cancellations.
• Admission control (quotas · deadlines · shedding) — The gate in front of the scarce resource. Checks the tenant token bucket, the request size limits and the current queue depth, then admits, queues or rejects with a retry hint. Rejecting early is cheap; discovering the overload after a prompt has been prefilled is expensive.
• Safety filters (input + streaming output) — Screens the prompt before admission and the generated text as it streams. Output screening works on a sliding window of partial text and can terminate a stream mid-generation, which is why the streaming path has to support a terminal error event after tokens have already been sent.
• Request queue (per model · priority tiers) — One queue per model and priority class, holding admitted requests waiting for a slot. Depth is the load signal for both shedding and autoscaling. Interactive requests jump batch-tier ones, and anything past its deadline is dropped before it reaches a GPU rather than after.
• Scheduler / router (capacity + prefix affinity) — Picks the replica for each request using live KV-memory headroom and current batch occupancy rather than round robin, and prefers a replica that already holds the prompt prefix in cache. Tracks per-replica capacity in tokens, not in requests, because that is the unit that actually runs out.
• Prefill pool (compute-bound · whole prompt) — Runs the prompt through the model in one parallel pass and produces the KV cache for the sequence plus the first token. Compute-bound, so it batches by total token count rather than by sequence count, and a long prompt here is what delays everyone else's first token if it is not chunked.
• Decode pool (continuous batching loop) — The steady-state loop: every step produces one token for every sequence in the batch. Memory-bandwidth-bound, so a larger batch is nearly free in time per step and is the reason throughput and batch size move together. Sequences join and leave the batch between steps rather than at batch boundaries.
• KV cache (paged blocks · prefix reuse) — GPU memory holding the attention state of every live sequence, allocated in fixed-size blocks so memory is not reserved for the maximum possible length. Shared prefixes (system prompts, few-shot examples, conversation history) point at the same blocks, which is the single biggest saving available in a chat workload.
• Weight store (object storage · versioned) — Model weights by version, pulled once per replica at start-up and cached on local NVMe so a restart does not re-download 70 GB. Distribution to a large fleet is itself a bandwidth problem, solved with peer-to-peer fan-out or a regional mirror rather than every worker reading from the same bucket.
• Model registry (versions · routing rules) — Which model versions exist, which replicas serve them, and how traffic is split during a rollout. Small, strongly consistent and read on the routing path through a cached view, because sending a request to a replica with the wrong weights is a silent correctness bug.
• Autoscaler (warm pool · queue depth) — Scales replicas from queue depth and time to first token rather than from GPU utilisation, which is misleading under batching. Because a cold start is minutes, it keeps a warm pool of loaded replicas and treats scaling down as the risky direction.
• Metering & billing (tokens per request) — Counts prompt and output tokens per request and emits a usage event even when the request fails or is cancelled midway, since the compute was spent either way. Feeds the quota buckets that admission control reads, so a tenant's spend and its rate limit come from the same numbers.
• Telemetry stream (per-request records) — One record per request with queue time, prefill time, token count, inter-token latency and the replica that served it. This is what makes the system debuggable: without per-stage timings, "the API is slow" cannot be attributed to queueing, prefill or decode.
• Dashboards & alerts (SLO views) — Time to first token and inter-token latency by model and tenant, queue depth, KV utilisation, preemption rate and cost per million tokens. The alerting metric is client-observed latency; GPU utilisation is a diagnostic, never the SLO.

1. One streaming completion, end to end — The path every request takes. The interesting part is that the response begins before the work is finished, so the connection, the batch slot and the caller stay bound together for the whole generation.
   1. Client posts a prompt and asks for a streamed response.
   2. Gateway authenticates the key, resolves the tenant, and screens the prompt.
   3. Admission control checks the tenant quota, the size limits and the current queue depth.
   4. The request is queued for its model and priority class, then picked up by the scheduler.
   5. The chosen replica prefills the prompt, writing the attention state into the KV cache.
   6. The sequence joins the running decode batch, which emits one token per step for every member.
   7. Tokens stream back through the gateway as they are produced, screened on the way out.
   8. On the last token the sequence leaves the batch, its KV blocks are freed, and usage is metered.
2. How a batch forms, changes and drains — The mechanism the whole design exists to support. The batch is not a fixed group of requests; it is a set that changes at every step, which is what keeps GPUs busy without making anyone wait for a batch window.
   1. The engine runs a step loop: every iteration produces exactly one token for every sequence in the batch.
   2. Between steps, finished sequences are evicted and their KV blocks returned to the pool.
   3. The scheduler admits waiting requests into the free slots, subject to KV memory rather than a slot count.
   4. Prefill for newly admitted sequences is interleaved with decode steps, in chunks.
   5. When KV memory runs short, the engine preempts a sequence rather than failing it.
   6. The batch size adapts continuously to the latency target.
3. Traffic doubles and the GPUs are already full — GPU capacity cannot be conjured in seconds, so overload is handled by deciding who waits, who is refused and what gets degraded — before the queue turns into a graveyard of timed-out requests.
   1. Queue depth and time to first token rise together; the queue is the first signal.
   2. Admission control tightens: batch-tier requests are queued longer or refused first.
   3. Tenants above their fair share are throttled before tenants below it.
   4. Requests whose deadline cannot be met are rejected at the head of the queue, not served late.
   5. The autoscaler promotes warm-pool replicas immediately and starts cold ones behind them.
   6. If the spike persists, the service degrades rather than fails: shorter outputs, a smaller model, or queued batch mode.
4. A GPU dies with thirty sequences in flight — Everything in the batch shares the failure, and the KV state that took a second of compute to build is gone. The system has to decide what is retryable and what the caller sees mid-stream.
   1. A worker in the tensor-parallel group stops responding and the replica fails its health check.
   2. The scheduler marks the replica unhealthy and stops placing new requests on it.
   3. Sequences that had not produced a token are re-queued and retried on another replica.
   4. Sequences already streaming cannot be silently retried, because partial text has been delivered.
   5. Usage is metered for the tokens actually produced, and the failure is attributed in telemetry.
   6. The autoscaler replaces the replica from the warm pool while the dead one reloads weights.
5. Ship a new model version — Two models, one fleet, no capacity to run both at full size. The rollout is a scheduling problem as much as a deployment one.
   1. The new version is published to the registry and its weights staged to the weight store.
   2. A small number of replicas load the new weights and warm up out of rotation.
   3. Routing sends a small share of traffic to the new version, keyed so a conversation stays on one version.
   4. Quality and latency are compared between versions on live traffic.
   5. The share is increased in steps, with drain rather than kill on the old replicas.
   6. If a regression shows up, routing flips back in one registry write.

Ask: Requests arrive one at a time and the GPU wants many at once. How do you group them?

Good answers name: Continuous (in-flight) batching, Static batching (fixed groups), Dynamic batching with a time window, One request per GPU, no batching.

Our pick: Continuous batching with chunked prefill, admission by projected KV memory rather than slot count, and an adaptive batch size bounded by a target inter-token latency.

1. Why does a bigger batch barely cost time per step?
   Decode is memory-bandwidth-bound: each step streams the model weights from HBM once, regardless of batch size, and the per-sequence work is small by comparison. Doubling the batch roughly doubles tokens per second until you reach the point where compute or KV memory becomes the limit. Prefill behaves the opposite way — it is compute-bound — which is why the two phases are scheduled differently.
2. What is chunked prefill and why is it needed?
   Prefilling a long prompt in a single pass occupies the GPU for hundreds of milliseconds, during which no sequence in the decode batch produces a token, so every other user sees a stall. Chunked prefill splits the prompt into pieces processed across several steps interleaved with decode, trading a slightly slower first token for that one request against stable inter-token latency for everyone else.
3. How do you choose the maximum batch size?
   Not as a constant. The engine holds a target inter-token latency (say 40 ms at p95) and admits sequences while the measured step time stays under it and KV memory allows. That makes the tradeoff explicit and automatic: during quiet periods batches grow and cost per token falls, during busy periods they are trimmed to protect the latency the product promised.
4. Two tenants, one batch — is that a problem?
   For latency, yes: they share the step time, so a large batch penalises everyone in it equally. For isolation and privacy, sequences in a batch do not share state beyond shared prefix blocks, and prefix sharing must be scoped per tenant so one tenant cannot detect another's prompt through a cache-hit timing difference.

Ask: GPU memory holds the model and the attention state of every live sequence. How is it managed?

Good answers name: Paged KV cache in fixed-size blocks, Contiguous per-sequence reservation, Prefix caching across requests, Offload KV to host memory or NVMe.

Our pick: Paged KV cache in fixed blocks with copy-on-write sharing, a tenant-scoped prefix cache in front of it, and host-memory offload used only for preempted or idle sequences.

1. Work through the memory maths for me.
   Per token, the KV state is 2 × layers × kv_heads × head_dim × dtype_bytes. For 80 layers, 8 grouped KV heads, head dimension 128, fp16: 2 × 80 × 8 × 128 × 2 = 320 KB per token for the full model, or 80 KB per GPU across four-way tensor parallelism. With 55 GB free per GPU after weights, that is about 700 k tokens, so roughly 500 concurrent sequences at 1 300 tokens each. Grouped-query attention is doing enormous work in that number — with full multi-head attention it would be eight times larger.
2. What happens when the cache fills mid-generation?
   The engine preempts: pick a victim by policy (usually the newest or the lowest priority), free its blocks, and either swap them to host memory or drop them and recompute from the prompt later. Recompute is often cheaper than swapping because prefill is fast and PCIe is slow. The request is not failed; it resumes when memory frees, and the preemption rate is a monitored metric because a high rate means the fleet is under-provisioned.
3. How does prefix caching interact with routing?
   The router hashes the prompt prefix and prefers a replica that already holds those blocks, which turns a 300 ms prefill into tens of milliseconds. It is affinity, not a hard constraint: if that replica is out of memory or overloaded, a cache miss elsewhere is better than queueing. This is the one place where the routing decision and the memory design are the same decision.
4. Does quantisation help here?
   Two separate savings. Quantising the weights (fp8, int8, or 4-bit) frees GPU memory for more KV and raises throughput because less data streams per step, at some quality cost that must be evaluated per model. Quantising the KV cache itself to fp8 halves the per-token cost and directly doubles concurrency. Both are legitimate answers; both need a quality evaluation rather than an assertion.

Ask: The two phases of generation have opposite performance characteristics. Should they share hardware?

Good answers name: Colocated with chunked prefill, Disaggregated prefill and decode pools, Separate replicas by workload shape instead, Prefill on CPU.

Our pick: Colocated prefill and decode with chunked prefill by default, plus dedicated replica pools for long-prompt traffic, keeping true disaggregation as the next step if the interconnect makes KV transfer cheap.

1. How large is the KV transfer in disaggregation?
   The whole prefilled state of the sequence: at 320 KB per token, a 2 000-token prompt is about 640 MB moved from the prefill machine to the decode machine. Over a 200 Gbps link that is tens of milliseconds, which is acceptable; over ordinary datacenter networking it is not. That single number decides whether disaggregation is viable in a given cluster.
2. How do you size the two phases?
   From the token mix, not from requests. Prompt tokens per second divided by prefill throughput gives the prefill capacity, output tokens per second divided by decode throughput gives the decode capacity, and the ratio between them follows the workload: a summarisation product is prefill-heavy, a chat product is decode-heavy. Publishing that ratio is how you justify buying different hardware for each.
3. What does time to first token actually decompose into?
   Queue wait plus prefill plus a step of scheduling. Under healthy load, prefill dominates and scales with prompt length; under load, queue wait dominates and scales with fleet saturation. Reporting those two components separately in telemetry is what lets you say whether the fix is more capacity or better scheduling.
4. Where does speculative decoding fit?
   A small draft model proposes several tokens and the large model verifies them in one pass, so an accepted run yields multiple tokens for one step — a 1.5–3× speedup on inter-token latency when acceptance is high. It costs memory for the draft model and gives back less when the batch is already large, because verification competes for the same compute. It is a latency optimisation for lightly loaded fleets more than a throughput one.

Ask: How do partial results get back to the client, and what happens when the connection breaks?

Good answers name: Server-sent events over HTTP, WebSocket, Polling for partial results, Asynchronous job with a callback or webhook.

Our pick: Server-sent events for the interactive API, with client disconnect propagated to the engine as a cancellation, and an asynchronous job API for the batch tier.

1. Why does cancellation matter so much?
   Because generation continues to consume the scarcest resource in the system for a caller who has gone. A user who closes the tab at token 20 of 500 leaves 480 tokens of GPU work with no consumer. The disconnect must therefore propagate from the gateway to the engine within a step or two, and the engine must remove the sequence and free its KV blocks. On a service where a noticeable share of streams are abandoned, this is several percent of total capacity.
2. How do you resume a broken stream?
   Honestly, mostly you do not: you surface an error and let the client retry, because restarting generation after the same prefix produces different text. If resumption is required, the client sends back the tokens it received and the server prefills prompt plus received tokens and continues — correct, but it pays a full prefill and needs the sampling state to be reproducible. Most APIs choose the simpler contract and make it explicit.
3. What sizes the gateway?
   Concurrent open connections, not request rate. At 280 requests per second each held for 12 seconds, that is roughly 3 400 simultaneous streams, plus headroom for slow clients. Each costs a socket, a buffer and a little state, so the gateway is a connection-management problem — and it must not buffer the response, or streaming silently becomes batch delivery.
4. How does output safety screening work on a stream?
   On a sliding window of recently emitted text rather than on the complete response, since there is no complete response until the end. It runs in parallel with generation so it does not add per-token latency, and when it trips, the stream ends with a terminal error event after partial text has already been sent. The client contract has to allow a stream to end that way, which is a design decision worth stating up front.

Ask: Capacity is fixed and demand is not. Who waits, who is refused, and how is that decided?

Good answers name: Token-bucket quotas per tenant, denominated in tokens, Priority queues with deadline dropping, Weighted fair queueing between tenants, First come, first served with a long queue.

Our pick: Token-denominated quotas per tenant at the edge, priority classes with deadline dropping in the queue, and weighted fair queueing between tenants inside each class, with queue depth bounded so that refusal beats unbounded waiting.

1. How can you enforce a token quota before you know the output length?
   Reserve an estimate at admission — prompt tokens plus max_tokens, or a per-tenant learned average — and reconcile against actual usage when the request finishes, returning the unused reservation to the bucket. Reserving the maximum is safe but under-utilises; reserving the average is efficient but occasionally over-admits. Either way the reconciliation is what keeps quota and billing consistent.
2. How deep should the queue be?
   Derive it from the deadline, not from memory: if the fleet drains 280 requests per second and the deadline is 10 seconds, a queue beyond roughly 2 800 requests is guaranteed to contain work that will expire. Bound it there and refuse beyond it, with a Retry-After that reflects the real drain time. A queue longer than the deadline is a mechanism for wasting capacity.
3. What stops the batch tier from being starved forever?
   A guaranteed floor — a minimum share of steps or a minimum number of replicas reserved for it — plus ageing, where a request's effective priority rises the longer it waits. Without both, "low priority" quietly means "never runs" during any sustained busy period, and the batch product becomes a lie.
4. Where do the quota counters live, and what if that store is down?
   A replicated in-memory store (Redis-style) holding per-tenant buckets, with each gateway keeping a small local lease of tokens so the common path does not make a network call. If the store is unreachable, fail open against the local lease for a bounded period and alert: briefly over-serving a tenant is much cheaper than refusing all traffic because a counter is unavailable.

Ask: GPUs are expensive and slow to start. How do you match capacity to demand without wasting either?

Good answers name: Scale on queue depth and time to first token, A warm pool of loaded, idle replicas, Scale on GPU utilisation, Backfill spare capacity with batch work and spot instances.

Our pick: Autoscale on queue depth and time to first token, keep a warm pool sized to the plausible doubling rate, fill troughs with batch-tier work on spot capacity, and treat scale-down as the risky direction with a long cooldown.

1. How big should the warm pool be?
   From how fast traffic can plausibly grow versus how long a cold start takes. If traffic can double in two minutes and a cold start is four, the pool must cover two cold-start windows of growth — in practice 15–25 % of the fleet. Cheaper alternatives that buy the same insurance: shorten the cold start with local NVMe weight caches and lazy loading, or run the pool on batch work so the capacity is not idle.
2. How do you make cold starts faster?
   Cache weights on local NVMe so a restart is a local read rather than a 70 GB download; distribute new versions peer-to-peer or from a regional mirror so a fleet-wide rollout does not saturate the network; keep the container image and CUDA graphs pre-built; and warm with synthetic requests before taking traffic. That takes a typical four-minute start under a minute for anything but a brand-new host.
3. Where does the money actually go, and what are the levers?
   Almost entirely to accelerator hours, so cost per million tokens is set by tokens produced per GPU-hour. The levers in order of effect: batch size (utilisation), prefix caching (avoided prefill), quantisation (more throughput and more KV headroom), routing cheap requests to a smaller model, and buying reserved or spot capacity. Squeezing the serving stack is worth more than any infrastructure saving elsewhere in this design.
4. Why is scaling down risky?
   Because removing a replica takes seconds while adding one takes minutes, so a premature scale-down during a brief trough leaves you unable to serve the recovery. Use a long cooldown, scale down one replica at a time, drain rather than kill, and never scale below the floor needed to survive the loss of an availability zone.