Design ChatGPT

An AI chat assistant for 100 M weekly users whose streamed answers survive dropped connections, call tools mid-answer, and share scarce GPUs across tiers.

Last updated 2026-10-26. Difficulty: hard. Patterns: streaming, tool-calling, context-management, rate-limiting, prompt-caching. Reported at OpenAI, Anthropic, Google, Meta, Microsoft, Amazon.

Functional requirements

• Multi-turn conversations. The user sends a message and the answer uses everything relevant from earlier turns. The server owns the history; the client sends only the new message and the id of the message it replies to.
• Streamed answers. Text appears token by token within a second of sending and keeps streaming for 10 to 60 seconds. Tool progress (searching, running code) appears in the same stream.
• Answers survive the connection. A dropped connection, a sleeping laptop or a second device must not lose or restart an answer. Generation continues server side and any client can resume from where it left off.
• Stop, regenerate and edit. Stop cancels the answer within a second and keeps what was written. Regenerating, or editing an earlier message, creates a branch the user can switch back from; nothing is overwritten.
• History on every device. List, reopen, rename and delete conversations from any device, including a conversation whose answer is still streaming somewhere else.
• Tools mid-answer. The model can call web search and code execution while answering, read the result and continue. Every tool call has a timeout and runs isolated from everything else.
• Memory and files. Saved memories, custom instructions and uploaded files are used as context when relevant, without sending a whole file to the model on every turn.
• Free and paid tiers with usage limits. Limits per model per time window. When a limit is reached, the user moves to a smaller model and is told when the limit resets, instead of being blocked.
• Safety checks on input and output. Messages and answers are checked against policy. A violation found mid-stream stops the answer and retracts what was already shown.
• Out of scope. Running the models themselves (the inference tier is a black box here; continuous batching and KV memory are the LLM Inference Service question), training, voice and image generation, billing, and enterprise admin.

Non-functional requirements

• Scale (100 M weekly users · 30 k msg/s peak). About 600 k answers streaming at once at peak. Open streams and token appends, not request rate, size the product layer.
• Time to first token (p95 < 1 s · our share < 150 ms). Most of it belongs to the inference tier (queueing and prefill). The product layer’s share (auth, limits, writes, context assembly) must stay under 150 ms, which rules out anything slow in front of the model call.
• No lost answers (every started answer completes or resumes). The requirement that drives the hardest trade-off: it decouples generation from the HTTP connection, which costs an extra hop and a store taking ~2.4 M appends a second, and buys resume, multi-device viewing and an explicit stop.
• Availability (99.9 % chat · 99.95 % history). Under a GPU shortage, degrade to a smaller model or a visible wait rather than an error. Reading old conversations must keep working while generation is impaired.
• Fairness under scarcity (paid latency holds at peak). Background work pauses first and free traffic falls back first. Limits are counted in tokens, because one request can cost a thousand times another.
• Safety exposure (cut within ~100 tokens of a violation). Checks run beside the stream, not in front of it, so the promise is a bounded exposure window, and that window is measured.
• Privacy (deletion within 30 days · no cross-user reuse). Conversations are private. Deletion reaches derived data (summaries, memories, file chunks). Cached prompt prefixes are reused only within a conversation or for the shared system prompt.
• Cost (≥ 40 % of prompt tokens from cache). GPUs dominate the bill, and the product layer controls the biggest lever: whether each turn’s prompt is a cache hit. Prompt layout and routing are cost decisions.

Back-of-envelope estimates

• Messages per second: ~12 k avg · ~30 k peak. 50 M daily users × 20 messages = 1 B messages/day; 1 B ÷ 86 400 ≈ 11.6 k/s. Evenings in the largest regions push peaks to about 2.5×, so ~30 k/s. Each message starts one answer, so this is also the rate of new runs and new streams.
• Concurrent answer streams: ~600 k at peak. 30 k new answers/s × 20 s average streaming time = 600 k open streams (Little’s law). At ~20 k active streams per gateway node that is ~30 nodes before headroom across zones. The gateway is sized by open connections, not by requests per second.
• Output tokens per second: ~15 M/s at peak. 30 k answers/s × 500 tokens = 15 M tokens/s. That is the inference tier’s load and the volume flowing through the token streams, which is why abandoned and duplicated generations are worth measuring.
• Token stream appends: ~2.4 M/s at peak. 600 k streams × 4 flushes a second (one every 250 ms, about 6 tokens) = 2.4 M appends/s, and each wakes a blocked reader, so ~5 M Redis ops/s. At ~150 k ops/s per shard that is ~35 shards. This is the single biggest write stream in the system, and the flush interval is the knob.
• Stream buffer memory: ~25 GB. One answer’s stream is about 6 KB (2 KB of text plus ~80 entries of overhead). Live: 600 k × 6 KB ≈ 3.6 GB. Kept 2 min after completion for late reconnects: 30 k/s × 120 s = 3.6 M streams × 6 KB ≈ 22 GB. Memory is small; the op rate, not the bytes, sets the shard count.
• Prefill avoided by conversation affinity: ~54 M tokens/s. 30 k messages/s × 4 k prompt tokens = 120 M prompt tokens/s at peak. About 3 k of each prompt is the previous turn’s prompt unchanged; with affinity routing ~60 % of turns find it cached: 30 k × 3 k × 0.6 ≈ 54 M tokens/s, about 45 % of all prefill. Without affinity the hit rate falls to a few percent.
• Conversation storage growth: ~3 TB/day · ~1.1 PB/year. 1 B exchanges a day (a message and its answer) × ~3 KB with metadata and tool steps = 3 TB/day ≈ 1.1 PB/year before replication. Reads are mostly the last few turns of one conversation, so the store is partitioned by conversation and read newest first.
• Concurrent code sandboxes: ~7.5 k executing. 10 % of 30 k messages/s use tools at 1.5 calls each = 4.5 k tool calls/s; a third of them run code for ~5 s: 1.5 k/s × 5 s = 7.5 k executing at once. Warm idle sessions kept for follow-up turns multiply the pool several times over.

Components

• Chat client (web · iOS · Android): Sends only the new message and the id of the message it replies to, never the history. Renders stream events by character offset, so a reconnect or a duplicated chunk never shows text twice, and resends the last event id when it reconnects. Stop is an explicit command, because closing the stream no longer cancels anything.
• Chat gateway (SSE · auth · resume): Terminates TLS, authenticates the session and holds the streaming responses: ~600 k at peak, ~20 k per node. It never talks to the orchestrator; it reads the message’s token stream from an entry id, so any node can serve any reconnect from any device. Disables proxy buffering and sends a heartbeat comment every 15 s so idle timeouts do not cut a stream during a long tool call.
• Chat service (conversations · branches): Owns the conversation API: send, edit, regenerate, stop, list, switch branch. Writes the user message and an empty assistant message before generation starts, reserves the estimated token cost against the user’s budget, picks the model or its fallback, and starts the run. Stateless: everything it knows is in the store and the limiter.
• Usage limiter (Redis · weighted tokens): Budgets per user, model and rolling window in price-weighted tokens: output counts four times input, cached input a tenth. Reserve at admission, settle when the run ends, so a short or stopped answer returns what it did not use. Keyed by user, so it shards trivially; if it is unreachable, the chat service fails open on a small per-node allowance.
• Conversation store (wide-column · by conversation): Messages as a tree: partition key conversation_id, sort key a time-ordered message id, each row with parent id, role, status, content, tool steps and usage. A per-user conversation index holds the title, updated_at and the current-leaf pointer, moved with a version check. Edits and regenerations add siblings and nothing is overwritten, which keeps branch switching and crash recovery simple.
• Orchestrator (run loop · lease · steps): Drives one run per assistant message under a lease with a fencing epoch: build the context, call the model, run tool calls, append tokens to the stream, persist each completed step. Stop and regenerate are commands to the run, never side effects of a connection. If an instance dies, another takes the lease and resumes from the last persisted step plus the text already streamed.
• Token streams (Redis Streams · per message): One stream key per in-flight answer: the orchestrator appends a chunk every 250 ms, gateways read from any entry id, and the entry id is the SSE event id. Kept 2 minutes after completion for late reconnects; after that the conversation store is the record. ~2.4 M appends a second at peak, so the op rate rather than the ~25 GB of memory sets the shard count. The run’s lease and epoch share the stream’s hash slot so the append script can fence stale writers.
• Context builder (token budget · stable prefix): Assembles the prompt within the model’s budget: system prompt, custom instructions and memory, rolling summary, recent turns verbatim, then this turn’s retrieved file chunks and the new message. Orders segments from most to least stable, so each turn’s prompt extends the previous one and the inference tier’s prefix cache can reuse it. Uses token counts stored per message instead of re-tokenizing history.
• Memory & files (vector index · blobs): Saved memories and custom instructions per user, and uploaded files chunked and embedded for retrieval. The context builder pulls a few hundred tokens of memory and the top file chunks for the current question, never a whole 300-page PDF. Retrieved text is marked as quoted data in the prompt and never placed in the system role.
• Model router (affinity · tiers · fallback): Chooses the model pool and the replica for each generation call. Hashes the conversation id onto the pool’s replicas with bounded load, so the next turn lands where the last turn’s KV cache lives, and spills to the next replica when one passes 1.25× the average. Keeps per-class priority queues (paid, free, background) with a reserved floor for paid, and sheds free traffic to a fallback model first.
• Inference tier (black box · per-model pools): The GPU fleets that run the models, treated here as a black box with a streaming generate call, a cancel and a prefix cache. Batching, paged KV memory and admission inside a pool are the LLM Inference Service question. What this design needs from it: honour the affinity hint, report queue time per class, and stop generating within a step of a cancel.
• Tool runner (microVM sandboxes · timeouts): Executes the tool calls the model emits: web search through a provider, and code in a microVM with no network, a CPU and memory cap and a 30-second limit, keyed by conversation so files persist between turns. Every call carries (run, step) as its idempotency key; a timeout becomes a tool result the model can reason about, not an error for the user.
• Search provider (web search API): A third-party web search API (or an in-house index) behind the search tool. Slow, from 300 ms to several seconds, rate-limited and billed per query, so a resumed run never repeats a search whose result is already persisted.
• Moderation (input + output classifiers): Small, fast classifiers over the user’s message and over overlapping ~100-token windows of the answer, about 150 k checks a second at peak. They run beside generation, never in front of it: the input check finishes inside the prefill time, and an output hit makes the orchestrator cancel the run and send a retract event. High-risk categories switch a run to a short lookahead buffer so nothing unscreened is shown.
• Event bus (Kafka · by conversation): message.completed and usage events, keyed by conversation id. Consumers write titles, refresh rolling summaries when history passes its budget, extract memories and meter usage. All of it runs at background priority and is the first work paused when GPUs are short; the bus simply holds the backlog.

User flows

1. Send a message and stream the answer. The core write path. The answer is produced by a run that is decoupled from the HTTP connection: the run writes tokens to a per-message stream and the gateway reads from it, which is what makes resume, stop and a second device possible.
   1. Client posts the new message with its parent id and a client message id; the response stays open as an event stream. The client sends only the new text and which message it replies to, never the history: the server owns the conversation. The POST response is itself the stream, so the first token does not wait for a second round trip. client_msg_id makes a retried POST join the same answer instead of starting another.
   2. Chat service reserves the estimated token cost against the user’s budget for the chosen model, or picks the fallback model. The estimate is the prompt’s tokens plus an expected answer (the user’s recent median, capped by max_tokens), weighted by model price, and it is settled to actual usage when the run ends. One atomic script checks and reserves, so ten parallel tabs cannot all pass on an empty counter. A spent budget routes to the smaller model and the first event says so, instead of failing.
   3. Chat service appends the user message and an empty assistant message to the conversation tree and moves the current-leaf pointer. Both rows exist before generation starts, so every device shows the answer in progress from the first moment. The assistant row starts as in_progress. The pointer moves with a conditional write on the conversation’s version, so two devices sending at once land as sibling branches instead of one overwriting the other.
   4. Chat service starts a run for the assistant message, and the gateway starts reading that message’s token stream. The run id is the assistant message id, and exactly one orchestrator holds its lease. The gateway never talks to the orchestrator: it blocks on the stream key, so the connection and the generation share only a name. If the gateway node dies, any other node can serve the same stream.
   5. The orchestrator has the context builder assemble the prompt from system prompt, memory, summary and recent turns, within the token budget. Stable parts come first and change rarely (system prompt, memory, summary, older turns), so this prompt is an append-only extension of the previous turn’s and the inference tier’s prefix cache can reuse it. File chunks retrieved for this turn go last, next to the new message. Token counts are stored per message, so assembly never re-tokenizes history.
   6. The orchestrator sends the prompt through the model router while the input check runs in parallel. The router hashes the conversation id onto the model’s replicas with bounded load, so this turn lands on the replica that probably still holds the last turn’s KV cache and only the new tokens are prefilled. The input classifier takes ~50 ms against a prefill of a few hundred, so it finishes before the first token; the orchestrator holds token release until it passes.
   7. Tokens are appended to the message’s stream every 250 ms, checked in windows, and forwarded by the gateway as SSE events. Each entry carries its character offset and the run’s lease epoch, and the append is a small script that rejects a stale epoch, so a zombie orchestrator cannot write after losing the lease. The SSE id is the Redis entry id, which is what the client sends back as Last-Event-ID on reconnect. Output moderation sees overlapping ~100-token windows alongside, never in the way of the stream.
   8. When the model finishes, the orchestrator writes the full answer to the store, settles the reservation and closes the stream. The final write is the durable copy. The stream key gets a 2-minute TTL so late reconnects can still read it, after which the store is the only source. Settlement charges actual weighted usage (1 100 uncached + 3 000 cached × 0.1 + 520 out × 4 ≈ 3 480) and returns the rest of the 6 100 reserved.
   9. A completion event goes on the bus; consumers write the title, refresh the rolling summary when history grows, and record usage. None of this is on the user’s path, so it runs at background priority and is paused first under load. The summary job rewrites the summary only when verbatim history passes its budget, because every rewrite changes the prompt prefix and costs a cache miss on the next turn.
2. Open the conversation on another device mid-answer. The read path, and the proof that an answer belongs to the conversation rather than to a connection: a phone opens a conversation the laptop started, renders the current branch, and joins the answer that is still being written.
   1. The phone loads the conversation list, newest first. One partition read on the user’s conversation index sorted by updated_at, 30 rows a page with a cursor. Titles come from the background title job, so a conversation started seconds ago shows a placeholder until its title lands.
   2. Opening a conversation reads its newest messages and walks parent links from the current leaf to get the visible path. Messages are one partition keyed by conversation id, read newest first in pages of 50. The visible path is the chain of parent ids from current_leaf; other branches appear only as a ‘2 / 3’ counter on the message where they split, with the sibling ids so the client can switch.
   3. The newest answer is still in progress, so the client opens its stream from the beginning. The phone has none of the answer yet, so it reads from 0-0: the stream holds every chunk written so far, so it catches up in one burst and then goes live. No orchestrator is involved and the laptop’s stream is unaffected; two readers are just two reads of one key.
   4. The laptop wakes from sleep and reconnects with the last event id it saw; the gateway resumes after that entry. Resume is a read from that id, exclusive. The reconnect can land on any gateway node, because no node holds state for a stream. Character offsets on each chunk let the client drop anything it already rendered.
   5. If the answer finished more than two minutes ago, the stream has expired and the client gets the final message from the store. The stream is a short-lived buffer, not the record. A missing key for a complete message means ‘send a snapshot, then done’. A missing key for an in-progress message is the failure case, handled by the run’s recovery in the crash flow.
   6. The user switches to an older branch on the phone; the pointer moves and other devices pick it up. Switching branches is a write to one field with a version check. Other devices learn of it from a light conversation-updated push or when they regain focus; nothing needs a socket held open just for this.
3. Peak hour: the flagship runs out of GPUs. GPU capacity cannot grow in minutes, so the product layer decides who gets the flagship model, who gets the smaller one, and who waits. The rule: degrade free before paid, and never turn a slow answer into a lost one.
   1. The router sees queue time for the flagship pool rise past its target for the free class. The signal is time in queue per priority class, not GPU utilisation, which looks the same at healthy and saturated load under continuous batching. The router keeps one queue per model and class: paid, free and background.
   2. Background work (titles, summaries, memory extraction) is paused first and its events wait on the bus. Nobody notices a missing title for ten minutes, and Kafka simply holds the backlog until capacity returns. This frees a few percent of the pool at no user-visible cost, which is why background is its own class rather than mixed with free traffic.
   3. The router sheds new free-tier flagship requests; the orchestrator rebuilds the prompt for the smaller model and tells the user. A free user gets a slightly weaker answer in two seconds instead of a better one in thirty, which is the better product trade. Changing model loses the prefix cache for that conversation and may shrink the context window, so the context builder re-assembles for the smaller model’s budget.
   4. Paid requests keep their reserved share of the flagship pool and wait briefly in priority order if they must. Paid traffic has a floor of the pool (say 60 %) and may borrow anything idle; free traffic borrows from paid only when paid is idle, so the scheme wastes nothing at night. Within a class, fair queueing by user stops one heavy user’s parallel tabs from crowding out everyone else.
   5. Users who have spent their flagship budget go to the fallback model until their window resets. Budgets are in weighted tokens over a rolling window, because a one-line question and a 100-page file analysis differ by a thousand times in GPU cost. Saying exactly when the window resets turns a hard wall into a predictable limit.
   6. If the fallback pool also saturates, free requests wait with a visible position and a deadline instead of failing. A request that cannot start within its deadline (20 s for free) is refused with a retry time, because holding it longer only means the user retries on top of it. The paid class is the last to see this, and when it does it is an incident, not a policy.
   7. Settled usage, shed counts and minutes in fallback per tier feed the capacity plan. Shed rate and fallback minutes per tier are the business numbers here: they are what product uses to choose between buying GPUs, raising the paid floor and tightening free limits.
4. A tool-using answer survives an orchestrator crash. The model searches the web and runs code mid-answer, and the machine driving that loop dies between steps. Per-step persistence and a fenced run lease let another orchestrator resume without repeating the search or restarting the visible text.
   1. The model emits a web search call; the orchestrator records the step before running it. Tool calling is a standard model API feature: instead of more text, the output ends with a structured call (a name and JSON arguments). The step is written as started with the idempotency key (run, step), so a resumed run knows the step exists and whether it finished.
   2. The tool runner runs the search with a 10-second timeout and the orchestrator persists the result. Results are trimmed to a token cap before they enter the context. A timeout is not a failure of the answer: the model receives ‘search timed out’ as the tool result and decides what to do. The user sees a ‘Searching the web’ event in the same stream.
   3. The result goes back to the model, which writes some text and then asks to run code. The second model call is the same prompt plus the call and its result, so with conversation affinity it is almost entirely a prefix-cache hit. The code runs in a microVM with no network, capped CPU and memory and a 30-second limit, in a session keyed by conversation so later turns can use the files it creates.
   4. The orchestrator crashes while the code runs; its lease expires and the next owner bumps the epoch. The lease is renewed every 2 s with a 10 s expiry, and lives in the same Redis hash slot as the stream. Bumping the epoch on takeover means that if the old owner was only paused, not dead, its next append is rejected. Clients see a pause in the stream, not an error.
   5. A sweeper hands the run to another orchestrator, which reloads the persisted steps and the text already streamed. Step 1 is done in the store, so the search is neither repeated nor billed again. Step 2 is started but not done; the sandbox has no network and no outside effects, so running it again is safe. The text the user has already seen is read back from the stream.
   6. It re-runs the code step, then asks the model to continue from the text already shown, appending under the new epoch. Continuation means the prompt carries the partial answer as the start of the assistant turn, so the model picks up mid-sentence instead of starting over. The client never sees text twice: offsets on each chunk let it ignore anything duplicated at the seam.
   7. The answer completes and the stored message records every step, so the conversation shows what was searched and run. Steps belong to the message, which is how the UI shows ‘Searched 4 sites’ and how the next turn’s context includes the code output without running anything again. A run that passes 10 tool steps or its token budget is asked for a final answer with tools disabled, so a looping model cannot spend unbounded GPU and search quota.
5. Stop an answer, then regenerate it. Closing the tab must not cancel an answer, but pressing Stop must cancel it within a second. Stop and regenerate are commands to the run, and regenerate never overwrites: it adds a sibling.
   1. The user presses Stop; the client sends a stop command for the message rather than closing the stream. Generation is decoupled from the connection, so closing the stream would only stop this device watching. Stop is an explicit, idempotent POST, and it works from any device, including one that never opened the stream.
   2. Chat service delivers the stop to the orchestrator holding the run’s lease. The lease names its owner, so the stop is a direct call to that instance. If the call fails, a cancel flag beside the lease is checked at every 250 ms flush, so the stop lands within one flush either way.
   3. The orchestrator cancels the inference request, which frees the GPU slot, and ends the stream with a stopped status. Cancelling inside the inference tier is the point: every token generated after Stop is GPU time with no reader. The partial answer is kept, because the user may want its first half.
   4. The partial answer is saved and usage is settled for the tokens actually produced. The user is charged for what was generated, not for the reservation, so stopping early gives budget back. The message status records that it was stopped, so the UI can offer to continue it.
   5. The user clicks Regenerate; chat service adds a new assistant message as a sibling under the same user message. The old answer stays in the tree and the user can flip between ‘1 / 2’ and ‘2 / 2’. Editing an earlier user message works the same way one level up: a new user message under the same parent, starting a new branch with its own answer.
   6. A new run starts with the same prompt as the first attempt, so it is a cache hit. Same conversation, same path up to m_42, same replica by affinity: the whole prompt is already in the KV cache and the first token comes back quickly. Everything else is an ordinary send.

Deep dives

The theory behind it

• Building LLM features: serving, latency and cost: System design for LLM-powered products: streaming responses, the cost and latency of tokens, batching and KV caches in inference, prompt caching, rate limits and fallbacks, guardrails, evaluation, and when to use RAG, tools or fine-tuning.
• WebSockets vs Server-Sent Events vs long polling: Choosing how a server pushes updates to clients: short polling, long polling, Server-Sent Events and WebSockets compared on direction, latency, overhead, proxies, scaling and mobile behaviour, plus WebTransport and push notifications, with a decision guide for interviews.
• Designing AI agent systems: The architecture behind LLM agents that use tools: the agent loop, tool definitions and execution, sandboxing, memory and context management, orchestration and multi-agent patterns, durable long-running tasks, guardrails and permissions, evaluation, cost control and observability.

Related

• Design Uber
• Design a Notification System
• Design an LLM Inference Service
• Design Zapier
• All 83 system design interview questions
• Run this question for a friend: the interviewer kit