Design ChatGPT

• Multi-turn conversations: The user sends a message and the answer uses everything relevant from earlier turns. The server owns the history; the client sends only the new message and the id of the message it replies to.
• Streamed answers: Text appears token by token within a second of sending and keeps streaming for 10 to 60 seconds. Tool progress (searching, running code) appears in the same stream.
• Answers survive the connection: A dropped connection, a sleeping laptop or a second device must not lose or restart an answer. Generation continues server side and any client can resume from where it left off.
• Stop, regenerate and edit: Stop cancels the answer within a second and keeps what was written. Regenerating, or editing an earlier message, creates a branch the user can switch back from; nothing is overwritten.
• History on every device: List, reopen, rename and delete conversations from any device, including a conversation whose answer is still streaming somewhere else.
• Tools mid-answer: The model can call web search and code execution while answering, read the result and continue. Every tool call has a timeout and runs isolated from everything else.
• Memory and files: Saved memories, custom instructions and uploaded files are used as context when relevant, without sending a whole file to the model on every turn.
• Free and paid tiers with usage limits: Limits per model per time window. When a limit is reached, the user moves to a smaller model and is told when the limit resets, instead of being blocked.
• Safety checks on input and output: Messages and answers are checked against policy. A violation found mid-stream stops the answer and retracts what was already shown.
• Out of scope: Running the models themselves (the inference tier is a black box here; continuous batching and KV memory are the LLM Inference Service question), training, voice and image generation, billing, and enterprise admin.

• Scale (100 M weekly users · 30 k msg/s peak): About 600 k answers streaming at once at peak. Open streams and token appends, not request rate, size the product layer.
• Time to first token (p95 < 1 s · our share < 150 ms): Most of it belongs to the inference tier (queueing and prefill). The product layer’s share (auth, limits, writes, context assembly) must stay under 150 ms, which rules out anything slow in front of the model call.
• No lost answers (every started answer completes or resumes): The requirement that drives the hardest trade-off: it decouples generation from the HTTP connection, which costs an extra hop and a store taking ~2.4 M appends a second, and buys resume, multi-device viewing and an explicit stop.
• Availability (99.9 % chat · 99.95 % history): Under a GPU shortage, degrade to a smaller model or a visible wait rather than an error. Reading old conversations must keep working while generation is impaired.
• Fairness under scarcity (paid latency holds at peak): Background work pauses first and free traffic falls back first. Limits are counted in tokens, because one request can cost a thousand times another.
• Safety exposure (cut within ~100 tokens of a violation): Checks run beside the stream, not in front of it, so the promise is a bounded exposure window, and that window is measured.
• Privacy (deletion within 30 days · no cross-user reuse): Conversations are private. Deletion reaches derived data (summaries, memories, file chunks). Cached prompt prefixes are reused only within a conversation or for the shared system prompt.
• Cost (≥ 40 % of prompt tokens from cache): GPUs dominate the bill, and the product layer controls the biggest lever: whether each turn’s prompt is a cache hit. Prompt layout and routing are cost decisions.

• Messages per second: ~12 k avg · ~30 k peak. 50 M daily users × 20 messages = 1 B messages/day; 1 B ÷ 86 400 ≈ 11.6 k/s. Evenings in the largest regions push peaks to about 2.5×, so ~30 k/s. Each message starts one answer, so this is also the rate of new runs and new streams.
• Concurrent answer streams: ~600 k at peak. 30 k new answers/s × 20 s average streaming time = 600 k open streams (Little’s law). At ~20 k active streams per gateway node that is ~30 nodes before headroom across zones. The gateway is sized by open connections, not by requests per second.
• Output tokens per second: ~15 M/s at peak. 30 k answers/s × 500 tokens = 15 M tokens/s. That is the inference tier’s load and the volume flowing through the token streams, which is why abandoned and duplicated generations are worth measuring.
• Token stream appends: ~2.4 M/s at peak. 600 k streams × 4 flushes a second (one every 250 ms, about 6 tokens) = 2.4 M appends/s, and each wakes a blocked reader, so ~5 M Redis ops/s. At ~150 k ops/s per shard that is ~35 shards. This is the single biggest write stream in the system, and the flush interval is the knob.
• Stream buffer memory: ~25 GB. One answer’s stream is about 6 KB (2 KB of text plus ~80 entries of overhead). Live: 600 k × 6 KB ≈ 3.6 GB. Kept 2 min after completion for late reconnects: 30 k/s × 120 s = 3.6 M streams × 6 KB ≈ 22 GB. Memory is small; the op rate, not the bytes, sets the shard count.
• Prefill avoided by conversation affinity: ~54 M tokens/s. 30 k messages/s × 4 k prompt tokens = 120 M prompt tokens/s at peak. About 3 k of each prompt is the previous turn’s prompt unchanged; with affinity routing ~60 % of turns find it cached: 30 k × 3 k × 0.6 ≈ 54 M tokens/s, about 45 % of all prefill. Without affinity the hit rate falls to a few percent.
• Conversation storage growth: ~3 TB/day · ~1.1 PB/year. 1 B exchanges a day (a message and its answer) × ~3 KB with metadata and tool steps = 3 TB/day ≈ 1.1 PB/year before replication. Reads are mostly the last few turns of one conversation, so the store is partitioned by conversation and read newest first.
• Concurrent code sandboxes: ~7.5 k executing. 10 % of 30 k messages/s use tools at 1.5 calls each = 4.5 k tool calls/s; a third of them run code for ~5 s: 1.5 k/s × 5 s = 7.5 k executing at once. Warm idle sessions kept for follow-up turns multiply the pool several times over.

• Chat client (web · iOS · Android): Sends only the new message and the id of the message it replies to, never the history. Renders stream events by character offset, so a reconnect or a duplicated chunk never shows text twice, and resends the last event id when it reconnects. Stop is an explicit command, because closing the stream no longer cancels anything.
• Chat gateway (SSE · auth · resume): Terminates TLS, authenticates the session and holds the streaming responses: ~600 k at peak, ~20 k per node. It never talks to the orchestrator; it reads the message’s token stream from an entry id, so any node can serve any reconnect from any device. Disables proxy buffering and sends a heartbeat comment every 15 s so idle timeouts do not cut a stream during a long tool call.
• Chat service (conversations · branches): Owns the conversation API: send, edit, regenerate, stop, list, switch branch. Writes the user message and an empty assistant message before generation starts, reserves the estimated token cost against the user’s budget, picks the model or its fallback, and starts the run. Stateless: everything it knows is in the store and the limiter.
• Usage limiter (Redis · weighted tokens): Budgets per user, model and rolling window in price-weighted tokens: output counts four times input, cached input a tenth. Reserve at admission, settle when the run ends, so a short or stopped answer returns what it did not use. Keyed by user, so it shards trivially; if it is unreachable, the chat service fails open on a small per-node allowance.
• Conversation store (wide-column · by conversation): Messages as a tree: partition key conversation_id, sort key a time-ordered message id, each row with parent id, role, status, content, tool steps and usage. A per-user conversation index holds the title, updated_at and the current-leaf pointer, moved with a version check. Edits and regenerations add siblings and nothing is overwritten, which keeps branch switching and crash recovery simple.
• Orchestrator (run loop · lease · steps): Drives one run per assistant message under a lease with a fencing epoch: build the context, call the model, run tool calls, append tokens to the stream, persist each completed step. Stop and regenerate are commands to the run, never side effects of a connection. If an instance dies, another takes the lease and resumes from the last persisted step plus the text already streamed.
• Token streams (Redis Streams · per message): One stream key per in-flight answer: the orchestrator appends a chunk every 250 ms, gateways read from any entry id, and the entry id is the SSE event id. Kept 2 minutes after completion for late reconnects; after that the conversation store is the record. ~2.4 M appends a second at peak, so the op rate rather than the ~25 GB of memory sets the shard count. The run’s lease and epoch share the stream’s hash slot so the append script can fence stale writers.
• Context builder (token budget · stable prefix): Assembles the prompt within the model’s budget: system prompt, custom instructions and memory, rolling summary, recent turns verbatim, then this turn’s retrieved file chunks and the new message. Orders segments from most to least stable, so each turn’s prompt extends the previous one and the inference tier’s prefix cache can reuse it. Uses token counts stored per message instead of re-tokenizing history.
• Memory & files (vector index · blobs): Saved memories and custom instructions per user, and uploaded files chunked and embedded for retrieval. The context builder pulls a few hundred tokens of memory and the top file chunks for the current question, never a whole 300-page PDF. Retrieved text is marked as quoted data in the prompt and never placed in the system role.
• Model router (affinity · tiers · fallback): Chooses the model pool and the replica for each generation call. Hashes the conversation id onto the pool’s replicas with bounded load, so the next turn lands where the last turn’s KV cache lives, and spills to the next replica when one passes 1.25× the average. Keeps per-class priority queues (paid, free, background) with a reserved floor for paid, and sheds free traffic to a fallback model first.
• Inference tier (black box · per-model pools): The GPU fleets that run the models, treated here as a black box with a streaming generate call, a cancel and a prefix cache. Batching, paged KV memory and admission inside a pool are the LLM Inference Service question. What this design needs from it: honour the affinity hint, report queue time per class, and stop generating within a step of a cancel.
• Tool runner (microVM sandboxes · timeouts): Executes the tool calls the model emits: web search through a provider, and code in a microVM with no network, a CPU and memory cap and a 30-second limit, keyed by conversation so files persist between turns. Every call carries (run, step) as its idempotency key; a timeout becomes a tool result the model can reason about, not an error for the user.
• Search provider (web search API): A third-party web search API (or an in-house index) behind the search tool. Slow, from 300 ms to several seconds, rate-limited and billed per query, so a resumed run never repeats a search whose result is already persisted.
• Moderation (input + output classifiers): Small, fast classifiers over the user’s message and over overlapping ~100-token windows of the answer, about 150 k checks a second at peak. They run beside generation, never in front of it: the input check finishes inside the prefill time, and an output hit makes the orchestrator cancel the run and send a retract event. High-risk categories switch a run to a short lookahead buffer so nothing unscreened is shown.
• Event bus (Kafka · by conversation): message.completed and usage events, keyed by conversation id. Consumers write titles, refresh rolling summaries when history passes its budget, extract memories and meter usage. All of it runs at background priority and is the first work paused when GPUs are short; the bus simply holds the backlog.

1. Send a message and stream the answer: The core write path. The answer is produced by a run that is decoupled from the HTTP connection: the run writes tokens to a per-message stream and the gateway reads from it, which is what makes resume, stop and a second device possible.
   1. Client posts the new message with its parent id and a client message id; the response stays open as an event stream.
   2. Chat service reserves the estimated token cost against the user’s budget for the chosen model, or picks the fallback model.
   3. Chat service appends the user message and an empty assistant message to the conversation tree and moves the current-leaf pointer.
   4. Chat service starts a run for the assistant message, and the gateway starts reading that message’s token stream.
   5. The orchestrator has the context builder assemble the prompt from system prompt, memory, summary and recent turns, within the token budget.
   6. The orchestrator sends the prompt through the model router while the input check runs in parallel.
   7. Tokens are appended to the message’s stream every 250 ms, checked in windows, and forwarded by the gateway as SSE events.
   8. When the model finishes, the orchestrator writes the full answer to the store, settles the reservation and closes the stream.
   9. A completion event goes on the bus; consumers write the title, refresh the rolling summary when history grows, and record usage.
2. Open the conversation on another device mid-answer: The read path, and the proof that an answer belongs to the conversation rather than to a connection: a phone opens a conversation the laptop started, renders the current branch, and joins the answer that is still being written.
   1. The phone loads the conversation list, newest first.
   2. Opening a conversation reads its newest messages and walks parent links from the current leaf to get the visible path.
   3. The newest answer is still in progress, so the client opens its stream from the beginning.
   4. The laptop wakes from sleep and reconnects with the last event id it saw; the gateway resumes after that entry.
   5. If the answer finished more than two minutes ago, the stream has expired and the client gets the final message from the store.
   6. The user switches to an older branch on the phone; the pointer moves and other devices pick it up.
3. Peak hour: the flagship runs out of GPUs: GPU capacity cannot grow in minutes, so the product layer decides who gets the flagship model, who gets the smaller one, and who waits. The rule: degrade free before paid, and never turn a slow answer into a lost one.
   1. The router sees queue time for the flagship pool rise past its target for the free class.
   2. Background work (titles, summaries, memory extraction) is paused first and its events wait on the bus.
   3. The router sheds new free-tier flagship requests; the orchestrator rebuilds the prompt for the smaller model and tells the user.
   4. Paid requests keep their reserved share of the flagship pool and wait briefly in priority order if they must.
   5. Users who have spent their flagship budget go to the fallback model until their window resets.
   6. If the fallback pool also saturates, free requests wait with a visible position and a deadline instead of failing.
   7. Settled usage, shed counts and minutes in fallback per tier feed the capacity plan.
4. A tool-using answer survives an orchestrator crash: The model searches the web and runs code mid-answer, and the machine driving that loop dies between steps. Per-step persistence and a fenced run lease let another orchestrator resume without repeating the search or restarting the visible text.
   1. The model emits a web search call; the orchestrator records the step before running it.
   2. The tool runner runs the search with a 10-second timeout and the orchestrator persists the result.
   3. The result goes back to the model, which writes some text and then asks to run code.
   4. The orchestrator crashes while the code runs; its lease expires and the next owner bumps the epoch.
   5. A sweeper hands the run to another orchestrator, which reloads the persisted steps and the text already streamed.
   6. It re-runs the code step, then asks the model to continue from the text already shown, appending under the new epoch.
   7. The answer completes and the stored message records every step, so the conversation shows what was searched and run.
5. Stop an answer, then regenerate it: Closing the tab must not cancel an answer, but pressing Stop must cancel it within a second. Stop and regenerate are commands to the run, and regenerate never overwrites: it adds a sibling.
   1. The user presses Stop; the client sends a stop command for the message rather than closing the stream.
   2. Chat service delivers the stop to the orchestrator holding the run’s lease.
   3. The orchestrator cancels the inference request, which frees the GPU slot, and ends the stream with a stopped status.
   4. The partial answer is saved and usage is settled for the tokens actually produced.
   5. The user clicks Regenerate; chat service adds a new assistant message as a sibling under the same user message.
   6. A new run starts with the same prompt as the first attempt, so it is a cache hit.

Ask: The user’s phone drops off the network twenty seconds into a forty-five second answer. What happens to the answer?

Good answers name: Proxy the model’s stream straight through the request, Run decoupled from the connection, tokens appended to a per-message stream, SSE resume by entry id, Buffer in the gateway’s memory, with a sticky WebSocket per session, Write partial text to the conversation store and let clients poll.

Our pick: Generation is a run owned by an orchestrator under a lease, keyed by the assistant message id and never by the HTTP request. The orchestrator appends a chunk every 250 ms to stream:{message_id} in Redis Streams, each entry carrying its character offset and the run’s epoch. The gateway serves SSE by blocking reads on that key, uses the entry id as the SSE id, and on reconnect resumes with a read from Last-Event-ID, so any node can serve any device. The POST that sends a message returns the stream directly to save a round trip, and GET /messages/{id}/stream is the resume path. Keys expire 2 minutes after the run completes; after that the conversation store holds the final answer and a reconnect gets a snapshot event. Stop is POST /messages/{id}/stop, routed to the lease holder, which cancels its inference request within one flush. SSE’s Last-Event-ID was designed for exactly this kind of resume, and browsers resend it on reconnect without any code.

1. The Redis shard holding a live stream fails over and loses the last second of chunks. What does the user see?
   Replication is asynchronous, so a failover can drop up to about a second of entries. A client that already received them is unaffected, because it resumes from an id and later entries still arrive. A client that reconnects into the gap sees a jump in the character offset and asks for a snapshot; the orchestrator holds the whole answer so far in memory and appends a snapshot entry. Only if the orchestrator dies at the same moment does the run fall back to its last persisted step, with a reset event telling clients to discard text after that point.
2. Why not Kafka for the token streams?
   Kafka is built for a modest number of long-lived partitions, and here 30 k short-lived streams a second each need reading from an arbitrary offset by message id. Multiplexing them onto partitions means every reader filters other messages’ tokens, and resuming one message means seeking inside a partition with no index by message. Redis gives a key per message, a read from any id, a TTL, and creation for free. Kafka is the right tool one layer down, for completion and usage events that consumers process in bulk.
3. The user closes the tab and never comes back. Do you keep generating?
   By default, yes: the answer is saved and waiting when they reopen the app, which is what users expect, and on average the remaining cost is half an answer. Under GPU pressure on the free tier, a run with no reader attached for 60 seconds stops at a sentence boundary and is saved with a Continue button. Paid runs always finish. The abandoned-generation rate and the GPU time it costs are tracked, so this stays a measured policy rather than a guess.
4. How many streams can one gateway node hold, and what breaks first?
   With an event loop and a few KB of buffer per connection, tens of thousands of active streams per node is realistic; I planned 20 k, so about 30 nodes for 600 k plus headroom across zones. What breaks first is usually not memory but intermediaries: proxies that buffer responses and idle timeouts that cut a stream during a long tool call. So disable buffering on every hop, send a comment heartbeat every 15 seconds, and batch reads so one call serves many streams on the same shard.
5. How do you know resume actually works?
   Measure it from the client: reconnects per stream, the share that resumed versus fell back to a snapshot, and offset gaps detected. On the server, count stream keys missing while their message is still in progress; that should be near zero and is the alarm for a lost shard or a broken run. And test it on purpose: a staging job that kills gateway nodes and flips client networks, then checks that each answer’s stored text equals what the client rendered.

Ask: A user edits their third message in a forty-turn conversation, then regenerates an answer twice. What do you store?

Good answers name: Immutable messages with parent ids, plus a current-leaf pointer per conversation, A linear list where an edit truncates and overwrites, The whole conversation as one document, Sharded relational database keyed by user id.

Our pick: Two tables in a wide-column store (DynamoDB, or Cassandra / Scylla). messages: partition key conversation_id, sort key a time-ordered message id (ULID), with parent_id, role, status (in_progress, complete, stopped, moderated, error), content, tool steps, model and usage. conversations: partition key user_id, sort key conversation_id, with title, current_leaf, version, a reference to the rolling summary and a deleted flag, plus an index on (user_id, updated_at) for the sidebar. Send, edit and regenerate insert children; the pointer moves with a conditional write on version, and a writer that loses re-reads and points it at its own newer message, so the latest write wins the pointer while both branches survive. Opening a conversation reads its partition newest first and walks parent links from current_leaf; other branches show as counters where they split. Partitioning messages by conversation rather than by user keeps a heavy user’s 10 k conversations from becoming one giant partition.

1. Two devices send to the same conversation at the same instant. Walk me through it.
   Both send the same parent_id, so both user messages are valid children of that leaf, each with its own empty assistant reply, and both answers generate. The pointer update is conditional on the conversation’s version: the first wins, the second fails, re-reads and points it at its own newer message. Nothing is lost, each device watches its own answer, and the message where they split shows ‘2 / 2’. That beats both a 409 and a silent overwrite.
2. A conversation has 3 000 messages. How do you open it quickly on a phone?
   Do not read all of it. Read the partition newest first in pages of 50; ids are time-ordered and the current path is usually the newest messages, so walking parent links from current_leaf rarely needs more than a page or two. The client renders that and loads older path segments as the user scrolls up. The context builder does not need the whole path either: it reads the rolling summary and the recent turns.
3. The user shares a conversation by link and keeps chatting. What does the link show?
   A snapshot, not a live view: sharing copies the current path at that moment into an immutable shared object with its own id. Later messages, other branches and anything the user deletes afterwards never leak through it. Deleting the conversation or revoking the link deletes the snapshot too. This is the one place where storing a whole document is the right shape.
4. A user deletes a conversation. What has to happen, and where?
   Mark the conversation deleted at once so it disappears from every device, then purge asynchronously: the message partition, derived summaries, memories extracted from it, uploaded file chunks, any live stream keys, and a search index if there is one. Backups age out on their stated retention. The purge is idempotent and tracked per conversation, so a crash resumes it, and an audit job samples deleted ids to prove nothing survived.
5. Why keep tool steps inside the message instead of as separate hidden messages?
   The steps are part of one answer: they are created, retried and deleted together, and the UI shows them inline. Keeping them in the message row, or in child rows under the same message id, keeps the tree about turns the user sees, so branching and path walks are not cluttered with internal steps. The context builder still reads them when it assembles the next prompt, so the model knows what it searched and ran.

Ask: Every turn sends the model around 4 k tokens of context, most of it unchanged since the last turn. Who builds that prompt, and how do you avoid paying to prefill it again?

Good answers name: Server-side assembly with a stable, append-only prefix, routed by conversation affinity with bounded load, Client sends the full history each turn; route to the least-loaded replica, Slide the window or re-summarise on every turn, Hard sticky sessions: pin each conversation to one replica.

Our pick: The context builder owns the prompt and lays it out from most to least stable: the system prompt (shared by everyone on a model version), custom instructions and memory, the rolling summary, verbatim turns, then this turn’s retrieved file chunks and the new message, with volatile details such as the current time at the very end. History has a budget per tier (say 8 k tokens free, 32 k paid); when verbatim turns pass it, a background job folds the oldest half into the summary once, costing one cache miss, instead of sliding the window every turn. The orchestrator passes conversation_id as the affinity key, and the router maps it with consistent hashing with bounded load (no replica above 1.25× the pool average), so a follow-up finds its prefix cached about 60 % of the time. That avoids ~54 M prefill tokens a second at peak, nearly half of all prompt tokens. The public APIs from OpenAI and Anthropic both offer prompt caching with cheaper cached input tokens, which is the same economics offered to developers.

1. Why not let the client send the history, like the API does?
   Because in the product the server owns the conversation. Uploading a 100-turn history with every message is ~300 KB on a mobile link, the client could rewrite earlier assistant turns or inject a system prompt, and two devices would each send their own version. With history on the server, the client sends one message and a parent id, and truncation, summaries and caching are decided in one place. The stateless API is right for developers precisely because they own their history; our users do not want to.
2. A user attaches a 300-page PDF. What goes into the prompt?
   Not the PDF. On upload it is chunked, embedded and indexed under the conversation; each turn the context builder retrieves the top chunks for the current question, up to the file budget of about 8 k tokens, and places them after the history, next to the new message. Putting them last means a different set of chunks next turn does not invalidate the long cached prefix. A request that genuinely needs the whole document, such as a full summary, runs as a map-reduce over chunks in tool steps rather than one enormous prompt.
3. The replica holding a popular conversation’s cache dies. What does it cost?
   Its conversations rehash to the next replicas on the ring and each pays one full prefill on its next turn: about 4 k tokens and a few hundred milliseconds of first-token latency, once. Across a replica with a few thousand active conversations, that is a short burst of extra prefill the pool absorbs. The load bound matters here: without it, every conversation from the dead replica would land on its single neighbour.
4. How do you count tokens without re-tokenizing the history every turn?
   Tokenize each message once when it is written and store its token count per model family, along with the summary’s count. Assembly is then arithmetic over stored counts, and only the new message is tokenized on the hot path. When a model with a new tokenizer launches, counts are backfilled lazily, with a safety margin until they are.
5. Retrieved pages and files can contain instructions like ‘ignore previous instructions’. Does the layout help?
   Partly. Untrusted content never goes in the system role and is wrapped as quoted data with its source, so the model is told it is material to read, not instructions to follow. Tool results are size-capped and stripped of markup that imitates role boundaries. That reduces prompt injection but does not eliminate it, so the second defence is in the tools: anything with an effect outside the sandbox needs explicit user confirmation.

Ask: Mid-answer, the model decides to search the web and then run some code. Who runs those, and what happens if the machine driving it dies halfway?

Good answers name: Orchestrator loop with per-step persistence and a fenced run lease, Let the model server execute tools inline, A general durable workflow engine per answer (Temporal, Step Functions), Keep the loop in memory and restart the answer on failure.

Our pick: Each assistant message is a run owned by one orchestrator under a 10-second lease renewed every 2 seconds, with an epoch that fences out a paused or zombie owner. When the model emits a tool call, the orchestrator writes the step as started with the idempotency key (run, step), sends a status event to the stream, calls the tool runner with a timeout (10 s for search, 30 s for code), trims the result to a token cap, writes the step as done, and calls the model again with the prompt plus the call and its result. With conversation affinity, that second call is almost entirely a prefix-cache hit. A sweeper reassigns runs whose lease expired; the new owner reloads completed steps from the store and the streamed text from the stream, re-runs only the step in flight, and asks the model to continue from the text already shown. A run is capped at 10 tool steps and a token budget, so a looping model cannot burn unbounded GPU or search quota.

1. The code step is running when the orchestrator dies. Is it safe to run again?
   Yes, because of how the sandbox is built: a microVM with no network, keyed by conversation, whose only effects are files inside it, so running the code again produces the same files. The new owner sees the step as started but not done and repeats it with the same idempotency key. For any tool with an outside effect, such as sending an email, the step needs user confirmation and the provider call carries the (run, step) key, so a repeat is deduplicated downstream.
2. A search takes 25 seconds. What does the user see, and what does the model do?
   The stream shows ‘Searching the web’ within a few hundred milliseconds of the call, so the wait is visible, and heartbeats keep the connection alive. At the 10-second timeout the tool runner returns ‘search timed out’ as the tool result rather than an error, and the model decides whether to answer from what it knows or try a narrower query. The user gets an answer instead of a spinner that ends in a failure.
3. How does continuing ‘from the text already shown’ work?
   The resumed model call includes the prompt, the completed steps and the partial answer as the beginning of the assistant turn, and the model generates what comes next. The words will differ from what the dead run would have written, but nothing after the cut was ever shown, so the user sees a pause and then a sentence that carries on. Character offsets on each chunk let the client discard anything duplicated at the seam.
4. What stops a model from calling tools forever?
   A cap of 10 tool steps per answer, a token budget per run and a wall-clock limit, all enforced by the orchestrator, plus per-user quotas on billed tools such as search. When a cap is hit, the orchestrator asks the model for a final answer with tools disabled, so the user still gets a response. The cap-hit rate is a metric, because a spike usually means a prompt or model regression.
5. Code sessions keep files between turns. Where does that state live?
   In the conversation’s sandbox, which stays warm for an idle timeout (say 20 minutes) and is then snapshotted to object storage or discarded. A later turn reattaches to the warm sandbox or restores the snapshot. That is why sandboxes are keyed by conversation rather than by message, and why the warm pool is sized from active conversations that use code, not only from executions per second.

Ask: Most users are on the free tier, and at 7 pm the flagship model is out of GPUs. How do you enforce limits and decide who gets served?

Good answers name: Per-user, per-model budgets in price-weighted tokens over a rolling window, reserved then settled, Priority classes at the router: paid floor, free shed to fallback first, background paused first, Requests per minute per user, Separate GPU fleets per tier.

Our pick: The usage limiter keeps a budget per (user, model, 5-hour rolling window) in weighted tokens (output counts four times input, cached input a tenth), in Redis keyed by user. The chat service reserves the prompt tokens plus an estimated answer (the user’s recent median, capped by max_tokens) with one atomic script, and the orchestrator settles to actual usage at the end; a spent budget sends the request to the fallback model with a notice saying when the window resets. At the router each model pool has three classes: paid, with a 60 % floor and the right to borrow anything idle; free, with whatever is left; and background, which pauses first. When the flagship’s free-class queue time passes its target, new free requests are shed to the smaller model, and only if that pool also saturates do they queue with a visible wait and a 20-second deadline. Per-user concurrency (1 stream free, 3 paid) and fair queueing by user within a class stop one person’s parallel tabs from crowding others out.

1. The limiter’s Redis is unreachable. Fail open or closed?
   Fail open, with a small local allowance per chat-service node per user, and alert. Refusing all chat because a counter is down turns a minor incident into a full outage, while over-serving for a few minutes costs a little GPU, and the router’s classes still protect paid capacity. Usage events are on the bus, so budgets are corrected once the limiter is back.
2. How do you stop one person with fifty free accounts?
   Per-account limits do not, so add limits keyed on things that are expensive to multiply: device, phone verification for heavier free use, payment method, and IP reputation, with stricter defaults for new accounts. Behavioural signals such as identical prompts across accounts and automation patterns feed an abuse model that can drop an account to a much lower budget. The budget mechanism is unchanged; abuse handling just changes which key a cost is charged to.
3. Doesn’t falling back to a smaller model mid-conversation cause problems?
   Two, both manageable. The prefix cache is per model, so the first fallback turn pays a full prefill, and the smaller model may have a smaller context window, so the context builder re-assembles with a shorter history. Answer quality changes, which is why the UI says so and lets a paid user wait for the flagship instead. When the window resets, the conversation returns to the flagship.
4. How do you choose the 60 % floor?
   From data: the paid class’s peak demand at the latency target, plus headroom, as a share of the pool. If paid peak demand is 45 % of flagship capacity, a 60 % floor protects it with margin while guaranteeing free users 40 % and anything paid leaves idle. Revisit it weekly from shed rate and minutes in fallback per tier, which are the numbers product uses to choose between buying GPUs and tightening free limits.
5. Why reserve tokens up front instead of just counting afterwards?
   Because counting afterwards lets a user start ten long requests in parallel while the counter still says zero, and every one is admitted. A reservation makes the budget check atomic with admission. The estimate is wrong for any single request, but settlement corrects it, and on average the reservation is close to actual usage.

Ask: Every message and every answer has to be checked against policy. Do you hold the answer back until it passes?

Good answers name: Parallel checks: input during prefill, output on rolling windows, retract on a hit, Block until the complete answer is checked, Lookahead buffer: release text only after its window passes, Rely on the model’s own training to refuse.

Our pick: The input check runs in parallel with prefill: it takes about 50 ms against a prefill of a few hundred, so the orchestrator simply holds the first token until it returns and the user never notices. Output is checked on overlapping windows of about 100 tokens, sent asynchronously while streaming continues. On a hit, the orchestrator cancels the run, appends a retract event to the stream and marks the message moderated; clients replace the text with a notice, and the original goes to a restricted review store and is excluded from all future context. If the input check, the conversation or the account puts a request in a high-risk category, the orchestrator switches that run to lookahead mode and releases each window only after it passes. Thresholds are per category, and every action is logged so precision, recall and appeals can be measured.

1. You already showed the text. What does retracting actually achieve?
   It cannot make someone unsee a sentence, so the goal is bounded exposure, not zero. It removes the text from the screen and from the stored conversation, so it is not reread, shared by link or fed into later context. It also stops generation, which is what limits exposure to about one window. For categories where any exposure is unacceptable, that is exactly why lookahead exists.
2. How do you keep 150 k checks a second cheap?
   Use a small distilled classifier, batched on its own accelerators, and choose the window deliberately: 100 tokens is a few seconds of text. Skip re-checking text quoted verbatim from a source that was already checked. Send only the windows the small model scores as borderline to a larger, slower model, which is a small fraction of traffic.
3. The output moderation service is down. What happens?
   Fail toward safety in proportion to risk. For ordinary traffic, keep generating but switch to lookahead with a cheaper fallback check, accepting slower output; for high-risk categories, stop starting new answers. Never stream unchecked text for long, and page someone, because moderation being down is a policy incident, not only an outage.
4. How do you know moderation is working?
   Sample traffic for human labelling to estimate precision and recall per category, and track the false retraction rate, because wrongly retracting a harmless answer is a user-facing bug. Measure the time from an offending token to the cut at p95, which is the exposure the design promises. And red-team continuously, with adversarial prompts run against every new model and policy change before it ships.