Design GitHub Copilot

Inline AI code suggestions for 10 M developers, shown within 300 ms of a pause in typing, when most requests are never used.

Last updated 2026-10-27. Difficulty: hard. Patterns: latency-budget, cancellation, prefix-caching, context-assembly, code-completion. Reported at Microsoft, OpenAI, Anthropic, Google, Meta, Databricks.

Functional requirements

• Inline suggestions as the developer types. Grey ghost text at the cursor after a pause in typing, accepted with Tab (or word by word) and dismissed with Esc or by typing something else. A single line by default; a whole block when the cursor sits on an empty line that opens one.
• Fill in the middle. The suggestion fits between the code before and after the cursor: completing a call whose result the next line already uses, or a body whose closing brace already exists.
• Relevant to the repository. Context from open tabs, recently edited files and the definitions of symbols near the cursor, so suggestions use the project’s own names and helpers rather than generic ones.
• Typing through a suggestion. Typing characters that match the ghost text shrinks it without a new request; backspacing to a state already answered shows that suggestion again.
• Stale requests stop costing. A keystroke cancels the request in flight, and the cancellation reaches the GPU, so generation for a suggestion nobody will see stops within a step.
• Organisation policies. Per organisation: no retention of prompts or suggestions, content exclusion by repository and path, a filter that blocks suggestions matching public code, and which features are enabled.
• Sensible, safe output. Suggestions are trimmed to a syntactically sensible unit and filtered for secrets before they are shown.
• Quality measurement and model rollouts. Acceptance and code retained after acceptance, per model, language and IDE. New models ship through A/B experiments with automatic rollback.
• Out of scope. Chat and agents (a separate path to larger models; the ChatGPT question covers chat), training the models, the inference engine’s internals (the LLM Inference Service question), next-edit prediction, code review and billing.

Non-functional requirements

• Latency from the last keystroke to ghost text (p50 < 300 ms · p95 < 800 ms). The requirement behind the hardest trade-off. A suggestion that arrives after the next keystroke is wasted, so the budget is set by typing speed, and it forces a small model, nearby regions, short outputs and prefix caching. The server’s share is about 120 ms at p50.
• Scale (10 M daily · 3 M at peak · 600 k req/s). A very high request rate for an LLM product, but each request is small: about 1 500 prompt tokens and 24 output tokens. Prompt tokens, not output tokens, size the fleet.
• GPU efficiency (≥ 80 % of prompt tokens from cache). About 85 % of requests produce nothing the developer keeps, so the design is judged by GPU time per accepted character. A cancelled request must stop decoding within one step.
• Availability (99.9 % · fail silent). A failure is a missing suggestion, never an error dialog or a blocked editor. Clients do not retry a failed completion; the next pause is the retry.
• Privacy (no retention for business orgs). Prompts and suggestions are processed in memory and never logged, stored or used for training. Prefix caches are scoped to one session and never shared across users or organisations.
• Policy enforcement (server side within 1 min). Content exclusion is enforced on the client and again at the edge, so a stale or modified client cannot send excluded files. The public-code filter fails closed for organisations that require it.
• Client footprint (< 10 ms per request, off the UI thread). The extension does real work (indexing, snippet scoring, tokenising) and none of it may make typing lag. A slow keystroke costs more goodwill than a missing suggestion.

Back-of-envelope estimates

• Requests per second at peak: ~600 k/s (750 k pauses/s). 3 M active developers × one debounced pause every 4 s = 750 k pauses/s. The suggestion cache answers ~20 % of them on the client (typing through, backspacing to an answered state), so ~600 k requests/s reach the servers. That is ~20 B requests a day (~230 k/s average): a huge request rate, but every request is tiny.
• Requests that end in kept code: ~15 %. 40 % of requests are cancelled by the next keystroke (240 k/s). Of the 360 k/s that complete, ~15 % are empty or filtered, leaving ~300 k/s shown, and ~30 % of those are accepted: ~90 k/s. 0.6 × 0.85 × 0.3 ≈ 15 %, about one request in seven. Waste is the normal case, so the design is judged by what a wasted request costs.
• Prompt tokens to prefill: ~900 M/s raw · ~180 M/s after the cache. 600 k/s × 1 500 prompt tokens (prefix ~900, suffix ~200, snippets and signatures ~350, header ~50) = 900 M tokens/s, sixty times the output. With ~80 % of prompt tokens served from the prefix cache, ~180 M/s are actually computed. Unlike chat, prefill is the dominant GPU load.
• Output tokens per second: ~14 M/s run to the stop · ~10 M/s with cancellation. Line mode (75 % of requests) stops at the newline after ~12 tokens and block mode (25 %) after ~60, so ~24 on average: 600 k × 24 = 14.4 M tokens/s. Cancelled requests (40 %) that stop within a step of the cancel produce about a quarter of their tokens: 14.4 M × (0.6 + 0.4 × 0.25) ≈ 10 M/s. Propagating the cancel saves ~4 M tokens/s, about 400 GPUs.
• GPUs at peak: ~6 000 (vs ~25 000 naive). One GPU per replica with a small model (~8 GB of weights at 8 bits): ~50 k prompt tokens/s of prefill and ~10 k output tokens/s of decode at a batch of ~80 with 8 ms steps. 180 M ÷ 50 k = 3 600 plus 10 M ÷ 10 k = 1 000 is 4 600, ≈ 6 000 with 30 % headroom. Without the prefix cache and cancellation: 900 M ÷ 50 k + 14.4 M ÷ 10 k ≈ 19 400, ≈ 25 000 with headroom. Each point of prefix hit rate is worth ~180 GPUs.
• Warm-prefix memory per GPU: ~50 GB. With 32 layers, 8 KV heads of 128 dimensions and an 8-bit KV cache: 2 × 32 × 8 × 128 × 1 B = 64 KB per token, so a 1 500-token prompt is ~100 MB. 3 M developers ÷ 6 000 GPUs = 500 sessions per GPU, so ~50 GB. That fits in the ~64 GB an 80 GB GPU has left after 8 GB of weights and ~8 GB for the running batch, with host RAM as the overflow tier.
• Server share of the p50 budget: ~120 ms of ~240 ms. From the last keystroke: debounce 75, client context 10, client ↔ edge round trip 30, gateway 5, edge ↔ region 20, queue 10, prefill of ~300 uncached tokens 15, decode 60 (a line of ~14 tokens at 8 ms a step, roughly halved by speculative decoding), post-processing 10, render 5: 240 ms, leaving 60 ms of the 300 ms target. The server’s part, gateway to post-processing, is ~120 ms, half of it decode.
• Public-code index per region: ~800 GB. Fingerprint the ~1 T tokens of deduplicated training code as hashes of 20-token runs, keeping only the smallest hash in each window of 21 (winnowing): ~2 ÷ 22 ≈ 1 in 10 positions, which still catches any shared run of 40 tokens (~150 characters). 100 B fingerprints × 8 B ≈ 800 GB, sharded in RAM across a few hosts per region, or a ~100 GB Bloom filter in front of an SSD store.

Components

• Editor extension (VS Code · JetBrains · Neovim): Watches keystrokes, debounces them (about 75 ms, adapted to the developer’s rhythm) and decides whether to ask at all. Sends each request as one stream on a long-lived HTTP/2 connection and resets that stream on the next keystroke, which is where cancellation starts. Renders ghost text only if the document is still in the state the request was made for, and measures what happens to accepted text afterwards.
• Context engine (in the extension · token budget): Builds the prompt parts in under 10 ms off the UI thread: prefix and suffix around the cursor, snippets from open tabs and recent edits chosen by identifier similarity, and signatures of nearby symbols, packed by priority into a 2 048-token budget. Snippets are re-chosen when the cursor moves to a new region, not on every keystroke, which keeps prompts stable for the prefix cache.
• Workspace index (in memory · tabs + recent edits): An in-memory index of ~20-line windows from open tabs and recently edited files with their identifier sets, plus symbol signatures from the language server, updated incrementally as the developer edits. Files matching the organisation’s exclusion rules are never indexed, so they can never be chosen as context.
• Suggestion cache (in the extension · by doc state): Recent suggestions keyed by document state (a hash of the text around the cursor). Typing characters that match shown ghost text consumes it with no request, and backspacing to a state already answered shows its suggestion again. It answers about 20 % of pauses, and it is the only path where a suggestion costs no GPU at all.
• Edge gateway (HTTP/2 · auth · policy · quota): Runs in many points of presence close to developers and holds their long-lived HTTP/2 connections, so no request pays a handshake. Verifies a short-lived signed token locally, applies the organisation’s cached policy (exclusion re-checked on every path, retention, public-code filter), enforces per-user quotas and picks the nearest GPU region with headroom. Turns a client stream reset into a cancel toward the region.
• Org policy store (versioned document per org): Each organisation’s settings as one versioned document: retention off, content-exclusion patterns per repository, the public-code filter (block or allow) and enabled features. Edges cache it for 60 s, and every response carries the policy version, so a client holding an older one refreshes at once.
• Region scheduler (affinity · deadlines · arms): One per GPU region. Routes each request by its session key to the replica most likely to hold its prompt prefix, with bounded-load hashing so a hot replica spills to its neighbour, and resolves the user’s model arm from the experiment config. Answers a request that has waited past its deadline with an empty suggestion instead of serving it late, and forwards cancels to the replica.
• Experiment config (model arms · ramps · degrade): Maps (user hash, language) to a model arm with each arm’s ramp, and holds the degradation settings a region applies under load: debounce hint, block mode, draft length. Read on every request from a local copy, so a rollback or a degradation step is one config write that takes effect in seconds.
• Completion model (small FIM model · 1 GPU each): Replicas of a small fill-in-the-middle code model (~8 GB at 8 bits), one GPU each, continuously batched at about 80 sequences so a step stays near 8 ms. Renders the prompt suffix-first so typing only appends to it, applies the request’s stop rule (end of line or end of block) as it decodes, drafts tokens speculatively, and checks for cancellation at every step. One pool per model arm; chat runs on larger models in other pools.
• Prefix cache (KV blocks · HBM + host RAM): The attention state of each session’s last prompt, kept on its replica so the next request prefills only what changed; about 80 % of prompt tokens come from here. Roughly 100 MB per session and 500 sessions per GPU, with idle sessions moved to host RAM and back in a few milliseconds. Entries are scoped to one session key and never shared across users.
• Post-processor (trim · parse · filters): Turns raw output into something worth showing in about 10 ms: trims overlap with the text after the cursor, drops repeats of the previous line, rejects completions that add syntax errors and replaces strings shaped like credentials. Runs the public-code check for organisations that require it. Lives on the server so every IDE gets the same rules from one place.
• Public-code index (winnowed fingerprints · ~800 GB): Fingerprints of the deduplicated training code: hashes of 20-token runs, winnowed to about one position in ten, which still catches any shared run of 40 tokens (~150 characters). Sharded in RAM per region, about 1 ms per lookup, rebuilt and versioned with each training set. For organisations that block public-code matches, an unavailable index means no suggestion, never an unchecked one.
• Telemetry stream (Kafka · no code content): Client outcome events (shown, accepted, dismissed, retained at 30 s, 2 min and 5 min) and server timings per request: about 20 B client events a day, ~4 TB. Events carry ids, lengths, timings and similarity ratios, never code or paths, so the stream is the same for every customer and needs no exception for no-retention organisations.
• Quality pipeline (per arm · language · IDE): Joins events by suggestion id and computes the funnel per opportunity: sent, cancelled, shown, accepted, retained. Compares arms on accepted and retained characters per opportunity with latency and filter guardrails, and writes ramp or rollback decisions to the experiment config.

User flows

1. A pause in typing becomes ghost text. The main path, inside about 300 ms from the last keystroke. Every hop is chosen for latency: the prompt is built on the developer’s machine, the request goes to the nearest region over a warm connection, lands on the replica that already holds its prefix, and stops at the end of the line.
   1. A pause in typing fires the debounce, and the extension checks its suggestion cache before asking. The debounce is about 75 ms, set from the developer’s own gaps between keystrokes and clamped to 50–150 ms. A cache hit (the document is in a state already answered) shows a suggestion with no request at all. A cheap local gate also skips positions where suggestions are rarely accepted, such as the middle of an identifier.
   2. The context engine assembles the prompt parts within the token budget. About 900 tokens of prefix and 200 of suffix around the cursor, plus ~350 tokens of snippets and symbol signatures that were chosen when the cursor entered this region; this step only reads them. It all runs off the UI thread in under 10 ms, so typing never lags. Excluded files are not in the index, so they cannot be picked.
   3. The extension sends the parts as one stream on its warm HTTP/2 connection to the nearest edge. The client sends parts, not a rendered prompt, so the server owns the model’s template and can change it with a model release. One connection per editor and one stream per request means a cancel is a stream reset, with no handshake and no new request. The session key lets the region route by affinity without the server keeping any session state.
   4. The edge verifies the token, applies the organisation’s policy and the quota, and picks a GPU region. The token is signed and carries the user, plan and organisation, so it is checked locally with no network call. The policy is cached for 60 s: every path in the request is re-checked against the exclusion rules, and the retention and public-code settings travel with the request. The region is the nearest one with headroom, from a capacity feed each region publishes every second. All of it costs about 5 ms.
   5. The region scheduler resolves the user’s model arm and routes by session key to the replica holding this file’s prefix. Bounded-load consistent hashing on the session key, the scheme in the ChatGPT question: the same document goes to the same replica unless that replica is above 1.25× the average load, in which case it spills to the next one and pays one cold prefill. The request carries a deadline, and one that has waited too long in the queue is answered empty rather than late.
   6. The replica prefills only the tokens after the cached prefix, then decodes with a draft until the line ends. The prompt is rendered suffix-first, so the typing since the last request only appended to it: here 1 456 of 1 480 tokens (91 full blocks) are already cached and prefill takes a few milliseconds. Line mode stops at the newline, checked as tokens are produced, so the GPU never decodes past the end of the statement. A drafter proposes a few tokens per step and the model verifies them in one pass.
   7. The post-processor trims the completion to fit its context and runs the filters. It drops a closing bracket that already follows the cursor, rejects a repeat of the previous line, parses prefix + completion + suffix to make sure no new syntax error appears, and replaces anything shaped like a credential. This organisation blocks public-code matches, so the completion plus the ~40 tokens before it are fingerprinted and checked against the regional index in about a millisecond.
   8. The suggestion returns through the edge and is rendered as ghost text if the document has not moved on. The client compares doc_version. If the developer typed in the last few milliseconds and the characters match the start of the completion, it shows the rest; otherwise it drops it. The suggestion goes into the cache keyed by document state. At p50 this is about 240 ms after the last keystroke, half of it on the server.
2. Type through, accept, and measure what stayed. Much of what makes completions feel instant happens with no request at all: the client consumes a shown suggestion as the developer types it. Acceptance is only the start of measurement; what counts is whether the code is still there minutes later.
   1. Ghost text is on screen; once it has been visible long enough to read, the extension counts it as shown. A suggestion overtaken by typing within ~300 ms was never really seen, and counting it as shown and then rejected would make every fast typist look like a reluctant one. The suggestion id, model arm and time shown stay on the client until the suggestion is resolved.
   2. The developer types the first characters of the suggestion; the cache consumes them and the ghost text shrinks with no request. Each matching keystroke removes a character from the front of the ghost text and no debounce fires, so typing through costs nothing and feels instant. A keystroke that does not match dismisses the suggestion, and the normal path resumes at the next pause. This is most of the ~20 % of pauses the cache answers.
   3. The developer presses Tab; the text is inserted and an accepted event is queued for telemetry. Events are batched and sent every 30 s on the same connection. They describe the suggestion (id, arm, language, length, time shown, characters typed through) but never contain its text or the file path, so the stream is identical for every customer, with or without retention.
   4. At 30 s, 2 min and 5 min the extension measures how much of the accepted text is still there. The editor’s change events move the accepted range as the developer edits around it, and the extension computes the edit similarity between what was inserted and what that range holds now. Only the ratio leaves the machine. Acceptance counts a Tab; retention counts code that stayed, which is much closer to whether it helped.
   5. The quality pipeline joins the events by suggestion id and computes the funnel per opportunity for each arm. Per pause: sent, cancelled, shown, accepted, and characters retained at 5 minutes, sliced by arm, language and IDE. Per opportunity rather than per shown suggestion, because a model that shows fewer, safer suggestions raises its acceptance rate while helping less, and a slower model is shown less often.
   6. An arm that wins on retained characters with its latency guardrails held is ramped up in the experiment config. The ramp goes 1, 5, 25, 50 and 100 % over days and per language, because a model can be better overall and worse for one language. A guardrail breach (p95 latency, empty rate, filter hits) sets the arm back to 0 % automatically.
3. Fast typing: cancel in flight, keep the prefix. A developer types a line in bursts, and most requests are overtaken by the next keystroke. The design has to stop paying for each one within a decode step and still reuse the prefill it already did.
   1. A 90 ms pause after “for item in” fires the debounce, and a request goes out. The developer is mid-line, typing in bursts with about 150 ms between keystrokes, so this request has a good chance of being overtaken before it returns. Sending it anyway is the bet the product makes: if the developer stops here, the suggestion is ready.
   2. The scheduler places it on the session’s replica, which prefills the new tokens and starts decoding. Prefill runs first and is short. The KV blocks it writes describe the prompt as it stands, which is exactly the prefix the next request will need, so this is the part of the request worth keeping whatever happens next.
   3. The developer types three more characters 110 ms later, and the extension resets the request’s stream. This is how 40 % of requests end. A stream reset is one small frame on a connection that is already open: nothing to authenticate, route or deduplicate, and it cannot race the original request down a different path.
   4. The edge turns the reset into a cancel, and the replica drops the sequence at the next step boundary. Each hop maps the cancelled stream to a cancelled context downstream, and the engine checks for cancellations between steps, so decoding stops within about 8 ms. Without this the replica would decode the rest of the line for nobody: across 240 k cancels a second, ~4 M tokens a second, or about 400 GPUs.
   5. The replica keeps r1’s prompt blocks as a cached prefix instead of freeing them. Cancelled work is not wasted work: the next prompt is r1’s prompt plus the characters just typed. If those characters happen to be what r1 had started to generate, even its generated tokens match the new prompt and are reused. Only blocks past the point where the two texts differ are freed.
   6. The next pause fires r2; it lands on the same replica and prefills eight tokens. Same session key, same replica, and a prompt that differs from r1’s only at its end. Time to the suggestion is then almost all decode, which is how a line can appear about 240 ms after the last keystroke even though every request is built from scratch on the client.
   7. At regional peak, responses carry a longer debounce hint, which removes requests before they exist. The cheapest request is the one never sent. Raising the debounce from about 75 to 150 ms at peak removes roughly a quarter of requests, mostly ones that would have been cancelled anyway, at the cost of suggestions appearing ~75 ms later. The hint comes from the region’s degradation config and lifts when load falls.
4. A GPU region fails at peak. Losing a region removes an eighth of the capacity and every warm prefix in it at the worst moment. Its developers move to neighbouring regions with a longer round trip and cold caches, and the system sheds and degrades by deadline instead of queueing suggestions nobody will see.
   1. A GPU region stops publishing capacity, and the edges stop sending requests to it. Each region’s scheduler publishes free capacity and health every second, and an edge treats a region that misses three updates, or whose error rate spikes, as down. With eight regions this removes about 750 GPUs and the warm prefixes of ~375 k developers at once.
   2. Requests from those developers go to the next-nearest regions, with a longer round trip. Failover is per request, because nothing about a session lives anywhere a request must return to, so there is no state to migrate. The round trip grows by about 30 ms and p50 moves from ~240 to ~270 ms, still inside the target.
   3. Every moved session arrives with a cold prefix and pays one full prefill. 375 k sessions × 1 500 tokens is ~560 M tokens of extra prefill, most of it in the next ten seconds as each developer pauses: about a third more prefill in each receiving region for that window. After it, the steady extra load is ~15 % per region, inside the 30 % headroom.
   4. Schedulers answer requests that cannot meet their deadline with an empty suggestion instead of queueing them. A suggestion that arrives after the developer has moved on is worth nothing and still costs a full generation. Dropping at the head of the queue keeps the queue short, so the requests that are served are served in time. The client sees nothing, which is exactly what a late suggestion would have looked like anyway.
   5. If load stays high, the receiving regions step down a degradation ladder kept in their config. In order: raise the debounce hint (about a quarter fewer requests), switch block mode to line mode (block requests are a quarter of requests but ~60 % of decode tokens), shorten the draft if steps slow down, and finally shed free-plan traffic before business traffic. Each step is chosen for GPU time saved per point of quality lost, and each is undone automatically as headroom returns.
   6. Clients never retry a failed or empty completion; the next pause is the retry. A retry would arrive after its moment had passed and would double the load exactly when capacity is short. Errors are silent in the editor, and the status icon shows a degraded state only after many consecutive failures, so a regional incident looks like slightly fewer suggestions, not a broken tool.
   7. When the region recovers, the edges shift its traffic back gradually so its caches warm without a stampede. Moving all 375 k sessions back at once would repeat the cold-prefill spike in the other direction. The region’s weight ramps from 10 to 100 % over about 5 minutes, so each slice of sessions pays its one cold prefill while the rest stay warm where they are.
5. An organisation excludes files and keeps nothing. Policy has to hold even when a client is old or wrong. Exclusion is enforced twice, on the client (excluded files are never read into context) and at the edge (every path in a request is checked), and no retention is a property of the whole request path, not a flag on a log line.
   1. An organisation admin adds exclusion rules and turns on the public-code filter; the policy version goes up. Policy is one versioned document per organisation, with exclusion patterns per repository. Business plans have no retention of prompts or suggestions by default, so that part needs no setting: it is how the serving path is built.
   2. Edges pick up version 43 within a minute, and each extension refreshes as soon as a response shows the version changed. Edges cache each organisation’s policy for 60 s. Every completion response carries the policy version, so a client holding version 42 refetches on its next request rather than waiting for its 10-minute poll.
   3. The extension drops excluded files from its index and makes no request while the cursor is in one. This is the first line of defence and the strongest one: text that is never read into context cannot be sent. Rules match repository-relative paths, so the same rule works in every IDE and on every machine.
   4. The edge re-checks every path in each request against the rules and strips or refuses what is excluded. An old or modified client could still send excluded text, so the server enforces the rule too: snippets tagged with an excluded path are removed, and a request whose current file is excluded is refused. The server can only check the paths a client reports, so organisations with exclusion rules refuse clients older than the version that supports them.
   5. The request is processed in memory only: nothing about its content is logged or stored. Logs carry ids, sizes and timings, never text or paths. Prefix-cache blocks live in GPU or host RAM, are scoped to the session key and are evicted after about 10 idle minutes, and core dumps on GPU hosts are disabled. No retention is a property of every component on the path, because one debug log line that prints prompts would break it.
   6. The post-processor suppresses any suggestion that matches public code for this organisation. The check covers the completion plus the ~40 tokens before it, since a short completion can extend a public passage the developer started typing. A match means nothing is shown, not a trimmed variant. If the index is unreachable, this organisation gets no suggestion rather than an unchecked one.
   7. The organisation’s usage and filter counts reach its admin dashboard through the same code-free telemetry. Admins see active users, acceptance and how often the public-code filter fired, aggregated per organisation. Because events never contain code, the analytics path needs no exception for no-retention customers.

Deep dives

The theory behind it

• Serving LLMs on GPUs: The mechanics behind fast, affordable model serving: prefill and decode phases, the KV cache and paged attention, continuous batching, quantisation, tensor and pipeline parallelism, speculative decoding, prefix caching, routing and autoscaling GPU fleets, and the metrics that matter.
• Tail latency, percentiles and queueing: Why p99 matters more than the average, how fan-out amplifies tail latency, Little's law and queueing near full utilisation, hedged requests, timeouts and deadlines, and how to set and measure latency budgets.
• Building LLM features: serving, latency and cost: System design for LLM-powered products: streaming responses, the cost and latency of tokens, batching and KV caches in inference, prompt caching, rate limits and fallbacks, guardrails, evaluation, and when to use RAG, tools or fine-tuning.

Related

• All 83 system design interview questions
• Run this question for a friend: the interviewer kit