Design GitHub Copilot

• Inline suggestions as the developer types: Grey ghost text at the cursor after a pause in typing, accepted with Tab (or word by word) and dismissed with Esc or by typing something else. A single line by default; a whole block when the cursor sits on an empty line that opens one.
• Fill in the middle: The suggestion fits between the code before and after the cursor: completing a call whose result the next line already uses, or a body whose closing brace already exists.
• Relevant to the repository: Context from open tabs, recently edited files and the definitions of symbols near the cursor, so suggestions use the project’s own names and helpers rather than generic ones.
• Typing through a suggestion: Typing characters that match the ghost text shrinks it without a new request; backspacing to a state already answered shows that suggestion again.
• Stale requests stop costing: A keystroke cancels the request in flight, and the cancellation reaches the GPU, so generation for a suggestion nobody will see stops within a step.
• Organisation policies: Per organisation: no retention of prompts or suggestions, content exclusion by repository and path, a filter that blocks suggestions matching public code, and which features are enabled.
• Sensible, safe output: Suggestions are trimmed to a syntactically sensible unit and filtered for secrets before they are shown.
• Quality measurement and model rollouts: Acceptance and code retained after acceptance, per model, language and IDE. New models ship through A/B experiments with automatic rollback.
• Out of scope: Chat and agents (a separate path to larger models; the ChatGPT question covers chat), training the models, the inference engine’s internals (the LLM Inference Service question), next-edit prediction, code review and billing.

• Latency from the last keystroke to ghost text (p50 < 300 ms · p95 < 800 ms): The requirement behind the hardest trade-off. A suggestion that arrives after the next keystroke is wasted, so the budget is set by typing speed, and it forces a small model, nearby regions, short outputs and prefix caching. The server’s share is about 120 ms at p50.
• Scale (10 M daily · 3 M at peak · 600 k req/s): A very high request rate for an LLM product, but each request is small: about 1 500 prompt tokens and 24 output tokens. Prompt tokens, not output tokens, size the fleet.
• GPU efficiency (≥ 80 % of prompt tokens from cache): About 85 % of requests produce nothing the developer keeps, so the design is judged by GPU time per accepted character. A cancelled request must stop decoding within one step.
• Availability (99.9 % · fail silent): A failure is a missing suggestion, never an error dialog or a blocked editor. Clients do not retry a failed completion; the next pause is the retry.
• Privacy (no retention for business orgs): Prompts and suggestions are processed in memory and never logged, stored or used for training. Prefix caches are scoped to one session and never shared across users or organisations.
• Policy enforcement (server side within 1 min): Content exclusion is enforced on the client and again at the edge, so a stale or modified client cannot send excluded files. The public-code filter fails closed for organisations that require it.
• Client footprint (< 10 ms per request, off the UI thread): The extension does real work (indexing, snippet scoring, tokenising) and none of it may make typing lag. A slow keystroke costs more goodwill than a missing suggestion.

• Requests per second at peak: ~600 k/s (750 k pauses/s). 3 M active developers × one debounced pause every 4 s = 750 k pauses/s. The suggestion cache answers ~20 % of them on the client (typing through, backspacing to an answered state), so ~600 k requests/s reach the servers. That is ~20 B requests a day (~230 k/s average): a huge request rate, but every request is tiny.
• Requests that end in kept code: ~15 %. 40 % of requests are cancelled by the next keystroke (240 k/s). Of the 360 k/s that complete, ~15 % are empty or filtered, leaving ~300 k/s shown, and ~30 % of those are accepted: ~90 k/s. 0.6 × 0.85 × 0.3 ≈ 15 %, about one request in seven. Waste is the normal case, so the design is judged by what a wasted request costs.
• Prompt tokens to prefill: ~900 M/s raw · ~180 M/s after the cache. 600 k/s × 1 500 prompt tokens (prefix ~900, suffix ~200, snippets and signatures ~350, header ~50) = 900 M tokens/s, sixty times the output. With ~80 % of prompt tokens served from the prefix cache, ~180 M/s are actually computed. Unlike chat, prefill is the dominant GPU load.
• Output tokens per second: ~14 M/s run to the stop · ~10 M/s with cancellation. Line mode (75 % of requests) stops at the newline after ~12 tokens and block mode (25 %) after ~60, so ~24 on average: 600 k × 24 = 14.4 M tokens/s. Cancelled requests (40 %) that stop within a step of the cancel produce about a quarter of their tokens: 14.4 M × (0.6 + 0.4 × 0.25) ≈ 10 M/s. Propagating the cancel saves ~4 M tokens/s, about 400 GPUs.
• GPUs at peak: ~6 000 (vs ~25 000 naive). One GPU per replica with a small model (~8 GB of weights at 8 bits): ~50 k prompt tokens/s of prefill and ~10 k output tokens/s of decode at a batch of ~80 with 8 ms steps. 180 M ÷ 50 k = 3 600 plus 10 M ÷ 10 k = 1 000 is 4 600, ≈ 6 000 with 30 % headroom. Without the prefix cache and cancellation: 900 M ÷ 50 k + 14.4 M ÷ 10 k ≈ 19 400, ≈ 25 000 with headroom. Each point of prefix hit rate is worth ~180 GPUs.
• Warm-prefix memory per GPU: ~50 GB. With 32 layers, 8 KV heads of 128 dimensions and an 8-bit KV cache: 2 × 32 × 8 × 128 × 1 B = 64 KB per token, so a 1 500-token prompt is ~100 MB. 3 M developers ÷ 6 000 GPUs = 500 sessions per GPU, so ~50 GB. That fits in the ~64 GB an 80 GB GPU has left after 8 GB of weights and ~8 GB for the running batch, with host RAM as the overflow tier.
• Server share of the p50 budget: ~120 ms of ~240 ms. From the last keystroke: debounce 75, client context 10, client ↔ edge round trip 30, gateway 5, edge ↔ region 20, queue 10, prefill of ~300 uncached tokens 15, decode 60 (a line of ~14 tokens at 8 ms a step, roughly halved by speculative decoding), post-processing 10, render 5: 240 ms, leaving 60 ms of the 300 ms target. The server’s part, gateway to post-processing, is ~120 ms, half of it decode.
• Public-code index per region: ~800 GB. Fingerprint the ~1 T tokens of deduplicated training code as hashes of 20-token runs, keeping only the smallest hash in each window of 21 (winnowing): ~2 ÷ 22 ≈ 1 in 10 positions, which still catches any shared run of 40 tokens (~150 characters). 100 B fingerprints × 8 B ≈ 800 GB, sharded in RAM across a few hosts per region, or a ~100 GB Bloom filter in front of an SSD store.

• Editor extension (VS Code · JetBrains · Neovim): Watches keystrokes, debounces them (about 75 ms, adapted to the developer’s rhythm) and decides whether to ask at all. Sends each request as one stream on a long-lived HTTP/2 connection and resets that stream on the next keystroke, which is where cancellation starts. Renders ghost text only if the document is still in the state the request was made for, and measures what happens to accepted text afterwards.
• Context engine (in the extension · token budget): Builds the prompt parts in under 10 ms off the UI thread: prefix and suffix around the cursor, snippets from open tabs and recent edits chosen by identifier similarity, and signatures of nearby symbols, packed by priority into a 2 048-token budget. Snippets are re-chosen when the cursor moves to a new region, not on every keystroke, which keeps prompts stable for the prefix cache.
• Workspace index (in memory · tabs + recent edits): An in-memory index of ~20-line windows from open tabs and recently edited files with their identifier sets, plus symbol signatures from the language server, updated incrementally as the developer edits. Files matching the organisation’s exclusion rules are never indexed, so they can never be chosen as context.
• Suggestion cache (in the extension · by doc state): Recent suggestions keyed by document state (a hash of the text around the cursor). Typing characters that match shown ghost text consumes it with no request, and backspacing to a state already answered shows its suggestion again. It answers about 20 % of pauses, and it is the only path where a suggestion costs no GPU at all.
• Edge gateway (HTTP/2 · auth · policy · quota): Runs in many points of presence close to developers and holds their long-lived HTTP/2 connections, so no request pays a handshake. Verifies a short-lived signed token locally, applies the organisation’s cached policy (exclusion re-checked on every path, retention, public-code filter), enforces per-user quotas and picks the nearest GPU region with headroom. Turns a client stream reset into a cancel toward the region.
• Org policy store (versioned document per org): Each organisation’s settings as one versioned document: retention off, content-exclusion patterns per repository, the public-code filter (block or allow) and enabled features. Edges cache it for 60 s, and every response carries the policy version, so a client holding an older one refreshes at once.
• Region scheduler (affinity · deadlines · arms): One per GPU region. Routes each request by its session key to the replica most likely to hold its prompt prefix, with bounded-load hashing so a hot replica spills to its neighbour, and resolves the user’s model arm from the experiment config. Answers a request that has waited past its deadline with an empty suggestion instead of serving it late, and forwards cancels to the replica.
• Experiment config (model arms · ramps · degrade): Maps (user hash, language) to a model arm with each arm’s ramp, and holds the degradation settings a region applies under load: debounce hint, block mode, draft length. Read on every request from a local copy, so a rollback or a degradation step is one config write that takes effect in seconds.
• Completion model (small FIM model · 1 GPU each): Replicas of a small fill-in-the-middle code model (~8 GB at 8 bits), one GPU each, continuously batched at about 80 sequences so a step stays near 8 ms. Renders the prompt suffix-first so typing only appends to it, applies the request’s stop rule (end of line or end of block) as it decodes, drafts tokens speculatively, and checks for cancellation at every step. One pool per model arm; chat runs on larger models in other pools.
• Prefix cache (KV blocks · HBM + host RAM): The attention state of each session’s last prompt, kept on its replica so the next request prefills only what changed; about 80 % of prompt tokens come from here. Roughly 100 MB per session and 500 sessions per GPU, with idle sessions moved to host RAM and back in a few milliseconds. Entries are scoped to one session key and never shared across users.
• Post-processor (trim · parse · filters): Turns raw output into something worth showing in about 10 ms: trims overlap with the text after the cursor, drops repeats of the previous line, rejects completions that add syntax errors and replaces strings shaped like credentials. Runs the public-code check for organisations that require it. Lives on the server so every IDE gets the same rules from one place.
• Public-code index (winnowed fingerprints · ~800 GB): Fingerprints of the deduplicated training code: hashes of 20-token runs, winnowed to about one position in ten, which still catches any shared run of 40 tokens (~150 characters). Sharded in RAM per region, about 1 ms per lookup, rebuilt and versioned with each training set. For organisations that block public-code matches, an unavailable index means no suggestion, never an unchecked one.
• Telemetry stream (Kafka · no code content): Client outcome events (shown, accepted, dismissed, retained at 30 s, 2 min and 5 min) and server timings per request: about 20 B client events a day, ~4 TB. Events carry ids, lengths, timings and similarity ratios, never code or paths, so the stream is the same for every customer and needs no exception for no-retention organisations.
• Quality pipeline (per arm · language · IDE): Joins events by suggestion id and computes the funnel per opportunity: sent, cancelled, shown, accepted, retained. Compares arms on accepted and retained characters per opportunity with latency and filter guardrails, and writes ramp or rollback decisions to the experiment config.

1. A pause in typing becomes ghost text: The main path, inside about 300 ms from the last keystroke. Every hop is chosen for latency: the prompt is built on the developer’s machine, the request goes to the nearest region over a warm connection, lands on the replica that already holds its prefix, and stops at the end of the line.
   1. A pause in typing fires the debounce, and the extension checks its suggestion cache before asking.
   2. The context engine assembles the prompt parts within the token budget.
   3. The extension sends the parts as one stream on its warm HTTP/2 connection to the nearest edge.
   4. The edge verifies the token, applies the organisation’s policy and the quota, and picks a GPU region.
   5. The region scheduler resolves the user’s model arm and routes by session key to the replica holding this file’s prefix.
   6. The replica prefills only the tokens after the cached prefix, then decodes with a draft until the line ends.
   7. The post-processor trims the completion to fit its context and runs the filters.
   8. The suggestion returns through the edge and is rendered as ghost text if the document has not moved on.
2. Type through, accept, and measure what stayed: Much of what makes completions feel instant happens with no request at all: the client consumes a shown suggestion as the developer types it. Acceptance is only the start of measurement; what counts is whether the code is still there minutes later.
   1. Ghost text is on screen; once it has been visible long enough to read, the extension counts it as shown.
   2. The developer types the first characters of the suggestion; the cache consumes them and the ghost text shrinks with no request.
   3. The developer presses Tab; the text is inserted and an accepted event is queued for telemetry.
   4. At 30 s, 2 min and 5 min the extension measures how much of the accepted text is still there.
   5. The quality pipeline joins the events by suggestion id and computes the funnel per opportunity for each arm.
   6. An arm that wins on retained characters with its latency guardrails held is ramped up in the experiment config.
3. Fast typing: cancel in flight, keep the prefix: A developer types a line in bursts, and most requests are overtaken by the next keystroke. The design has to stop paying for each one within a decode step and still reuse the prefill it already did.
   1. A 90 ms pause after “for item in” fires the debounce, and a request goes out.
   2. The scheduler places it on the session’s replica, which prefills the new tokens and starts decoding.
   3. The developer types three more characters 110 ms later, and the extension resets the request’s stream.
   4. The edge turns the reset into a cancel, and the replica drops the sequence at the next step boundary.
   5. The replica keeps r1’s prompt blocks as a cached prefix instead of freeing them.
   6. The next pause fires r2; it lands on the same replica and prefills eight tokens.
   7. At regional peak, responses carry a longer debounce hint, which removes requests before they exist.
4. A GPU region fails at peak: Losing a region removes an eighth of the capacity and every warm prefix in it at the worst moment. Its developers move to neighbouring regions with a longer round trip and cold caches, and the system sheds and degrades by deadline instead of queueing suggestions nobody will see.
   1. A GPU region stops publishing capacity, and the edges stop sending requests to it.
   2. Requests from those developers go to the next-nearest regions, with a longer round trip.
   3. Every moved session arrives with a cold prefix and pays one full prefill.
   4. Schedulers answer requests that cannot meet their deadline with an empty suggestion instead of queueing them.
   5. If load stays high, the receiving regions step down a degradation ladder kept in their config.
   6. Clients never retry a failed or empty completion; the next pause is the retry.
   7. When the region recovers, the edges shift its traffic back gradually so its caches warm without a stampede.
5. An organisation excludes files and keeps nothing: Policy has to hold even when a client is old or wrong. Exclusion is enforced twice, on the client (excluded files are never read into context) and at the edge (every path in a request is checked), and no retention is a property of the whole request path, not a flag on a log line.
   1. An organisation admin adds exclusion rules and turns on the public-code filter; the policy version goes up.
   2. Edges pick up version 43 within a minute, and each extension refreshes as soon as a response shows the version changed.
   3. The extension drops excluded files from its index and makes no request while the cursor is in one.
   4. The edge re-checks every path in each request against the rules and strips or refuses what is excluded.
   5. The request is processed in memory only: nothing about its content is logged or stored.
   6. The post-processor suppresses any suggestion that matches public code for this organisation.
   7. The organisation’s usage and filter counts reach its admin dashboard through the same code-free telemetry.

Ask: A developer types five characters a second. Why not ask for a completion on every keystroke, and what stops most of your GPU time going to suggestions nobody sees?

Good answers name: A request on every keystroke, stale responses dropped on the client, Fixed debounce with client-side discard only, Adaptive debounce, a client gate and cache, cancellation propagated to the engine, prefill kept as cache, Prefetch the next completion while one is shown.

Our pick: The extension debounces with a delay set from the developer’s own rhythm (around the 75th percentile of their gaps between keystrokes, clamped to 50–150 ms), and a cheap local gate skips positions where suggestions are rarely accepted, such as the middle of an identifier or a spot where a suggestion was just dismissed. Before sending, it checks the suggestion cache keyed by document state: typing the characters of shown ghost text consumes it, backspacing to an answered state shows it again, and about 20 % of pauses end there. Each request is one stream on a long-lived HTTP/2 connection; the next keystroke resets that stream, the edge and the scheduler propagate the cancel as a cancelled context, and the engine checks for cancellations at every step boundary, so decode stops within ~8 ms. The engine keeps the cancelled request’s prompt blocks as a cached prefix, so the request a keystroke later prefills only a few tokens. At peak, propagation saves ~4 M decode tokens a second (about 400 GPUs), and the reuse is a large part of why the prefix hit rate holds near 80 %. The numbers to watch are cancels per request, tokens decoded after a cancel arrived, and pauses answered by the cache.

1. The response arrives 5 ms after the developer typed another character. What does the client do?
   Compare versions: the request was for document version v and the editor is now at v + 1. If the typed character matches the start of the completion, show the rest, which is the typing-through rule applied to a response that raced the keystroke. Otherwise drop it and let the debounce for v + 1 fire. Never render a suggestion computed for a document state that no longer exists; ghost text that disagrees with the code is worse than none.
2. Why is an HTTP/2 stream reset better than a separate cancel request?
   It is free and immediate: one small frame on a connection that is already open, nothing to authenticate or route, and every proxy that speaks HTTP/2 or gRPC turns it into a cancelled context downstream. A separate cancel call could race the original on a different path, reach a different edge, and need its own idempotency. The cost is that every hop must actually honour cancellation: one proxy that buffers, or an engine that only checks between batches, silently turns cancellation back into a client-side discard.
3. How do you choose the debounce?
   As an experiment rather than a constant: arms at 50, 75, 100 and 150 ms and the adaptive rule, compared on accepted characters per opportunity, end-to-end latency and GPU tokens per accepted character. A shorter debounce shows suggestions sooner but cancels more; a longer one saves GPU but feels sluggish. The adaptive rule usually wins because typing rhythm varies several times between people: a fixed 75 ms means constant cancellations for a fast typist and a lag for a slow one.
4. Ten times the developers. What changes in this layer?
   Nothing breaks here, because debounce, gate and cache scale with the developers’ own machines. What grows is the GPU fleet, and the client is where the cheapest savings are: a gate or cache that is 10 % better removes 10 % of requests before any GPU sees them. At that scale I would train the gate as a tiny on-device model on acceptance data, and spend effort on the client before the server.
5. How do you know cancellation actually reaches the GPU?
   Measure the gap. For every cancelled request the engine logs the tokens decoded after the cancel arrived and the time from the client’s reset to the engine’s abort; both should be about one step, 8 ms and zero or one token. A rising number means some hop stopped propagating, such as a proxy upgrade that buffers or an engine change that checks less often. That is a GPU-cost regression users never notice, which is exactly why it needs its own alarm.

Ask: The model sees about 1 500 tokens. Which 1 500, who chooses them, and how do you make suggestions repository-aware without breaking the latency budget?

Good answers name: Prefix only, completed left to right, A client-built fill-in-the-middle prompt: prefix, suffix, similar snippets from open tabs and recent edits, symbol signatures, packed by priority, A server-side repository index with embedding retrieval, Send whole files up to a large context window.

Our pick: The context engine in the extension keeps a workspace index of ~20-line windows from open tabs and recently edited files of the same language, with their identifier sets, and gets signatures of symbols near the cursor from the language server. When the cursor enters a new region (a new function, a jump, another file) it scores the windows by Jaccard similarity of identifiers against the ~20 lines before the cursor and keeps the best, off the typing path; GitHub’s exact ranking is not public, so this is my design. Each request packs a 2 048-token budget by priority: the ~900 tokens of prefix nearest the cursor, then ~200 of suffix, then signatures, then snippets, then more distant prefix, each snippet tagged with its path. The client sends the parts, not a rendered prompt, so the server renders the model’s template and can change it with a model release. Excluded files never enter the index. Priority decides what gets in; the server’s layout decides the order, which matters for caching.

1. Why Jaccard similarity of identifiers rather than embeddings?
   It is cheap enough to run on every cursor move inside the IDE over a few thousand windows in a millisecond or two, needs no model on the client, and code that shares identifiers is usually the code that helps: the same API called the same way. Embeddings capture more meaning, but they need a model and an index, and their gain for a single-line completion is smaller than for chat retrieval. I would run embeddings as an experiment arm against Jaccard on accepted characters per opportunity rather than assume either way.
2. An organisation excludes secrets/** and a developer has secrets/keys.py open in a tab. Walk me through it.
   The context engine never indexes that file, so none of its windows can be chosen as a snippet, and no request is made while the cursor is in it. If an old or modified client sent a snippet tagged with that path anyway, the edge strips it, and a request whose current file is excluded is refused. What neither layer can catch is text copied from that file into a file that is not excluded: exclusion governs files, and the documentation should say so plainly.
3. How do you count tokens on the client when the tokenizer belongs to the model?
   Ship the tokenizer with the extension, since BPE tables run fine in process, versioned with the model family, and have the server return the tokenizer version it expects. Count with a small safety margin. If the server still finds a prompt over budget, say after a model change, it trims the far end of the prefix at a line boundary rather than failing the request.
4. A function’s definition is in a file that is not open. Is it missed?
   Mostly not: the language server resolves symbols near the cursor wherever they are defined, and the engine includes their signatures, which is most of what a completion needs. The bodies of unopened files are missed, and that is the case for a server-side index. I would add one only for repositories where offline evaluation shows a gain, and run its retrieval on cursor moves so it is never on the keystroke path.
5. How would you know a context change helps?
   Offline first: replay held-out edits from internal repositories, mask the code that was actually written, and compare exact match and edit similarity with the old and new context. Then online as an A/B arm on accepted and retained characters per opportunity, with prompt tokens and latency as guardrails, because more context that helps a little but adds 20 % to prefill may not be worth its GPUs.

Ask: Consecutive requests from one developer share about 97 % of their prompt. How do you stop paying to prefill it again 600 thousand times a second?

Good answers name: Least-loaded routing with any layout, A cache-friendly layout plus affinity by session key with bounded load, A server-side session: the client sends edits as deltas and the server keeps the document, Pin each developer to one replica for the day.

Our pick: The server renders prompts suffix first: header, snippets and signatures, then the suffix, then the prefix, then the marker for the middle, so text typed at the cursor only appends to the prompt. The FIM paper trains and evaluates both the prefix-first and suffix-first orders, so this is a training choice rather than a hack. Snippets are sticky (re-chosen only when the cursor moves to a new region, in a deterministic order), and the prefix window’s start snaps to a 32-line boundary instead of sliding a line at a time, trading slightly longer prompts for a start that rarely moves. The region scheduler hashes the session key, a hash of (installation, document), onto replicas with bounded-load consistent hashing capped at 1.25× the average, the scheme from the ChatGPT question; a spill costs one cold prefill. Idle sessions’ blocks move from GPU memory to host RAM and back in a few milliseconds, and every entry is scoped to its session key. The result is a hit rate around 80 % and a fleet of ~6 000 GPUs instead of ~25 000.

1. Why key affinity by document rather than by developer?
   Consecutive requests share a prefix only within one document; a developer who switches files gets a different header, snippets and text, so keeping them on one replica buys nothing and concentrates a heavy user’s load. Keying by document spreads one developer across replicas and lets each hold one file’s prefix. When the developer switches back to a file within a few minutes, its prefix is often still warm on that file’s replica.
2. A replica dies. What does it cost?
   Its ~500 sessions rehash to their next replicas on the ring, with the load bound spreading them out, and each pays one full prefill of ~1 500 tokens on its next request: about 35 ms instead of 15, once. In total that is ~750 k tokens, about fifteen GPU-seconds of prefill spread over several neighbours, which nobody notices. Losing a whole region is the expensive version, because it moves hundreds of thousands of sessions at once.
3. Does putting the suffix first hurt quality?
   It is a property of training, not of serving: the model has to be trained with the suffix-first format, and then evaluation decides. If it showed a loss, the fallback is the prefix-first order with a short suffix, accepting that the suffix and everything after the change are recomputed each time: a few hundred tokens per request rather than the whole prompt, so a lower but still useful hit rate.
4. How much memory do warm prefixes need, and what happens when it runs out?
   About 100 MB per session (1 500 tokens × 64 KB with an 8-bit KV cache) and ~500 sessions per GPU, so ~50 GB, which fits beside 8 GB of weights and ~8 GB for the running batch on an 80 GB GPU. Beyond that, idle sessions move to host RAM, where a host has a terabyte or more, and come back over PCIe in a few milliseconds. Eviction is least recently used; a developer idle for ten minutes loses the prefix and pays one cold prefill.
5. Could two developers share cache entries, say for a popular open-source file?
   The bytes might be identical, but sharing across users creates a timing side channel (a fast response reveals that someone else had that exact prompt), and a business customer’s code must not influence another tenant’s requests in any way. The only safely shareable part is the template’s fixed tokens, which are tiny. So entries are scoped to the session key and cross-session reuse is limited to those fixed tokens.

Ask: Your best model writes better code. Why not use it for inline completions?

Good answers name: The large chat model for everything, A small FIM model on one GPU, short max length, stop at end of line or block, speculative decoding, nearest region, A cascade: the small model first, a large model for blocks on long pauses, Run the model on the developer’s machine.

Our pick: The p50 budget from the last keystroke: debounce 75 ms, context on the client 10, client ↔ edge 30, gateway 5, edge ↔ region 20, queue 10, prefill 15, decode 60, post-processing 10, render 5, which is 240 ms with 60 ms of slack, and the server’s share is ~120 ms. The model is a small FIM-trained code model (~8 GB at 8 bits) on one GPU per replica, with batches held near 80 sequences so a step stays around 8 ms. Each request carries a mode: line (stop at the newline, at most 64 tokens) or block (on an empty line that opens a block; stop when the indentation returns to the block’s level or the brackets balance, at most 256), and the engine checks the stop rule as tokens are produced so the GPU stops early. Speculative decoding drafts a few tokens per step with a tiny draft model or with n-grams copied from the prompt, since code repeats its own identifiers, roughly halving the steps on code; the draft length is lowered as batches grow, because verification then competes for compute, and the 60 ms of slack covers losing it at peak. There is no streaming to the client: a line is ~12 tokens and fits in one small response. Every hop’s time is logged per request so a p95 regression can be pinned to a step.

1. If p50 is 240 ms, where does a p95 of 800 ms come from?
   Mostly four places: block completions (~60 tokens, ~250 ms of decode), cold prefixes after a file switch or a replica move (a full 1 500-token prefill in a busy batch), queueing at peak, and the developer’s own network (hotel Wi-Fi, a VPN hairpin, 150 ms or more of round trip). Each has a different fix, which is why per-step timings exist: capacity for queueing, sticky snippets and affinity for cold prefixes, nearer regions for the network, and nothing at all for blocks if 800 ms is acceptable for them.
2. Why one GPU per replica rather than a big tensor-parallel replica?
   The model fits in 8 GB, so tensor parallelism would only add communication to every step with no memory benefit, and it would make each replica a bigger failure domain. With one GPU per replica there are about 6 000 small, identical units, bounded-load hashing across them is simple, and losing one touches ~500 sessions rather than thousands.
3. How does speculative decoding work here, and when would you turn it off?
   A cheap drafter proposes the next few tokens, either a tiny model or a lookup of n-grams that already appear in the prompt, which works well on code because names repeat, and the main model verifies them all in one step, keeping the longest correct run, so the output is unchanged. It cuts steps per suggestion, which is latency. At large batch sizes verifying several tokens per sequence makes each step compute-bound and slower, so the scheduler shortens the draft as batches grow and turns it off if step time rises.
4. Isn’t stopping at the end of the line too conservative?
   Mid-line it is what the developer wants: they are finishing a statement. Block mode covers the case where a whole body is useful, and it is triggered by position (an empty line after a signature or a block opener) rather than by default, because a 60-token suggestion takes five times the decode and is accepted less often per token. The split is itself an experiment: the share of block mode and its retained characters decide whether to widen it.
5. What would you change at ten times the developers?
   The per-request design holds and the fleet grows to tens of thousands of GPUs, so efficiency compounds: a better client gate, a higher prefix hit rate (each point is ~180 GPUs today), 4-bit weights where quality allows, and distilled per-language models where a language’s volume justifies one. And more regions closer to developers, since network is about 50 ms of the 240 ms p50 and only geography fixes it.

Ask: The model returned 40 tokens. What do you actually show, and how do you make sure it is not someone else’s code or a leaked key?

Good answers name: Show raw output and let the developer edit, Trim to a syntactic unit, remove suffix overlap, check the parse, A public-code filter on winnowed fingerprints in a regional index, A model-based security reviewer on every suggestion.

Our pick: The engine applies the mode’s stop rule while it decodes, so most outputs already end where they should. The post-processor then trims overlap with the suffix (a completion ending in the “)” that already follows the cursor loses it), drops a completion that repeats the previous line, trims trailing whitespace, and parses prefix + completion + suffix with an incremental parser, rejecting a completion that adds syntax errors. For the public-code filter, the index holds fingerprints of the deduplicated training set: a hash of every run of 20 tokens, keeping only the smallest hash in each window of 21 (winnowing), which stores about one position in ten and guarantees catching any shared run of 40 tokens, roughly 150 characters. A suggestion is checked together with the ~40 tokens of prefix before it, and any match means nothing is shown for organisations that block matches. Secrets are caught by patterns for known credential formats plus an entropy check on string literals, and a hit replaces the literal with a placeholder or drops the suggestion. The rules and the index are versioned with the model they protect, and the rules live on the server so every IDE changes at once.

1. Why check the context along with the completion?
   Because a 13-token completion cannot match 40 tokens of anything by itself, yet it can be the continuation of a public passage the developer is retyping or pasted the start of. Checking the window that spans the end of the prefix and the completion catches a suggestion that would extend a verbatim copy. The cost is false positives on boilerplate everyone writes, which is why the window is long and the index is built from deduplicated training data.
2. The index is 800 GB. Where does it live, and what if it is down?
   Sharded by fingerprint across a few memory-optimised hosts per region, or as a ~100 GB Bloom filter on each post-processing host with exact checks only on Bloom hits, which are rare. If it is unavailable, organisations that block matches get no suggestions (fail closed) rather than unfiltered ones, because the setting is a promise; everyone else is unaffected. It is rebuilt offline whenever the training set changes.
3. How do you trim to a sensible unit in a language you cannot parse?
   Fall back to generic rules: stop at the newline in line mode, and in block mode track bracket balance and indentation relative to the line where the block started, which works for most C-family and indentation-based languages. The parse check is skipped where there is no grammar. Because the rules live on the server, adding a grammar improves every IDE at once.
4. A suggestion contains what looks like an AWS key. Where did it come from, and what do you do?
   Either it was memorised from training data or it was copied from the developer’s own context, such as a key in an open file. Either way it should not be suggested: known formats and high-entropy literals are replaced with a placeholder or the suggestion is dropped. Prevention is upstream too: the context engine skips files such as .env and key files by default, and training data is scrubbed of secrets. Counting hits by source tells you which upstream fix matters most.
5. How do you know the filters are not hiding good suggestions?
   Log how often each filter fires and at what kind of position, and run holdbacks where a small share of traffic has a non-policy filter (a trimming rule) relaxed, comparing retained characters. The public-code and secret filters are promises, so they are never relaxed online; their false positives are measured offline on labelled samples.

Ask: You trained a new completion model. How do you know it is better, and how do you ship it to 10 million developers without making their editors worse?

Good answers name: Offline benchmarks only, Acceptance rate per shown suggestion as the ship metric, A/B by user on accepted and retained characters per opportunity, with latency and filter guardrails, Shadow traffic: run the new model on live requests without showing it.

Our pick: A new model first passes offline infilling and held-out-repository evaluations, then shadows ~1 % of requests to measure latency and GPU cost on real prompts, then enters an A/B experiment: users are assigned by hashing their id with the experiment’s salt (the A/B Testing Platform question covers the mechanics), so each developer sees one model consistently, and each arm gets its own replica pool because prefix caches are per model. The primary metric is accepted characters per opportunity and the decision metric is characters still present at 5 minutes per opportunity, measured on the client as edit similarity over the accepted range and sent only as numbers. Guardrails are p50 and p95 latency, empty and filtered rates, cancel rate and errors, sliced by language and IDE, and a breach sets the arm to 0 % in the experiment config, which the scheduler reads on every request, so rollback takes seconds. The ramp is 1, 5, 25, 50 and 100 % per language over days, and the old model’s replicas stay warm until the new one has held at 100 % for several days.

1. Why measure retention on the client rather than the server?
   Only the editor sees what happens to the code after it is inserted: edits, deletions, undo. The server sees nothing until the file is committed, if ever, and for no-retention organisations it must not see the code at all. The extension follows the inserted range through change events and sends a similarity ratio at fixed times, so the measurement needs no code to leave the machine.
2. Faster models get accepted more. How do you separate quality from speed?
   Decide on the combination developers actually get, because that is the product, and per-opportunity metrics fold latency in naturally. To understand why an arm wins, slice by latency bucket: comparing suggestions shown at the same latency isolates quality. Sometimes the right decision is the faster, slightly weaker model, and the metric should allow that.
3. A new model is better overall but worse for one language. Ship it?
   Not as one decision. The experiment config maps language to arm, so ship the new model where it wins and keep the old one where it loses while the training team looks at why. Language is a first-class slice in the guardrails, because a regression in a smaller language disappears inside a global average.
4. How do you roll back in the middle of the day?
   One write to the experiment config, which maps (user hash, language) to an arm and is read by the scheduler on every request, so it takes effect within seconds. The old model’s replicas are kept warm until the new one has held at 100 % for days, so there is capacity to roll back to. Nothing needs draining, since each request lives about a hundred milliseconds.
5. What is a sign the metric is being gamed?
   Acceptance rising while characters per opportunity or retention fall, which means the model is showing shorter, safer suggestions. Or suggestions shown per pause falling, which means it is slower or returns empty more often. Watching the whole funnel per opportunity (sent, cancelled, shown, accepted, retained) rather than one ratio makes those shifts visible.