Design a RAG System

An assistant that answers employees’ questions from their company’s documents, with citations, only from what each user may open, minutes after edits.

Last updated 2026-10-26. Difficulty: hard. Patterns: rag, vector-search, hybrid-search, permissions, multi-tenancy. Reported at OpenAI, Anthropic, Google, Microsoft, Databricks, Notion.

Functional requirements

• Ask a question, get a cited answer. The answer starts streaming within about 2 seconds, and every claim links to the passage it came from: document title, section and a deep link into the source app. No citation, no claim.
• Connect a company’s sources. An admin authorises wikis, ticket trackers, drive folders and chat (Confluence, Jira, Google Drive, SharePoint, Slack) once; we backfill everything, then keep it in sync. Every source has its own API, auth, change feed and rate limits.
• Answer only from what this user may open. Permissions come from the sources and the identity provider: page restrictions, folder sharing, private channels, nested groups. The user learns nothing, not even that a document exists, that they could not open in the source app.
• Reflect edits, deletes and permission changes within minutes. An answer that quotes last year’s travel policy or a deleted page is worse than no answer. Revocations matter more than edits, because a stale permission is a leak and a stale edit is only a wrong answer.
• Follow-up questions. “What about contractors?” only means something after the previous turn, so a follow-up is rewritten into a standalone question before retrieval. Storing conversations is the ChatGPT question and is not repeated here.
• Say so when the documents do not answer. The model must not fill gaps from its training data. “I couldn’t find this in the documents you can access” is a correct answer, and the eval set includes questions whose right answer is exactly that.
• Feedback and admin insight. Thumbs up or down with a reason on every answer, and a per-tenant report of unanswered questions, which tells an admin which documentation is missing.
• Out of scope. Serving the model (the LLM Inference Service question treats it in depth), resumable streams and conversation trees (ChatGPT), training or fine-tuning, agents that act in source apps, and web search.

Non-functional requirements

• Never reveal what the user cannot open (0 leaks). The requirement behind the hardest tradeoff. Permissions differ per user and change constantly, so retrieval filters by the user’s principals inside the index, and every chunk is re-checked against current ACLs before it reaches a prompt. It also rules out sharing answers between users.
• Time to first token (p95 < 2 s). About 200 ms for everything we own (embedding, both searches, re-ranking, checks); the rest is the model’s prefill of a ~6 k-token prompt. Any extra model call before retrieval has to earn its place in this budget.
• Freshness (edits p95 < 5 min · ACLs < 1 min). Measured from the change in the source to the change being answerable, once the source has told us. Webhooks arrive in seconds, polling at the sync interval, so each source’s change feed sets the floor.
• Answer quality (recall@50 ≥ 0.9 · faithfulness ≥ 0.95). Measured offline on golden question sets with known sources: did the right chunk reach the re-ranker, and is each sentence supported by the chunk it cites. A regression blocks a release like a failing test.
• Tenant isolation and deletion (no cross-tenant reads · purge ≤ 30 days). A query can only touch its own tenant’s partitions, by routing rather than by a filter. Offboarding deletes every chunk, vector and raw file, and destroys the tenant’s key.
• Scale (5 k tenants · 2 B chunks · 2 k q/s). Extremely skewed: the largest tenant has 200 M chunks while the median has tens of thousands. The index layout has to suit both ends.
• Availability (99.9 %). Degrade rather than fail: without the re-ranker, answer from the fused order; without the vector index, from BM25; without the model, return the top passages as search results.
• Cost per question (~2.5 ¢, budgeted per tenant). Nearly all of it is LLM tokens, so tokens per prompt is a design parameter, and per-tenant token quotas protect both the fleet and the bill.

Back-of-envelope estimates

• Questions per day: ~50 M (~600/s avg · 2 000/s peak). Peak is given as 2 000/s. Questions cluster in working hours across a few time zones, so the daily average is about a third of that, ~570/s, and 570 × 86 400 ≈ 50 M/day. Each question is one retrieval, one re-rank and one LLM call, so this is also the model call rate.
• Vector index in RAM: ~2 TB per copy · 40 shards · 120 servers. Per chunk: a 768-dimension vector as int8 is 768 B, HNSW links at M = 16 add 128 B (32 bottom-layer neighbours × 4 B), and ids, version and ACL references ~100 B: ~1 KB. 2 B chunks × 1 KB = 2 TB per copy. Full float32 vectors (3 KB each, 6 TB) stay on SSD for rescoring. At 50 M chunks (~50 GB) per shard that is 40 shards, × 3 replicas = 120 servers. Memory, not query rate, sets the count.
• Re-ranker GPUs at peak: ~100 GPUs. 2 000 questions/s × 50 candidates = 100 k (question, chunk) pairs/s, each ~550 tokens. A BERT-base-sized cross-encoder scores ~1 000 pairs/s per GPU, so ~100 GPUs at peak. Cost is linear in the candidate count, which is why it sees 50 candidates and not 500. Embedding 2 000 short questions a second, by contrast, takes a couple of GPUs.
• Prompt size and LLM cost: ~6 k tokens · ~2.4 ¢ per question. Grounding rules ~500 tokens + 8 chunks × ~550 (text plus title and heading path) ~4 400 + recent conversation ~600 + the question ≈ 6 k input tokens; answers average ~400 output tokens. At an assumed $3 per M input and $15 per M output: $0.018 + $0.006 ≈ $0.024, so ~$1.2 M/day at 50 M questions. Retrieval, re-ranking and the indexes add well under a tenth of a cent. Each extra chunk in the prompt costs ~$80 k a day.
• Steady-state re-embedding: ~40 M chunks/day · ~1 GPU. 2 B chunks at ~8 per document is 250 M documents. About 2 % change on a working day: 5 M edits × 8 chunks = 40 M chunks/day ≈ 460/s. At ~1 000 chunks/s per GPU, one GPU keeps up, and skipping chunks whose content hash did not change cuts it further. Steady-state ingestion is cheap; backfills and model changes size the bulk lane.
• Onboarding the largest tenant: ~6 days, set by the source API. 200 M chunks ÷ 8 = 25 M documents. If the source allows our app ~50 requests/s for this tenant, fetching takes 25 M ÷ 50 = 500 k s ≈ 6 days. Embedding the 200 M chunks needs 200 M ÷ 1 000/s = 56 GPU-hours, under 3 hours on 20 GPUs. The source API, not the GPUs, sets onboarding time, so backfill fetches recently edited and most-viewed documents first.
• Re-embedding for a new model: ~560 GPU-hours · 2× index RAM during the switch. 2 B chunks ÷ ~1 000 chunks/s per GPU = 2 M GPU-seconds ≈ 560 GPU-hours: about a day on 25 GPUs, ~$1–2 k at $2–3 per GPU-hour. Chunk text is in our store, so nothing is re-fetched from sources. The real cost is running old and new indexes side by side, another 2 TB of RAM per copy, so tenants migrate in waves.
• Our share of time to first token: ~200 ms of the 2 s. Auth and principals ~15 ms, query embedding ~15 ms, dense and BM25 searches in parallel across shards ~60 ms, fusion and chunk fetch ~15 ms, re-ranking 50 pairs ~80 ms, final ACL check and prompt assembly ~20 ms: ~200 ms. The model’s prefill of a 6 k-token prompt plus its queue, 0.5–1 s, is the rest. Retrieval is never why the answer is slow; a second model call before it would be.

Components

• Assistant client (web · Slack app · extension): Where employees ask: a web app, a chat bot and a browser extension on one API. Renders the streamed answer and turns citation markers into links that open the source at the cited section, where the source’s own login applies again. Sends thumbs and reasons as feedback.
• API gateway (SSO · tenant · SSE): Authenticates the employee through the company’s SSO and takes tenant and user from the session, never from the request body, because every permission decision downstream trusts them. Applies per-user and per-tenant rate limits before any GPU work and holds the streaming response.
• Answer service (retrieve · re-rank · cite): Runs the query pipeline: resolve the user’s principals, rewrite a follow-up into a standalone question, embed it, search the dense and keyword indexes in parallel, fuse with reciprocal rank fusion, re-rank the top 50, re-check the survivors against current ACLs, and assemble a ~6 k-token prompt of numbered sources. Stateless; it owns the per-stage latency budget and degrades a stage rather than failing the answer.
• Query cache (Redis · vectors + principals): Two small caches that are safe to keep. Query embeddings keyed by (tenant, embedding model version, normalised text), and each user’s resolved principal set with its membership version, kept 60 s. It never holds answers or retrieved chunks for reuse by another user, because those depend on who is asking.
• Embedding service (GPU · query + bulk lanes): Runs the embedding model (768 dimensions here) on GPUs in two lanes: a query lane with tiny batches and a ~15 ms budget, and a bulk lane that packs ingestion chunks into large batches at ~1 000 chunks/s per GPU. Each model version is its own deployment, because a question must be embedded by the same model as the index it searches.
• Vector index (HNSW per tenant · int8): Per-tenant partitions of an ANN index: HNSW over int8 vectors in RAM (~1 KB per chunk with links and metadata), full-precision vectors on SSD for rescoring. Small tenants are packed hundreds to a shard; the 200 M-chunk tenant spans four 50 M-chunk shards. Each chunk carries its ACL as principal ids, and each search chooses between a filtered graph walk and an exact scan of the allowed chunks by how selective the filter is.
• Keyword index (BM25 · same partitions): An inverted index over the same chunks, partitioned the same way, scored with BM25 with title and heading fields boosted. It finds what embeddings blur: error codes, ticket ids, product code names and people. ACL principals are indexed as terms, so the permission filter is a posting-list intersection and exact at any selectivity.
• Re-ranker (cross-encoder · GPU · top 50): A cross-encoder that reads the question and each candidate chunk together and scores how well the chunk answers it, far more precise than comparing two separately computed vectors. Too expensive for more than ~50 candidates: 100 k pairs a second at peak is ~100 GPUs. A calibrated score threshold decides which chunks go into the prompt at all.
• LLM tier (black box · streaming): The model that writes the answer, a hosted API or our own fleet, treated as a black box with a streaming generate call (the LLM Inference Service question covers the inside). What this design needs from it: follow the grounding rules, cite source ids, reuse the cached prefix of a stable system prompt, and enforce per-tenant token quotas.
• Source apps & IdP (Confluence · Drive · Slack): The tenant’s systems of record and its identity provider (Okta, Entra ID). Each has its own API, webhook support, permission model and rate limits, shared by every sync we run for that tenant. They are the truth for content and access; our copy is minutes old at best.
• Connectors (hooks · delta sync · buckets): One worker type per source and one sync state per (tenant, source): cursor, last full listing, credentials, and a token bucket sized to the source’s limits. A webhook only says what changed, so the connector fetches the current version itself; a delta sync every 10 minutes catches lost webhooks, and a weekly full listing finds deletions no API reported. Permission changes and deletes go ahead of edits and backfill.
• Change queue (Kafka · key = tenant:doc): Document changes, permission changes and deletions, keyed by (tenant, doc id) so one document’s events stay in order. Separate topics per priority lane (ACLs and deletes, edits, backfill), so a 25 M-document onboarding never delays someone’s revocation. It absorbs backfill bursts the GPUs and indexes would otherwise see directly.
• Indexer (chunk · embed · upsert): Parses each document by type, splits it at headings and paragraphs into chunks of up to ~500 tokens with ~50 tokens of overlap, and prefixes each with its title and heading path. Re-embeds only chunks whose content hash changed, in large GPU batches, writes the store first, then upserts both indexes keyed by (tenant, doc, chunk number) with the document version, so a replay is a no-op and a shorter new version deletes its trailing chunks.
• Content & ACL store (Postgres by tenant · blobs): The source of truth we control: documents with version, deleted flag and ACL; chunk rows with text, heading path, content hash and model version; group memberships synced from the IdP with a version per user. Raw files sit in object storage. Every index can be rebuilt from it without touching a source API, and the final permission check on every answer reads it.
• Eval & feedback (golden sets · recall@k): Golden question sets with the documents that should answer them, replayed on every change of chunker, embedding model, re-ranker, prompt or LLM: recall@k at each retrieval stage, citation precision and faithfulness from a judge model calibrated against human labels. Also stores traces (chunk keys and scores per stage, tokens, citations) and online feedback, which become new golden items.

User flows

1. Ask a question, get a cited answer. The read path. Most of it is retrieval the user never sees, and every stage is filtered by who is asking, which is why nothing in it can be shared across users.
   1. The client sends the question; the gateway authenticates the user through SSO and resolves the tenant. Tenant and user come from the session, never the body, because every filter after this point trusts them. The response is a server-sent event stream the gateway holds open for the length of the answer.
   2. The answer service loads the user’s principals: their own id plus every group they belong to. Groups are expanded at query time, nested ones included, so a membership change never requires reindexing a chunk. A typical employee has 50 to 200 principals; the set is cached 60 s with its membership version, which the final check confirms.
   3. The question is embedded, from the cache if someone in the tenant asked the same thing recently. A follow-up is first rewritten into a standalone question by a small model, ~150 ms, paid only by turns that have history. The cache key is (tenant, model version, normalised text); a miss costs ~15 ms on the query lane, where batches are tiny so nobody waits for one to fill.
   4. Dense and keyword searches run in parallel on the tenant’s partitions, each filtered by the user’s principals. Each returns its top 100. The vector index finds paraphrases (“time off after having a baby” matches “parental leave”); BM25 finds exact tokens that embeddings blur, like “ERR_4031” or a project code name. Both apply the permission filter inside the search, so the 100 results are all chunks the user may open.
   5. The two lists are merged with reciprocal rank fusion into 50 candidates. RRF scores each chunk by the sum of 1 / (60 + rank) over the lists it appears in (Cormack, Clarke and Büttcher, SIGIR 2009). It needs no calibration between a cosine similarity and a BM25 score, which live on different scales, and a chunk that both retrievers rank well rises to the top.
   6. The re-ranker scores each candidate against the question and keeps the best eight above a threshold. A cross-encoder reads question and chunk together, so it can tell a chunk about contractor benefits in general from the one paragraph on contractor parental leave. 50 pairs take ~80 ms. The threshold matters as much as the order: fewer, stronger chunks make a cheaper prompt and give the model less to go wrong with.
   7. The survivors are re-checked against the current document rows, and any chunk that is deleted, outdated or no longer visible is dropped. The index can lag a delete or an ACL change by up to a minute; the store is updated first, so this check closes the window. It reads at most eight document rows plus the user’s membership version, a few milliseconds. Only chunks that pass are ever placed in a prompt.
   8. The prompt is assembled within its token budget, each chunk labelled as a numbered source, and sent to the model. About 6 k tokens: grounding rules first (stable, so the provider’s prefix cache hits), then sources S1 to S8, each with title, heading path and last-updated date, then the recent conversation and the question. The rules: answer only from the sources, cite [S#] after each claim, and say so when the sources do not cover the question.
   9. The answer streams back, citation markers are resolved to source links, and the trace is logged. A marker naming a source that was not in the prompt is dropped rather than shown, and an answer with no valid citation is flagged. The trace (principal-set version, chunk keys, scores at each stage, tokens, citations) is what lets anyone explain later why an answer said what it said.
2. An edited page is answerable minutes later. The write path from a source to both indexes. It must be idempotent end to end, because webhooks repeat, workers crash mid-document, and the same edit often arrives by webhook and again by the next sync.
   1. The wiki sends a webhook that a page changed; the connector verifies it and records a fetch to do. The webhook is a hint, not data: it says page 4411 is now at version 42. The connector replies 200 at once and fetches the content itself, because payloads can be partial, stale, or forged when signatures are not checked.
   2. The connector takes a token from the tenant’s bucket for that source and fetches the page with its permissions. Limits are per app and per tenant at the source, shared by every sync for that tenant. Permissions come with the content: the page’s own restrictions plus what it inherits from its space, mapped to principal ids such as g-finance or u-8812.
   3. The raw content goes to object storage and a change event is published, keyed by tenant and document. Keying by (tenant, doc id) keeps one document’s events in order on one partition. The event carries the source version and the permissions and points at the content rather than carrying it, so a 40 MB PDF never travels through Kafka.
   4. The indexer skips the event if it already holds this version, then parses the page and splits it by structure. Sections split at headings, then paragraphs, into chunks of up to ~500 tokens with ~50 tokens of overlap, each prefixed with the title and heading path (“Travel policy › Per diems › Europe”). A 4 000-token page becomes about 9 chunks; a table keeps its header row in every chunk of rows.
   5. Only chunks whose content hash changed are sent to the embedding service, batched with other documents’ chunks. A one-paragraph edit usually changes one or two chunks; the rest keep their vectors because their hash and the model version are unchanged. The bulk lane packs a few hundred chunks from many documents per GPU batch, ~1 000 chunks/s per GPU, and never competes with the query lane.
   6. The new version’s document row and chunk rows are written to the store in one transaction. The store moves first so it is always the truth: the final check on every answer reads it, and indexes can be rebuilt from it. Chunk rows beyond the new chunk count are deleted in the same transaction, which is what removes text that was cut from the page.
   7. Both indexes upsert the chunks by (tenant, doc, chunk number) and delete chunk numbers the new version no longer has. Deterministic keys make every write an overwrite: replaying the event, or getting the same edit from the next delta sync, changes nothing. Writes carry the version and apply only if newer. Searchable within a second of the write; webhook to answerable is typically one to two minutes.
3. A narrow-access user in the largest tenant. The scale-breaking case: 200 M chunks and a user who may open under a tenth of a percent of them. Vector search followed by a permission filter returns nothing, and a filtered graph walk crawls, so the index has to change plan.
   1. A contractor in the 200 M-chunk tenant asks a question; their principals open about 160 k chunks, 0.08 % of the tenant. The principal set is tiny: the user, a project group and a contractors group. The narrowness is in the documents, since nearly everything else in the company is restricted. Per-principal chunk counts kept by the index say how narrow before any search runs.
   2. The search fans out to the tenant’s four dedicated shards, each holding about 50 M chunks. The tenant is sharded by hash(doc id), so a document’s chunks live on one shard and an upsert or delete touches one place. A question must ask all four and merge; latency follows the slowest shard, so each has replicas and slow calls are hedged.
   3. Each shard estimates how many chunks the filter allows before choosing how to search. Here ~40 k of 50 M per shard. A post-filter over the usual top 100 would keep 0.08 of a chunk on average: nothing. A filtered HNSW walk steps over about a thousand disallowed nodes for every allowed one, and the allowed nodes may not even be connected to each other in the graph, so recall collapses. That is the recall cliff.
   4. Below about 1 % of the shard, the shard skips the graph and scans the allowed chunks exactly. The allowed set is the union of per-principal bitmaps, and its int8 vectors are 40 k × 768 B ≈ 31 MB: a few milliseconds across cores, with perfect recall. Above the threshold the filtered graph walk is fast and accurate, with ef raised as the allowed fraction falls. The threshold is tuned by measuring recall against exact search in each selectivity band.
   5. The keyword index applies the same principals as a filter clause, which is exact at any selectivity. ACL principals are indexed as terms on each chunk, so the filter is an intersection of posting lists: the narrower the user, the less work, the opposite of the graph. It is one more reason to keep a lexical path beside the dense one.
   6. Shard results are merged, fused, re-ranked and re-checked against the store, as for any other user. Merging four top-100 lists per retriever is trivial, and from here the path is identical to everyone else’s. The shard layout is invisible above the index.
   7. If nothing clears the re-ranker’s threshold, the user is told their documents do not cover it, without a model call. The wording matters: “I couldn’t find this in documents you can access”, never “there is a document about this you cannot open”, which would leak that it exists. Padding the prompt with weak chunks to have something to say is how wrong answers with real-looking citations happen.
4. Lost webhooks, throttling and silent deletes. Sources drop webhooks, rate-limit us and often never report deletions. The design treats webhooks as a latency optimisation and makes periodic sync and reconciliation the correctness mechanism.
   1. A tenant’s webhook subscription quietly expires and edits stop arriving as events. Subscriptions expire, admins misconfigure endpoints, and providers drop deliveries during incidents. None of it produces an error on our side: the only symptom is silence, which looks exactly like a quiet day.
   2. The delta sync, every 10 minutes per (tenant, source), asks for everything changed since its cursor and finds the edits. Freshness degrades from about a minute to the sync interval, never to never. The cursor advances only after the changes are published, so a crash replays a window instead of skipping one, and the indexer’s version check makes the replay harmless.
   3. The connector sees changes that no webhook announced, re-registers the subscription, and alerts if that fails. The share of changes found by sync but not by webhook is a per-connection metric. A few percent is normal loss; most of them means the subscription is gone, and it is recreated automatically.
   4. The source starts returning 429s; the connector backs off and spends what budget it has on permissions and deletes first. One bucket per (tenant, source) is shared by webhook fetches, delta sync and backfill. When it is short, backfill pauses, edits wait, and permission changes and deletions go first, because a stale answer is a quality problem and a stale permission is a leak.
   5. The weekly reconciliation lists every document the source still has, and the indexer compares the list with the store. Many APIs report edits but not deletions, and a permission change on a parent folder may produce no event for the 10 000 files under it. The listing carries only ids, versions and ACL hashes, so it is cheap, and every difference becomes an ordinary change event.
   6. Documents we hold that the source no longer lists are tombstoned in the store and removed from both indexes. The store row is marked deleted first, which hides the document at the final check at once; then its chunks are deleted by (tenant, doc) from both indexes. HNSW marks the nodes deleted and compaction rebuilds the graph later; purge from disk follows within the deletion deadline.
   7. An indexer that crashes halfway through a document re-reads the event, and the keyed upserts make the rerun harmless. The consumer commits its offset only after both indexes acknowledge, and the rerun overwrites the same keys. Chunks carry their document version, and the final check drops any chunk older than the document row, so a half-applied edit is never cited as if it were the new version.
5. Access is revoked. The privacy path, with two different changes: a person leaves a group, and a document is restricted. Neither may leave a window in which the assistant quotes something the user can no longer open.
   1. IT removes a user from the finance group, and the identity provider pushes the change to the connector. Directories push membership changes over SCIM or a change feed; where they cannot, group lists are polled every few minutes. Source-specific groups (a wiki group, a private chat channel) arrive from their own connectors the same way.
   2. The change goes out on the priority lane, and the store updates the user’s memberships and their version. Priority topics mean this never waits behind a backfill. Nothing in either index changes: chunks carry group ids and groups are expanded per question, which is exactly why a group of 10 000 people can change without rewriting a single chunk.
   3. On the user’s next question, the final check sees a newer membership version than the cached set and re-filters with the new one. The 60 s cache could otherwise let finance chunks through the index filter for up to a minute; the version read in the final check catches it in the same request. Candidates are re-filtered with the new set, and retrieval re-runs only if too few survive. Revocation applies as soon as it reaches the store.
   4. Separately, an owner restricts a page to two people; the connector fetches the new permissions and publishes an ACL-only change. Restricting a page does not change its text, so nothing is re-chunked or re-embedded. The event says so, and it travels on the same priority lane as membership changes.
   5. The indexer updates the document’s ACL in the store first, then rewrites the ACL field on its chunks in both indexes. An attribute update on nine chunks, no GPU work, done in seconds. The order matters: store first, so the final check enforces the new ACL while the indexes still carry the old one.
   6. In the gap, a stale index can still return the page as a candidate, and the final check drops it before any prompt is built. This is the asymmetry the design relies on: the index may be stale in either direction, and stale in the dangerous direction is harmless because the check reads the truth. The remaining exposure is upstream: nothing can act on a change the source has not reported, which is why permission feeds get the shortest sync intervals.

Deep dives

The theory behind it

• Vector search (IVF): an interactive playground. Cluster vectors with k-means, search the nearest lists for a query, and see how the number of lists and probes trades recall against the vectors compared.
• Vector search, embeddings and RAG: Embeddings and similarity, approximate nearest-neighbour indexes (HNSW, IVF, product quantisation), vector databases versus adding vectors to your existing store, hybrid search, and retrieval-augmented generation (RAG) pipelines end to end.
• Approximate nearest neighbour indexes (HNSW, IVF, PQ): How vector databases find similar embeddings fast: why exact search does not scale, HNSW graphs, IVF clustering, product quantisation and other compression, the recall versus latency versus memory trade-off, filtering, updates and deletes, sharding, and choosing parameters.
• Building LLM features: serving, latency and cost: System design for LLM-powered products: streaming responses, the cost and latency of tokens, batching and KV caches in inference, prompt caching, rate limits and fallbacks, guardrails, evaluation, and when to use RAG, tools or fine-tuning.

Related

• Design Zapier
• Design a Recommendation System
• Design a Distributed Logging System
• Design a CDN
• All 83 system design interview questions
• Run this question for a friend: the interviewer kit