Design a Vector Database

• Create an index: Name, dimension (768 or 1536 here), distance metric (cosine, dot product or Euclidean), the metadata fields to make filterable, an optional embedding-model tag, and a performance or cost class. Dimension and metric are fixed for the index’s life.
• Upsert vectors with metadata: Batches of up to 1,000 of { id, values, metadata }; an upsert of an existing id replaces it. The response carries a log position the client can pass back for read-your-writes.
• Delete: By id, by filter, or a whole namespace. A deleted vector must stop matching within seconds and be physically gone within a bounded time, because customers delete for legal reasons.
• Top-k query with metadata filters: A query vector, k up to 1,000, and a filter such as category = shoes and price < 100: equality, set membership and numeric ranges combined with and/or. High recall even when the filter allows 0.1 % of the index.
• Namespaces: Partitions inside an index by a key every query carries, such as an end user or a project, so a query only touches that slice. The cheapest filter there is, and the one customers should use for high-cardinality keys.
• Bulk import: Load hundreds of millions of vectors from Parquet files in the customer’s bucket without going through the upsert API, for first loads and for re-embedding with a new model.
• Aliases: A stable name that points at an index and switches atomically, so a tenant can move to an index built with a new embedding model without its clients ever querying a half-built one.
• Out of scope: Producing embeddings (customers bring vectors; the RAG System question designs the pipeline that does), keyword and hybrid BM25 search, cross-region replication, and billing.

• Scale and skew (10 k tenants · ~20 B vectors): From a few thousand vectors to 2 B in one index, at 768 or 1536 dimensions. About a hundred tenants send most of the queries and most of the rest are idle most of the hour, so one layout has to suit both ends.
• Latency at a recall target (p99 < 50 ms · recall@10 ≥ 0.95): The requirement behind the hardest trade-off. Recall, latency and memory cannot all be maximised, so the index type, the quantisation and where the bytes live are chosen to hit both numbers at the lowest cost, and recall is measured continuously rather than assumed.
• Throughput (50 k queries/s · 100 k upserts/s): At peak, fleet-wide. One index takes 5 k queries a second on its own; upserts arrive in batches and include bulk loads and re-embeddings.
• Freshness (searchable p99 < 2 s): From the acknowledged upsert to appearing in results, and the same for a deleted vector disappearing. Building a graph index takes minutes, so freshness cannot wait for one.
• Durability (0 acknowledged writes lost): An upsert is acknowledged once two of three zone replicas of the log have it, and it reaches object storage within seconds. Every query node can be lost without losing a vector.
• Availability (99.95 % for queries): A dead node or zone costs capacity, not answers: hot shards have replicas in two zones, and a shard that still misses its deadline is left out with the response marked partial.
• Cost (idle index ≈ storage only): An idle index should cost cents per million vectors a month, not dollars. That rules out keeping every index in RAM and makes object storage the source of truth.
• Tenant isolation (no cross-tenant reads): A query can only open its own namespace’s segments, enforced by routing rather than by a filter, and one tenant’s burst cannot push the others past their p99.

• Bytes per 768-d vector: 3 KB raw · ~900 B int8 + graph · 96 B PQ. float32: 768 × 4 = 3,072 B. Scalar int8: 768 B (4×). Product quantisation into 96 one-byte codes, or one bit per dimension: 96 B (32×). HNSW with M = 16 keeps 32 neighbours at the bottom layer, 32 × 4 B = 128 B, plus a few percent for the upper layers. The default hot form, int8 vectors plus the graph, is ~900 B; 1536-d vectors double every figure but the graph.
• The largest index: ~12 TB raw · 3.4 TB of RAM as int8 HNSW · 0.4 TB as an SSD graph. 2 B × 1536 × 4 B = 12.3 TB of float32, so a single exact query would read 12 TB. As int8 + HNSW: 2 B × (1,536 + 128 + ~50 B of ids) ≈ 3.4 TB of RAM per copy, seven 512 GB machines. As a graph on SSD: 192-byte PQ codes in RAM, ~0.4 TB, and full vectors with 64-neighbour lists on NVMe, 2 B × 6.4 KB ≈ 13 TB. At ~100 M vectors a shard, 20 shards.
• The whole fleet: ~80 TB raw · ~170 TB of RAM if kept naively. 20 B vectors at an average 1,024 dimensions (two-thirds 768, one-third 1536) × 4 B = 4 KB each, 80 TB. Held as float32 HNSW in RAM with two copies: ~4.2 KB × 20 B × 2 ≈ 170 TB, ~$850 k a month at an assumed $5 per GB-month. This is the number the design exists to avoid.
• Cost per million 768-d vectors a month: ~$16 float32 RAM · ~$5.5 int8 RAM · ~$1.5 SSD graph · ~9 ¢ idle. At assumed prices of $5 per GB-month for RAM, $0.25 for NVMe and $0.02 for object storage, one copy. float32 + HNSW: 3.2 GB of RAM, ~$16. int8 + HNSW: ~0.95 GB of RAM ($4.75) plus 3 GB of float32 on NVMe for re-scoring ($0.75), ~$5.5. Graph on SSD: ~0.15 GB of RAM ($0.75) plus ~3.3 GB of NVMe ($0.8), ~$1.5. Object storage alone, every file included (~4.3 GB), ~9 ¢: about 60× less than the hot int8 form. Idle indexes belong there.
• Query CPU at peak: ~145 k shard searches/s · ~435 cores busy. Of 50 k queries/s, 5 k go to the 20-shard index (100 k shard searches) and 45 k to single-shard indexes: 145 k/s. A shard search covers 10 to 15 segments, graph walks of a few tenths of a millisecond on the big ones and exact scans of the small ones, ~3 ms of CPU in all: 145 k × 3 ms ≈ 435 cores busy, ~1,100 provisioned at 40 % for headroom and hedges. The hot RAM tier needs ~10 TB for two copies, ~20 machines of 512 GB, so memory, not CPU, sets its size.
• Write path: ~440 MB/s into the log at peak. 100 k upserts/s × (4 KB of float32 at 1,024 dimensions + ~300 B of id and metadata) ≈ 440 MB/s, ×3 zone replicas ≈ 1.3 GB/s of log writes. At an average ~30 k/s that is ~11 TB of log a day, kept 24 hours. Appends are sequential and spread over hundreds of partitions: writes are a log-sizing problem, not an index problem.
• Index build compute: ~300 cores at peak. An HNSW insert at ef_construction 200 is ~5,000 distance computations, ~1 ms of one core at 768 d. Compaction rebuilds graphs at size tiers of ~100 k, ~1 M and ~10 M vectors, so each vector is indexed ~3 times: 100 k/s × 3 × 1 ms = 300 cores at peak, ~90 on average, on stateless builders that can run on spot capacity. A 10 M-vector segment is ~10,000 core-seconds, ~3 minutes on 64 cores. Building is cheap next to holding vectors in RAM.
• An idle tenant: ~1.7 ¢ a month · first query ~0.5 s. 200 k vectors at 768 d × ~4.3 KB in object storage (float32, int8, graph, metadata) ≈ 0.86 GB × $0.02 per GB-month ≈ 1.7 ¢. Waking it fetches ~200 MB of int8 vectors, graph and bitmaps with parallel ranged GETs at ~1 GB/s, ~0.2 s, so the first query takes ~0.5 s and the next ~10 ms. Kept hot, the same tenant would cost ~$1 a month of RAM per copy.

• Customer app (RAG · search · recommendations): The tenant’s own service: a RAG pipeline (the RAG System question is one such customer), a product search or a recommender. It embeds documents and queries with its own model, upserts in batches with metadata and queries by vector with filters. It must embed queries with the same model as the index it searches, which nothing in the numbers can check.
• Embedding model (the customer’s · v1 → v2): Whatever model the tenant uses, hosted or its own, outside our system. It fixes the dimension and the space: vectors from two models are not comparable, so changing models means building a new index. We never see the text, only the vectors.
• API gateway (API key · quotas · aliases): Authenticates the API key to a tenant, resolves index names and aliases from the catalog, validates dimensions and the declared model tag, and charges per-tenant token buckets for queries and upserts before any work is done. Sends writes to the write service, queries to the coordinator, and imports to the control plane.
• Write service (stateless · hash(id) → shard): Splits each batch by hash of the id into the index’s 256 virtual buckets, maps buckets to shards and appends each part to that shard’s log partition, returning the highest log position. Routing by id means all writes for one id are ordered by one log. It never touches an index, which is why writes scale with the log.
• Write-ahead log (partition per shard · 3 zones): A replicated log of the Kafka or BookKeeper class, one partition per shard, acknowledged at two of three zone replicas in ~5 ms. It is the durable copy of every write until builders have sealed it into object storage, and the stream every query replica tails for freshness. Kept 24 hours; recovery replays from the offset recorded in the shard’s manifest.
• Query coordinator (fan-out · merge · hedge): Resolves (tenant, index, namespace) to shards and replicas from a cached catalog version and sends the query to one replica of each shard, choosing by fewest requests in flight. Hedges a shard that is slower than its own p95, merges the shards’ top k, and marks the answer partial if a shard misses the deadline. Logs a sample of queries, with the manifest versions they read, for the recall audit.
• Query nodes (segments · filters · re-score): Each hosts replicas of many shards: per shard, a growing segment per namespace built by tailing the log, the sealed segments of the current manifest, and their deletion bitmaps. A search turns the filter into a bitmap per segment, picks an exact scan or a filtered graph walk from the allowed count, re-scores candidates on full vectors and returns the shard’s top k. Nothing on a node is the only copy of anything.
• Segment cache (RAM · local NVMe · LRU by index): The query nodes’ memory hierarchy over object storage. RAM holds the int8 vectors, HNSW graphs, PQ codes and bitmaps of indexes queried in the last ~30 minutes (~5 TB per copy fleet-wide); NVMe holds float32 vectors for re-scoring, graphs on SSD and recently used segments; misses are parallel ranged GETs. Eviction is per index, by recency weighted by query rate, with a pinned tier for customers who pay never to be cold.
• Object storage (S3 class · immutable segments): The source of truth for everything sealed. Per segment: float32 and int8 vectors, metadata columns and their bitmaps, the sorted id table and Bloom filter, the graph or PQ files, and the deletion bitmap; per shard, every manifest version. ~110 TB for 20 B vectors, ~$2 k a month. Files are written once and never modified, which is what lets any node load them and any audit reproduce a query.
• Catalog (strongly consistent · etcd class): Small and strongly consistent: tenants and API keys; indexes with dimension, metric, model tag, filterable fields and class; aliases; the bucket-to-shard map; each shard’s current manifest version; replica placements and node leases. Every change is a compare-and-set on a version, so an alias switch or a manifest swap is one atomic step that every coordinator and node sees in the same order.
• Index builders (seal · compact · import): Stateless workers. They consume log partitions and cut the same offset ranges into sealed segments; run compactions that merge segments by size tier, drop deleted rows and build a fresh HNSW or SSD graph (~1 ms of a core per vector); and build segments straight from Parquet for bulk imports. Every new segment is checked with self-queries against an exact answer before it is offered for commit, and output goes to new object keys, so a crashed job leaves nothing visible.
• Control plane (placement · manifests · scaling): Decides where everything runs: shard replicas onto query nodes by RAM, NVMe and load; eviction of idle indexes and their warm-up; replica counts from query rates; tenant moves from pooled to dedicated shards; shard splits. Plans compactions per shard, hands them to builders, and commits each result as a new manifest version by compare-and-set, so it is the only writer of manifests.
• Recall auditor (batched exact search): Recomputes the exact answers of sampled queries on the very segments they searched, in one batched scan per index, and reports recall@10 per index, per plan and per filter-selectivity bucket. Its numbers drive the controller’s ef_search and scan thresholds and are shown to customers, because a quiet drop in recall is invisible from the outside.

1. Upsert a batch, searchable in a second: The write path runs on three clocks. An upsert is durable when two zones have it in the log (~5 ms), searchable when the query replicas have tailed it into a growing segment (well under a second), and cheap to search once a builder has sealed and indexed it (minutes). None of them waits for a graph.
   1. The client embeds 500 product descriptions with its model and upserts them with metadata.
   2. The gateway resolves the key, the index and its dimension from the catalog, and charges the tenant’s upsert quota.
   3. The write service splits the batch by hash of the id and appends each part to its shard’s log partition.
   4. Every query replica of the shard tails the partition and appends the vectors to the namespace’s growing segment.
   5. A query that carries the returned LSN is answered only by replicas that have applied it.
   6. At 10 k vectors or 5 s, a builder seals the same log range into a segment in object storage.
   7. The control plane commits a new manifest; replicas load the segment and drop those records from their growing segment.
   8. Later, compaction merges small segments into a larger one and builds its graph off the query path.
2. Filtered top-k on one shard: The read path for a typical index: one shard, a dozen segments, and a filter that allows 3 % of the rows. Each segment picks its own plan from an exact count of allowed rows, and int8 candidates are re-scored on full vectors before the top 10 leave the node.
   1. The client embeds the query with the same model and asks for the 10 nearest shoes under 100.
   2. The coordinator maps the index to its shard and picks the replica with the fewest requests in flight.
   3. On the query node, each segment turns the filter into a bitmap of allowed rows, minus the deleted ones.
   4. Each segment picks a plan: an exact scan when few rows pass, a filtered graph walk when many do.
   5. The node re-scores its best 40 candidates on float32 vectors from NVMe and returns its exact top 10 with metadata.
   6. The coordinator returns the merged top 10, and a sampled query is logged for the recall audit.
3. The 2 B-vector index at peak: The scale case: one index of 2 B vectors at 1536 dimensions taking 5 k queries a second. It is 20 shards of graphs on SSD, every query asks all 20 and waits for the slowest, so the tail and the replica count are the design problem, not the search itself.
   1. The coordinator sends the query to one replica of each of the 20 shards in parallel.
   2. Each shard walks a graph stored on NVMe, steering with compressed codes held in RAM.
   3. Shard 7’s node is busy filling its cache, so after that shard’s p95 the coordinator hedges to the other replica.
   4. The coordinator merges 200 candidates and answers; a shard that missed the deadline is left out and flagged.
   5. Traffic rises 4× for an hour, and the control plane adds replicas that fill from object storage.
4. A query node dies at peak: Query nodes hold nothing that is not also in the log and object storage, so a dead node costs capacity, never data. What matters is the minute between the crash and its replacement being warm and caught up.
   1. Node qn-31 stops renewing its lease, and after 10 s the control plane removes it from every replica list.
   2. Queries in flight to qn-31 time out at the shard deadline and are retried on the other replica.
   3. The control plane assigns qn-31’s shards to healthy nodes, which fetch the segments from object storage.
   4. Each new replica replays the log from its manifest’s checkpoint to rebuild growing segments and deletions.
   5. Once within a second of the log head, the replica is marked ready and coordinators start using it.
5. Switch to a new embedding model: A tenant moves 200 M vectors from a 768-d model to a 1024-d one. Vectors from two models live in different spaces and cannot share an index, so the switch is a second index filled by dual writes and a bulk import, checked, and put behind the alias in one step.
   1. The tenant creates products-v2 with the new dimension and model tag, and starts writing every change to both indexes.
   2. The tenant re-embeds its corpus offline and asks us to import the files from its bucket.
   3. Builders read the files, partition rows by hash of the id into the new shards, and build large segments directly.
   4. Committing the import tombstones whichever copy of each id has the lower version.
   5. The recall auditor checks v2 against exact search while the tenant runs its own relevance evaluation.
   6. The tenant switches the alias, and queries move to v2 as each client instance switches its query model.
   7. products-v1 stops getting queries, drops out of RAM and costs only object storage until it is deleted a week later.

Ask: One tenant has 2 B vectors at 1536 dimensions and the fleet has 20 B. Which index do you build, and where do its bytes live?

Good answers name: HNSW over int8 vectors in RAM, float32 on NVMe for re-scoring, A graph on SSD, DiskANN-style: PQ codes in RAM, full vectors and neighbour lists on NVMe, IVF-PQ in RAM (FAISS-style), re-scored from full vectors, HNSW over float32 in RAM.

Our pick: The index type is a property of a segment, chosen by the system from the segment’s size and the index’s class; customers pick latency or cost, not an algorithm. Segments under ~50 k vectors, growing ones and small tenants’, are scanned exactly, because a 38 MB int8 scan is ~3 ms and needs no build. Sealed segments of the performance class get HNSW with M = 16 and ef_construction = 200 over int8 vectors (per-dimension ranges trained on the segment), ef_search set per index by the recall auditor, and each shard’s best 40 candidates re-scored on float32 from local NVMe. Shards of the cost class, and any index past ~100 M vectors a shard, use a graph on SSD in the style of DiskANN (Subramanya et al., NeurIPS 2019): a Vamana graph of degree 64 with each node’s full vector and neighbour list in one sector-aligned record on NVMe, steered by 192-byte PQ codes in RAM. The paper reports serving a billion vectors from one machine with 64 GB of RAM at single-digit milliseconds and over 95 % recall; I would size our shards from our own benchmarks. For the 2 B-vector index that is ~0.4 TB of RAM and ~13 TB of NVMe per copy instead of 3.4 TB of RAM, and binary or PQ-only tiers stay per-index options, enabled only where the auditor shows they hold the target.

1. Why not binary quantisation everywhere? It is 32× smaller and the distance is a popcount.
   Because how much it loses depends on the model. Some embedding models, especially high-dimensional ones trained with it in mind, keep most of their neighbourhood structure at one bit a dimension; others lose a lot. With an oversampled first pass, say 400 candidates for a top 10, and a float32 re-score, binary can reach the target on the right model for a fraction of the RAM. So it is a per-index option switched on only when the auditor shows it holds 0.95 on that index’s own queries.
2. Where exactly does int8 lose recall, and what does re-scoring cost?
   Rounding each dimension to 256 levels adds a small error to every distance, which mostly reorders near ties around the cut-off. Taking the best 40 by int8 distance and recomputing them on float32 fixes the order: 40 random 3 KB reads from NVMe, tens of microseconds in parallel, plus 40 × 768 multiply-adds. It recovers nearly all of the loss, and the auditor measures what remains per index rather than us assuming it.
3. The big index’s traffic grows 4×. What breaks first on the SSD graph?
   IOPS. Each shard search is ~100 random 8 KB reads, so 20 k queries a second over 20 shards is ~40 M reads a second across the replicas, on the order of 80 NVMe drives. First, cache the graph nodes nearest the entry point in RAM, since every walk passes through them and a few GB removes the first several rounds of reads. Then add replicas, which are cache fills from object storage. If the index stays that hot, RAM HNSW for it becomes cheaper per query than SSDs, and that is the honest answer.
4. Pinterest serves 10 B vectors with IVF-PQ. Why not here?
   Pinterest’s visual index is rebuilt weekly, takes a few hundred queries a second and targets recall@100 with a deep re-rank. This index takes 5 k a second at recall@10 of 0.95 and changes continuously. IVF needs a large nprobe to reach that recall at 1536 dimensions, which means scanning hundreds of thousands of codes per shard per query, and its centroids drift as data is appended. A graph reaches the target with far fewer distance computations, so IVF-PQ stays an option for bulk-loaded, rarely changed indexes.
5. How do you choose M and ef_search?
   M sets memory and build cost per vector, so it is fixed per index class: 16 is ~128 B a vector and enough for 768 to 1536 dimensions, and harder data gets 32. ef_search is a query-time dial, so it is tuned continuously: the auditor measures recall@10 per index and the controller raises ef until the target holds, then trims it while it still holds, because every unnecessary candidate is latency. Filtered queries get their own ef, scaled by selectivity.

Ask: Building an HNSW graph over 10 M vectors takes minutes, and deleting from one is hard. How is an upsert searchable within a second, and a delete gone just as fast?

Good answers name: Log, growing segment scanned exactly, sealed segments built in the background, deletion bitmaps, compaction, One mutable HNSW per shard, inserted in place, deletes marked, Rebuild the whole index periodically, plus a small fresh buffer, Index synchronously in the write path before acknowledging.

Our pick: An upsert is routed by hash of the id to its shard’s log partition, replicated to three zones and acknowledged at two, ~5 ms, and the response carries the log position. Every query replica tails the partition into a growing segment per namespace, float32 vectors plus metadata columns scanned exactly, sealed at ~10 k vectors or 5 s; builders consume the same partition and cut the same offset ranges into sealed segments in object storage, so replicas never disagree about what a segment holds. A shard manifest, the segment list plus the log offset it covers, is swapped by compare-and-set, and a query reads one manifest version plus the growing records past its checkpoint. A delete, or the old copy behind an upsert, becomes a bit in its segment’s deletion bitmap, found through per-segment id Bloom filters and sorted id tables, and every search ANDs NOT that bitmap. Size-tiered compaction at ~100 k, ~1 M and ~10 M vectors merges segments, drops deleted rows and builds a fresh graph each time, and rewrites any segment on its own once 20 % of it is deleted. Milvus’s public architecture describes a similar split of growing and sealed segments over a log and object storage; the thresholds here are mine.

1. A builder crashes halfway through a compaction. What happens?
   Nothing visible. Its output was being written under new segment ids that no manifest references, so the inputs are still live and queries are unaffected. The controller sees the job’s lease expire and reschedules it, and a sweeper deletes objects that no manifest has referenced for 48 hours. Commit-last is the whole trick: the manifest swap is the only step that makes work visible.
2. An upsert replaces sku-81231. How does the old copy stop matching?
   While applying the upsert, each replica checks the shard’s segments’ Bloom filters for the id, then the sorted id table of the one or two that might hold it, and sets that row’s bit in the deletion bitmap; the new copy goes into the growing segment. Bloom filters at ~10 bits a vector cost ~1.25 B per vector in RAM and rule out the other segments in nanoseconds. Because both happen while applying one log record, there is no moment when both copies match, or neither.
3. Why not remove deleted nodes from the HNSW graph properly?
   Because the graph’s quality comes from neighbours chosen while building, and a removed node leaves each of its in-neighbours with a hole that must be refilled by re-running neighbour selection, about the cost of a fresh insert, while queries walk the graph. Marked deletions avoid that but tax every query, and the candidate list fills with dead nodes, so ef must rise to hold recall. Rewriting the segment without them is the same work done once, in the background, on data that no longer changes.
4. A tenant must have deleted vectors physically gone within 14 days. Can you promise that?
   Yes, as a scheduled guarantee rather than a side effect. A deleted vector stops matching within a second through the bitmap. The controller forces a compaction of any segment holding a delete older than 7 days, superseded segments are kept only 48 hours after their manifest is replaced, and the log ages out in 24 hours. So the bytes are gone from every object well within 14 days, and the oldest pending delete per tenant is a reported metric.
5. A bulk load pushes writes to 300 k a second. What happens?
   The log absorbs it, since it is sequential appends over many partitions, but growing segments seal three times as often and builders fall behind, so shards collect small segments and queries slow down. The defence is to send bulk loads down the import path, which builds large segments straight from files, and to cap each tenant’s upsert rate per shard. If a backlog builds anyway, the controller merges the smallest segments first, because segment count is what hurts queries.

Ask: The largest index does not fit on one machine. Do you split it by id or by region of the vector space, and how do you keep a 20-way fan-out fast?

Good answers name: Shard by hash of the id, ask every shard, merge the top k; replicas for throughput, hedging for the tail, Partition by cluster: k-means centroids, route each query to its nearest 3 shards, Shard by a customer key such as user or project, One very large machine per copy.

Our pick: Hash sharding over 256 virtual buckets per index, so a shard is a range of buckets and splitting one moves half its buckets: builders re-partition its segments by the next hash bit in the background and the controller flips the bucket map in one catalog version. Shards target ≤ 50 M vectors on the RAM tier and ≤ 100 M on the SSD tier, so the big index is 20 shards. The coordinator sends each query to one replica of every shard, chosen by fewest requests in flight, asks each for its re-scored top 10 and merges 200 candidates; a shard that has not answered by its own p95 (~8 ms) gets a hedged request to another replica, and one still missing at the 30 ms deadline is left out and the response marked partial. Replicas exist for throughput and zone failure, not durability, so the controller scales each shard’s replica count with its load, two at minimum and three for the largest indexes. Cluster partitioning stays a per-index option for customers who choose lower recall for a lower price, and the auditor shows them what it costs.

1. Why ask each shard for only its top 10?
   Because with hash sharding the global top 10 is contained in the union of the shards’ top 10s: a vector in the global top 10 is necessarily in its own shard’s top 10. So k per shard is exact for the merge, given shard lists with comparable scores, and every shard re-scores on full vectors before answering, so they are. Asking for more only helps when shards return approximate scores on different scales.
2. If each shard is slow 1 % of the time, how often is a query slow?
   With 20 shards, 1 − 0.99^20 ≈ 18 % of queries wait on at least one slow shard, so the per-shard p99 becomes roughly the query’s p82. Hedging after the per-shard p95 sends ~5 % extra requests and means a query is slow only when both replicas of some shard are slow at once. It is also an argument for fewer, bigger shards where memory allows, since the tail grows with the shard count.
3. The tenant doubles to 4 B vectors. What do you do?
   Split every shard by moving half its virtual buckets to a new shard. Builders read each shard’s segments from object storage and write two sets partitioned by the next hash bit while the old shard keeps serving, and new log partitions start at a cutover offset. The controller loads the new shards, waits until they have applied the log head, and flips the bucket map in one catalog version. Nothing is copied between query nodes; it is a background batch.
4. A replica of shard 7 is a few seconds behind the log. Can a query use it?
   For an ordinary query, only if it is within ~1 s of the head, which is the freshness promise, so the coordinator skips laggards. For a query carrying min_lsn, only a replica that has applied that position may answer. A replica that stays behind is taken out of rotation by the controller until it catches up, because lag usually means an overloaded node.
5. Could the coordinator stop early once it has good enough results?
   Not safely with hash sharding: any shard might hold the best match, and nothing in the scores already received says what is missing. What it can do is degrade explicitly: at the deadline it returns what it has and marks the response partial, which costs about 5 % of recall per missing shard for that one query. With cluster partitioning early termination is natural, which is part of its appeal.

Ask: “category = shoes and price < 100” matches 3 % of the index. Do you filter before the vector search, after it, or during it?

Good answers name: Per-segment bitmaps and an exact count; scan the allowed rows when few, filtered graph walk when many, Post-filter: search unfiltered for k × oversample, then drop disallowed results, Pre-filter, then always scan the allowed rows exactly, A separate index per value of a common filter field.

Our pick: Customers declare filterable fields when they create the index, and at seal time each segment builds a compressed (roaring) bitmap per value of each keyword field, capped at ~10 k values (beyond that the field is stored but filtered by scanning its column), and ~64 quantile buckets per numeric field, with exact checks on the rows of the boundary bucket. A query’s filter compiles into bitmap ANDs and ORs plus an AND NOT with the deletion bitmap, giving each segment an exact allowed count. Below ~20 k allowed rows or ~1 % of the segment, the segment scans those rows’ int8 vectors exactly; above it, the HNSW walk still steps through every neighbour for navigation but admits only allowed nodes to the result list, with ef raised roughly as k divided by selectivity up to a cap, and below ~10 % it also expands through a disallowed neighbour to that neighbour’s neighbours, so the allowed nodes stay connected. Post-filtering is used only when more than ~90 % of rows pass. The thresholds are per index, set from the auditor’s recall by selectivity bucket, and a field every query filters on by equality, such as an end-user id, should be a namespace rather than a filter: we tell customers so.

1. The filter allows 0.01 % of the 2 B-vector index. What does a query cost?
   About 200 k allowed rows across 20 shards, ~10 k per shard and a few hundred per segment. Every segment falls below the scan threshold, so each shard scores its ~10 k allowed rows on the PQ codes in RAM, ~2 MB of codes and well under a millisecond, and re-scores the best 40 from NVMe. That is cheaper than an unfiltered query and close to exact. Very selective filters are the easy case once you stop using the graph for them.
2. What is anticorrelation, and why does it hurt even at 10 % selectivity?
   It is when the allowed rows sit far from the query, like a sandals query filtered to furniture. The walk heads towards sandals, where almost nothing is allowed, and spends its budget there before it reaches the allowed region, so recall drops even though a tenth of the rows pass. Selectivity alone does not predict it, which is why the auditor reports recall per selectivity bucket per index, and why the fix is a higher ef for filtered queries or an earlier switch to the scan.
3. A customer filters on user_id, with a million distinct users.
   Then a bitmap per value is a million bitmaps per segment and every query allows a sliver of the index, the worst shape for a graph. It should be a namespace: each user’s vectors in their own segments, so a query touches only that user’s data, which is usually small enough to scan exactly. We spot high-cardinality equality filters in the query logs and recommend the change, because it is a data-model fix, not an index setting.
4. How much RAM do the bitmaps cost?
   Each row sits in exactly one value’s bitmap per field, so a field costs between an eighth of a byte and about two bytes per row, depending on how clustered its values are, and a numeric field’s quantile buckets cost the same. Five filterable fields are ~10 B a row at worst, about 1 % of an int8 768-d vector. The real limit is cardinality, which is why bitmaps are capped per field.
5. Why not build the filter fields into the graph, so neighbours share attributes?
   You can build edges that respect a field, and filter-aware graph indexes do something like that for a known set of labels. It works for a few equality fields, but customers combine arbitrary predicates with ranges, and there cannot be a graph per combination. Bitmaps plus a plan per segment handle any predicate the bitmaps can express, with a walk that only has to stay connected. For one dominant field, namespaces give the same benefit with none of the complexity.

Ask: A hundred tenants send most of the queries, most of the rest send a few an hour, and one holds 2 B vectors. How do you serve them all without paying RAM for the idle ones or letting one hurt the others?

Good answers name: Object storage as the truth, query nodes as RAM and NVMe caches; small tenants as namespaces in pooled shards, big ones on dedicated shards; per-tenant quotas and fair scheduling, Every index always resident on its own provisioned nodes, One shared index for everyone, with a tenant_id filter, Local disks as the source of truth, with replica-to-replica copies.

Our pick: Object storage holds every segment, index file, deletion bitmap and manifest; the log holds the last seconds to minutes; query nodes hold nothing that cannot be refetched. A tenant’s index lives in namespaces: tenants under ~1 M vectors share pooled shards, hundreds to a shard, but every segment belongs to exactly one namespace and the coordinator routes by (tenant, index, namespace) taken from the API key, so no query opens another tenant’s files and there is no tenant filter to get wrong. Namespaces under ~50 k vectors are simply scanned; tenants over ~50 M vectors get dedicated shards, and the largest get dedicated nodes. RAM caches the int8 vectors, graphs and bitmaps of indexes queried in the last ~30 minutes (~5 TB per copy for ~4 B hot vectors), NVMe caches full vectors and recent segments, and eviction is per index by recency weighted by query rate, with a pinned tier for customers who pay never to be cold. Per-tenant token buckets for queries and upserts sit at the gateway, and query nodes schedule searches by weighted fair queuing across tenants with a cap on each tenant’s concurrent searches, priced by an estimated cost (segments × plan × k), so one tenant’s heavy filtered scans cannot take a node’s cores. A tenant that outgrows its pool is moved by placing new shards, warming them from object storage and flipping its routing. This is broadly the shape managed serverless vector databases describe publicly; the numbers and details are my design.

1. A tenant idle for a week sends a query. Walk me through it.
   The coordinator finds no warm replica for its namespace, so the controller assigns it to a pooled node with spare RAM, and that node reads the manifest and fetches the int8 vectors, graphs and bitmaps with parallel ranged GETs: ~200 MB for 200 k vectors, ~0.2 s at ~1 GB/s. The first query is answered in ~0.5 s, re-scoring from int8 until the float32 file reaches NVMe, and the next one in ~10 ms. Customers who cannot accept that pay to stay pinned, or send a warm-up query when their service starts.
2. One tenant on a pooled node starts sending 5 k filtered queries a second that each scan 20 k rows. What protects the others?
   First its quota at the gateway, which caps its queries a second. Then each search’s estimated cost, rows scanned plus graph work, is charged against the tenant’s share on the node, and the scheduler serves tenants in weighted round robin with a cap on each one’s concurrent searches, so the heavy tenant queues behind itself. If it stays heavy, the controller moves its namespace to its own shard, a cache fill from object storage, and its bill reflects the capacity.
3. How do you delete a tenant completely?
   Remove its catalog entries so routing stops at once, then delete its object-storage prefixes; because segments never mix namespaces, a tenant is a set of prefixes, not rows scattered through shared files. Pooled log partitions age out within 24 hours. With a per-tenant encryption key on its objects, destroying the key makes any copy that lingers in a backup unreadable immediately.
4. What is the floor of the storage bill for the whole fleet?
   Every vector is stored once in object storage as float32 plus int8, its share of the graph and its metadata: ~5.5 KB at an average 1,024 dimensions, ~110 TB for 20 B vectors, about $2 k a month at $0.02 per GB-month. Superseded segments kept 48 hours add a few percent. Everything above that floor is RAM and NVMe, spent only on what is being queried.
5. The tenant migrating to a new embedding model doubles its RAM. Who pays?
   For the overlap, the tenant does, because both indexes are genuinely in use while clients roll over. But it is short: the old index is evicted ~30 minutes after its last query and from then on costs only object storage, a few dollars a week for 200 M vectors. Storage and compute separation is what turns a migration’s cost from a second permanent copy into an hour or two of double RAM.

Ask: You promise recall@10 of 0.95. Deletes, filters, new segments and new data all move it. How do you know you are keeping the promise, per index, today?

Good answers name: Sample real queries with their manifest versions, recompute exact answers in batches on the same segments, report recall per index, plan and selectivity, and tune ef from it, Public benchmark datasets in CI, Self-queries: use vectors sampled from each new segment as queries and check them against an exact answer on that segment, Rely on the customer’s relevance metrics.

Our pick: Coordinators log up to ~500 queries per index a day, sampled uniformly, each with its vector, filter and k, the manifest version of every shard it read and each segment’s results, to a samples topic. Daily per index, hourly for the largest, the auditor groups the samples by segment and runs one batched exact search over the float32 vectors of each segment that was read, plus the logged growing records replayed from the log, applying the same filter and deletion bitmaps; superseded segments are retained 48 hours so this is always possible. That is about one pass over the fleet’s ~80 TB of full vectors a day, under 1 GB/s, on spare builder capacity. It reports recall@10 per index, per plan (graph walk, SSD graph, scan) and per selectivity bucket; the controller raises ef_search or lowers the scan threshold where a bucket is under 0.95 and trims ef where recall has headroom. Builders also gate every new segment with self-queries before it is committed, and customers see their own measured recall on their dashboard.

1. Recall for one tenant fell from 0.96 to 0.89 overnight. How do you find out why?
   Break it down by what the auditor logs. If it is one plan, say filtered walks in one selectivity bucket, a new filter pattern is anticorrelated and the threshold or ef needs to move. If it is one kind of segment, look at deleted fractions and quantiser ranges on recent segments, since new data outside the trained int8 ranges gets clipped. If it is everywhere at once, the query vectors changed, often a client embedding with a new model, which shows as a shift in the query distribution and is the customer’s to fix.
2. Isn’t exact search over 80 TB a day expensive?
   It is mostly a read: ~80 TB a day is under 1 GB/s across the fleet, from object storage or NVMe caches. The arithmetic is a matrix multiply of each index’s samples against its vectors, ~500 queries × 20 B vectors × 1,024 dimensions ≈ 10^16 multiply-adds a day, a few machine-hours of vectorised CPU. Doing the same queries one at a time would read each index once per query, so batching is the whole trick.
3. The tenant switches embedding models. Does the auditor tell them whether the new one is better?
   No, and it matters not to pretend otherwise. The auditor says whether the index finds the true nearest neighbours of whatever vectors it holds; whether the new model’s neighbours are better answers is a relevance question only the tenant’s labelled evaluation can settle. What we do is audit both indexes the same way during the dual-write period, so a recall difference between them is never mistaken for a model difference.
4. Why log per-segment results rather than just the final top 10?
   Because the final top 10 mixes the effects of every segment, the merge and the re-score. Per-segment results let the auditor compute recall for each segment’s plan separately and point at the one that is failing, say a 9 M-vector segment with 30 % of its rows deleted, instead of reporting one blurred number per query. They cost a few hundred bytes per sample.
5. Could you measure recall online, per query, instead?
   Only by running the exact search alongside, which costs orders of magnitude more than the query for anything but tiny namespaces. Namespaces under ~50 k vectors are already searched exactly, so their recall is 1 by construction. For the rest, a daily batched measurement with alerts is the right granularity, because recall drifts over hours and days, not from one request to the next.