System design interview question

Design a Recommendation System

Q: You have a billion videos and 200 ms. Why not score a big candidate set with your best model, and how do you split the budget?

Six sources run in parallel with 30 ms deadlines and return ~2,500 candidates: long-term ANN 600, session ANN 400, co-watch 500, subscriptions 400, trending 300, fresh 300. After de-duplication and filters, ~2,000 go to an in-process pre-ranker (~20 cheap features, distilled from the heavy ranker) that keeps 500; the feed service fetches ~800 feature keys and the GPU ranker scores the 500 with multi-task heads in ~50 ms; the re-ranker picks 50 from the top ~150. The budget is 10 ms user context, 30 retrieval, 15 merge and pre-rank, 20 features, 50 heavy rank, 20 re-rank and respond, leaving ~55 ms of headroom for network and tail. Pre-rank recall is measured by scoring all 2,000 with the heavy ranker on 0.1 % of requests, off the critical path. Google’s 2016 paper “Deep Neural Networks for YouTube Recommendations” describes the same split: candidate generation narrows millions to hundreds, and a ranking network with richer features orders them, trained for expected watch time rather than click-through.

Q: At 100 k requests a second, what do you precompute, what do you cache, and what happens when part of the pipeline is slow?

Cache each input for exactly as long as it is valid. The long-term ANN result depends only on the daily user vector and the daily index, so it is cached all day under cand:{user}:{index_version} (~5 KB) and precomputed at low priority in the hour before each heavy user’s usual open; about 200 M users open daily and predictably, and light users are computed on request. The ranked remainder of each feed is cached for 10 minutes for scrolling, trending and popular lists live in memory on every feed host with a 60 s refresh, and video features sit in a per-host cache with a 60 s TTL, which cuts store reads from ~80 M to ~35 M a second. The final ranking always runs live. Under load the ladder is: drop slow sources at their deadlines, narrow the heavy-ranked set from 500 to 300 to 150, serve the pre-ranker’s order, serve the cached session list, serve popular in region, with policy filters on every rung. Each rung is recorded in the served log, and the popular path is the everyday path for signed-out users, so it is always warm.

Q: Features come from a warehouse when you train and from a key-value store when you serve, and some changed seconds ago. How do you keep them fresh in production and identical in training?

Every feature is defined once in a registry with its source, window, default and owner. Batch pipelines write the slow features daily (90-day topic affinities, video quality scores, and the user vectors from the daily build); a Flink job keyed by user id updates the last 200 watches, (user, channel) affinities and the session vector within seconds; per-video counters are pre-aggregated per minute. Both write the online store, a Bigtable, Cassandra or Aerospike class store keyed so that all of a user’s keys share a shard, and the offline store, from the same code. The feed service fetches the features, passes them to the ranker and logs that exact tensor (~21 KB per request) to feed.served, and the ranker trains on served logs joined with labels by request id. New features are backfilled with point-in-time joins until a few weeks of logs exist, and a daily job compares served values with offline recomputation per feature. Feature age is logged too, so a stale stream shows up on a dashboard instead of in the A/B.

Q: You retrain the two-tower model every day. Why can’t you update the item vectors in place, and what happens to a video uploaded an hour after the build?

The daily job trains, embeds the catalogue, builds 8 IVF-PQ shards and writes them to blob storage as v813, with the matching long-term user vectors written beside v812’s under versioned keys; the stream processor computes session vectors in both spaces during the overlap. Index servers load v813 alongside v812 (2 × ~9 GB per shard), and nothing serves until the checks pass: recall@600 against exact search on sampled user vectors, the distribution of cluster sizes, then 1 % of traffic as a canary on watch-time metrics. One config change then flips the pointer; each request reads it once and uses that version for every vector, query and cache key, so cached candidates under v812 simply stop being read. v812 stays loaded for an hour for rollback. New uploads go to the fresh index instead: HNSW over the last 48 h (~10 M videos, ~6 GB), embedded with the live version’s item tower from content features and inserted within minutes; 48 h covers a build cycle plus one failed build.

Q: A user installs the app and opens it for the first time, and a creator publishes their first video. What does each one get, and what does it cost you?

New videos: the stream processor embeds each video from its content within minutes of review and inserts it into the fresh index, so it reaches users whose session vectors are near it. Up to 3 of 50 slots in a feed are exploration slots for new videos, chosen by Thompson sampling over predicted engagement with priors from the channel’s history: about 4 % of impressions, ~2 B a day, or ~400 per new video on average. Budgets are set per channel and scaled by trust, so a channel uploading 10,000 junk videos does not get 10,000 budgets, and a video graduates after ~1,000 impressions, when its own rates are reliable. New users: country, language, device, time and optional onboarding topics pick a first page from popular-in-region lists plus exploration slots spread across topics, and a “has history” input lets the ranker learn separate behaviour for newcomers instead of reading empty history as bad history. After the first meaningful watch the session vector exists, and session ANN and co-watch take over on the next refresh; the next daily build adds a long-term vector. Measure new-creator first-week views, time to a new video’s first 100 views, and new-user day-1 and day-7 retention.

Q: You train on what people clicked in the feed your own model produced. Why is that a trap, and what do you log to get out of it?

The client logs an impression when at least half a tile is visible for a second, with request id, position, row layout and device; the served log already holds source, slot and model version. The ranker takes position and device through a small separate tower that is used in training and switched off at serving, so the main heads learn a video’s appeal without its slot; YouTube’s 2019 RecSys paper on its multitask ranking system describes this shallow-tower approach to position bias alongside multi-task heads. Objectives combine click, watch time, likes, dismissals and a satisfaction head trained on in-app survey answers, so a video that is clicked and then abandoned or dismissed scores badly. Two details from Google’s 2016 YouTube paper are worth borrowing: a fixed number of training examples per user, so heavy viewers do not dominate the loss, and an example-age feature set to zero at serving, so the model can learn how strongly users prefer fresh videos. A sub-1 % slice with adjacent positions swapped calibrates the position tower, and exploration slots carry their selection probabilities into the logs.

The “recommended for you” video home feed: 50 personalised videos picked from billions for 500 M daily users in under 200 ms.

Last updated 2026-10-26. Difficulty: hard. Patterns: recommendations, ranking, vector-search, feature-store, real-time. Reported at Google, Meta, TikTok, Netflix, Amazon, Pinterest.

Walk through a strong candidate's answer, turn by turn.

The interviewer asks, the candidate answers and draws, and you press Next. Skip to any stage or key moment, and ask the AI Mentor anything along the way.

Walk through it Take the mock interview

Or sit it in the style of Google · Meta · TikTok · Netflix, run as that company’s round is reported to go.

The short answer

How to design Recommendation System in a system design interview, in five steps:

Scope it. Home feed of 50 ranked videos on every open, personalised by long-term taste and by this session, candidates from several sources, diverse and not repetitive, safe by policy, A good first feed for new users, A fair start for new videos, feedback controls, learn from what was shown.
Set the targets. Latency: p99 < 200 ms server side; Scale: 500 M DAU · ~100 k feed requests/s peak; Freshness of user signals: a watch shapes the next refresh < 1 min; Freshness of content: new video recommendable < 15 min after review.
Do the numbers. Home-feed requests: ~58 k/s avg · ~100 k/s peak; Engagement events: ~20 B/day · ~400 k/s peak; Candidates per request: ~2.5 k → 2 k → 500 → 50.
Draw the design. API gateway, Home feed service, Candidate cache, Ranking service, Embedding ANN index, Candidate lists, Fresh-video index, Online feature store, Video catalog.
Go deep where it is hard.
- The funnel and its latency budget → Several retrieval sources, a light pre-ranker, a heavy ranker on 500, then rule-based re-ranking, each with a deadline.
- Serving 100 k requests per second → Live ranking over cached slow inputs: day-exact long-term candidates, precomputed for heavy users, shared lists in process, a brownout ladder.
- Features that are fresh and identical → Online and offline stores from shared definitions, batch plus stream pipelines, and training on features logged at serving time.
- Daily embeddings and the atomic index swap → Versioned daily build loaded beside the old one, checked, then swapped with one pointer; a separate fresh index for the last 48 h.
- Cold start for new users and new videos → Content embeddings and context first, per-channel exploration budgets for videos, fast session adaptation for users.
- Feedback loops and position bias → Viewport impressions with position and slot, position learned as a training-only effect, several objectives including satisfaction.

Functional requirements

Home feed of 50 ranked videos on every open. One call returns 50 videos and a feed token. Scrolling pages through the same ranked session list; pull-to-refresh asks for a new one.
Personalised by long-term taste and by this session. Watch history, likes, subscriptions and searches over months, plus what was watched minutes ago. Three cooking videos in a row should change the next refresh.
Candidates from several sources. Embedding similarity, “people who watched X watched Y”, subscriptions, trending in the region and fresh uploads. Each catches what the others miss, and the mix is tunable per experiment.
Diverse and not repetitive. At most 2 videos per channel in the first 10, a spread of topics, and nothing the user has finished or dismissed.
Safe by policy. Removed videos never appear; age-restricted and borderline ones are filtered or demoted for the viewer’s context. A takedown applies within minutes, including to cached lists.
A good first feed for new users. From country, language, device and optional onboarding topics, then personal within a few watches.
A fair start for new videos. An upload can be recommended within minutes of clearing review, before it has any watch history, through content signals and a capped exploration budget.
Feedback controls. “Not interested” and “Don’t recommend channel” apply on the next refresh; paused or cleared watch history stops influencing the feed.
Learn from what was shown. Every impression with its position and source, and what followed (click, watch time, like, dismiss, survey answer), feeds training and evaluation.
Out of scope. Search, the watch-next panel, ads, upload and transcoding (see Design YouTube), and model architectures beyond naming them. The round is about the system around the models.

Non-functional requirements

Latency (p99 < 200 ms server side). Split before anything is built: ~10 ms user context, 30 ms retrieval, 15 ms merge and pre-rank, 70 ms features and heavy ranking, 20 ms re-rank and respond, ~55 ms headroom. This drives the hardest tradeoff in the design: every extra candidate raises recall and costs feature reads and GPU time inside a fixed budget.
Scale (500 M DAU · ~100 k feed requests/s peak). Plus ~400 k events/s at peak into the stream. The fleet is sized by scoring work (candidates × requests), not by request count.
Freshness of user signals (a watch shapes the next refresh < 1 min). Typically seconds. The client also sends its last watched ids with the request, so a lagging stream degrades aggregates, not the obvious “I just watched this”.
Freshness of content (new video recommendable < 15 min after review). The main embedding index is rebuilt daily, so new uploads need a path that does not wait for the build.
Availability (99.95 % of opens get a feed). Degrade instead of failing: a less personal feed from a cache or a popular list beats an error on the app’s front door. Fewer than 1 % of feeds should be degraded on a normal day.
Policy correctness (0 removed videos served 5 min after takedown). Policy is re-checked at serve time against current labels, never trusted from a cached candidate list or a day-old index.
Quality measured on long-term outcomes (A/B on satisfied watch time + retention). Clicks alone reward clickbait. Primary metrics are watch time weighted by satisfaction and 7-day retention, with guardrails on dismissals, diversity and new-creator exposure.
Privacy (paused history ignored within 1 min). Paused or deleted watch history must leave online features, cached candidates and future training data, not just the history page.

Back-of-envelope estimates

Home-feed requests: ~58 k/s avg · ~100 k/s peak. 500 M DAU × 10 feed loads a day (opens, refreshes, scroll pages) = 5 B/day ÷ 86 400 ≈ 58 k/s. The evening peak runs ~1.7× the average: ~100 k/s.
Engagement events: ~20 B/day · ~400 k/s peak. 500 M users × ~40 events a day (one impression batch per feed load, plays, progress at 25/50/75 %, likes, dismissals) = 20 B/day ≈ 230 k/s average, ~400 k/s at peak. The impression batches carry ~50 B impressions a day. This is the stream the freshness requirement rides on.
Candidates per request: ~2.5 k → 2 k → 500 → 50. Six sources in parallel: long-term ANN 600, session ANN 400, co-watch of the last 10 watches 10 × 50, subscriptions up to 400, trending 300, fresh uploads 300 = 2.5 k. Duplicates and filters remove ~20 %: 2 k reach the pre-ranker, 500 the heavy ranker, 50 are served.
Heavy-ranker GPUs: ~500 at peak. 100 k requests/s × 500 candidates = 50 M scores/s. A batched multi-task ranker does ~100 k scores/s per GPU, so ~500 GPUs at peak before regional redundancy. The ranker is the most expensive box on the diagram, and the candidate count is its dial.
Feature lookups: ~80 M keys/s peak. ~800 keys per request (5 user keys, 500 video keys, ~300 user × channel keys) × 100 k/s = 80 M/s. Video features are shared across users, so a per-host cache absorbs ~90 % of the 50 M/s video reads; the online store still serves ~35 M/s, mostly user-keyed.
Main ANN index: 512 GB raw · ~70 GB compressed. ~1 B recommendable videos × 128 dims × 4 B = 512 B each = 512 GB raw. Product quantisation to 64 B per vector plus an 8 B id ≈ 72 GB: 8 shards of ~9 GB held in RAM, replicated for query rate. Full vectors stay on SSD for re-scoring.
Online feature store: ~5 TB · ~15 TB replicated. 1 B monthly users × ~4 KB (last 200 watches, two 128-dim fp16 vectors, topic and channel affinities) = 4 TB, plus 1 B videos × ~1 KB of counters, quality scores and policy flags = 1 TB. Three replicas: ~15 TB, small enough to hold entirely in memory or on local flash.
Serving log (features as served): ~100 TB/day. 5 B requests × (1 KB of user and context features + 50 served videos × ~400 B of features) ≈ 5 B × 21 KB ≈ 105 TB/day, ~3 PB kept for 30 days. It is the ranker’s training set, so it is not sampled: expensive, and still the cheapest fix for training/serving skew there is.

Components

Client app (mobile · TV · web): Asks for the home feed on open and pull-to-refresh, sending the ids of the last few videos it played as a hint. Renders 50 tiles and pages through them. Logs viewport impressions (video, position, visible time) in batches, because what was seen and skipped says as much as what was clicked.
API gateway (auth · experiments · deadline): Authenticates, resolves the experiment arms (which ranking model, which source mix) and forwards to the home feed service with a 200 ms deadline. Stamps the request id that every impression and log line carries back.
Home feed service (funnel orchestrator): Runs one request end to end: read user context, call six candidate sources in parallel with a 30 ms deadline each, merge and filter, pre-rank 2 k to 500 in process, fetch features and call the ranker, re-rank for diversity, freshness and policy, log what was served. It owns the latency budget and every fallback, so a slow stage narrows the funnel instead of failing the request. It fetches the ranker’s features itself, so the tensor that was scored is exactly the tensor that is logged.
Candidate cache (Redis · per user): Two entries per user. The long-term ANN result (600 ids, ~5 KB), keyed by user and index version: it depends only on the daily user vector and the daily index, so it is valid all day, and it is precomputed off-peak for heavy users. And the ranked remainder of the current feed, with a 10-minute TTL, so scrolling is a read rather than a new funnel.
Ranking service (GPU · multi-task heads): Pure model serving. Takes feature tensors for 500 candidates and returns predictions from multi-task heads: click, expected watch time, like, dismiss and survey satisfaction. Batches rows from many concurrent requests on each GPU. It fetches no features, so it scales, fails and is canaried as compute alone.
Embedding ANN index (IVF-PQ · ~1 B × 128-d): Item-tower embeddings of every recommendable video, compressed to 64 B and held in RAM across 8 shards with replicas (FAISS or ScaNN class). Queried with the long-term and session user vectors for the nearest few hundred. Rebuilt daily and swapped atomically together with the user vectors of the same training run, because vectors from two runs live in different spaces.
Candidate lists (co-watch · subs · trending): Precomputed lists keyed by something the request already knows: video id to its top 50 co-watched videos, channel id to recent uploads, region and language to trending and popular. One batched multi-get per source. The popular lists are pre-filtered for policy and double as the fallback feed when personal data is unavailable.
Fresh-video index (HNSW · last 48 h): A small ANN index of videos published in the last 48 hours (~10 M, ~6 GB), embedded from content features and inserted within minutes of an upload clearing review. It covers the gap until the next main build and is where new-video exploration candidates come from. Kept separate so the main index stays an immutable, verified artefact.
Online feature store (KV · user + video keys): Current feature values keyed by user, video and (user, channel): long-term and session vectors, last 200 watches, topic affinities, video engagement rates and policy flags, in a Bigtable, Cassandra or Aerospike class store. Daily batch values and seconds-fresh stream values are written from the same definitions as the offline store. All of a user’s keys hash to one shard, so the user half of a request is one round trip.
Video catalog (metadata · policy labels): Source of truth for video metadata, publish state and policy labels (removed, age-restricted, borderline). Every change is an event: publishes feed the fresh index, and takedowns reach a denylist held in memory on every feed host within a minute, which every request is filtered against.
Event stream (Kafka · key=user_id): Impression batches, watches and feedback from clients, served logs from the feed service, and catalog changes: ~400 k events/s at peak. User events are keyed by user id, so one consumer applies one user’s events in order, and no key is hot because one person produces a few events a second at most.
Stream processor (Flink · seconds): Three jobs. Updates each user’s recent history, (user, channel) affinities and session vector within seconds of a watch, plus per-video counters pre-aggregated per minute. Embeds newly published videos from their content and inserts them into the fresh index. Lands raw events and served logs in the offline store.
Offline store (Iceberg · served log + labels): The offline half of the feature store: the full history of served logs (features exactly as scored), impressions and watches, partitioned by hour, plus batch feature tables with point-in-time snapshots. Training examples are served logs joined with what happened next, by request id.
Training & builds (daily · versioned): Daily: retrains the two-tower model and the ranker, embeds the catalogue, builds the next index version, writes the matching user vectors and batch features, and rebuilds co-watch and popular lists. Every artefact carries one version tag, and nothing goes live until recall and canary checks pass.

User flows

Open the app: the funnel in 200 ms. The main read path. Six sources narrow a billion videos to 2,000, two models narrow those to 500 and then 50, and every stage has its own slice of the 200 ms.
1. Client asks for the home feed, sending the ids of the last videos it played. The gateway resolves the experiment arms and attaches a 200 ms deadline that every downstream call inherits. The recent-watch hint lets the feed reflect a watch even when the event stream is a minute behind.
2. Feed service reads the user’s context in one round trip. Long-term vector (daily), session vector (updated seconds after each watch), last 200 watches, dismissed channels and age setting, all on one shard, ~5 ms. Both vectors carry the index version they belong to, and the request pins that version for every later step.
3. Six candidate sources run in parallel, each with a 30 ms deadline. Long-term ANN (600, read from the candidate cache when present), session ANN (400), co-watch of the last 10 watches (10 × 50), subscriptions’ recent uploads (up to 400), trending in region and language (300) and fresh uploads (300). A source that misses its deadline is dropped, not waited for, and the served log records which sources answered.
4. Candidates are merged, de-duplicated and filtered to about 2,000. A video found by three sources keeps all three tags, which are themselves pre-rank features. Filters drop finished and dismissed videos, blocked channels, region and age restrictions, and anything on the takedown denylist held in memory on the host. Filtering before ranking keeps GPU time off videos that can never be shown.
5. An in-process pre-ranker scores the 2,000 and keeps 500. A light model on ~20 cheap features that travel with each candidate (retrieval scores, source tags, video age, duration, channel, global click rate), ~2 µs per candidate, under 10 ms. It is distilled from the heavy ranker, and its recall is measured: of the 50 the heavy ranker would serve from all 2,000, how many survive the cut to 500.
6. Feed service fetches the heavy features for 500 and calls the ranker. ~800 keys in one batched multi-get, most video keys from a per-host cache, ~20 ms. The ranker batches rows from many requests on each GPU and returns per-head predictions in ~50 ms. The feed service combines the heads into one value with weights owned by product and set per experiment.
7. The re-ranker builds the final 50 for diversity, freshness and policy. Greedy selection from the top ~150: at most 2 per channel in the first 10, a penalty for topic clusters already on the page, a boost for subscribed channels’ uploads under 24 h old, demotion of borderline videos, and up to 3 exploration slots for new videos from the fresh source. Rules live after the model so they can change without retraining.
8. The response returns, the remainder is cached for scrolling, and the served log is written. The next ~100 ranked videos are cached for 10 minutes under the feed token, so a scroll is a read. The served log carries every feature the ranker saw for the 50, their positions and sources, and the model and index versions: about 21 KB per request.
A watch shapes the next refresh. Fresh signals within a session. The long-term vector changes once a day; history, the session vector and counters change seconds after a watch, and the request hint covers any lag.
1. Client logs the impressions it showed and the watch that followed. Impressions are logged when a tile is actually visible (half the tile on screen for a second), not when it is served: a video served at position 40 that nobody scrolled to is not a negative. Each carries the request id and position, which is what joins it to the served log.
2. Events land on the stream, keyed by user. Keying by user puts one user’s events on one partition, so one consumer updates their history in order. No user is a hot key: a person produces a few events a second at most, even at ~400 k/s overall.
3. The stream processor updates the user’s history and session vector. A meaningful watch (over 30 s, or half of a short video) is appended to the last-200 history, bumps the (user, channel) affinity and moves the session vector: a recency-weighted mean of the item embeddings of the last 20 meaningful watches, with a 30-minute half-life. It is written with the index version of the embeddings it used; p99 under a minute, typically ~5 s.
4. Video counters update, pre-aggregated per minute. Impressions, clicks and watch time per video in 1 h and 24 h windows feed the ranker’s engagement features. The processor sums per video per minute before writing, so a video taking 50 k impressions a second costs one write a minute instead of a hot key.
5. The user pulls to refresh, and the session sources bring in related videos. The session ANN query uses the new session vector, and co-watch seeds from the last 10 watches, so three cooking videos bring cooking candidates. If the hint names a video newer than the session vector’s timestamp, the feed service seeds co-watch with it directly and nudges the vector by its embedding, so the refresh is right even when the stream is behind.
6. Long-term candidates come from the cache; ranking re-runs with the fresh features. One watch does not change long-term taste, so the 600 long-term ANN candidates are read back from the candidate cache instead of recomputed. The heavy ranker re-runs on the new mix with the fresh session features, so even cached candidates move.
Evening peak: 100 k requests a second. The scale-breaking case. Retrieval moves off the peak where it can, shared keys stay off the network, and the width of the funnel becomes the load-shedding knob.
1. Before the regional peak, a low-priority pass precomputes long-term candidates for heavy users. About 200 M users open the app every day at predictable times. Running their long-term ANN query in the hour before their usual open moves the heaviest index queries off the peak, and their first open reads 600 cached ids in ~1 ms. Light users are computed on request: precomputing for someone who opens twice a month wastes the work on most days.
2. A peak-hour open reads the precomputed candidates instead of querying the main index. The long-term ANN result is a pure function of the daily user vector and the daily index, so the cached copy is exact, not an approximation, until the next index version. Keys include the version, so the swap invalidates every entry at once without a purge.
3. Trending lists are read from an in-process copy. Every request in a region reads the same trending and popular lists: tens of thousands of reads a second on one key would be the hottest key in the system. Each feed host keeps the ~2,000 region and language lists in memory and refreshes them every 60 s, so they cost no network reads.
4. Video features come mostly from a per-host cache; user keys go to the store. Of ~80 M key reads a second, 50 M are video features shared by everyone. The ~10 M videos that take ~90 % of impressions fit in a few GB per host with a 60 s TTL, so the store sees ~35 M reads a second, spread evenly by user id.
5. The ranker batches rows across requests on each GPU. Dynamic batching waits up to 5 ms to fill a batch of ~4,000 rows (8 requests × 500), which is what gets one GPU to ~100 k scores/s: ~500 GPUs at peak. Capacity scales ahead of the daily curve, because the curve is predictable and a GPU replica takes minutes to load a model.
6. When GPU queueing rises, the feed service narrows the funnel instead of failing. A brownout ladder driven by the ranker’s queue time: heavy-rank 500, then 300, then 150 candidates, keeping the pre-ranker’s best. Each step costs a fraction of a percent of quality and saves 40 to 70 % of GPU time. Only past the last step does traffic skip the heavy ranker, and background prefetches are deferred first.
A dependency fails: degrade, never blank. Failure handling. Every stage has a cheaper fallback that still applies policy, and the served log records which fallback ran so training is not misled by it.
1. The fresh index misses its 30 ms deadline, and the request goes on without it. No retry inside the request: a retry doubles load on a struggling dependency and cannot finish in the budget anyway. The feed is slightly less fresh for a few seconds, and sources_ok in the served log shows it. A source failing across a whole cell is circuit-broken for 10 s.
2. The ranker times out, so the pre-ranker’s order goes through the re-ranker. Diversity, policy and the denylist still apply: the fallback skips the expensive model, never the safety rules. The served log marks the ranker as "prerank-fallback", so training does not learn from these impressions as if the heavy model had chosen them.
3. The feature store is unreachable for this user’s shard, so the cached session list is served. Without features there is nothing to rank with. If the user had a feed in the last 10 minutes, serve its cached remainder minus what they have since seen: personal and a few minutes old, which beats a generic list.
4. With no personal data at all, popular in region and language is served. The popular lists are already filtered for policy and safe for an unknown age. The same path serves signed-out users and the first open of new users, so it runs every second, not only during incidents, and is known to work.
5. The client gets a full feed with a degraded flag. Never a 5xx on the home screen for a ranking problem. The client shows the feed and quietly refreshes after 30 s when the flag is set, which usually lands on a recovered path.
6. Degraded feeds are counted by reason and alerted on. sources_ok, funnel_width and the fallback reason in the served log give per-minute dashboards: the share of feeds degraded, by reason and by cell. Normal is under 1 %. The SLO is 99.95 % of opens getting any feed; the degraded share is the early warning.
A new upload’s first day. The background path, and item cold start. Content gives a new video an embedding within minutes, exploration earns it real data, and the daily build turns that data into a learned place in the main index.
1. A video clears review and the catalog publishes an event. Only after automated policy review: an unreviewed video is never eligible, however fresh. The same topic carries takedowns, which reach every feed host’s denylist within a minute.
2. The stream processor embeds it from its content and inserts it into the fresh index. The item tower runs on content alone: title and description text, thumbnail and transcript embeddings, channel, language, duration. It was trained with the video-id embedding dropped for a share of examples, so a video with no history still lands near similar videos. Searchable within minutes; p99 under 15.
3. A viewer with a nearby session vector gets it as a fresh candidate, and it wins an exploration slot. Up to 3 of 50 slots go to new videos chosen by Thompson sampling over predicted engagement, with priors from the channel’s history. Each new video has an impression budget, about 400 on average and more for strong priors, and the served log records the slot and its selection probability.
4. Its impressions and watches flow back as counters within a minute. Click and watch rates are shrunk toward the channel’s average until the video has enough impressions of its own, so 3 clicks in 5 impressions does not read as a 60 % click rate. After roughly 1,000 impressions the ranker’s engagement features carry it and the exploration boost ends.
5. The day’s logs land in the offline store and the nightly job joins them with labels. Training examples are served logs joined with impressions and watches by request id, over a window long enough for late events (a watch finished hours later). The ranker trains on the logged features; the two-tower model trains on watches, so the new video now gets a learned id embedding.
6. The build writes a new index version, matching user vectors and new co-watch lists. Item embeddings for ~1 B videos, new IVF centroids and PQ codebooks, and 8 shards written to blob storage as v813. User vectors for v813 are written under their own keys beside v812’s, and the co-watch lists now include the new video.
7. Index servers load v813 beside v812, the checks pass, and the version pointer flips. Each shard loads the new version alongside the old (2 × ~9 GB), recall@600 against exact search is checked on sampled user vectors, and 1 % of traffic runs on v813. Then one config change flips the pointer; each request reads it once and uses one version for every vector and query. The ranker model ships the same way, by canary.
8. From now on it comes from the main index and co-watch, and the fresh index drops it at 48 h. The new video now competes on a learned embedding and real engagement, with no exploration boost. The fresh index keeps 48 h, enough to cover a build cycle plus one failed build, and trims older entries hourly.

Deep dives

The funnel and its latency budget

You have a billion videos and 200 ms. Why not score a big candidate set with your best model, and how do you split the budget?

Scoring is the cost. A heavy multi-task ranker does ~100 k scores a second per GPU. At 100 k requests a second, scoring 10,000 candidates each would be 1 B scores a second, or 10,000 GPUs; scoring the whole catalogue is not even a question. Retrieval has the opposite problem: cheap per item, but coarse, since a nearest-neighbour query knows about the user only what one vector holds.

The insight is that each stage only has to keep what the next stage would want, and that is measurable. Cheap stages see many items with few features; expensive stages see few items with many. The budget is set per stage before choosing components, and each stage’s recall is measured against the stage after it, so widening or narrowing the funnel is a data decision rather than a guess.

Several retrieval sources, a light pre-ranker, a heavy ranker on 500, then rule-based re-ranking, each with a deadline chosen
One embedding retrieval, then the heavy ranker situational: an early product with a small catalogue, or a single surface such as “more like this video”
Heavy-rank all 2,000 retrieved candidates, with no pre-ranker situational: at a tenth of the traffic, where GPU cost matters less than simplicity
Precompute each user’s ranked feed offline rejected

The answer: Six sources run in parallel with 30 ms deadlines and return ~2,500 candidates: long-term ANN 600, session ANN 400, co-watch 500, subscriptions 400, trending 300, fresh 300. After de-duplication and filters, ~2,000 go to an in-process pre-ranker (~20 cheap features, distilled from the heavy ranker) that keeps 500; the feed service fetches ~800 feature keys and the GPU ranker scores the 500 with multi-task heads in ~50 ms; the re-ranker picks 50 from the top ~150. The budget is 10 ms user context, 30 retrieval, 15 merge and pre-rank, 20 features, 50 heavy rank, 20 re-rank and respond, leaving ~55 ms of headroom for network and tail. Pre-rank recall is measured by scoring all 2,000 with the heavy ranker on 0.1 % of requests, off the critical path. Google’s 2016 paper “Deep Neural Networks for YouTube Recommendations” describes the same split: candidate generation narrows millions to hundreds, and a ranking network with richer features orders them, trained for expected watch time rather than click-through.

How do you know the pre-ranker is not throwing away videos the heavy ranker would have served?

Measure it. On 0.1 % of requests, score all 2,000 candidates with the heavy ranker asynchronously, after the response has gone, and check how many of its top 50 survived the pre-ranker’s cut to 500; the target is above 95 %. The pre-ranker is distilled from the heavy ranker’s scores, so a drop usually means the two drifted apart after a heavy-ranker release. The fix is to retrain the pre-ranker as part of every heavy-ranker release.

Your p99 is 260 ms. Where do you look first?

The per-stage timings in the served log, at p99, not the average. The usual culprit is the feature fetch: a request that reads 800 keys waits for the slowest, so its p99 is roughly the store’s p99.9. Fixes are fewer keys (the per-host video cache), grouping keys by shard, and hedged reads after the store’s p95. Next is GPU queueing at peak, which the brownout ladder handles, then a slow ANN shard, which replicas with hedging cover.

Heavy-ranking 1,000 candidates instead of 500 improves watch time by 0.3 %. Do you ship it?

It is a cost question with a number on both sides: ~500 more GPUs at peak against 0.3 % of watch time on a platform this size, which may well be worth it. First check cheaper routes to the same gain: a better pre-ranker that puts those videos inside the 500, or a better source. If it ships, the brownout ladder is adjusted so the extra 500 are the first thing shed at peak.

Why does each source get its own deadline instead of one deadline for retrieval?

With one shared deadline the slowest source sets the latency for all of them, and one sick dependency takes the whole stage down with it. Per-source deadlines let the merge go ahead with whatever arrived. Losing trending for a few seconds is invisible; losing the feed is not. The served log records which sources answered, so the quality cost is measurable.

Why keep diversity and policy as rules after the model instead of features inside it?

Policy cannot be a probability: a removed video needs zero chance, which a weight cannot promise. Diversity is a property of the page, while the ranker scores videos independently. Rules after the model are also fast to change: a new limit on a topic ships as config the same day, while retraining takes days.

Serving 100 k requests per second

At 100 k requests a second, what do you precompute, what do you cache, and what happens when part of the pipeline is slow?

Each request runs six sources, ~800 feature reads and 500 GPU scores, which makes the home feed the most expensive request on the platform. It has to reflect the last few minutes, so the final ranking cannot be prepared in advance. But much of the work feeding it changes far more slowly: the long-term user vector and the main index change once a day, trending once a minute, and video features are the same for every user.

So cache by how fast each input changes, and degrade along the same lines. Anything that is a pure function of daily inputs can be computed once a day, ahead of the user, for users who will almost certainly open the app. Anything shared by many users should never cross the network per request. And under load, the funnel’s width is a better shedding knob than rejecting requests.

Live ranking over cached slow inputs: day-exact long-term candidates, precomputed for heavy users, shared lists in process, a brownout ladder chosen
Compute everything on every request rejected
Cache the final feed per user for minutes situational: for scrolling within one feed (the ranked remainder, 10 minutes) and as a fallback when features are unavailable
Reject excess requests at the gateway when saturated situational: as the last resort, and first for background prefetches, after the brownout ladder is exhausted

The answer: Cache each input for exactly as long as it is valid. The long-term ANN result depends only on the daily user vector and the daily index, so it is cached all day under cand:{user}:{index_version} (~5 KB) and precomputed at low priority in the hour before each heavy user’s usual open; about 200 M users open daily and predictably, and light users are computed on request. The ranked remainder of each feed is cached for 10 minutes for scrolling, trending and popular lists live in memory on every feed host with a 60 s refresh, and video features sit in a per-host cache with a 60 s TTL, which cuts store reads from ~80 M to ~35 M a second. The final ranking always runs live. Under load the ladder is: drop slow sources at their deadlines, narrow the heavy-ranked set from 500 to 300 to 150, serve the pre-ranker’s order, serve the cached session list, serve popular in region, with policy filters on every rung. Each rung is recorded in the served log, and the popular path is the everyday path for signed-out users, so it is always warm.

The candidate cache loses a node at the peak. What happens?

Those users miss and fall through to a live long-term ANN query, which is the normal path for light users anyway. The risk is load: a node’s worth of heavy users adds index queries during the busiest hour. The ANN tier is sized to absorb one cache node’s share, and if it cannot, the long-term source misses its 30 ms deadline and the feed is built from the other five sources. Nothing about correctness depends on the cache.

How do you decide which users are worth precomputing for?

Predict the probability of an open in the next few hours from the last 28 days of opens by hour of day, and precompute when that probability times the retrieval saved beats the cost of the precompute; in practice that is anyone above about 50 %. Track the waste, entries that expire unread, and raise the threshold if it climbs above about 30 %.

At 10x the traffic, what breaks first?

GPU cost, because the ranker grows with requests times candidates: 10x is ~5,000 GPUs. The levers are a narrower heavy set behind a better pre-ranker, distilling the ranker into a smaller model, and caching ranker outputs within a session for candidates whose features did not change. Feature reads come next and shard horizontally by user. Retrieval is the easiest: more index replicas.

Would you cache the whole response for a few seconds?

Yes, for a few seconds: a 5-second cache keyed by user and request parameters absorbs double taps, retries and app restarts with no visible staleness. That is different from caching the feed for minutes, which would make pull-to-refresh look broken. Responses served from it are marked in the served log so their impressions are not double counted.

How do you make sure the fallbacks still work when you need them?

Run them all the time. The popular path serves signed-out users every second. The pre-ranker-only path serves a small permanent holdout, which also measures what the heavy ranker is worth. And game days in one cell switch off the ranker or the feature store on purpose, checking that the degraded share and latency stay inside the SLO.

Features that are fresh and identical

Features come from a warehouse when you train and from a key-value store when you serve, and some changed seconds ago. How do you keep them fresh in production and identical in training?

The ranker reads ~800 feature keys per request, 80 M a second at peak, inside a 20 ms slice. Some are daily aggregates; some must move within seconds of a watch. If training rebuilds the same features from the warehouse through different code (a SQL window instead of a streaming one, NULL where serving returns 0, a value computed after the event instead of before it), the model learns from numbers it never sees in production. Offline metrics look good, the A/B is flat, and nothing errors.

The way out is to stop reconstructing what the model saw and record it. If the feed service logs the exact feature tensor it sent to the ranker, training on that log is free of skew by construction, including for stream features read at a particular second. Freshness then becomes a separate question: which features need the stream, and how fast.

Online and offline stores from shared definitions, batch plus stream pipelines, and training on features logged at serving time chosen
Recompute training features from the warehouse with point-in-time joins situational: to backfill a new feature for its first weeks, until enough logged history exists
Daily batch features only rejected
Let the model server compute features from raw events at request time rejected

The answer: Every feature is defined once in a registry with its source, window, default and owner. Batch pipelines write the slow features daily (90-day topic affinities, video quality scores, and the user vectors from the daily build); a Flink job keyed by user id updates the last 200 watches, (user, channel) affinities and the session vector within seconds; per-video counters are pre-aggregated per minute. Both write the online store, a Bigtable, Cassandra or Aerospike class store keyed so that all of a user’s keys share a shard, and the offline store, from the same code. The feed service fetches the features, passes them to the ranker and logs that exact tensor (~21 KB per request) to feed.served, and the ranker trains on served logs joined with labels by request id. New features are backfilled with point-in-time joins until a few weeks of logs exist, and a daily job compares served values with offline recomputation per feature. Feature age is logged too, so a stale stream shows up on a dashboard instead of in the A/B.

You add a new feature today. How do you train with it tomorrow?

Backfill it with a point-in-time join over the offline store: for each logged request, the feature’s value as of that request’s timestamp. Train on the backfill, and start logging the feature at serving at once, before any model uses it. After two to four weeks, switch training to the logged values and compare the two; any gap is the skew the backfill had.

The stream processor falls 20 minutes behind. What does the user see, and how do you know?

Session vectors and recent history go stale, so a refresh reflects the last few watches less. The request hint still seeds co-watch with the video just watched, so the obvious case keeps working. You know because every feature carries its update time and the served log records feature age; an alert fires when the p99 age of session features passes a few minutes, next to the processor’s consumer lag.

Why not log features for 1 % of requests and save 99 TB a day?

Because the rare cases are the ones the model needs most: new videos, small creators, new users. A 1 % sample keeps 1 % of a new video’s few hundred impressions. Better savings are to drop served items that never entered the viewport and to down-sample unclicked impressions with a recorded weight, keeping every positive. Next to ~500 GPUs, ~3 PB of storage for 30 days is cheap.

How do you keep 800 key reads inside 20 ms?

Fewer reads and fewer round trips. Video features are cached on the feed host, so most of the 500 video reads never leave the machine. All user keys, including the ~300 (user, channel) keys, hash by user id to one shard, so the user half is one batched call. Then hedge any shard that has not answered by its p95.

A feature returns 0 online when missing and NULL in the warehouse. How would you have caught it?

Training on logged features never sees the warehouse version, so a logged feature is safe; a backfilled one is not. The daily comparison catches it: it recomputes a sample of served features offline and compares distributions, and a spike of zeros online against nulls offline stands out. The real fix is in the registry, where the default is part of the feature’s definition and used by both paths.

Daily embeddings and the atomic index swap

You retrain the two-tower model every day. Why can’t you update the item vectors in place, and what happens to a video uploaded an hour after the build?

A daily build embeds ~1 B videos, trains new IVF centroids and PQ codebooks, and writes 8 shards: ~512 GB of raw vectors becoming ~70 GB of index, a few hours of work. Each training run produces its own vector space. A user vector from today’s run queried against yesterday’s index returns neighbours that are close only by accident, and nothing errors. Mixing versions during a rollout, or inside a cache, is a silent quality outage.

So the unit of release is not the index. It is the index plus every user vector, session vector and cached candidate list in the same space, under one version tag. And a daily cadence means a video uploaded just after a build waits a day or more for an embedding, which is exactly when a new video most needs a chance.

Versioned daily build loaded beside the old one, checked, then swapped with one pointer; a separate fresh index for the last 48 h chosen
Update item vectors in place as the model retrains rejected
Append new uploads to the main index under the current version situational: when there is no exploration requirement and running one index matters more
Backward-compatible training that keeps new vectors comparable with old ones situational: when vectors are stored in many systems that cannot all switch at once

The answer: The daily job trains, embeds the catalogue, builds 8 IVF-PQ shards and writes them to blob storage as v813, with the matching long-term user vectors written beside v812’s under versioned keys; the stream processor computes session vectors in both spaces during the overlap. Index servers load v813 alongside v812 (2 × ~9 GB per shard), and nothing serves until the checks pass: recall@600 against exact search on sampled user vectors, the distribution of cluster sizes, then 1 % of traffic as a canary on watch-time metrics. One config change then flips the pointer; each request reads it once and uses that version for every vector, query and cache key, so cached candidates under v812 simply stop being read. v812 stays loaded for an hour for rollback. New uploads go to the fresh index instead: HNSW over the last 48 h (~10 M videos, ~6 GB), embedded with the live version’s item tower from content features and inserted within minutes; 48 h covers a build cycle plus one failed build.

The new build has a bug and its recall dropped by a third. How do you catch it before users do?

The recall check is the gate: before any traffic, compare the index’s top 600 with exact search on a few thousand sampled user vectors, and a drop below the threshold blocks the flip. Cluster sizes catch degenerate centroids. If something gets past both, the canary watches watch time and the share of feeds whose long-term source returned results, and the flip is reverted automatically while v812 is still loaded.

Halfway through loading, half the shards have v813. A query fans out to all shards. What does it see?

Only v812, because the pointer has not flipped and every query names its version. Each shard answers for the version it is asked for from whichever copy it holds, and the flip waits until every shard reports v813 loaded. No request can be served half from one space and half from the other.

Why not embed new uploads straight into the main index?

You could: IVF-PQ can append vectors using the existing centroids and codebooks. I keep them separate for two reasons. The main index stays an immutable artefact that was verified and can be rolled back. And a separate source guarantees new videos a quota of candidates, whereas inside the main index a content-only vector rarely wins against vectors learned from millions of watches.

How long until a new video reaches the main index, and does that matter?

Up to a day for the next build to start plus a few hours to build, so 6 to 30 hours. It does not matter because the fresh index covers it within minutes, and by the time it reaches the main index it has real watches, so its learned embedding is better than its content one.

The daily build fails. What happens?

Nothing visible: v812 keeps serving, and its user vectors and cached candidates stay valid. Embeddings age by a day, which costs a little quality, and the fresh index still covers new uploads because it keeps 48 hours. Two failures in a row page someone, because uploads older than 48 h with no main-index embedding then start to drop out of retrieval.

Cold start for new users and new videos

A user installs the app and opens it for the first time, and a creator publishes their first video. What does each one get, and what does it cost you?

Both have no history, and history is what the strongest signals are made of: the user tower’s input, a video’s learned id embedding and its engagement rates. At 5 M uploads a day, giving every new video 1,000 impressions to measure it would cost 5 B impressions a day, a tenth of all 50 B. Giving them none means the feed only ever shows what it already knows, and new creators stop uploading.

Neither is unknown; both are unobserved. A new user has a country, a language, a device, a time of day and maybe a few topic picks; a new video has a title, a thumbnail, a transcript and a channel. The design question is how far that content gets them, and how to buy the rest with a bounded, measured amount of exploration. Contractor Matching covers how exploration traffic is logged and reweighted; here the problem is sizing and placing it at this volume.

Content embeddings and context first, per-channel exploration budgets for videos, fast session adaptation for users chosen
Popular only, until there is enough data rejected
Uniform random exploration rejected
A long onboarding questionnaire situational: on TVs and shared devices with no other signal, kept to one optional screen of topics

The answer: New videos: the stream processor embeds each video from its content within minutes of review and inserts it into the fresh index, so it reaches users whose session vectors are near it. Up to 3 of 50 slots in a feed are exploration slots for new videos, chosen by Thompson sampling over predicted engagement with priors from the channel’s history: about 4 % of impressions, ~2 B a day, or ~400 per new video on average. Budgets are set per channel and scaled by trust, so a channel uploading 10,000 junk videos does not get 10,000 budgets, and a video graduates after ~1,000 impressions, when its own rates are reliable. New users: country, language, device, time and optional onboarding topics pick a first page from popular-in-region lists plus exploration slots spread across topics, and a “has history” input lets the ranker learn separate behaviour for newcomers instead of reading empty history as bad history. After the first meaningful watch the session vector exists, and session ANN and co-watch take over on the next refresh; the next daily build adds a long-term vector. Measure new-creator first-week views, time to a new video’s first 100 views, and new-user day-1 and day-7 retention.

Exploration slots get half the click rate of normal slots. How do you justify them?

With a long-term holdout: a small share of users and creators with exploration switched off, measured over months on creator upload rates, catalogue coverage and user retention. The short-term cost is visible, about 2 % of the clicks in those positions. The long-term cost of not exploring shows up as fewer new creators and a staler feed, and the share is tuned on that experiment rather than argued.

A spammer uploads 10,000 videos a day to farm exploration impressions.

Budgets are per channel, not per video, so 10,000 uploads share one channel’s budget. Channel trust (age, past engagement, policy strikes) scales the budget, and a new channel starts small. Videos must also clear automated review before they are eligible at all, and a low predicted quality from the content model shrinks the budget further.

A signed-out viewer on a shared TV. What do they get?

There is no identity, so the device and the session are the user: popular in region and language for the first page, then session-only adaptation from what is watched, discarded afterwards. Safety settings default to conservative because the viewer’s age is unknown, and nothing from that session leaks into a signed-in profile later.

How fast does a new user’s feed become personal?

On the first refresh after their first meaningful watch, since the session vector and co-watch seeds exist seconds later. After about three watches the session sources dominate the candidate mix, and the long-term vector arrives with the next daily build. The metric is engagement per feed for new users by number of prior watches, which should climb steeply over the first five.

Feedback loops and position bias

You train on what people clicked in the feed your own model produced. Why is that a trap, and what do you log to get out of it?

At ~50 B impressions a day the training data is plentiful and biased three ways. The top tile gets clicked more because it is at the top. A video never shown has no label at all. And a click is a cheap signal: a model trained only on clicks learns that misleading thumbnails work. Each bias feeds the next model, which shows more of the same, which produces more of the same labels.

The way out is in the logs. Log what was seen, not what was served or what was clicked: each impression with its position, its source and whether it filled an exploration slot. Then the position effect can be learned apart from the video’s appeal, the objectives can include signals clickbait cannot fake, and a sliver of traffic can be kept honest on purpose.

Viewport impressions with position and slot, position learned as a training-only effect, several objectives including satisfaction chosen
Train on clicks from the served feed as they are rejected
Weight each example by its estimated chance of being examined situational: for models that cannot take position as an input, such as the pre-ranker’s training set
Randomise the order of a share of feeds situational: on a tiny slice, well under 1 %, ideally as swaps of adjacent positions to calibrate the position model

The answer: The client logs an impression when at least half a tile is visible for a second, with request id, position, row layout and device; the served log already holds source, slot and model version. The ranker takes position and device through a small separate tower that is used in training and switched off at serving, so the main heads learn a video’s appeal without its slot; YouTube’s 2019 RecSys paper on its multitask ranking system describes this shallow-tower approach to position bias alongside multi-task heads. Objectives combine click, watch time, likes, dismissals and a satisfaction head trained on in-app survey answers, so a video that is clicked and then abandoned or dismissed scores badly. Two details from Google’s 2016 YouTube paper are worth borrowing: a fixed number of training examples per user, so heavy viewers do not dominate the loss, and an example-age feature set to zero at serving, so the model can learn how strongly users prefer fresh videos. A sub-1 % slice with adjacent positions swapped calibrates the position tower, and exploration slots carry their selection probabilities into the logs.

How do you know the position correction works?

On the swap slice, a video’s click rate should differ between position 3 and position 4 only by what the position tower predicts. Offline, compare rankings with and without the tower against the swap data. Online, the A/B of the corrected model should lift watch time from lower positions, not only reshuffle the top.

Watch time favours long videos. Is that a problem?

Yes, if it is the only objective: a 40-minute video watched for 8 minutes outranks a 3-minute video watched to the end. So the heads include completion and satisfaction, and the value weights are tuned per surface and by video length. The guardrail is the share of short videos, which should not collapse when a new ranker ships.

Why log impressions on the client? The server knows what it served.

The server knows what it sent, not what was seen. Of 50 served videos a user typically sees about 10, and training on the other 40 as negatives teaches the model that everything below the fold is unappealing, which is position bias in its worst form. Only the client knows what entered the viewport and for how long.

What stops a user’s feed collapsing into one topic?

Three things outside the model: sources with different reasons to watch (subscriptions, trending, fresh), the re-ranker’s topic penalty on the page, and exploration slots. And one inside: the satisfaction head, trained on survey answers, which tends to reward variety that clicks alone do not. Topic entropy per user over time is a guardrail metric.

Survey answers are rare. How does a head train on them?

Shared layers learn from the dense signals (click, watch time), and the satisfaction head sits on top with few parameters, so a few million answers a day are enough. Answers are weighted to correct for who tends to answer surveys, and the head’s predictions are checked for calibration against fresh survey results every week.

The theory behind it

Vector search (IVF): an interactive playground. Cluster vectors with k-means, search the nearest lists for a query, and see how the number of lists and probes trades recall against the vectors compared.
Multi-armed bandits: an interactive playground. Serve three headlines with an even A/B split, epsilon-greedy and Thompson sampling, and compare clicks earned, clicks lost while learning and how often each picks the real winner.
Recommendation systems: How large recommendation systems are built: candidate generation, ranking and re-ranking, embeddings and two-tower models, features and feature stores, offline and online serving, feedback loops, cold start, and how to evaluate them.
Feature stores and serving machine learning models: The infrastructure behind production machine learning: offline and online feature stores, training-serving skew, point-in-time correct training data, real-time features from streams, model serving with latency budgets, batching, shadow and canary deployments, and monitoring drift.
Vector search, embeddings and RAG: Embeddings and similarity, approximate nearest-neighbour indexes (HNSW, IVF, product quantisation), vector databases versus adding vectors to your existing store, hybrid search, and retrieval-augmented generation (RAG) pipelines end to end.