Design a News Feed

• Create a post — Text, images, video. Visibility: public or followers-only.
• Follow and unfollow users — Asymmetric follow graph. Some accounts have 100 M followers.
• View the home feed — Posts from followed accounts, newest first or ranked. Infinite scroll with stable pagination.
• View a user's profile timeline — That user's own posts, newest first.
• Like, comment, share — Counters shown on every post; engagement feeds ranking.
• Feed freshness — A new post from someone you follow appears within seconds on refresh.
• Out of scope — Search, DMs, notifications, ads insertion (mention where it would plug in), content moderation pipeline.

• Scale (500 M DAU) — Reads dominate: each user opens the feed several times a day; most never post.
• Feed load latency (p99 < 300 ms) — First page of the feed must be a cache read, never a query over the follow graph.
• Post visibility delay (< 5 s for most, < 60 s for celebrity posts) — Eventual consistency is fine; users tolerate a short delay but not missing posts.
• Availability (99.99 % reads) — Writes can degrade (post queued) as long as reading the feed works.
• No missing or duplicate posts — Pagination must be stable while new posts arrive.
• Privacy — Followers-only posts never leak to non-followers, including through caches after an unfollow or block.
• Cost — Fan-out is the dominant cost. The design must bound work per post regardless of follower count.

• Posts per second: ~6 k avg · 30 k peak — 500 M DAU, ~1 % post daily, ~1 post each = 5 M/day, plus reshares ≈ ~6 k/s. Peaks around events 5×. Writes are cheap; what they trigger is not.
• Feed reads per second: ~60 k avg · 300 k peak — 500 M DAU × 10 feed opens/day = 5 B/day ≈ 58 k/s, peak 5×. Each open reads one page (20–50 posts). Every one must be a cache hit.
• Average followers per user: ~200 (median ~50) — Heavy-tailed. Mean around 200, median under 100, and a few thousand accounts with over 1 M followers. The mean sizes total fan-out; the tail breaks naive designs.
• Fan-out writes per second: ~1.2 M — 6 k posts/s × 200 followers = 1.2 M feed inserts/s average, if every post fans out to every follower. One celebrity post with 100 M followers is 100 M writes on its own, which is why celebrities are handled differently.
• Feed cache size: ~2–5 TB (ids only) — Cache the most recent 500 post ids per user (8 B each ≈ 4 KB with metadata): 500 M × 4 KB = 2 TB for active users. Materialised hydrated posts are separate. Realistically 2–5 TB of Redis for feed lists; the 400 TB figure is what you would need if you cached full post bodies per user, which is why you do not.
• Post storage growth: ~2 PB/year (media excluded) — 5 M posts/day × ~1 KB metadata × 365 ≈ 1.8 TB/yr of rows; media is ~2 MB average × 40 % of posts ≈ 1.5 PB/yr in object storage. Media dominates storage; metadata dominates query load.
• Ranking inference per second: ~15 M candidates — 300 k feed opens/s at peak × ~50 candidates scored each = 15 M scores/s. Per-candidate inference must be sub-millisecond, batched, and cached per (user, session).

• Client — Web and mobile apps. Renders the feed, prefetches the next page, dedupes on post id, and records impressions and engagement for ranking.
• API gateway — Auth, rate limiting, request routing. Stateless.
• Post service — Creates posts, uploads media via pre-signed URLs, writes the post row, and publishes a post-created event. Owns the profile timeline read path.
• Feed service (read path) — Serves the home feed: reads the user's precomputed feed list, merges in celebrity posts fetched at read time, hydrates post bodies, calls ranking, and returns a page with a stable cursor.
• Fan-out workers — Consume post-created events, look up followers, and push the post id into each follower's feed list. Skip fan-out for celebrity authors and inactive followers. Horizontally scaled; the biggest compute pool in the system.
• Ranking service (ML inference) — Scores candidate posts for a user with a model over user, author, post, and engagement features. Returns an ordering plus diversity rules (no three posts from one author in a row).
• Social graph service — Follows, followers, blocks, mutes. Answers "followers of X" (for fan-out) and "does A follow B" (for privacy checks) from a cache in front of a sharded store.
• Event bus (Kafka) — post.created, post.deleted, engagement events, follow/unfollow. Fan-out workers, counters, and ranking feature pipelines consume from here.
• Feed cache (Redis · list per user) — For each active user, a sorted list of the most recent ~500 post ids from non-celebrity followees, keyed by user id. The entire first page of the feed is one ZREVRANGE. Evicted for users inactive for weeks and rebuilt on demand.
• Post cache (Redis · post id → body) — Hydrated post bodies and counters by post id. Hit rate is very high because recent posts are read by all their followers within hours. Backed by the posts DB.
• Posts DB (Cassandra / sharded MySQL) — Post rows keyed by post id (time-ordered), plus a per-author timeline table keyed by (author id, time) for profile pages and celebrity pulls. Append heavy.
• Graph DB (sharded by user id) — Follower and following adjacency lists, both directions materialised, sharded by user id so "followers of X" is one shard read. Celebrity follower lists are chunked.
• Media storage + CDN — Images and video uploaded directly from the client with pre-signed URLs, transcoded asynchronously, served via CDN. Posts reference media by id.
• Counter service (likes · comments · views) — Aggregates engagement events into per-post counters with approximate, eventually consistent values. Writes to the post cache; persists periodically.

1. Create a post and fan it out — The write path. The post is durable in milliseconds; delivery to followers' feeds happens asynchronously and is bounded regardless of follower count.
   1. Client uploads media directly to storage, then creates the post.
   2. Post service writes the post row and the author timeline entry.
   3. Post service publishes post.created and acks the client.
   4. Fan-out worker checks the author's follower count: celebrity or not?
   5. For a normal author, the worker fetches followers in chunks and pushes the post id into each active follower's feed list.
   6. Counters and ranking features consume the same event.
2. Load the home feed — The read path. One cache read for the precomputed list, one small query for celebrity posts, hydrate, rank, paginate. No follow-graph traversal on the hot path.
   1. Client requests the first page of the feed.
   2. Feed service reads the user's precomputed feed list from the cache.
   3. Feed service fetches recent posts from the celebrities this user follows.
   4. Feed service hydrates candidate post ids from the post cache.
   5. Feed service sends candidates to Ranking and gets an ordering.
   6. Feed service returns the page with a cursor; client renders and prefetches the next page.
3. A celebrity with 100 M followers posts — The case that breaks fan-out on write. Here the post costs nothing to publish and is pulled by readers, which spreads the cost over time and cache hits.
   1. Celebrity creates a post; it is written and the event is published like any other.
   2. Fan-out worker sees the celebrity flag and skips fan-out entirely.
   3. Followers open their feed; the Feed service pulls the celebrity's recent posts at read time.
   4. Post cache absorbs the read storm; counters update approximately.
   5. Delete: the celebrity removes the post; the delete propagates through the same path.
4. Scroll: stable pagination while new posts arrive — Offset pagination breaks the moment a new post lands. Cursors anchored on time-ordered ids keep the scroll stable and duplicate-free.
   1. Client requests the next page with the cursor from the previous response.
   2. Feed service reads the precomputed list below the cursor position.
   3. Celebrity posts are continued from their per-author cursor positions.
   4. Ranking is applied within the page, and already-shown posts are excluded.
   5. When the precomputed list is exhausted, fall back to on-demand assembly from followees' timelines.
5. Unfollow, block, and privacy — Precomputed feeds are copies. Every copy must respect the current follow graph and visibility, which means invalidation and read-time checks.
   1. User A unfollows B (or blocks B).
   2. Fan-out worker removes B's recent posts from A's feed list.
   3. Read-time safety check filters anything the cache missed.
   4. B switches their account to private.
   5. A deleted post is tombstoned and disappears from every feed.

Ask: Do you build each user's feed when a post is created, or when the user opens the app?

Good answers name: Hybrid: push for normal authors, pull for celebrities, Pure fan-out on write, Pure fan-out on read.

Our pick: Hybrid with a celebrity threshold around 1 M followers, tuned by measuring fan-out latency and read latency. Push writes post ids (not bodies) into a capped Redis sorted set per active follower; skip followers inactive for more than a few weeks and rebuild their feed lazily on return. Celebrities' posts are pulled from their author_timeline at read time, which is a single hot key per celebrity that caches perfectly. The Feed service merges both sources by time before ranking. Facebook and Instagram use variants of this; Twitter moved to it after the pure-push era.

1. Where exactly do you set the celebrity threshold and what happens as an account crosses it?
   Measure two curves: fan-out completion time versus follower count, and the read-time cost of one more pulled author. Pick the count where fan-out exceeds your visibility SLO (say 60 s). Crossing it is a flag flip on the author: new posts are pulled; old posts already fanned out stay in lists. Going back down is the same in reverse. Hysteresis avoids flapping.
2. A normal user with 500 k followers posts. How long until all followers see it?
   Chunk followers into tasks of 5 000; 100 tasks run in parallel across the worker fleet. Each task does 5 000 pipelined ZADDs in well under a second. End to end a few seconds, dominated by queueing. If the fleet is saturated, prioritise by author engagement so the posts most likely to be seen soon go first.
3. Why store post ids in the feed list rather than the full post?
   Storage (ids are 8 bytes, bodies are kilobytes), consistency (an edit or delete updates one place), and freshness of counters. The cost is a hydration step, which is a batched MGET against a cache with a very high hit rate. Storing bodies is the mistake that produces the 400 TB cache estimate.
4. How do you rebuild a feed for a user returning after two months?
   Their list was evicted. On open, take their following list, fetch the last N posts from each followee's author_timeline (parallel, bounded), merge by time, write the result into a new feed list, and serve it. That is fan-out on read for exactly one request, acceptable because it is rare, and it warms the list for subsequent opens.
5. The fan-out worker crashes halfway through a 500 k-follower post. What happens?
   Kafka redelivers the event to another worker. Each chunk task is idempotent because ZADD with the same member is a no-op, so followers already done are unaffected and the rest complete. If you track chunk completion in a small table you can skip finished chunks, but idempotent writes make that an optimisation, not a correctness requirement.

Ask: Reverse chronological is simple and honest. Why rank, and how do you do it without making the feed slow?

Good answers name: Two-stage: cheap candidates + learned scoring + rules, with a chronological fallback, Reverse chronological only, Precompute a ranked feed per user offline.

Our pick: Two-stage ranking inside the Feed request. Candidates: the ~100 most recent ids from the precomputed list plus celebrity pulls. Features: user embedding and recent activity from an online feature store, author and post features from the post cache, engagement counters, and pairwise affinity between reader and author. Model: a multi-task neural net predicting probability of like, comment, share, dwell, and hide, combined into a score with weights that encode product goals. Then rules: author diversity, freshness boost, demotion of borderline content. Hard 50 ms budget with a reverse-chronological fallback. Log features and outcomes for training so serving and training see identical inputs.

1. How do you avoid the feed showing the same top-ranked post every time the user refreshes?
   Track impressions: once shown, a post gets a decaying penalty in the session and is excluded after a couple of impressions. The cursor and a per-user recent-impressions set (short TTL) carry this. Also inject some freshness so a refresh reliably shows something new.
2. Where do the features come from within 50 ms?
   An online feature store (Redis or similar) keyed by user id and post id, populated by streaming jobs from engagement events. The request fetches the user's feature vector once and the candidates' feature vectors in one batched call, then runs inference on a GPU or optimised CPU model in a few milliseconds. Anything not in the store is not a feature.
3. A new post has no engagement yet. How does it ever get ranked high?
   Cold start is handled by content and author features (author's historical engagement rate with this reader, post type, predicted quality from text and image models) plus an explicit freshness prior. Some systems run a small exploration budget: show new posts to a slice of followers to gather signal quickly.
4. How would you measure whether ranking is helping?
   A/B test against chronological on a holdout: engagement (likes, comments, time spent), retention over weeks, and negative metrics (hides, reports, unfollows). Retention is the honest metric; engagement alone can be gamed by outrage. Keep a permanent small holdout on chronological to detect long-term drift.

Ask: What exactly is stored per user, where, how big, and what happens when it is lost?

Good answers name: Redis sorted set of post ids per user, capped, rebuildable, Per-user feed table in a wide-column store (Cassandra), Store hydrated post bodies per user.

Our pick: Redis Cluster, one sorted set per active user, member and score both the time-ordered post id, trimmed to 500 on every insert. Shard by user id. Replicas for read availability, no persistence: a lost shard is rebuilt lazily on read with a per-user lock to prevent duplicate rebuilds, and a background job re-warms recently active users to avoid a stampede. Evict users idle for more than ~30 days. Estimate ~10 KB per user, ~200 M active lists ≈ 2 TB across the cluster. Metrics that matter: hit rate on feed open (target above 99 %), rebuild rate, p99 ZREVRANGE latency.

1. A Redis shard holding 5 M users' feeds dies at peak. Walk through the next five minutes.
   Replica promotes in seconds; if there is no replica, every feed open on that shard misses. Each miss triggers a rebuild: 200 author_timeline reads and a merge. Bound it with a per-user rebuild lock, a global rebuild rate limit that serves a degraded chronological page directly from author timelines when exceeded, and a background warmer that rebuilds users in order of recent activity. Users see slower feeds for a few minutes, not errors.
2. Why cap at 500 and not 5 000?
   Users rarely scroll past a few hundred posts, memory scales linearly with the cap, and trims get slower. 500 covers hours to days of content for most users. Beyond it, fall back to on-demand assembly, which is slower but rare.
3. How does unfollow removal work without an index by author?
   Read the whole list (500 members with scores) and filter by author client-side, which needs the author id. Either encode author id into the member (post_id:author_id) or resolve via the post cache. Then ZREM the matches. It is a few hundred elements, so it is cheap; the simplicity beats maintaining a secondary index.
4. Would you use Redis Streams or Lists instead of sorted sets?
   Lists cannot do range-by-score or remove-by-member efficiently. Streams are append-only and ordered but removing arbitrary entries is awkward and range-by-id works only on stream ids. Sorted sets match the operation set exactly, which is why they are the usual choice.

Ask: One post read by ten million people in five minutes. What melts first?

Good answers name: Key replication + local cache + single-flight on miss, Bigger cache nodes, Serve celebrity posts from a CDN as static JSON.

Our pick: Three layers. Detect hot keys by sampling access counts per key in the Feed service. For hot keys, write N copies (post:{id}:{0..N-1}) and read a random one; invalidate all N on update. In front of Redis, each Feed server keeps a small in-memory cache with a 2 second TTL for post bodies and author timelines; at 300 k feed opens per second across a few hundred servers, this alone cuts Redis load on a viral key by orders of magnitude. On a cache miss, single-flight (one in-flight DB read per key per process) prevents the stampede. Counters for hot posts are updated in batches and displayed approximately.

1. How do you know a key is hot before it takes down a node?
   Sample: each Feed server counts accesses per key in a sliding window using a small top-K sketch and reports keys above a threshold. Or proactively: any post by an author above the celebrity threshold is treated as hot from creation. In practice both, since virality also happens to normal users.
2. The like count on a viral post shows 1.2 M on one refresh and 1.1 M on the next. Acceptable?
   It is a symptom of reading different replicas or local caches with different staleness. Acceptable in magnitude, but monotonic display is a nicer experience: the client can keep the max it has seen for a post within a session. Exact counts are not worth the coordination cost.
3. Does the same problem exist for the follow graph?
   Yes: "followers of celebrity X" is a 100 M-element list, and "does A follow X" is asked constantly. Store follower lists chunked, never load them whole except in fan-out (which is skipped for celebrities anyway), and answer follow checks from a per-reader following set, which is small.

Ask: What can a user actually rely on? Which delays and anomalies are acceptable, and which are bugs?

Good answers name: Eventual consistency for delivery, strong checks for privacy and deletion at read time, Strongly consistent feed (transactional fan-out), No read-time checks; rely on cache invalidation.

Our pick: State the guarantees explicitly. Delivery: at-least-once into feed lists within seconds (normal) or on next open (celebrity), deduped by post id at the client and cursor. Ordering: by time-ordered post id within a source, then re-ranked; the same session never shows a post twice. Privacy: visibility and block checks happen at hydration against current state, every time, so a stale cache copy can never leak. Deletion: tombstones at hydration, so a deleted post vanishes within the post cache TTL (seconds). Author: their own post appears immediately on their profile and is injected into their own feed at read time. Anything outside these guarantees is documented as a known and acceptable anomaly.

1. A user posts, immediately opens their home feed, and does not see their post. Bug?
   By the guarantees, yes: the author should see their own post. The fix is cheap: the Feed service injects the reader's own most recent posts at read time from their author_timeline, which is a single hot-for-them key. Never rely on fan-out to deliver an author's post to themselves.
2. How do you prevent the same post showing twice in one scroll session?
   Time-ordered ids and a cursor with a strict less-than boundary handle the chronological source. Ranking can pull an older post forward, so the cursor also carries a bloom filter of ids already served this session; hydration drops matches. The client dedupes by id as a final guard.
3. What does "new posts available" do and why not just insert them at the top?
   Inserting shifts content under the user's thumb. The client polls a lightweight "count of posts newer than my top id" endpoint, shows a pill, and only on tap fetches and prepends. It also gives the ranking system a natural boundary for a new session.
4. Blocked user B still saw A's post via a reshare by C. Whose bug?
   A product decision that must be explicit. Most networks: a block hides A's content from B everywhere, including reshares, so the hydration check must consider the original author of a reshared post, not just the resharer. If the check only looks at the top-level author, that is a bug in the design.