Design a News Feed
The home timeline of Twitter, Instagram or Facebook: posts from people you follow, ranked, fresh, and fast for hundreds of millions of users.
Last updated 2026-09-22. Difficulty: hard. Patterns: fan-out, ranking, caching, social. Reported at Anthropic and 6 more with Pro.
The interviewer asks, the candidate answers and draws, and you press Next. Pause to answer yourself at the key decisions, and ask the coach anything along the way.
Functional requirements
- Create a post. Text, images, video. Visibility: public or followers-only.
- Follow and unfollow users. Asymmetric follow graph. Some accounts have 100 M followers.
- View the home feed. Posts from followed accounts, newest first or ranked. Infinite scroll with stable pagination.
- View a user's profile timeline. That user's own posts, newest first.
- Like, comment, share. Counters shown on every post; engagement feeds ranking.
- Feed freshness. A new post from someone you follow appears within seconds on refresh.
- Out of scope. Search, DMs, notifications, ads insertion (mention where it would plug in), content moderation pipeline.
Non-functional requirements
- Scale (500 M DAU). Reads dominate: each user opens the feed several times a day; most never post.
- Feed load latency (p99 < 300 ms). First page of the feed must be a cache read, never a query over the follow graph.
- Post visibility delay (< 5 s for most, < 60 s for celebrity posts). Eventual consistency is fine; users tolerate a short delay but not missing posts.
- Availability (99.99 % reads). Writes can degrade (post queued) as long as reading the feed works.
- No missing or duplicate posts. Pagination must be stable while new posts arrive.
- Privacy. Followers-only posts never leak to non-followers, including through caches after an unfollow or block.
- Cost. Fan-out is the dominant cost. The design must bound work per post regardless of follower count.
Back-of-envelope estimates
- Posts per second: ~6 k avg · 30 k peak. 500 M DAU, ~1 % post daily, ~1 post each = 5 M/day, plus reshares ≈ ~6 k/s. Peaks around events 5×. Writes are cheap; what they trigger is not.
- Feed reads per second: ~60 k avg · 300 k peak. 500 M DAU × 10 feed opens/day = 5 B/day ≈ 58 k/s, peak 5×. Each open reads one page (20–50 posts). Every one must be a cache hit.
- Average followers per user: ~200 (median ~50). Heavy-tailed. Mean around 200, median under 100, and a few thousand accounts with over 1 M followers. The mean sizes total fan-out; the tail breaks naive designs.
- Fan-out writes per second: ~1.2 M. 6 k posts/s × 200 followers = 1.2 M feed inserts/s average, if every post fans out to every follower. One celebrity post with 100 M followers is 100 M writes on its own, which is why celebrities are handled differently.
- Feed cache size: ~2–5 TB (ids only). Cache the most recent 500 post ids per user (8 B each ≈ 4 KB with metadata): 500 M × 4 KB = 2 TB for active users. Materialised hydrated posts are separate. Realistically 2–5 TB of Redis for feed lists; the 400 TB figure is what you would need if you cached full post bodies per user, which is why you do not.
- Post storage growth: ~2 PB/year (media excluded). 5 M posts/day × ~1 KB metadata × 365 ≈ 1.8 TB/yr of rows; media is ~2 MB average × 40 % of posts ≈ 1.5 PB/yr in object storage. Media dominates storage; metadata dominates query load.
- Ranking inference per second: ~15 M candidates. 300 k feed opens/s at peak × ~50 candidates scored each = 15 M scores/s. Per-candidate inference must be sub-millisecond, batched, and cached per (user, session).
Components
- Client: Web and mobile apps. Renders the feed, prefetches the next page, dedupes on post id, and records impressions and engagement for ranking.
- API gateway: Auth, rate limiting, request routing. Stateless.
- Post service: Creates posts, uploads media via pre-signed URLs, writes the post row, and publishes a post-created event. Owns the profile timeline read path.
- Feed service (read path): Serves the home feed: reads the user's precomputed feed list, merges in celebrity posts fetched at read time, hydrates post bodies, calls ranking, and returns a page with a stable cursor.
- Fan-out workers: Consume post-created events, look up followers, and push the post id into each follower's feed list. Skip fan-out for celebrity authors and inactive followers. Horizontally scaled; the biggest compute pool in the system.
- Ranking service (ML inference): Scores candidate posts for a user with a model over user, author, post, and engagement features. Returns an ordering plus diversity rules (no three posts from one author in a row).
- Social graph service: Follows, followers, blocks, mutes. Answers "followers of X" (for fan-out) and "does A follow B" (for privacy checks) from a cache in front of a sharded store.
- Event bus (Kafka): post.created, post.deleted, engagement events, follow/unfollow. Fan-out workers, counters, and ranking feature pipelines consume from here.
- Feed cache (Redis · list per user): For each active user, a sorted list of the most recent ~500 post ids from non-celebrity followees, keyed by user id. The entire first page of the feed is one ZREVRANGE. Evicted for users inactive for weeks and rebuilt on demand.
- Post cache (Redis · post id → body): Hydrated post bodies and counters by post id. Hit rate is very high because recent posts are read by all their followers within hours. Backed by the posts DB.
- Posts DB (Cassandra / sharded MySQL): Post rows keyed by post id (time-ordered), plus a per-author timeline table keyed by (author id, time) for profile pages and celebrity pulls. Append heavy.
- Graph DB (sharded by user id): Follower and following adjacency lists, both directions materialised, sharded by user id so "followers of X" is one shard read. Celebrity follower lists are chunked.
- Media storage + CDN: Images and video uploaded directly from the client with pre-signed URLs, transcoded asynchronously, served via CDN. Posts reference media by id.
- Counter service (likes · comments · views): Aggregates engagement events into per-post counters with approximate, eventually consistent values. Writes to the post cache; persists periodically.
User flows
- Create a post and fan it out. The write path. The post is durable in milliseconds; delivery to followers' feeds happens asynchronously and is bounded regardless of follower count.
- Client uploads media directly to storage, then creates the post. Media bytes never touch application servers. The create call references the media id. An idempotency key on the create call prevents duplicate posts on retry.
- Post service writes the post row and the author timeline entry. Two writes: posts by id, and author_timeline by (author_id, post_id). The profile page and celebrity read-time pulls read the second table.
- Post service publishes post.created and acks the client. Published via an outbox so the event cannot be lost after the DB commit. The author sees their post immediately on their own profile; the feed delivery is asynchronous.
- Fan-out worker checks the author's follower count: celebrity or not? Above a threshold (say 1 M followers) the post is not fanned out at all; it will be pulled at read time. This bounds the worst case. The threshold is a tuning knob between write cost and read latency.
- For a normal author, the worker fetches followers in chunks and pushes the post id into each active follower's feed list. Chunks of a few thousand followers per task so a 500 k-follower author is parallelised across workers. Followers inactive for weeks are skipped; their feed is rebuilt on their next visit. Pipelined Redis writes keep this at millions of inserts per second across the fleet.
- Counters and ranking features consume the same event. Engagement counters start at zero; feature pipelines register the new post so ranking can score it once it appears in candidate sets.
- Load the home feed. The read path. One cache read for the precomputed list, one small query for celebrity posts, hydrate, rank, paginate. No follow-graph traversal on the hot path.
- Client requests the first page of the feed. No cursor on the first page. Subsequent pages carry an opaque cursor that encodes the position in both the precomputed list and the celebrity merge.
- Feed service reads the user's precomputed feed list from the cache. One ZREVRANGE returns the latest ~100 post ids from non-celebrity followees. If the key is missing (inactive user, cache loss), fall back to building it on the fly from the following list and each followee's recent posts, then repopulate.
- Feed service fetches recent posts from the celebrities this user follows. Most users follow only a handful of celebrities. Their recent posts are read from author_timeline, which is hot in cache since millions of users read the same rows. Merge into the candidate set by time.
- Feed service hydrates candidate post ids from the post cache. One MGET for ~120 post ids. Misses go to the posts DB and are written back. Deleted posts and posts from users who since blocked this reader are filtered here.
- Feed service sends candidates to Ranking and gets an ordering. Ranking scores each candidate with the user's features and applies diversity rules. Budget ~50 ms. On timeout, fall back to reverse chronological so the feed always loads.
- Feed service returns the page with a cursor; client renders and prefetches the next page. The cursor encodes the smallest post id served from the precomputed list and per-celebrity positions, so the next page continues below it even as new posts arrive at the top. The client dedupes by post id in case of overlap.
- A celebrity with 100 M followers posts. The case that breaks fan-out on write. Here the post costs nothing to publish and is pulled by readers, which spreads the cost over time and cache hits.
- Celebrity creates a post; it is written and the event is published like any other. Nothing special on the write side. The post row and author_timeline entry are the only writes.
- Fan-out worker sees the celebrity flag and skips fan-out entirely. Instead of 100 M feed inserts, zero. The worker only warms the post cache with the hydrated post so the first readers do not stampede the DB.
- Followers open their feed; the Feed service pulls the celebrity's recent posts at read time. The author_timeline row for this celebrity is read by millions of feed loads within minutes; it is a single hot key served from cache after the first read. Read-time merge costs one extra small query per celebrity followed, typically under five.
- Post cache absorbs the read storm; counters update approximately. The hydrated post is a hot key. Replicate hot keys across several cache nodes (key suffixing) to avoid a single-node bottleneck. Like counts are eventually consistent and batched; nobody needs an exact count of 2.3 M likes.
- Delete: the celebrity removes the post; the delete propagates through the same path. Because nothing was fanned out, there is nothing to retract from 100 M lists. The post is tombstoned, the cache entry is replaced by a tombstone, and hydration filters it out. This is a hidden advantage of pull for high-fan-out authors.
- Scroll: stable pagination while new posts arrive. Offset pagination breaks the moment a new post lands. Cursors anchored on time-ordered ids keep the scroll stable and duplicate-free.
- Client requests the next page with the cursor from the previous response. The cursor is opaque to the client and signed so it cannot be tampered with.
- Feed service reads the precomputed list below the cursor position. A range query by score strictly less than max_id. New posts added at the top do not shift this window.
- Celebrity posts are continued from their per-author cursor positions. Each celebrity followed has its own position in the cursor. Merge by time with the precomputed list.
- Ranking is applied within the page, and already-shown posts are excluded. Ranking may pull a slightly older but higher-scored post into this page. To avoid it reappearing later, the cursor carries a compact set (bloom filter) of ids already shown in this session.
- When the precomputed list is exhausted, fall back to on-demand assembly from followees' timelines. Deep scrolls past 500 posts are rare. Beyond the cached window, build older pages by merging followees' author_timelines, which is slower but acceptable at that depth.
- Unfollow, block, and privacy. Precomputed feeds are copies. Every copy must respect the current follow graph and visibility, which means invalidation and read-time checks.
- User A unfollows B (or blocks B). The graph service updates both adjacency directions and publishes a follow.removed or block.created event.
- Fan-out worker removes B's recent posts from A's feed list. Scan A's list for posts by B and remove them. The list is short (500) so this is cheap. Without this, A keeps seeing B for hours.
- Read-time safety check filters anything the cache missed. During hydration the Feed service checks each post's author against A's current block list and, for followers-only posts, that A still follows the author. This is the last line of defence and is cheap because the block and follow-check are cached per reader.
- B switches their account to private. Existing public posts become followers-only. No copies need updating because visibility is checked at hydration time against the post's current visibility, not the visibility at fan-out time.
- A deleted post is tombstoned and disappears from every feed. Feed lists still contain the id, but hydration returns a tombstone and the post is dropped from the page. Lists self-clean as they are trimmed. This is why feeds store ids, not bodies.
Deep dives
Fan-out on write vs fan-out on read
Do you build each user's feed when a post is created, or when the user opens the app?
Fan-out on write (push) precomputes every follower's feed at post time: reads are a single list lookup, writes cost O(followers). Fan-out on read (pull) computes the feed on open by merging the recent posts of everyone the user follows: writes cost O(1), reads cost O(followees) queries. Twitter's famous lesson was that pure push dies on celebrities and pure pull dies on read latency.
The numbers decide it. Reads outnumber writes 10:1 and must be fast; the average user follows 200 accounts but a handful of accounts have 100 M followers. So push for the many, pull for the few, and merge at read time.
- Hybrid: push for normal authors, pull for celebrities chosen
- Pure fan-out on write rejected
- Pure fan-out on read rejected: small networks or when every user follows very few accounts
The answer: Hybrid with a celebrity threshold around 1 M followers, tuned by measuring fan-out latency and read latency. Push writes post ids (not bodies) into a capped Redis sorted set per active follower; skip followers inactive for more than a few weeks and rebuild their feed lazily on return. Celebrities' posts are pulled from their author_timeline at read time, which is a single hot key per celebrity that caches perfectly. The Feed service merges both sources by time before ranking. Facebook and Instagram use variants of this; Twitter moved to it after the pure-push era.
Where exactly do you set the celebrity threshold and what happens as an account crosses it?
Measure two curves: fan-out completion time versus follower count, and the read-time cost of one more pulled author. Pick the count where fan-out exceeds your visibility SLO (say 60 s). Crossing it is a flag flip on the author: new posts are pulled; old posts already fanned out stay in lists. Going back down is the same in reverse. Hysteresis avoids flapping.
A normal user with 500 k followers posts. How long until all followers see it?
Chunk followers into tasks of 5 000; 100 tasks run in parallel across the worker fleet. Each task does 5 000 pipelined ZADDs in well under a second. End to end a few seconds, dominated by queueing. If the fleet is saturated, prioritise by author engagement so the posts most likely to be seen soon go first.
Why store post ids in the feed list rather than the full post?
Storage (ids are 8 bytes, bodies are kilobytes), consistency (an edit or delete updates one place), and freshness of counters. The cost is a hydration step, which is a batched MGET against a cache with a very high hit rate. Storing bodies is the mistake that produces the 400 TB cache estimate.
How do you rebuild a feed for a user returning after two months?
Their list was evicted. On open, take their following list, fetch the last N posts from each followee's author_timeline (parallel, bounded), merge by time, write the result into a new feed list, and serve it. That is fan-out on read for exactly one request, acceptable because it is rare, and it warms the list for subsequent opens.
The fan-out worker crashes halfway through a 500 k-follower post. What happens?
Kafka redelivers the event to another worker. Each chunk task is idempotent because ZADD with the same member is a no-op, so followers already done are unaffected and the rest complete. If you track chunk completion in a small table you can skip finished chunks, but idempotent writes make that an optimisation, not a correctness requirement.
Ranking the feed
Reverse chronological is simple and honest. Why rank, and how do you do it without making the feed slow?
Chronological feeds favour prolific posters and bury the posts a user would most want to see. Ranked feeds improve engagement dramatically, which is why every large network moved to them, and they create obligations: users need to understand and control it, and the system must not amplify harmful content. Architecturally, ranking is a candidate generation and scoring pipeline that runs inside a 300 ms request.
The standard structure is two-stage: candidate generation produces a few hundred posts cheaply (the precomputed feed list and celebrity pulls are the candidate generator here), then a heavier model scores them and a final layer applies business rules and diversity.
- Two-stage: cheap candidates + learned scoring + rules, with a chronological fallback chosen
- Reverse chronological only situational: early stage, as a user-selectable option, or for regulatory requirements
- Precompute a ranked feed per user offline rejected
The answer: Two-stage ranking inside the Feed request. Candidates: the ~100 most recent ids from the precomputed list plus celebrity pulls. Features: user embedding and recent activity from an online feature store, author and post features from the post cache, engagement counters, and pairwise affinity between reader and author. Model: a multi-task neural net predicting probability of like, comment, share, dwell, and hide, combined into a score with weights that encode product goals. Then rules: author diversity, freshness boost, demotion of borderline content. Hard 50 ms budget with a reverse-chronological fallback. Log features and outcomes for training so serving and training see identical inputs.
How do you avoid the feed showing the same top-ranked post every time the user refreshes?
Track impressions: once shown, a post gets a decaying penalty in the session and is excluded after a couple of impressions. The cursor and a per-user recent-impressions set (short TTL) carry this. Also inject some freshness so a refresh reliably shows something new.
Where do the features come from within 50 ms?
An online feature store (Redis or similar) keyed by user id and post id, populated by streaming jobs from engagement events. The request fetches the user's feature vector once and the candidates' feature vectors in one batched call, then runs inference on a GPU or optimised CPU model in a few milliseconds. Anything not in the store is not a feature.
A new post has no engagement yet. How does it ever get ranked high?
Cold start is handled by content and author features (author's historical engagement rate with this reader, post type, predicted quality from text and image models) plus an explicit freshness prior. Some systems run a small exploration budget: show new posts to a slice of followers to gather signal quickly.
How would you measure whether ranking is helping?
A/B test against chronological on a holdout: engagement (likes, comments, time spent), retention over weeks, and negative metrics (hides, reports, unfollows). Retention is the honest metric; engagement alone can be gamed by outrage. Keep a permanent small holdout on chronological to detect long-term drift.
Feed cache design
What exactly is stored per user, where, how big, and what happens when it is lost?
The precomputed feed is the heart of the read path, so its data structure and failure behaviour matter more than most design details. It must support: append at the head, read the top N, read a range below a cursor, remove specific entries (unfollow), and cap the size. A Redis sorted set scored by time-ordered post id does all of this in O(log N) with N capped at a few hundred.
It is a cache, not a source of truth. Everything in it can be rebuilt from the posts DB and the follow graph. That framing determines the replication and persistence choices: you want it fast and cheap to rebuild, not durable.
- Redis sorted set of post ids per user, capped, rebuildable chosen
- Per-user feed table in a wide-column store (Cassandra) situational: when memory cost is prohibitive or feeds must survive cache loss without rebuild
- Store hydrated post bodies per user rejected
The answer: Redis Cluster, one sorted set per active user, member and score both the time-ordered post id, trimmed to 500 on every insert. Shard by user id. Replicas for read availability, no persistence: a lost shard is rebuilt lazily on read with a per-user lock to prevent duplicate rebuilds, and a background job re-warms recently active users to avoid a stampede. Evict users idle for more than ~30 days. Estimate ~10 KB per user, ~200 M active lists ≈ 2 TB across the cluster. Metrics that matter: hit rate on feed open (target above 99 %), rebuild rate, p99 ZREVRANGE latency.
A Redis shard holding 5 M users' feeds dies at peak. Walk through the next five minutes.
Replica promotes in seconds; if there is no replica, every feed open on that shard misses. Each miss triggers a rebuild: 200 author_timeline reads and a merge. Bound it with a per-user rebuild lock, a global rebuild rate limit that serves a degraded chronological page directly from author timelines when exceeded, and a background warmer that rebuilds users in order of recent activity. Users see slower feeds for a few minutes, not errors.
Why cap at 500 and not 5 000?
Users rarely scroll past a few hundred posts, memory scales linearly with the cap, and trims get slower. 500 covers hours to days of content for most users. Beyond it, fall back to on-demand assembly, which is slower but rare.
How does unfollow removal work without an index by author?
Read the whole list (500 members with scores) and filter by author client-side, which needs the author id. Either encode author id into the member (post_id:author_id) or resolve via the post cache. Then ZREM the matches. It is a few hundred elements, so it is cheap; the simplicity beats maintaining a secondary index.
Would you use Redis Streams or Lists instead of sorted sets?
Lists cannot do range-by-score or remove-by-member efficiently. Streams are append-only and ordered but removing arbitrary entries is awkward and range-by-id works only on stream ids. Sorted sets match the operation set exactly, which is why they are the usual choice.
Hot keys and celebrity read storms
One post read by ten million people in five minutes. What melts first?
Pull-based delivery for celebrities moves the load from writes to reads, concentrated on a few keys: the celebrity's author_timeline row and the hydrated post body. A single Redis node handles maybe 100–200 k ops per second; a viral post can exceed that on one key. The database behind it cannot take even a fraction of the misses.
This is the hot key problem, and it has standard answers at each layer: replicate the key across nodes, add a local in-process cache with a short TTL, and make sure a miss never becomes a stampede.
- Key replication + local cache + single-flight on miss chosen
- Bigger cache nodes rejected
- Serve celebrity posts from a CDN as static JSON situational: public posts of very large accounts, with counters fetched separately
The answer: Three layers. Detect hot keys by sampling access counts per key in the Feed service. For hot keys, write N copies (post:{id}:{0..N-1}) and read a random one; invalidate all N on update. In front of Redis, each Feed server keeps a small in-memory cache with a 2 second TTL for post bodies and author timelines; at 300 k feed opens per second across a few hundred servers, this alone cuts Redis load on a viral key by orders of magnitude. On a cache miss, single-flight (one in-flight DB read per key per process) prevents the stampede. Counters for hot posts are updated in batches and displayed approximately.
How do you know a key is hot before it takes down a node?
Sample: each Feed server counts accesses per key in a sliding window using a small top-K sketch and reports keys above a threshold. Or proactively: any post by an author above the celebrity threshold is treated as hot from creation. In practice both, since virality also happens to normal users.
The like count on a viral post shows 1.2 M on one refresh and 1.1 M on the next. Acceptable?
It is a symptom of reading different replicas or local caches with different staleness. Acceptable in magnitude, but monotonic display is a nicer experience: the client can keep the max it has seen for a post within a session. Exact counts are not worth the coordination cost.
Does the same problem exist for the follow graph?
Yes: "followers of celebrity X" is a 100 M-element list, and "does A follow X" is asked constantly. Store follower lists chunked, never load them whole except in fan-out (which is skipped for celebrities anyway), and answer follow checks from a per-reader following set, which is small.
Consistency and freshness guarantees
What can a user actually rely on? Which delays and anomalies are acceptable, and which are bugs?
A feed is an eventually consistent view assembled from caches. Interviewers probe whether you know which inconsistencies you have accepted and which you have prevented. The acceptable ones: a post appears a few seconds late; a like count is approximate; two devices show slightly different orders. The unacceptable ones: a private post seen by a non-follower; a deleted post still visible; a post missing forever; the same post shown twice in one scroll.
The design gets there with read-your-writes for the author, idempotent fan-out, tombstones checked at hydration, visibility checked at hydration, and cursor-based pagination with dedupe.
- Eventual consistency for delivery, strong checks for privacy and deletion at read time chosen
- Strongly consistent feed (transactional fan-out) rejected
- No read-time checks; rely on cache invalidation rejected
The answer: State the guarantees explicitly. Delivery: at-least-once into feed lists within seconds (normal) or on next open (celebrity), deduped by post id at the client and cursor. Ordering: by time-ordered post id within a source, then re-ranked; the same session never shows a post twice. Privacy: visibility and block checks happen at hydration against current state, every time, so a stale cache copy can never leak. Deletion: tombstones at hydration, so a deleted post vanishes within the post cache TTL (seconds). Author: their own post appears immediately on their profile and is injected into their own feed at read time. Anything outside these guarantees is documented as a known and acceptable anomaly.
A user posts, immediately opens their home feed, and does not see their post. Bug?
By the guarantees, yes: the author should see their own post. The fix is cheap: the Feed service injects the reader's own most recent posts at read time from their author_timeline, which is a single hot-for-them key. Never rely on fan-out to deliver an author's post to themselves.
How do you prevent the same post showing twice in one scroll session?
Time-ordered ids and a cursor with a strict less-than boundary handle the chronological source. Ranking can pull an older post forward, so the cursor also carries a bloom filter of ids already served this session; hydration drops matches. The client dedupes by id as a final guard.
What does "new posts available" do and why not just insert them at the top?
Inserting shifts content under the user's thumb. The client polls a lightweight "count of posts newer than my top id" endpoint, shows a pill, and only on tap fetches and prepends. It also gives the ranking system a natural boundary for a new session.
Blocked user B still saw A's post via a reshare by C. Whose bug?
A product decision that must be explicit. Most networks: a block hides A's content from B everywhere, including reshares, so the hydration check must consider the original author of a reshared post, not just the resharer. If the check only looks at the top-level author, that is a bug in the design.