System design interview question

Design Google News

Q: Why not poll every feed once a minute? And if publishers can push, why poll at all?

Each feed has a state row: next poll time, ETag and Last-Modified, the last 200 item guids, any WebSub lease, and a publishing rate per hour of the week, kept as a decayed count so both the 9 a.m. rush and the quiet night are learned. The interval is about half the expected gap between items, clamped to between a floor (1 minute for authoritative sources, 5 for the long tail) and 60 minutes, and a feed that just produced items is polled again sooner, because news arrives in bunches. That gives roughly 5 k feeds at 1 minute, 45 k at 5 and 150 k at 30: about 320 polls a second, ~80 % of them 304s. Where a feed advertises a hub, we subscribe, renew well before the lease ends, verify every delivery’s X-Hub-Signature, and relax the backstop poll to 15 minutes; pings to our own endpoint are authenticated per publisher, carry no content, and only trigger an immediate poll, at most one every 30 seconds. Every request goes through the crawler’s fetch tier, which already does politeness, robots.txt and DNS caching, so this component decides when to fetch, never how.

Q: Four hundred outlets cover the same earthquake in different words. How do you put them in one story within seconds, without comparing every article with every other?

Embed title plus the first three sentences with a multilingual sentence encoder into 384 dimensions, and compute a 100-hash MinHash of the body’s 5-word shingles. For each article, first look it up in 20 LSH bands of 5 rows: a match with estimated Jaccard above 0.8 is a syndicated copy, joins that article’s story and is flagged so it counts once for coverage. Otherwise query an in-memory HNSW index over the centroids of stories active in the last 48 hours in that language for the 10 nearest, score each as cosine similarity plus a bonus for shared linked entities minus a penalty for hours since the story’s last article, and join the best if it clears a threshold around 0.78, tuned on rater-labelled pairs to about 98 % precision; otherwise start a new story. The centroid is a running mean weighted toward recent articles, so a story that develops (the quake, then aftershocks, then the death toll) follows its own coverage. One writer per language consumes the normalised-article topic in order; English, the largest, peaks at ~100 articles a second, which a single process handles with room to spare. Precision beats recall: one event shown as two stories looks like a small duplicate on the page, while two events in one story put the wrong headline over someone’s reporting.

Q: The repair pass decides two stories are one and another is really two. What happens to the story URLs people have shared, the counters, and the pages already cached at the edge?

Story ids are 64-bit time-ordered ids minted by the language’s writer. The repair pass reads a snapshot, proposes merges (centroid cosine above about 0.88 with overlapping entities and times) and splits (two sub-clusters, each cohesive, well separated and with at least 3 sources), and hands them to the writer, which applies them in sequence with assignments and publishes them to story.changes, keyed by language, each with a change id. On a merge the older story survives, which is almost always the one already linked from front pages, and the other becomes an alias row, alias_of = survivor, which the API answers with a 301 and the ranker collapses. On a split, the half containing the current representative keeps the id and the other half gets a new one; counters such as originals per 5 minutes and distinct sources are recomputed from member rows. A story is repaired at most once per 30 minutes, so nothing flaps. Consumers apply changes idempotently by change id, so a replay after a crash is harmless.

Q: What makes a story “top” in the UK edition right now, and how does an earthquake reach the top within five minutes without the list jumping around every minute?

For each story and edition, coverage is log(1 + Σ authority) over distinct publishers with an original article in the last 6 hours, where authority is an offline per-source, per-topic score built from original reporting, citations by other publishers and corrections history; syndicated copies add nothing. Velocity is original articles in the last 30 minutes against the story’s own previous rate. Freshness decays with the age of the latest original article, with a half-life of about 4 hours, so a long-running story stays only while it keeps developing. Regional relevance is the share of coverage from publishers in the edition’s region plus geo entities in it. The ranker keeps these aggregates in memory from story.changes and scores the ~300 k stories with an article in the last 24 hours for 70 editions every minute (about 20 M scores, a few cores). It smooths scores over the last 5 cycles and swaps two stories only when the gap exceeds 10 %, then applies diversity rules: at most 3 of the top 10 from one section, and never two stories whose centroids have a cosine similarity above 0.85 (a likely missed merge). A burst overrides smoothing: at least 10 original articles in 5 minutes, at least 5 publishers with authority above 0.5, and a Poisson tail probability below 10⁻⁶ against the baseline for a new story in that topic. The story is then labelled Breaking in the editions where it is relevant and pinned at the top for up to 2 hours.

Q: A big event hits and 200,000 people a second open the home page. Why not personalise the whole page, and what stops the origin from being trampled?

The ranker writes each page as an immutable versioned object (pages/en-US/top/v48212) and then moves a latest pointer; origin servers watch the pointers and hold the current version of all 840 pages in memory (~35 MB), so a request never reads a store. The edge caches GET /v1/editions/{edition}/{section} with s-maxage=60, stale-while-revalidate=60 and stale-if-error=3600, coalesces concurrent misses in every PoP, and refreshes through ~20 shield locations, so the origin sees about one request per page per shield per minute, ~280 a second, however large the audience. Stale-while-revalidate means no reader waits on a refresh, and stale-if-error keeps the last good page up for an hour if the whole pipeline is down. Signed-in clients fetch For you separately with Cache-Control: private: the service reads the reader’s profile (follows, hides, a 64-dimension interest vector and the last 200 reads, ~5 KB) and re-ranks the edition’s pool of ~1,000 candidate stories, which the ranker publishes with each snapshot and every For-you server holds in memory, in about 2 ms of CPU; at 80 k requests a second that is ~160 cores. Under overload For you sheds first and the page renders without it, because the shared page is the one that must never fail.

Q: Which of 400 articles gets the headline and the click, and how do you stop the same three big publishers from winning every story?

Per edition, the candidates are the story’s original (non-copy) articles from the last 6 hours in the edition’s language. Each is scored on cosine similarity to the centroid (is it about the core of the story), the publisher’s authority for the topic, freshness, a headline check (a clickbait classifier, length, no all-caps), and a boost for the publisher that reported the story first and for local publishers on local stories. The top candidate wins unless others are within 5 %; then the pick rotates among them by hash(story, edition, hour), so close calls share the traffic. Each publisher’s share of representative impressions in an edition is capped, at say 10 % a day, and beyond the cap its near-ties go to others. Four supporting links are chosen by maximal marginal relevance, each maximising relevance minus similarity to those already picked, with at most one per publisher, one copy of any wire story, and opinion and fact-checks labelled. The headline is always the publisher’s own, linked to the publisher’s page, and concentration is tracked as a Herfindahl index of representative impressions per edition, next to clicks and rater quality.

Collect 2 M articles a day from 50,000 publishers, group them into stories, rank them per edition, and serve 300 M daily readers.

Last updated 2026-10-26. Difficulty: hard. Patterns: clustering, ingestion, ranking, edge-caching, deduplication. Reported at Google, Meta, Microsoft, Apple, LinkedIn, X.

Walk through a strong candidate's answer, turn by turn.

The interviewer asks, the candidate answers and draws, and you press Next. Skip to any stage or key moment, and ask the AI Mentor anything along the way.

Walk through it Take the mock interview

Or sit it in the style of Google · Meta · Microsoft · Apple, run as that company’s round is reported to go.

The short answer

How to design Google News in a system design interview, in five steps:

Scope it. Ingest articles from 50,000 publishers, normalise and deduplicate articles, group articles about the same event into stories, top stories per edition and section, breaking news, for you, full Coverage of a story.
Set the targets. Freshness for breaking news: Breaking on top < 5 min; Discovery latency: p90 < 2 min (top 5k feeds); Read scale and latency: 200 k/s peak · p99 < 200 ms; Availability of edition pages: 99.95 %.
Do the numbers. Articles ingested: ~23/s avg · ~200/s peak; Feed polls: ~320/s, ~80 % are 304s; Page requests: ~10 k/s avg · 200 k/s event peak.
Draw the design. Edge cache, News API (origin), For-you service, Reader profiles, Page snapshots, Story ranker, Story store, Story clusterer, Article store.
Go deep where it is hard.
- Polling feeds and publisher push → Adaptive polling per feed, plus WebSub and pings, with polling as the backstop.
- Grouping articles into stories as they arrive → Online assignment: MinHash LSH for copies, then the nearest active centroid by embedding, one writer per language.
- Merges, splits and stable story ids → Stable ids; merges become aliases and splits give the minority a new id, sequenced through the writer.
- Ranking stories and spotting breaking news → A score on independent coverage, authority, velocity, freshness and regional relevance, every 60 s, smoothed, with a burst rule.
- Serving 200 k requests a second → Shared pages precomputed and cached at the edge, plus a separate For-you request that re-ranks a cached pool.
- Headline choice, source diversity and fairness → Score candidates for centrality and quality, rotate near-ties, cap exposure, pick supporting links for diversity.

Functional requirements

Ingest articles from 50,000 publishers. RSS and Atom feeds, news sitemaps, WebSub hubs and publisher pings. Fetching itself (politeness, robots.txt, DNS) is the existing crawler fetch tier; this design decides what to fetch and when.
Normalise and deduplicate articles. Canonical URL, publisher, published and first-seen time, language, title, lede, image and linked entities. The same article seen through a feed, a ping and a sitemap, or under AMP and print URLs, is stored once; a corrected headline is a new version, not a new article.
Group articles about the same event into stories. Within seconds of ingest: 300 reprints of one wire report and 100 independent reports of the same earthquake become one story, with a representative article, a headline, and a stable id and URL.
Top stories per edition and section. Ranked lists for ~70 editions (region plus language) with ~12 sections each: World, Business, Technology, Sports, Local and so on. Stories are ranked, not articles.
Breaking news. A story whose coverage bursts is labelled Breaking and moved to the top of the editions where it matters within minutes, and an event is emitted for the notification system to alert followers.
For you. A personalised list from followed topics, sources and reading history, with controls (follow, hide a source, fewer like this). Built on top of the shared candidates, not instead of the shared page.
Full Coverage of a story. A page per story: a timeline of articles, a range of publishers, local sources, opinion and fact-checks labelled, and links to coverage of the same event in other languages.
Out of scope. The crawler’s fetch internals (see Design a Web Crawler), full-text news search, delivering push notifications, ads, comments and publisher payments. We show headlines, snippets and links; readers read on the publisher’s site.

Non-functional requirements

Freshness for breaking news (Breaking on top < 5 min). From the first authoritative reports to the label on the edition page: about a minute to discover, a minute to rank, a minute at the edge. This drives the hardest trade-off in the design: clustering must decide online and greedily, and be corrected later.
Discovery latency (p90 < 2 min (top 5k feeds)). Publication to first seen, measured against the publisher’s timestamp or an audit poll. The long tail may take up to an hour; nobody waits on a weekly magazine.
Read scale and latency (200 k/s peak · p99 < 200 ms). Normal load is ~10 k page requests a second; a big event brings a 20× spike within a minute, synchronised on the same pages. Almost all of it must be answered by the edge.
Availability of edition pages (99.95 %). A page a minute stale is acceptable; a blank page is not. The last good snapshot stays servable for an hour even if ingestion, clustering and ranking all stop.
Clustering precision (≥ 98 % same-event pairs). Rater-labelled sample pairs, weighted by impressions. Merging two events puts the wrong headline over someone’s reporting, which is worse than showing one event as two stories, so thresholds favour precision.
Fairness to publishers (≤ 10 % of headlines per publisher). Per edition per day, measured on representative impressions. Syndicated copies count once, original reporting is rewarded, and close calls rotate rather than always going to the biggest brand.
Good citizen to publishers (≤ 1 poll/min per feed). Conditional GETs, politeness and robots.txt through the shared fetch tier, and push wherever a publisher offers it.
Privacy (no personal data in shared pages). Reading history powers For you only, can be paused or deleted, and never enters a cached page: a shared page is identical bytes for every reader in an edition.

Back-of-envelope estimates

Articles ingested: ~23/s avg · ~200/s peak. 2 M ÷ 86 400 ≈ 23/s. Mornings and big events bunch publishing into bursts of ~10×, so plan for ~200/s. One machine could parse this: writes are not the hard part.
Feed polls: ~320/s, ~80 % are 304s. 50 k sources × ~4 feeds (main, sections, news sitemap) = 200 k feeds. Adaptive tiers: 5 k busy feeds every minute (83/s), 45 k every 5 minutes (150/s), 150 k every 30 minutes (83/s) ≈ 320 polls/s. Most return 304 Not Modified, so polling costs a few MB/s; latency, not cost, is why we add push.
Page requests: ~10 k/s avg · 200 k/s event peak. 300 M daily readers × ~3 page loads = 900 M/day ÷ 86 400 ≈ 10 k/s. A normal morning peaks at ~3× that; a big event brings 200 k/s within a minute, a 20× spike. The read side is the scale problem.
Origin requests for shared pages: ~280/s. 70 editions × 12 sections = 840 shared pages, ~40 KB each (~35 MB in all). With a 60 s edge TTL, coalesced misses and ~20 shield locations that each refresh every page once a minute: 840 × 20 ÷ 60 ≈ 280/s at the origin, whether readers send 10 k/s or 200 k/s.
For-you requests at peak: ~80 k/s · ~160 cores. If 40 % of page loads are signed-in readers who get For you: 200 k/s × 0.4 = 80 k/s, the only per-reader work in the system. Re-ranking a ~1,000-story pool in memory costs ~2 ms of CPU, so ~160 cores: about ten 32-core servers at half load.
Clustering working set: ~6 GB of vectors. A 48-hour window × 2 M/day = 4 M live articles × a 384-dimension float32 embedding (1.5 KB) = 6 GB, plus 400 B of MinHash each (1.6 GB) and ~600 k story centroids (0.9 GB). English is ~40 %, about 3 GB: one language fits in one machine’s memory, so each language can have a single writer.
Storage growth: ~7 TB/yr articles · 1.5 TB profiles. ~10 KB an article (metadata, compressed text, embedding, MinHash) × 2 M/day = 20 GB/day ≈ 7 TB/yr. Reader profiles: 300 M × 5 KB = 1.5 TB. Storage is small; the design is driven by latency and read spikes.

Components

Reader app / web (web · iOS · Android): Fetches the shared edition page from the edge and, for signed-in readers, the For-you slice in a second request the page can render without. A tap opens the publisher’s own page; a beacon records the read.
Edge cache (CDN · 60 s TTL · coalescing): Caches shared pages keyed by edition and section only, with no cookies in the key, for 60 s with stale-while-revalidate and stale-if-error. Coalesces concurrent misses in each PoP and refreshes through ~20 shield locations. This is what turns a 200 k/s event spike into a few hundred origin requests a second.
News API (origin) (pages held in memory): Stateless origin for edition pages, Full Coverage pages, For you and read events. Holds the current version of every shared page in memory, preloaded when its version pointer moves, so a request never reads a store. Resolves merged story ids to their survivor with a 301.
For-you service (rerank ~1,000 candidates): Builds the personal slice by re-ranking the edition’s candidate pool, held in memory, against the reader’s profile, in about 2 ms of CPU. It is the first thing shed under overload. Also folds read events into profiles, batched per reader.
Reader profiles (Bigtable · by user id): Followed topics, entities and sources, hidden sources, a 64-dimension interest vector that decays toward recent reads, and the last 200 stories read: ~5 KB a reader, ~1.5 TB in all, with a regional read cache. Paused or deleted on the reader’s request.
Page snapshots (versioned per edition/section): Immutable versioned snapshots of every shared page (pages/{edition}/{section}/v{n}) and each edition’s candidate pool, written by the ranker, plus a latest pointer per page moved by compare-and-set after validation. Object storage for versions, a small KV for pointers. Rolling back is moving a pointer.
Story ranker (every 60 s · per edition): Every minute, scores recently active stories for each of 70 editions on independent coverage, source authority, velocity, freshness and regional relevance; smooths ranks so lists do not flap; detects bursts; chooses each story’s representative per edition; and publishes the snapshots. Keeps story aggregates in memory from story.changes, so a cycle reads almost nothing.
Story store (Bigtable · by story id): Story rows keyed by story id: centroid, member articles with time, source and copy flag, 5-minute coverage counts, distinct sources, entities, topics, and alias_of for merged stories. Written only from story.changes, which the language’s clusterer produces. Members are kept so any counter can be recomputed after a merge or split.
Story clusterer (one writer per language): One writer per language consumes normalised articles in order. Catches syndicated copies with MinHash LSH, otherwise finds the nearest active story centroids in an in-memory HNSW index over a 48-hour window (~3 GB for English), and joins or starts a story. Runs a repair pass every 10 minutes that proposes merges and splits, applied in the same sequence. Commits decisions with its input offsets in one Kafka transaction and has a warm standby.
Article store (by canonical URL hash): One row per article keyed by a hash of the canonical URL, with versions, publisher, times, language, entities and story id, plus an index on content hash for the same text under another URL. ~10 KB an article, ~7 TB a year. A conditional write on the row is the exact-dedupe decision.
Parse & dedupe (canonical URL · hash · embed): Turns fetched pages into normalised articles: canonical URL (rel=canonical, og:url, tracking parameters stripped), publisher, published and first-seen time, language, title, lede, image, linked entities and topics. Decides new, update or duplicate against the article store, then embeds title and lede and computes a MinHash of the body. Stateless, scaled to bursts of ~200 articles a second.
Article log (Kafka · by URL, by language): Kafka. items.raw is keyed by URL hash, so two versions of one article stay in order; articles.normalised and story.changes are keyed by language, so one clusterer sees one language in a single order. Lets parsing and clustering lag during a burst without losing anything.
Feed poller (adaptive schedule · ETags): One state row per feed (~200 k): next poll time, ETag and Last-Modified, recent item guids, a publishing rate per hour of the week and any WebSub lease. Polls on an adaptive 1 to 60 minute interval with conditional GETs, fetches each new item’s page, and has a priority lane for pushed items. All requests go through the crawler’s shared fetch tier.
Push receiver (WebSub · publisher pings): Internet-facing endpoint for WebSub deliveries and publisher pings. Verifies the hub signature or the publisher’s key, answers in milliseconds, and turns each notification into a fetch-now request to the poller. It never treats pushed content as the article: the page is always fetched from the publisher.
Publishers (RSS/Atom · sitemaps · hubs): 50,000 news sites, wire services and blogs. They expose RSS or Atom feeds, news sitemaps, sometimes a WebSub hub, and article pages with canonical links and schema.org NewsArticle data of uneven quality. A wire service publishes every minute; most publishers a few times a day.

User flows

A polled article joins its story. The everyday write path: a feed poll finds a new item, the page is fetched, normalised and deduplicated, and the clusterer puts it in the story it belongs to. Seconds from poll to story; the ranker picks it up on its next 60-second cycle.
1. The poller finds a busy world-news feed due and sends a conditional GET with the ETag from its last poll. This feed has published ~25 items an hour today, so its interval is 2 minutes; a quiet local feed would be at 30. About 80 % of polls come back 304 with no body, which is why polling 200 k feeds costs a few MB/s. The request goes through the crawler’s shared fetch tier, which owns robots.txt, per-host politeness and DNS.
2. The poller diffs the items against the guids it has seen, finds two new ones, and fetches each article page. Feeds return the latest 20 to 50 items, so consecutive polls overlap almost entirely; diffing on guid, or link when a feed has no guids, finds the new ones. The feed item has a title and a link, but clustering needs the text, and the canonical URL and times come from the page’s own markup.
3. The raw item goes onto the article log. The log decouples fetching from parsing, so a burst of 200 articles a second queues for seconds instead of overloading the parsers. Keying by URL hash keeps two versions of the same article in order.
4. The parser extracts the canonical URL, publisher, times, language, title, lede, body and linked entities. Canonical URL comes from rel=canonical, then og:url, then the fetched URL with tracking parameters stripped. The page’s datePublished is kept for display, but freshness uses first-seen time, so a publisher that re-dates old articles gains nothing. Entities are linked to knowledge-graph ids, so "Tokyo" and "東京" are one entity.
5. The parser checks the article store by canonical URL and content hash: new, update or duplicate. Same canonical URL with the same content hash is a refetch: drop it. Same URL with a new hash is a new version, such as a corrected headline; it is re-embedded, and re-scored against its story only if the title or lede changed. A different URL with an identical content hash (an AMP or print copy) is linked to the original, not stored twice.
6. The parser embeds the title and lede, computes a MinHash of the body, and publishes the normalised article for its language. Two signatures for two kinds of sameness: the 384-dimension embedding catches independent reports of one event in different words, and the MinHash of 5-word shingles catches syndicated copies of the same text. Keying by language means one clusterer sees every English article in one order.
7. The clusterer looks for a syndicated copy in its LSH buckets, then for the 10 nearest active story centroids. 20 bands of 5 hashes find pairs with Jaccard similarity of 0.8 or more almost always and pairs below 0.3 rarely. No copy this time, so the HNSW index over the ~240 k English stories active in the last 48 hours returns 10 candidates in about a millisecond.
8. It scores the candidates, joins the best one above the threshold, and records the change. The score is cosine similarity to the centroid, plus a bonus for shared entities, minus a penalty for hours since the story’s last article; at 0.78 or more the article joins, otherwise it starts a new story. The story’s centroid, 5-minute counts and distinct sources update, and the ranker sees the change on its next cycle.
Open the home page in the US edition. The main read path. The shared page comes from the edge as the same bytes for every US reader; only the small For-you slice touches our servers per reader.
1. The client asks for US top stories, and the nearest edge location has a fresh copy. The cache key is edition and section only: no cookie, no user id. One object serves everyone in en-US, so the hit ratio is above 99.9 % and the request costs the origin nothing.
2. When the copy expires, the edge keeps serving it and sends one revalidation upstream for everyone. Concurrent misses in a PoP coalesce into one request, and PoPs refresh through ~20 shield locations, so the origin sees each page about 20 times a minute in all. No reader waits on the refresh: stale-while-revalidate answers from the old copy meanwhile.
3. The origin answers from the snapshot it already holds in memory. Origin servers check each page’s latest pointer every few seconds and preload the new immutable version, ~35 MB for all 840 pages. The request path never reads a store, so a slow store cannot slow a page.
4. A signed-in client asks for its For-you slice in a second request, which the edge passes through. Keeping it separate means the shared page stays a public cache entry and renders even when For you is slow or shed. The page draws first; the slice fills in when it arrives, or is left out after 300 ms.
5. The For-you service reads the reader’s profile. Follows, hidden sources, a 64-dimension interest vector and the last 200 stories read, ~5 KB. A regional read cache holds it for 5 minutes, since interests do not change between two page loads.
6. It re-ranks the edition’s candidate pool against the profile and returns 20 stories. The pool is ~1,000 stories the ranker publishes with each snapshot, held in memory on every For-you server. Score is story importance × (1 + affinity from follows and the interest vector), hidden sources removed; stories already read drop out unless they have developed since; at most 2 from one topic in a row. About 2 ms.
7. The reader opens a story’s Full Coverage page, cached at the edge per story. The origin builds it from the story row: a timeline, top coverage, local sources, labelled opinion and fact-checks, and linked coverage in other languages. Each server has at most one build per story in flight, and keeps the result for 10 seconds.
8. A tap opens the publisher’s article, and a beacon records the read for For you. We link out: the reader reads on the publisher’s site. Read events, ~15 k/s on average, update the profile asynchronously and batched per reader, so a lost beacon costs a little personalisation and nothing else.
An earthquake: 300 sources in 15 minutes. The scale-breaking case, on both sides at once. Coverage bursts into the pipeline and has to reach the top of the right editions within minutes, while readers arrive at 200 k requests a second.
1. Within a minute, wire services and large publishers push WebSub deliveries and pings. The receiver checks the signature with the secret we gave the hub when subscribing, answers in milliseconds and never fetches inline: hubs retry slow callbacks, and a burst of slow answers would back up into them.
2. The receiver asks the poller to fetch those items now, ahead of the schedule. Pushed work goes in a priority lane, so it never waits behind the 320 scheduled polls a second. A thin ping that names only the feed triggers an immediate conditional GET; a publisher can force at most one poll every 30 seconds.
3. The first article starts a new story, and the next forty join it, one decision at a time. Each language’s writer sees its articles in one order. With parallel workers sharing an index, several of the first articles would each find no story and each start one, splitting the quake into many small stories at exactly the moment it matters. One writer at ~5 ms an article handles the English peak of ~100 a second.
4. On its next cycle the ranker sees 60 original articles from 35 publishers in 5 minutes and flags a burst. The rule: at least 10 original articles in 5 minutes, at least 5 publishers with authority above 0.5, and a Poisson tail probability below 10⁻⁶ against the baseline for a new story in that topic. Copies and low-authority rewrites do not count, so one prolific site cannot set it off.
5. The ranker labels it Breaking, pins it at the top of the editions where it is relevant, and publishes new snapshots in the same cycle. Relevant means top stories in the Japan editions, the World section everywhere, and top stories wherever coverage is broad enough. A burst skips the usual smoothing. The ranker also emits story.breaking for the notification system, which is outside this design.
6. Readers pile in at 200 k requests a second; the edge serves them, with a soft purge of only the affected top-stories pages. A soft purge marks those few pages stale without dropping them, so each PoP keeps answering from the old copy while one coalesced request fetches the new one. The origin sees a few hundred requests a second however many readers arrive, and nothing is computed on a miss.
7. The quake’s Full Coverage page draws tens of thousands of requests a second and is served the same way. Each origin server builds the page at most once per story at a time (single flight) and keeps it for 10 seconds, and the edge keeps it for 30, so the story store sees a handful of reads a second for the busiest story in the world.
8. For you grows with signed-in traffic and sheds first if it cannot keep up. At 80 k requests a second For you is the only per-reader work, ~160 cores. If its p99 passes 150 ms or CPU passes 80 %, the API answers degraded and clients hide the slice; top stories without For you during an earthquake is the system working as intended.
The English clusterer dies mid-burst. Clustering is a single writer per language by design, so its failure has to be quick to recover from, and must never assign an article twice or let two writers decide at once.
1. The English writer stops renewing its lease, and the warm standby takes it about 10 seconds later. A lease in a small coordination service such as etcd or ZooKeeper decides who writes. The standby has been applying the writer’s committed decisions all along, so its in-memory window, ~3 GB for English, is already current.
2. The new writer opens a transactional producer with the language’s fixed id, which fences the old one. Kafka bumps the producer epoch for that transactional id, so if the old writer was only paused (a long GC, a network blip) and wakes up, its next write is rejected. The lease says who should write; fencing makes sure nobody else can.
3. It resumes reading English articles from the last committed input offset. Every decision was committed in one Kafka transaction with the input offset it consumed, so the two always agree: an article is either decided and behind the offset, or neither. Nothing is lost and nothing is decided twice.
4. It drains the backlog that built up during the failover. Ten seconds at the English peak of ~100 articles a second is ~1,000 articles; at ~5 ms each, the backlog is gone in about 5 to 10 seconds. The ranker’s next cycle includes them: a few stories were up to a minute late, and nobody else noticed.
5. Meanwhile edition pages kept being served from the last snapshots. Nothing on the read path depends on the clusterer: the ranker keeps publishing from its in-memory aggregates, and the origin serves snapshots. A clustering outage makes pages slightly stale; it never makes them fail.
6. For a small language with no standby, the new writer first rebuilds its window from the story store. It loads the stories active in the last 48 hours with their centroids and member signatures, rebuilds the HNSW and LSH indexes, then resumes from the committed offset. For a language with 20 k active stories that takes well under a minute, which is fine for a language that publishes a few articles a minute.
Repair merges two stories and splits one. Online assignment is greedy and makes mistakes in both directions. A slower pass over the last 6 hours fixes them every 10 minutes, without breaking story URLs or making the front page jump.
1. Every 10 minutes the English writer snapshots the stories updated in the last 6 hours and re-clusters their articles with average-link clustering. Average-link compares members with members, too slow to run per article but fine for ~30 k recently active stories once every 10 minutes, on a copy, off the critical path. Its output is a list of proposals, not a new set of clusters.
2. It finds two stories about the same court ruling, born a minute apart, and proposes a merge. The classic greedy mistake: in the first minutes neither story had enough articles for its centroid to pull in the other’s. Their centroids now have a cosine similarity of 0.89, with the same entities and times: above the 0.88 merge bar, which is stricter than the 0.78 join bar because a merge moves many articles at once.
3. It finds a story whose articles fall into two tight groups, a company’s results and a lawsuit against it, and proposes a split. Two sub-clusters, each cohesive, well separated from the other, and each with at least 3 sources; otherwise it is one story with a side angle. A story repaired in the last 30 minutes waits, so nothing flips back and forth.
4. The writer applies the merge in sequence with assignments: the older story survives and the other becomes an alias. Going through the single writer means an article cannot join a story at the same instant it is merged away. The survivor’s counters are recomputed from its members, which keep each article’s time, source and copy flag.
5. It applies the split: the half with the current representative keeps the id, and the other half gets a new one. Readers saw the representative’s headline, so keeping that half under the old id means the page they have open still shows the same story. The new story’s 5-minute counts and sources are rebuilt from its members.
6. The ranker’s next cycle shows one ruling story and two company stories, and the merged-away URL redirects. The ranker applies changes idempotently by change id, and repaired stories go through the usual smoothing, so a repair rarely moves a top story by more than a place. A request for the alias gets a 301 to the survivor; edge copies of the old page age out within 30 seconds.

Deep dives

Polling feeds and publisher push

Why not poll every feed once a minute? And if publishers can push, why poll at all?

Freshness is measured from the moment a publisher posts to the moment the article sits in a ranked story, and breaking news must be on top within 5 minutes. Polling 200 k feeds every minute is 3,300 requests a second, ten times what the adaptive schedule needs, and most of it lands on small publishers whose feeds change a few times a day: a local paper with 10 articles a day would get 1,440 polls for 10 changes. Polling everything every 30 minutes is cheap and misses the target six times over.

Publishing rates are extremely skewed and predictable: a wire service posts every minute all day, a weekly magazine twice a week, and most feeds follow a strong daily rhythm. So each feed’s interval should follow its own rate. Push (WebSub, pings) brings latency down to seconds when it works, but only some publishers run a hub, hubs are misconfigured, and pings get dropped, so push can make the fast path faster but cannot be the only path.

Adaptive polling per feed, plus WebSub and pings, with polling as the backstop chosen
The same fixed interval for every feed rejected
Push only: WebSub and publisher pings rejected
Poll section front pages or news sitemaps instead of feeds situational: A publisher has no feed, or a broken one: treat its news sitemap or front page as a feed on the same adaptive schedule

The answer: Each feed has a state row: next poll time, ETag and Last-Modified, the last 200 item guids, any WebSub lease, and a publishing rate per hour of the week, kept as a decayed count so both the 9 a.m. rush and the quiet night are learned. The interval is about half the expected gap between items, clamped to between a floor (1 minute for authoritative sources, 5 for the long tail) and 60 minutes, and a feed that just produced items is polled again sooner, because news arrives in bunches. That gives roughly 5 k feeds at 1 minute, 45 k at 5 and 150 k at 30: about 320 polls a second, ~80 % of them 304s. Where a feed advertises a hub, we subscribe, renew well before the lease ends, verify every delivery’s X-Hub-Signature, and relax the backstop poll to 15 minutes; pings to our own endpoint are authenticated per publisher, carry no content, and only trigger an immediate poll, at most one every 30 seconds. Every request goes through the crawler’s fetch tier, which already does politeness, robots.txt and DNS caching, so this component decides when to fetch, never how.

A quiet local paper suddenly has the biggest story in the country, and your model polls it every 30 minutes.

Three things pull it forward. Any new item shortens the feed’s next interval at once, so the second article is found within minutes even if the first took half an hour. When a story bursts, the ranker raises the poll priority of sources whose region or past coverage matches it, which turns a 30-minute feed into a 1-minute one for the next hour. And if the paper pings or runs a hub, it was never waiting on the schedule at all.

Why not trust the content of a WebSub delivery and skip the page fetch?

The delivery is authentic as far as the hub goes: it is signed with the secret we chose when subscribing. But a feed entry is a title, a link and maybe a summary, and we need the page for the canonical URL, the full text for clustering and the structured data for times. So push saves the wait for the next poll, and the page fetch still runs through the normal path. Pings to our own endpoint carry no content by design, so a forged ping costs one extra conditional GET.

The same article reaches you from a feed, a ping and a sitemap, under three different URLs.

Identity is the canonical URL: the page’s rel=canonical, then og:url, then the fetched URL with tracking parameters, fragments and redirect wrappers stripped. The article row is keyed by a hash of it, and a conditional write decides new, update or duplicate, so whichever path arrives first creates the row and the others become no-ops. A content-hash index catches the remaining cases, such as an AMP copy that declares no canonical.

A server returns a new ETag on every request, so every poll looks changed.

Some servers generate ETags per response or rebuild the feed with a fresh timestamp. Diffing on item guids means a changed ETag with no new guids costs one parse and produces nothing. We record that this feed’s validators are useless and stop sending them, and the rate model counts only genuinely new items, so noise never makes the feed get polled faster.

How do you know the schedule is good?

Measure discovery delay: for a sample of articles, compare when we first saw the item with the publisher’s timestamp where it is trustworthy, or with a high-frequency audit poll of a few hundred feeds. Track it per tier against the target of p90 under 2 minutes for the top 5 k feeds. The other side is waste: the share of polls that found nothing, per feed. A feed that is almost always 304 is over-polled; one whose items are always found at the very end of an interval is under-polled.

Grouping articles into stories as they arrive

Four hundred outlets cover the same earthquake in different words. How do you put them in one story within seconds, without comparing every article with every other?

At 2 M articles a day and a 48-hour window there are ~4 M live articles; comparing each newcomer with all of them is 4 M similarity checks per article, and the classic batch algorithms (agglomerative clustering, DBSCAN) are quadratic over the window. They give better clusters but take minutes, which breaks the breaking-news target. And wording varies: a wire report, a local angle, a live blog and an explainer about one quake share few exact phrases, while 300 sites reprinting one wire story share nearly all of them.

Two observations shape the answer. First, there are two kinds of sameness: syndicated copies (nearly identical text) and independent coverage of one event (same entities and time, different words). The first is caught exactly and cheaply with MinHash; the second needs a semantic representation. Second, the ingest rate is tiny, 23 articles a second on average, so each language can be clustered by one process making one decision at a time, which removes a whole class of races and lets every article see every story created before it.

Online assignment: MinHash LSH for copies, then the nearest active centroid by embedding, one writer per language chosen
Batch clustering of the whole window every few minutes situational: As the repair pass over the last 6 hours, where its quality is worth the cost and its output is applied as merges and splits to stable ids
Keyword or entity overlap rules rejected: As one feature in the score, never on its own
Ask an LLM whether each article belongs to each candidate story situational: For borderline scores on stories near the top of big editions, a few thousand checks a day, and for labelling training pairs

The answer: Embed title plus the first three sentences with a multilingual sentence encoder into 384 dimensions, and compute a 100-hash MinHash of the body’s 5-word shingles. For each article, first look it up in 20 LSH bands of 5 rows: a match with estimated Jaccard above 0.8 is a syndicated copy, joins that article’s story and is flagged so it counts once for coverage. Otherwise query an in-memory HNSW index over the centroids of stories active in the last 48 hours in that language for the 10 nearest, score each as cosine similarity plus a bonus for shared linked entities minus a penalty for hours since the story’s last article, and join the best if it clears a threshold around 0.78, tuned on rater-labelled pairs to about 98 % precision; otherwise start a new story. The centroid is a running mean weighted toward recent articles, so a story that develops (the quake, then aftershocks, then the death toll) follows its own coverage. One writer per language consumes the normalised-article topic in order; English, the largest, peaks at ~100 articles a second, which a single process handles with room to spare. Precision beats recall: one event shown as two stories looks like a small duplicate on the page, while two events in one story put the wrong headline over someone’s reporting.

Why compare with the centroid rather than with the nearest article?

Nearest-article assignment is single-link clustering, and it chains: A is close to B, B to C, C to D, and soon a story about an earthquake has absorbed one about insurers’ quarterly results. Comparing with the centroid asks whether the article is about what the story is mostly about. The cost is drift for stories that genuinely evolve, which is why the centroid leans toward recent articles and the repair pass checks cohesion.

Two articles about a brand-new event arrive in the same millisecond. Do you get two stories?

Not within a language: one writer consumes that language’s partition in order, so the second article is scored after the first has created its story, and finds it. With parallel workers sharing an index, both would see no match and both would create a story, and in breaking news that race is the norm, because dozens of articles land in the first minute. Across languages it does happen, on purpose: linking a Japanese and an English story is a separate step.

How do you link the same event across languages?

Keep clusters per language, since thresholds and entity extraction are tuned per language and editions are per language anyway. A job every few minutes compares centroids across languages with the same multilingual encoder, limited to stories with overlapping linked entities and times, and records links rather than merging. Full Coverage then shows international coverage, and a burst in one language can raise the poll priority of sources covering the linked story elsewhere.

A publisher stuffs every article with trending keywords to get into big stories.

The embedding is of the title and lede, not tags or keyword lists, so stuffing moves it little, and entity overlap is a bonus, not the decision. An article that sits far from the centroid but was pulled in by entities scores badly as a representative and is removed by the repair pass’s cohesion check. Repeat offenders lose authority, which costs them far more than any single assignment, since authority drives ranking and headline choice.

How do you measure clustering quality?

Raters label sampled pairs: pairs inside one story (is this the same event?) estimate precision, and pairs from different stories with high similarity estimate missed merges. Track both per language, weighted by impressions, because errors in top stories matter far more than in singletons. Between rating rounds, two cheap live signals help: how often the repair pass changes a story that is already on a front page, and readers’ “not about this story” reports.

Merges, splits and stable story ids

The repair pass decides two stories are one and another is really two. What happens to the story URLs people have shared, the counters, and the pages already cached at the edge?

Online assignment is greedy, so its mistakes are predictable: two stories form in parallel in the first minutes of an event before either has enough articles to attract the other, and a broad early article seeds a story that later turns out to be two events about the same company. The repair pass, average-link clustering over the last 6 hours every 10 minutes, finds a few hundred such cases a day. If it simply replaced the clusters with its own, every run would mint new ids, and every shared link, edge cache entry, reading history entry and velocity series keyed by story id would break every 10 minutes.

The insight is that a story id is a public, durable name, while cluster membership is an opinion that improves over time. So keep ids stable across repairs, express repairs as explicit changes (merge and split) applied by the same single writer as article assignments, and keep enough per-article data (time, source, copy flag) to recompute any story’s counters from its members after a change.

Stable ids; merges become aliases and splits give the minority a new id, sequenced through the writer chosen
Recompute clusters from scratch each run and match them to old ids by overlap situational: Offline, for the archive and as a clean reference to evaluate the online clusterer against
Never repair; accept greedy mistakes rejected
Let the repair job write to the story store directly rejected

The answer: Story ids are 64-bit time-ordered ids minted by the language’s writer. The repair pass reads a snapshot, proposes merges (centroid cosine above about 0.88 with overlapping entities and times) and splits (two sub-clusters, each cohesive, well separated and with at least 3 sources), and hands them to the writer, which applies them in sequence with assignments and publishes them to story.changes, keyed by language, each with a change id. On a merge the older story survives, which is almost always the one already linked from front pages, and the other becomes an alias row, alias_of = survivor, which the API answers with a 301 and the ranker collapses. On a split, the half containing the current representative keeps the id and the other half gets a new one; counters such as originals per 5 minutes and distinct sources are recomputed from member rows. A story is repaired at most once per 30 minutes, so nothing flaps. Consumers apply changes idempotently by change id, so a replay after a crash is harmless.

A reader shared a link to the absorbed story yesterday.

The alias row stays, so /stories/{old} answers 301 to the survivor and the reader lands on the merged story. Aliases are resolved to the end of the chain when written: when B merges into C, every alias pointing at B is rewritten to C, so a request never follows more than one hop. Alias rows are tiny and kept as long as the story is.

The writer applies a merge while the edge still has the old story page cached.

For up to its 30-second TTL the edge serves the old page, which is a slightly stale but valid story. On revalidation the origin answers with the redirect, and the ranker’s next snapshots reference only the survivor. Repairs do not purge: the staleness is bounded and small, and a purge per repair would be hundreds of purges a day for nothing a reader would notice.

Why does the half with the representative keep the id, and not the bigger half?

Because that is what readers saw. The headline and image on the front page belong to the representative, so keeping its half under the old id means the page people have open still shows the same story. Choosing by size would sometimes move the visible headline to a new id and leave the old URL showing a different event.

What if the repair pass is wrong and merges two different events?

The split check runs on merged stories too, so a bad merge is undone on a later pass, after the 30-minute limit. The merge bar (0.88) is higher than the join bar (0.78) because a merge moves many articles at once. For the top 10 of the biggest editions, a borderline merge can be held for an editor or an LLM check, which is a handful of decisions an hour.

How many changes is this, and can consumers keep up?

About 2 M assignments a day plus a few hundred repairs: tens of changes a second on average and low hundreds in a big event. Each consumer (the store writer, the ranker, the Full Coverage builder) handles that on one machine. Keying story.changes by language gives each language one total order, which is the simplest correct choice at these rates.

Ranking stories and spotting breaking news

What makes a story “top” in the UK edition right now, and how does an earthquake reach the top within five minutes without the list jumping around every minute?

We rank stories, not articles: ~600 k active stories, most of them a single article, and the useful signals live on the story. How many independent publishers cover it, how authoritative they are, how fast coverage is growing, how recent the latest development is, and where it matters. Article count alone is easy to game and dominated by syndicated copies; reader clicks reward the sensational and feed on themselves, since a story at the top is clicked partly because it is at the top.

The other constraint is time. Lists are recomputed every 60 seconds for 840 edition and section pages, and a breaking story must reach the top on its first or second cycle, while ordinary stories should hold their places for minutes rather than swapping with every new article. Breaking news is statistically obvious: a new story going from nothing to 60 original articles from 35 publishers in 5 minutes is far outside anything the baseline produces.

A score on independent coverage, authority, velocity, freshness and regional relevance, every 60 s, smoothed, with a burst rule chosen
Rank by article count in the last few hours rejected
Rank by reader engagement situational: As a modest feature with position-corrected click rates in sections like Entertainment and in For you; never the main driver of top stories
A learned model trained on rater judgements of importance situational: Once the formula is instrumented and labels exist; it uses the same features and keeps the burst override

The answer: For each story and edition, coverage is log(1 + Σ authority) over distinct publishers with an original article in the last 6 hours, where authority is an offline per-source, per-topic score built from original reporting, citations by other publishers and corrections history; syndicated copies add nothing. Velocity is original articles in the last 30 minutes against the story’s own previous rate. Freshness decays with the age of the latest original article, with a half-life of about 4 hours, so a long-running story stays only while it keeps developing. Regional relevance is the share of coverage from publishers in the edition’s region plus geo entities in it. The ranker keeps these aggregates in memory from story.changes and scores the ~300 k stories with an article in the last 24 hours for 70 editions every minute (about 20 M scores, a few cores). It smooths scores over the last 5 cycles and swaps two stories only when the gap exceeds 10 %, then applies diversity rules: at most 3 of the top 10 from one section, and never two stories whose centroids have a cosine similarity above 0.85 (a likely missed merge). A burst overrides smoothing: at least 10 original articles in 5 minutes, at least 5 publishers with authority above 0.5, and a Poisson tail probability below 10⁻⁶ against the baseline for a new story in that topic. The story is then labelled Breaking in the editions where it is relevant and pinned at the top for up to 2 hours.

A publisher republishes old articles with today’s date to look fresh.

Freshness and velocity use first-seen time, never the claimed date. A re-dated article has the same canonical URL, so it is a new version of a row whose first-seen time is last week, and it gains nothing. A publisher caught doing it repeatedly loses authority, which costs far more than it ever gained.

Fifty sites copy one viral social media post, and coverage spikes. Is that breaking news?

The burst rule counts original articles from distinct publishers above an authority floor, and fifty near-identical rewrites from low-authority sites mostly fail both tests: many are MinHash copies, and authority above 0.5 is earned. If ten authoritative newsrooms report it within five minutes, then it is news, wherever it started.

How do you evaluate a change to the ranking weights?

Offline, replay a week of story.changes through the new ranker and have raters compare the two top-10 lists side by side for sampled editions and times, which measures importance directly. Online, run the new weights for a slice of readers and watch position-corrected click-through on top stories, return visits and complaints. Also track time-to-top for stories raters later marked as major: a change that delays breaking news is a regression even if clicks rise.

Why every 60 seconds? Why not re-rank on every new article?

Because the page is cached for 60 seconds at the edge anyway, and bursts are judged on 5-minute counts, so ranking per article would change nothing a reader sees while multiplying snapshot writes. Sixty seconds keeps the end-to-end budget at about a minute to discover, a minute to rank and a minute at the edge, inside the 5-minute target. A confirmed burst skips smoothing and is published in the cycle that detects it.

How do you work out regional relevance for a story with no place names?

From who covers it: if 80 % of the authoritative coverage comes from publishers based in Brazil, the story is relevant to the Brazil edition whatever the text says. Geo entities add to that, and global topics such as a central bank decision gain relevance in many editions because publishers everywhere cover them. Local sections use the same idea at city level, with a stricter rule: the publisher or the entities must be local.

Serving 200 k requests a second

A big event hits and 200,000 people a second open the home page. Why not personalise the whole page, and what stops the origin from being trampled?

Normal load is ~10 k page requests a second; a big event brings 200 k, a 20× spike that arrives within a minute. A page built per reader would need a profile read, a candidate fetch and a ranking pass for every request, at the moment the stores are also busiest with ingestion. And the spike is synchronised: everyone wants the same new version of the same page at the same moment, the classic thundering herd if caches expire together or the origin computes on a miss.

But most of the page is the same for everyone in an edition: top stories for en-US are top stories for every en-US reader. There are only ~840 shared pages (70 editions × 12 sections), ~40 KB each, recomputed once a minute. Only the For-you slice depends on the reader, and its candidates are about a thousand live stories per edition, not millions of items, so personalising it is a cheap re-ordering of a list that already exists.

Shared pages precomputed and cached at the edge, plus a separate For-you request that re-ranks a cached pool chosen
Personalise the whole page on every request rejected
Precompute every reader’s page rejected
Purge the edge on every new version instead of short TTLs situational: For a confirmed breaking story: soft-purge only the top-stories pages of affected editions, so the label appears in seconds and stale-while-revalidate still protects the origin

The answer: The ranker writes each page as an immutable versioned object (pages/en-US/top/v48212) and then moves a latest pointer; origin servers watch the pointers and hold the current version of all 840 pages in memory (~35 MB), so a request never reads a store. The edge caches GET /v1/editions/{edition}/{section} with s-maxage=60, stale-while-revalidate=60 and stale-if-error=3600, coalesces concurrent misses in every PoP, and refreshes through ~20 shield locations, so the origin sees about one request per page per shield per minute, ~280 a second, however large the audience. Stale-while-revalidate means no reader waits on a refresh, and stale-if-error keeps the last good page up for an hour if the whole pipeline is down. Signed-in clients fetch For you separately with Cache-Control: private: the service reads the reader’s profile (follows, hides, a 64-dimension interest vector and the last 200 reads, ~5 KB) and re-ranks the edition’s pool of ~1,000 candidate stories, which the ranker publishes with each snapshot and every For-you server holds in memory, in about 2 ms of CPU; at 80 k requests a second that is ~160 cores. Under overload For you sheds first and the page renders without it, because the shared page is the one that must never fail.

Why not use the recommendation system’s full funnel for For you?

Because the corpus is different. A video recommender retrieves from billions of items with ANN and a cascade of rankers; here the live corpus per edition is about a thousand stories, already scored for importance, so retrieval is “take the pool” and one scoring pass over 1,000 items costs about a millisecond. If For you grew into an endless feed with a learned ranker, the funnel, feature store and served-feature logging from the recommendation design would apply as they are, with the edition pool as one candidate source.

The ranker publishes a bad snapshot, say an empty top stories list.

The publish step validates before it moves the pointer: a minimum story count, a representative for every story, and no edition losing more than half its stories in one cycle. A failed check leaves the previous version live and alerts. Because versions are immutable, rollback is moving the pointer back, and the edge picks it up within a TTL.

What does a reader see in the minute after a breaking story is ranked?

Without a purge, the previous version for up to 60 seconds. For a confirmed burst we soft-purge only the top-stories pages of the affected editions: they are marked stale, so the next request in each PoP triggers one coalesced revalidation while still being answered from the old copy, which cuts that minute to seconds. Apps that are already open check a tiny, edge-cached version endpoint every minute and show a “new stories” pill.

There are 5,000 local sections, most of them rarely read. Do you precompute them all every minute?

No. They are built on demand, at most once a minute each, from local candidates the ranker keeps in memory, and cached at the edge for 5 minutes, since local stories move slowly. Precomputing all 5,000 every minute would mostly produce pages nobody opens this hour.

How do you make sure personal data never ends up in the shared cache?

Shared endpoints never read cookies or authorization headers: the edge strips them before the cache lookup, and the origin rejects shared requests that still carry them. Responses are Cache-Control: public with no Set-Cookie, and a test checks that two different readers get identical bytes. For you lives on its own path with Cache-Control: private, so a misconfiguration fails safe, by not caching, rather than by caching someone’s page.

Headline choice, source diversity and fairness

Which of 400 articles gets the headline and the click, and how do you stop the same three big publishers from winning every story?

The representative article gets most of the clicks on a story card, so choosing it is choosing where the traffic goes, and publishers notice. The obvious rules each fail: the earliest article is often a two-line flash or a site that copied it; the most authoritative publisher sends nearly all traffic to a handful of national brands; the most clicked headline rewards sensationalism. And syndication means the same wire text may appear under 300 mastheads.

There are three separate goals: the headline must describe the story accurately, the card should show a range of sources (different publishers, a local outlet, analysis, labelled opinion), and exposure across publishers should not be winner-takes-all. Each is a decision on top of the cluster, and each can be measured.

Score candidates for centrality and quality, rotate near-ties, cap exposure, pick supporting links for diversity chosen
The earliest article wins rejected
The most authoritative publisher wins rejected
Generate a neutral headline with a model situational: For the title of a Full Coverage page on a long-running topic, reviewed; never as a replacement for an article’s headline on a card

The answer: Per edition, the candidates are the story’s original (non-copy) articles from the last 6 hours in the edition’s language. Each is scored on cosine similarity to the centroid (is it about the core of the story), the publisher’s authority for the topic, freshness, a headline check (a clickbait classifier, length, no all-caps), and a boost for the publisher that reported the story first and for local publishers on local stories. The top candidate wins unless others are within 5 %; then the pick rotates among them by hash(story, edition, hour), so close calls share the traffic. Each publisher’s share of representative impressions in an edition is capped, at say 10 % a day, and beyond the cap its near-ties go to others. Four supporting links are chosen by maximal marginal relevance, each maximising relevance minus similarity to those already picked, with at most one per publisher, one copy of any wire story, and opinion and fact-checks labelled. The headline is always the publisher’s own, linked to the publisher’s page, and concentration is tracked as a Herfindahl index of representative impressions per edition, next to clicks and rater quality.

A local paper broke the story; an hour later national outlets have fifty articles. Who gets the headline?

The first-report boost keeps the local paper in contention while its piece stays central and fresh. In the local edition it usually wins; in the national edition it is at least among the supporting links. If national coverage has moved the story on with new facts, those articles sit closer to the updated centroid and win, which is the right outcome for readers.

Two readers see different headlines for the same story. Is that a problem?

Only if it looks like an error. Rotation changes hourly per edition, not per request, so a reader who refreshes sees a stable card, and the edge cache keeps it stable anyway. Both headlines passed the same centrality and quality checks, so both describe the story accurately.

How do you decide that an article is opinion?

From the publisher’s own markup first: schema.org types and section paths such as /opinion/ are reliable for most large publishers. A classifier covers the rest, trained on publisher-labelled articles, with its threshold set for precision, because labelling news as opinion is the worse error. The label shows on the card so readers know what they are clicking.

Should a paywalled article be the representative?

It can be, when it is the best coverage, and much of the best reporting is paywalled. The card labels it, and when the best paywalled and the best free article are within the near-tie band, the shared page prefers the free one, since most readers cannot open the other. For you can prefer publishers the reader subscribes to.

How would you show publishers that this is fair?

Publish the rules, not the weights: original reporting is rewarded, copies count once, close calls rotate and no publisher exceeds a share cap. Give each publisher a dashboard of its impressions, headline wins and the reasons it lost close calls (a copy, older, less central). Internally, the concentration index per edition is a launch metric for any ranking change, next to rater quality.

The theory behind it

Web crawling at scale: How large crawlers fetch billions of pages: the URL frontier, politeness per host, prioritisation and recrawl scheduling, DNS and fetching, URL normalisation and deduplication, content fingerprints, robots.txt, traps and distributing the crawl.
Bloom filters, HyperLogLog and count-min sketch: Probabilistic data structures for system design interviews: Bloom filters for membership, HyperLogLog for counting distinct items, count-min sketch for frequencies and heavy hitters, with sizes, error rates and where each is used.
The thundering herd problem: When many clients act at the same moment and overwhelm a system: cache stampedes, retry storms, reconnect storms after outages, synchronised cron jobs and notification opens, cold starts after deploys, and the fixes: jitter, request coalescing, early refresh, backoff, admission control and staggering.