Design Google News

• Ingest articles from 50,000 publishers: RSS and Atom feeds, news sitemaps, WebSub hubs and publisher pings. Fetching itself (politeness, robots.txt, DNS) is the existing crawler fetch tier; this design decides what to fetch and when.
• Normalise and deduplicate articles: Canonical URL, publisher, published and first-seen time, language, title, lede, image and linked entities. The same article seen through a feed, a ping and a sitemap, or under AMP and print URLs, is stored once; a corrected headline is a new version, not a new article.
• Group articles about the same event into stories: Within seconds of ingest: 300 reprints of one wire report and 100 independent reports of the same earthquake become one story, with a representative article, a headline, and a stable id and URL.
• Top stories per edition and section: Ranked lists for ~70 editions (region plus language) with ~12 sections each: World, Business, Technology, Sports, Local and so on. Stories are ranked, not articles.
• Breaking news: A story whose coverage bursts is labelled Breaking and moved to the top of the editions where it matters within minutes, and an event is emitted for the notification system to alert followers.
• For you: A personalised list from followed topics, sources and reading history, with controls (follow, hide a source, fewer like this). Built on top of the shared candidates, not instead of the shared page.
• Full Coverage of a story: A page per story: a timeline of articles, a range of publishers, local sources, opinion and fact-checks labelled, and links to coverage of the same event in other languages.
• Out of scope: The crawler’s fetch internals (see Design a Web Crawler), full-text news search, delivering push notifications, ads, comments and publisher payments. We show headlines, snippets and links; readers read on the publisher’s site.

• Freshness for breaking news (Breaking on top < 5 min): From the first authoritative reports to the label on the edition page: about a minute to discover, a minute to rank, a minute at the edge. This drives the hardest trade-off in the design: clustering must decide online and greedily, and be corrected later.
• Discovery latency (p90 < 2 min (top 5k feeds)): Publication to first seen, measured against the publisher’s timestamp or an audit poll. The long tail may take up to an hour; nobody waits on a weekly magazine.
• Read scale and latency (200 k/s peak · p99 < 200 ms): Normal load is ~10 k page requests a second; a big event brings a 20× spike within a minute, synchronised on the same pages. Almost all of it must be answered by the edge.
• Availability of edition pages (99.95 %): A page a minute stale is acceptable; a blank page is not. The last good snapshot stays servable for an hour even if ingestion, clustering and ranking all stop.
• Clustering precision (≥ 98 % same-event pairs): Rater-labelled sample pairs, weighted by impressions. Merging two events puts the wrong headline over someone’s reporting, which is worse than showing one event as two stories, so thresholds favour precision.
• Fairness to publishers (≤ 10 % of headlines per publisher): Per edition per day, measured on representative impressions. Syndicated copies count once, original reporting is rewarded, and close calls rotate rather than always going to the biggest brand.
• Good citizen to publishers (≤ 1 poll/min per feed): Conditional GETs, politeness and robots.txt through the shared fetch tier, and push wherever a publisher offers it.
• Privacy (no personal data in shared pages): Reading history powers For you only, can be paused or deleted, and never enters a cached page: a shared page is identical bytes for every reader in an edition.

• Articles ingested: ~23/s avg · ~200/s peak. 2 M ÷ 86 400 ≈ 23/s. Mornings and big events bunch publishing into bursts of ~10×, so plan for ~200/s. One machine could parse this: writes are not the hard part.
• Feed polls: ~320/s, ~80 % are 304s. 50 k sources × ~4 feeds (main, sections, news sitemap) = 200 k feeds. Adaptive tiers: 5 k busy feeds every minute (83/s), 45 k every 5 minutes (150/s), 150 k every 30 minutes (83/s) ≈ 320 polls/s. Most return 304 Not Modified, so polling costs a few MB/s; latency, not cost, is why we add push.
• Page requests: ~10 k/s avg · 200 k/s event peak. 300 M daily readers × ~3 page loads = 900 M/day ÷ 86 400 ≈ 10 k/s. A normal morning peaks at ~3× that; a big event brings 200 k/s within a minute, a 20× spike. The read side is the scale problem.
• Origin requests for shared pages: ~280/s. 70 editions × 12 sections = 840 shared pages, ~40 KB each (~35 MB in all). With a 60 s edge TTL, coalesced misses and ~20 shield locations that each refresh every page once a minute: 840 × 20 ÷ 60 ≈ 280/s at the origin, whether readers send 10 k/s or 200 k/s.
• For-you requests at peak: ~80 k/s · ~160 cores. If 40 % of page loads are signed-in readers who get For you: 200 k/s × 0.4 = 80 k/s, the only per-reader work in the system. Re-ranking a ~1,000-story pool in memory costs ~2 ms of CPU, so ~160 cores: about ten 32-core servers at half load.
• Clustering working set: ~6 GB of vectors. A 48-hour window × 2 M/day = 4 M live articles × a 384-dimension float32 embedding (1.5 KB) = 6 GB, plus 400 B of MinHash each (1.6 GB) and ~600 k story centroids (0.9 GB). English is ~40 %, about 3 GB: one language fits in one machine’s memory, so each language can have a single writer.
• Storage growth: ~7 TB/yr articles · 1.5 TB profiles. ~10 KB an article (metadata, compressed text, embedding, MinHash) × 2 M/day = 20 GB/day ≈ 7 TB/yr. Reader profiles: 300 M × 5 KB = 1.5 TB. Storage is small; the design is driven by latency and read spikes.

• Reader app / web (web · iOS · Android): Fetches the shared edition page from the edge and, for signed-in readers, the For-you slice in a second request the page can render without. A tap opens the publisher’s own page; a beacon records the read.
• Edge cache (CDN · 60 s TTL · coalescing): Caches shared pages keyed by edition and section only, with no cookies in the key, for 60 s with stale-while-revalidate and stale-if-error. Coalesces concurrent misses in each PoP and refreshes through ~20 shield locations. This is what turns a 200 k/s event spike into a few hundred origin requests a second.
• News API (origin) (pages held in memory): Stateless origin for edition pages, Full Coverage pages, For you and read events. Holds the current version of every shared page in memory, preloaded when its version pointer moves, so a request never reads a store. Resolves merged story ids to their survivor with a 301.
• For-you service (rerank ~1,000 candidates): Builds the personal slice by re-ranking the edition’s candidate pool, held in memory, against the reader’s profile, in about 2 ms of CPU. It is the first thing shed under overload. Also folds read events into profiles, batched per reader.
• Reader profiles (Bigtable · by user id): Followed topics, entities and sources, hidden sources, a 64-dimension interest vector that decays toward recent reads, and the last 200 stories read: ~5 KB a reader, ~1.5 TB in all, with a regional read cache. Paused or deleted on the reader’s request.
• Page snapshots (versioned per edition/section): Immutable versioned snapshots of every shared page (pages/{edition}/{section}/v{n}) and each edition’s candidate pool, written by the ranker, plus a latest pointer per page moved by compare-and-set after validation. Object storage for versions, a small KV for pointers. Rolling back is moving a pointer.
• Story ranker (every 60 s · per edition): Every minute, scores recently active stories for each of 70 editions on independent coverage, source authority, velocity, freshness and regional relevance; smooths ranks so lists do not flap; detects bursts; chooses each story’s representative per edition; and publishes the snapshots. Keeps story aggregates in memory from story.changes, so a cycle reads almost nothing.
• Story store (Bigtable · by story id): Story rows keyed by story id: centroid, member articles with time, source and copy flag, 5-minute coverage counts, distinct sources, entities, topics, and alias_of for merged stories. Written only from story.changes, which the language’s clusterer produces. Members are kept so any counter can be recomputed after a merge or split.
• Story clusterer (one writer per language): One writer per language consumes normalised articles in order. Catches syndicated copies with MinHash LSH, otherwise finds the nearest active story centroids in an in-memory HNSW index over a 48-hour window (~3 GB for English), and joins or starts a story. Runs a repair pass every 10 minutes that proposes merges and splits, applied in the same sequence. Commits decisions with its input offsets in one Kafka transaction and has a warm standby.
• Article store (by canonical URL hash): One row per article keyed by a hash of the canonical URL, with versions, publisher, times, language, entities and story id, plus an index on content hash for the same text under another URL. ~10 KB an article, ~7 TB a year. A conditional write on the row is the exact-dedupe decision.
• Parse & dedupe (canonical URL · hash · embed): Turns fetched pages into normalised articles: canonical URL (rel=canonical, og:url, tracking parameters stripped), publisher, published and first-seen time, language, title, lede, image, linked entities and topics. Decides new, update or duplicate against the article store, then embeds title and lede and computes a MinHash of the body. Stateless, scaled to bursts of ~200 articles a second.
• Article log (Kafka · by URL, by language): Kafka. items.raw is keyed by URL hash, so two versions of one article stay in order; articles.normalised and story.changes are keyed by language, so one clusterer sees one language in a single order. Lets parsing and clustering lag during a burst without losing anything.
• Feed poller (adaptive schedule · ETags): One state row per feed (~200 k): next poll time, ETag and Last-Modified, recent item guids, a publishing rate per hour of the week and any WebSub lease. Polls on an adaptive 1 to 60 minute interval with conditional GETs, fetches each new item’s page, and has a priority lane for pushed items. All requests go through the crawler’s shared fetch tier.
• Push receiver (WebSub · publisher pings): Internet-facing endpoint for WebSub deliveries and publisher pings. Verifies the hub signature or the publisher’s key, answers in milliseconds, and turns each notification into a fetch-now request to the poller. It never treats pushed content as the article: the page is always fetched from the publisher.
• Publishers (RSS/Atom · sitemaps · hubs): 50,000 news sites, wire services and blogs. They expose RSS or Atom feeds, news sitemaps, sometimes a WebSub hub, and article pages with canonical links and schema.org NewsArticle data of uneven quality. A wire service publishes every minute; most publishers a few times a day.

1. A polled article joins its story: The everyday write path: a feed poll finds a new item, the page is fetched, normalised and deduplicated, and the clusterer puts it in the story it belongs to. Seconds from poll to story; the ranker picks it up on its next 60-second cycle.
   1. The poller finds a busy world-news feed due and sends a conditional GET with the ETag from its last poll.
   2. The poller diffs the items against the guids it has seen, finds two new ones, and fetches each article page.
   3. The raw item goes onto the article log.
   4. The parser extracts the canonical URL, publisher, times, language, title, lede, body and linked entities.
   5. The parser checks the article store by canonical URL and content hash: new, update or duplicate.
   6. The parser embeds the title and lede, computes a MinHash of the body, and publishes the normalised article for its language.
   7. The clusterer looks for a syndicated copy in its LSH buckets, then for the 10 nearest active story centroids.
   8. It scores the candidates, joins the best one above the threshold, and records the change.
2. Open the home page in the US edition: The main read path. The shared page comes from the edge as the same bytes for every US reader; only the small For-you slice touches our servers per reader.
   1. The client asks for US top stories, and the nearest edge location has a fresh copy.
   2. When the copy expires, the edge keeps serving it and sends one revalidation upstream for everyone.
   3. The origin answers from the snapshot it already holds in memory.
   4. A signed-in client asks for its For-you slice in a second request, which the edge passes through.
   5. The For-you service reads the reader’s profile.
   6. It re-ranks the edition’s candidate pool against the profile and returns 20 stories.
   7. The reader opens a story’s Full Coverage page, cached at the edge per story.
   8. A tap opens the publisher’s article, and a beacon records the read for For you.
3. An earthquake: 300 sources in 15 minutes: The scale-breaking case, on both sides at once. Coverage bursts into the pipeline and has to reach the top of the right editions within minutes, while readers arrive at 200 k requests a second.
   1. Within a minute, wire services and large publishers push WebSub deliveries and pings.
   2. The receiver asks the poller to fetch those items now, ahead of the schedule.
   3. The first article starts a new story, and the next forty join it, one decision at a time.
   4. On its next cycle the ranker sees 60 original articles from 35 publishers in 5 minutes and flags a burst.
   5. The ranker labels it Breaking, pins it at the top of the editions where it is relevant, and publishes new snapshots in the same cycle.
   6. Readers pile in at 200 k requests a second; the edge serves them, with a soft purge of only the affected top-stories pages.
   7. The quake’s Full Coverage page draws tens of thousands of requests a second and is served the same way.
   8. For you grows with signed-in traffic and sheds first if it cannot keep up.
4. The English clusterer dies mid-burst: Clustering is a single writer per language by design, so its failure has to be quick to recover from, and must never assign an article twice or let two writers decide at once.
   1. The English writer stops renewing its lease, and the warm standby takes it about 10 seconds later.
   2. The new writer opens a transactional producer with the language’s fixed id, which fences the old one.
   3. It resumes reading English articles from the last committed input offset.
   4. It drains the backlog that built up during the failover.
   5. Meanwhile edition pages kept being served from the last snapshots.
   6. For a small language with no standby, the new writer first rebuilds its window from the story store.
5. Repair merges two stories and splits one: Online assignment is greedy and makes mistakes in both directions. A slower pass over the last 6 hours fixes them every 10 minutes, without breaking story URLs or making the front page jump.
   1. Every 10 minutes the English writer snapshots the stories updated in the last 6 hours and re-clusters their articles with average-link clustering.
   2. It finds two stories about the same court ruling, born a minute apart, and proposes a merge.
   3. It finds a story whose articles fall into two tight groups, a company’s results and a lawsuit against it, and proposes a split.
   4. The writer applies the merge in sequence with assignments: the older story survives and the other becomes an alias.
   5. It applies the split: the half with the current representative keeps the id, and the other half gets a new one.
   6. The ranker’s next cycle shows one ruling story and two company stories, and the merged-away URL redirects.

Ask: Why not poll every feed once a minute? And if publishers can push, why poll at all?

Good answers name: Adaptive polling per feed, plus WebSub and pings, with polling as the backstop, The same fixed interval for every feed, Push only: WebSub and publisher pings, Poll section front pages or news sitemaps instead of feeds.

Our pick: Each feed has a state row: next poll time, ETag and Last-Modified, the last 200 item guids, any WebSub lease, and a publishing rate per hour of the week, kept as a decayed count so both the 9 a.m. rush and the quiet night are learned. The interval is about half the expected gap between items, clamped to between a floor (1 minute for authoritative sources, 5 for the long tail) and 60 minutes, and a feed that just produced items is polled again sooner, because news arrives in bunches. That gives roughly 5 k feeds at 1 minute, 45 k at 5 and 150 k at 30: about 320 polls a second, ~80 % of them 304s. Where a feed advertises a hub, we subscribe, renew well before the lease ends, verify every delivery’s X-Hub-Signature, and relax the backstop poll to 15 minutes; pings to our own endpoint are authenticated per publisher, carry no content, and only trigger an immediate poll, at most one every 30 seconds. Every request goes through the crawler’s fetch tier, which already does politeness, robots.txt and DNS caching, so this component decides when to fetch, never how.

1. A quiet local paper suddenly has the biggest story in the country, and your model polls it every 30 minutes.
   Three things pull it forward. Any new item shortens the feed’s next interval at once, so the second article is found within minutes even if the first took half an hour. When a story bursts, the ranker raises the poll priority of sources whose region or past coverage matches it, which turns a 30-minute feed into a 1-minute one for the next hour. And if the paper pings or runs a hub, it was never waiting on the schedule at all.
2. Why not trust the content of a WebSub delivery and skip the page fetch?
   The delivery is authentic as far as the hub goes: it is signed with the secret we chose when subscribing. But a feed entry is a title, a link and maybe a summary, and we need the page for the canonical URL, the full text for clustering and the structured data for times. So push saves the wait for the next poll, and the page fetch still runs through the normal path. Pings to our own endpoint carry no content by design, so a forged ping costs one extra conditional GET.
3. The same article reaches you from a feed, a ping and a sitemap, under three different URLs.
   Identity is the canonical URL: the page’s rel=canonical, then og:url, then the fetched URL with tracking parameters, fragments and redirect wrappers stripped. The article row is keyed by a hash of it, and a conditional write decides new, update or duplicate, so whichever path arrives first creates the row and the others become no-ops. A content-hash index catches the remaining cases, such as an AMP copy that declares no canonical.
4. A server returns a new ETag on every request, so every poll looks changed.
   Some servers generate ETags per response or rebuild the feed with a fresh timestamp. Diffing on item guids means a changed ETag with no new guids costs one parse and produces nothing. We record that this feed’s validators are useless and stop sending them, and the rate model counts only genuinely new items, so noise never makes the feed get polled faster.
5. How do you know the schedule is good?
   Measure discovery delay: for a sample of articles, compare when we first saw the item with the publisher’s timestamp where it is trustworthy, or with a high-frequency audit poll of a few hundred feeds. Track it per tier against the target of p90 under 2 minutes for the top 5 k feeds. The other side is waste: the share of polls that found nothing, per feed. A feed that is almost always 304 is over-polled; one whose items are always found at the very end of an interval is under-polled.

Ask: Four hundred outlets cover the same earthquake in different words. How do you put them in one story within seconds, without comparing every article with every other?

Good answers name: Online assignment: MinHash LSH for copies, then the nearest active centroid by embedding, one writer per language, Batch clustering of the whole window every few minutes, Keyword or entity overlap rules, Ask an LLM whether each article belongs to each candidate story.

Our pick: Embed title plus the first three sentences with a multilingual sentence encoder into 384 dimensions, and compute a 100-hash MinHash of the body’s 5-word shingles. For each article, first look it up in 20 LSH bands of 5 rows: a match with estimated Jaccard above 0.8 is a syndicated copy, joins that article’s story and is flagged so it counts once for coverage. Otherwise query an in-memory HNSW index over the centroids of stories active in the last 48 hours in that language for the 10 nearest, score each as cosine similarity plus a bonus for shared linked entities minus a penalty for hours since the story’s last article, and join the best if it clears a threshold around 0.78, tuned on rater-labelled pairs to about 98 % precision; otherwise start a new story. The centroid is a running mean weighted toward recent articles, so a story that develops (the quake, then aftershocks, then the death toll) follows its own coverage. One writer per language consumes the normalised-article topic in order; English, the largest, peaks at ~100 articles a second, which a single process handles with room to spare. Precision beats recall: one event shown as two stories looks like a small duplicate on the page, while two events in one story put the wrong headline over someone’s reporting.

1. Why compare with the centroid rather than with the nearest article?
   Nearest-article assignment is single-link clustering, and it chains: A is close to B, B to C, C to D, and soon a story about an earthquake has absorbed one about insurers’ quarterly results. Comparing with the centroid asks whether the article is about what the story is mostly about. The cost is drift for stories that genuinely evolve, which is why the centroid leans toward recent articles and the repair pass checks cohesion.
2. Two articles about a brand-new event arrive in the same millisecond. Do you get two stories?
   Not within a language: one writer consumes that language’s partition in order, so the second article is scored after the first has created its story, and finds it. With parallel workers sharing an index, both would see no match and both would create a story, and in breaking news that race is the norm, because dozens of articles land in the first minute. Across languages it does happen, on purpose: linking a Japanese and an English story is a separate step.
3. How do you link the same event across languages?
   Keep clusters per language, since thresholds and entity extraction are tuned per language and editions are per language anyway. A job every few minutes compares centroids across languages with the same multilingual encoder, limited to stories with overlapping linked entities and times, and records links rather than merging. Full Coverage then shows international coverage, and a burst in one language can raise the poll priority of sources covering the linked story elsewhere.
4. A publisher stuffs every article with trending keywords to get into big stories.
   The embedding is of the title and lede, not tags or keyword lists, so stuffing moves it little, and entity overlap is a bonus, not the decision. An article that sits far from the centroid but was pulled in by entities scores badly as a representative and is removed by the repair pass’s cohesion check. Repeat offenders lose authority, which costs them far more than any single assignment, since authority drives ranking and headline choice.
5. How do you measure clustering quality?
   Raters label sampled pairs: pairs inside one story (is this the same event?) estimate precision, and pairs from different stories with high similarity estimate missed merges. Track both per language, weighted by impressions, because errors in top stories matter far more than in singletons. Between rating rounds, two cheap live signals help: how often the repair pass changes a story that is already on a front page, and readers’ “not about this story” reports.

Ask: The repair pass decides two stories are one and another is really two. What happens to the story URLs people have shared, the counters, and the pages already cached at the edge?

Good answers name: Stable ids; merges become aliases and splits give the minority a new id, sequenced through the writer, Recompute clusters from scratch each run and match them to old ids by overlap, Never repair; accept greedy mistakes, Let the repair job write to the story store directly.

Our pick: Story ids are 64-bit time-ordered ids minted by the language’s writer. The repair pass reads a snapshot, proposes merges (centroid cosine above about 0.88 with overlapping entities and times) and splits (two sub-clusters, each cohesive, well separated and with at least 3 sources), and hands them to the writer, which applies them in sequence with assignments and publishes them to story.changes, keyed by language, each with a change id. On a merge the older story survives, which is almost always the one already linked from front pages, and the other becomes an alias row, alias_of = survivor, which the API answers with a 301 and the ranker collapses. On a split, the half containing the current representative keeps the id and the other half gets a new one; counters such as originals per 5 minutes and distinct sources are recomputed from member rows. A story is repaired at most once per 30 minutes, so nothing flaps. Consumers apply changes idempotently by change id, so a replay after a crash is harmless.

1. A reader shared a link to the absorbed story yesterday.
   The alias row stays, so /stories/{old} answers 301 to the survivor and the reader lands on the merged story. Aliases are resolved to the end of the chain when written: when B merges into C, every alias pointing at B is rewritten to C, so a request never follows more than one hop. Alias rows are tiny and kept as long as the story is.
2. The writer applies a merge while the edge still has the old story page cached.
   For up to its 30-second TTL the edge serves the old page, which is a slightly stale but valid story. On revalidation the origin answers with the redirect, and the ranker’s next snapshots reference only the survivor. Repairs do not purge: the staleness is bounded and small, and a purge per repair would be hundreds of purges a day for nothing a reader would notice.
3. Why does the half with the representative keep the id, and not the bigger half?
   Because that is what readers saw. The headline and image on the front page belong to the representative, so keeping its half under the old id means the page people have open still shows the same story. Choosing by size would sometimes move the visible headline to a new id and leave the old URL showing a different event.
4. What if the repair pass is wrong and merges two different events?
   The split check runs on merged stories too, so a bad merge is undone on a later pass, after the 30-minute limit. The merge bar (0.88) is higher than the join bar (0.78) because a merge moves many articles at once. For the top 10 of the biggest editions, a borderline merge can be held for an editor or an LLM check, which is a handful of decisions an hour.
5. How many changes is this, and can consumers keep up?
   About 2 M assignments a day plus a few hundred repairs: tens of changes a second on average and low hundreds in a big event. Each consumer (the store writer, the ranker, the Full Coverage builder) handles that on one machine. Keying story.changes by language gives each language one total order, which is the simplest correct choice at these rates.

Ask: What makes a story “top” in the UK edition right now, and how does an earthquake reach the top within five minutes without the list jumping around every minute?

Good answers name: A score on independent coverage, authority, velocity, freshness and regional relevance, every 60 s, smoothed, with a burst rule, Rank by article count in the last few hours, Rank by reader engagement, A learned model trained on rater judgements of importance.

Our pick: For each story and edition, coverage is log(1 + Σ authority) over distinct publishers with an original article in the last 6 hours, where authority is an offline per-source, per-topic score built from original reporting, citations by other publishers and corrections history; syndicated copies add nothing. Velocity is original articles in the last 30 minutes against the story’s own previous rate. Freshness decays with the age of the latest original article, with a half-life of about 4 hours, so a long-running story stays only while it keeps developing. Regional relevance is the share of coverage from publishers in the edition’s region plus geo entities in it. The ranker keeps these aggregates in memory from story.changes and scores the ~300 k stories with an article in the last 24 hours for 70 editions every minute (about 20 M scores, a few cores). It smooths scores over the last 5 cycles and swaps two stories only when the gap exceeds 10 %, then applies diversity rules: at most 3 of the top 10 from one section, and never two stories whose centroids have a cosine similarity above 0.85 (a likely missed merge). A burst overrides smoothing: at least 10 original articles in 5 minutes, at least 5 publishers with authority above 0.5, and a Poisson tail probability below 10⁻⁶ against the baseline for a new story in that topic. The story is then labelled Breaking in the editions where it is relevant and pinned at the top for up to 2 hours.

1. A publisher republishes old articles with today’s date to look fresh.
   Freshness and velocity use first-seen time, never the claimed date. A re-dated article has the same canonical URL, so it is a new version of a row whose first-seen time is last week, and it gains nothing. A publisher caught doing it repeatedly loses authority, which costs far more than it ever gained.
2. Fifty sites copy one viral social media post, and coverage spikes. Is that breaking news?
   The burst rule counts original articles from distinct publishers above an authority floor, and fifty near-identical rewrites from low-authority sites mostly fail both tests: many are MinHash copies, and authority above 0.5 is earned. If ten authoritative newsrooms report it within five minutes, then it is news, wherever it started.
3. How do you evaluate a change to the ranking weights?
   Offline, replay a week of story.changes through the new ranker and have raters compare the two top-10 lists side by side for sampled editions and times, which measures importance directly. Online, run the new weights for a slice of readers and watch position-corrected click-through on top stories, return visits and complaints. Also track time-to-top for stories raters later marked as major: a change that delays breaking news is a regression even if clicks rise.
4. Why every 60 seconds? Why not re-rank on every new article?
   Because the page is cached for 60 seconds at the edge anyway, and bursts are judged on 5-minute counts, so ranking per article would change nothing a reader sees while multiplying snapshot writes. Sixty seconds keeps the end-to-end budget at about a minute to discover, a minute to rank and a minute at the edge, inside the 5-minute target. A confirmed burst skips smoothing and is published in the cycle that detects it.
5. How do you work out regional relevance for a story with no place names?
   From who covers it: if 80 % of the authoritative coverage comes from publishers based in Brazil, the story is relevant to the Brazil edition whatever the text says. Geo entities add to that, and global topics such as a central bank decision gain relevance in many editions because publishers everywhere cover them. Local sections use the same idea at city level, with a stricter rule: the publisher or the entities must be local.

Ask: A big event hits and 200,000 people a second open the home page. Why not personalise the whole page, and what stops the origin from being trampled?

Good answers name: Shared pages precomputed and cached at the edge, plus a separate For-you request that re-ranks a cached pool, Personalise the whole page on every request, Precompute every reader’s page, Purge the edge on every new version instead of short TTLs.

Our pick: The ranker writes each page as an immutable versioned object (pages/en-US/top/v48212) and then moves a latest pointer; origin servers watch the pointers and hold the current version of all 840 pages in memory (~35 MB), so a request never reads a store. The edge caches GET /v1/editions/{edition}/{section} with s-maxage=60, stale-while-revalidate=60 and stale-if-error=3600, coalesces concurrent misses in every PoP, and refreshes through ~20 shield locations, so the origin sees about one request per page per shield per minute, ~280 a second, however large the audience. Stale-while-revalidate means no reader waits on a refresh, and stale-if-error keeps the last good page up for an hour if the whole pipeline is down. Signed-in clients fetch For you separately with Cache-Control: private: the service reads the reader’s profile (follows, hides, a 64-dimension interest vector and the last 200 reads, ~5 KB) and re-ranks the edition’s pool of ~1,000 candidate stories, which the ranker publishes with each snapshot and every For-you server holds in memory, in about 2 ms of CPU; at 80 k requests a second that is ~160 cores. Under overload For you sheds first and the page renders without it, because the shared page is the one that must never fail.

1. Why not use the recommendation system’s full funnel for For you?
   Because the corpus is different. A video recommender retrieves from billions of items with ANN and a cascade of rankers; here the live corpus per edition is about a thousand stories, already scored for importance, so retrieval is “take the pool” and one scoring pass over 1,000 items costs about a millisecond. If For you grew into an endless feed with a learned ranker, the funnel, feature store and served-feature logging from the recommendation design would apply as they are, with the edition pool as one candidate source.
2. The ranker publishes a bad snapshot, say an empty top stories list.
   The publish step validates before it moves the pointer: a minimum story count, a representative for every story, and no edition losing more than half its stories in one cycle. A failed check leaves the previous version live and alerts. Because versions are immutable, rollback is moving the pointer back, and the edge picks it up within a TTL.
3. What does a reader see in the minute after a breaking story is ranked?
   Without a purge, the previous version for up to 60 seconds. For a confirmed burst we soft-purge only the top-stories pages of the affected editions: they are marked stale, so the next request in each PoP triggers one coalesced revalidation while still being answered from the old copy, which cuts that minute to seconds. Apps that are already open check a tiny, edge-cached version endpoint every minute and show a “new stories” pill.
4. There are 5,000 local sections, most of them rarely read. Do you precompute them all every minute?
   No. They are built on demand, at most once a minute each, from local candidates the ranker keeps in memory, and cached at the edge for 5 minutes, since local stories move slowly. Precomputing all 5,000 every minute would mostly produce pages nobody opens this hour.
5. How do you make sure personal data never ends up in the shared cache?
   Shared endpoints never read cookies or authorization headers: the edge strips them before the cache lookup, and the origin rejects shared requests that still carry them. Responses are Cache-Control: public with no Set-Cookie, and a test checks that two different readers get identical bytes. For you lives on its own path with Cache-Control: private, so a misconfiguration fails safe, by not caching, rather than by caching someone’s page.

Ask: Which of 400 articles gets the headline and the click, and how do you stop the same three big publishers from winning every story?

Good answers name: Score candidates for centrality and quality, rotate near-ties, cap exposure, pick supporting links for diversity, The earliest article wins, The most authoritative publisher wins, Generate a neutral headline with a model.

Our pick: Per edition, the candidates are the story’s original (non-copy) articles from the last 6 hours in the edition’s language. Each is scored on cosine similarity to the centroid (is it about the core of the story), the publisher’s authority for the topic, freshness, a headline check (a clickbait classifier, length, no all-caps), and a boost for the publisher that reported the story first and for local publishers on local stories. The top candidate wins unless others are within 5 %; then the pick rotates among them by hash(story, edition, hour), so close calls share the traffic. Each publisher’s share of representative impressions in an edition is capped, at say 10 % a day, and beyond the cap its near-ties go to others. Four supporting links are chosen by maximal marginal relevance, each maximising relevance minus similarity to those already picked, with at most one per publisher, one copy of any wire story, and opinion and fact-checks labelled. The headline is always the publisher’s own, linked to the publisher’s page, and concentration is tracked as a Herfindahl index of representative impressions per edition, next to clicks and rater quality.

1. A local paper broke the story; an hour later national outlets have fifty articles. Who gets the headline?
   The first-report boost keeps the local paper in contention while its piece stays central and fresh. In the local edition it usually wins; in the national edition it is at least among the supporting links. If national coverage has moved the story on with new facts, those articles sit closer to the updated centroid and win, which is the right outcome for readers.
2. Two readers see different headlines for the same story. Is that a problem?
   Only if it looks like an error. Rotation changes hourly per edition, not per request, so a reader who refreshes sees a stable card, and the edge cache keeps it stable anyway. Both headlines passed the same centrality and quality checks, so both describe the story accurately.
3. How do you decide that an article is opinion?
   From the publisher’s own markup first: schema.org types and section paths such as /opinion/ are reliable for most large publishers. A classifier covers the rest, trained on publisher-labelled articles, with its threshold set for precision, because labelling news as opinion is the worse error. The label shows on the card so readers know what they are clicking.
4. Should a paywalled article be the representative?
   It can be, when it is the best coverage, and much of the best reporting is paywalled. The card labels it, and when the best paywalled and the best free article are within the near-tie band, the shared page prefers the free one, since most readers cannot open the other. For you can prefer publishers the reader subscribes to.
5. How would you show publishers that this is fair?
   Publish the rules, not the weights: original reporting is rewarded, copies count once, close calls rotate and no publisher exceeds a share cap. Give each publisher a dashboard of its impressions, headline wins and the reasons it lost close calls (a copy, older, less central). Internally, the concentration index per edition is a launch metric for any ranking change, next to rater quality.