Design a Web Crawler

Fetch billions of pages a month politely, discover new links, skip duplicates, and keep the copy fresh without getting the crawler banned.

Last updated 2026-09-22. Difficulty: hard. Patterns: frontier, politeness, dedupe, scheduling, distributed-queues. Reported at Anthropic and 6 more with Pro.

Functional requirements

• Crawl from a seed set. Start from a list of URLs, fetch each page, extract links, and continue until a budget or scope limit is reached.
• Respect robots.txt and crawl politely. Honour disallow rules and crawl-delay, and never open more than a small number of concurrent connections to one host. A polite crawler is one that does not get blocked.
• Deduplicate URLs and content. Do not fetch the same URL twice in one cycle; detect near-duplicate pages (mirrors, tracking parameters, print versions) so the corpus is not inflated.
• Prioritise. Important or frequently changing pages are fetched first and more often. Priority comes from a score (link-based importance, change rate, freshness needs).
• Store the fetched content. Raw HTML with headers and fetch metadata, versioned, so downstream indexers and analysers can process it. Storage is append-only.
• Re-crawl for freshness. Pages are revisited on a schedule that adapts to how often they change.
• Handle traps and errors. Infinite calendars, session-id URLs, redirect loops, huge files, slow hosts. The crawler bounds its work per host and per page.
• Out of scope. Indexing and ranking, JavaScript rendering (mention the headless-browser lane), and the search product. We produce a corpus.

Non-functional requirements

• Throughput (1 B pages/month · ~400 pages/s). The number is modest per second; the difficulty is that it is spread over tens of millions of hosts with per-host limits.
• Politeness (≤ 1 request per host per second by default). The hardest constraint: it turns a throughput problem into a scheduling problem, because the frontier must always have a host ready that is not in cooldown.
• Freshness (news within hours · long tail within weeks). Drives the re-crawl scheduler and the priority model.
• Coverage and dedupe (< 5 % duplicate content). Duplicates waste fetch budget and storage and skew the corpus.
• Robustness (no single host can stall the crawl). Timeouts, per-host budgets and trap detection are mandatory, not optional.
• Durability (frontier survives crashes). The frontier is the crawler's state; losing it means restarting from seeds. It must be persisted.
• Scalability (horizontal on fetchers and frontier). Add fetchers to add throughput, as long as the frontier can partition hosts among them.

Back-of-envelope estimates

• Fetch rate: ~400 pages/s. 1 B pages/month ÷ 2.6 M s ≈ 385/s. Plan for 2× headroom (800/s) because retries, redirects and robots fetches add requests beyond page fetches.
• Concurrent connections: ~2 000. Average fetch takes ~2 s (DNS, connect, TLS, transfer, slow servers). 800/s × 2 s = 1 600 in flight; round to 2 000. A single fetcher machine handles thousands of idle connections, so a few dozen fetchers with headroom.
• Download bandwidth: ~1.6 Gbit/s. Average page with headers ~500 KB (HTML only; media not fetched). 400/s × 500 KB = 200 MB/s ≈ 1.6 Gbit/s. Bandwidth is not the bottleneck; politeness is.
• Storage per month: ~150 TB compressed. 1 B × 500 KB = 500 TB raw; HTML compresses ~3–4×, so ~150 TB/month, ~1.8 PB/year. Object storage with lifecycle rules; keep the latest version hot and older versions cold.
• URLs seen: ~10–20 B. Each page yields ~50 links; most point to already-known URLs. Over a year the seen-URL set reaches tens of billions. The URL-seen structure must hold ~15 B keys: as 8-byte fingerprints that is ~120 GB, feasible in a sharded key-value store or a large Bloom filter.
• Hosts in the frontier: ~50 M. The crawlable web is a few hundred million hosts; a 1 B-page crawl touches roughly 50 M of them. With a 1 req/s per-host limit, most hosts are idle at any moment: only ~400 need to be "ready" per second, so the scheduler must keep many hosts warm.
• DNS lookups: ~400/s → ~10/s with cache. One lookup per fetch without caching. Hosts repeat heavily, so a local DNS cache with a ~1 h TTL cuts external lookups by 95 %+. Run a caching resolver per fetcher cluster; public resolvers rate limit.

Components

• Seed / URL API: Where crawls start: seed lists, sitemaps, and an API for operators to inject or reprioritise URLs. Also receives external hints such as ping services and sitemap updates.
• URL frontier (per-host queues + priority): The heart of the crawler: the set of URLs to fetch, organised so that a fetcher can always get the next URL for a host that is not in cooldown. Two levels: priority queues that select which URLs are eligible, and per-host FIFO queues with a next-allowed-time that enforce politeness. Persisted, partitioned by host across frontier nodes.
• Host scheduler (min-heap of next-fetch times): Maintains, per frontier partition, a heap of hosts keyed by the earliest time they may be fetched again. Hands out (host, URL) leases to fetchers so that no host has more than one in-flight request and the per-host delay is respected.
• Fetcher fleet (async HTTP · many connections): Stateless workers that lease URLs, resolve DNS via a local cache, check robots rules, fetch with strict timeouts and size limits, and hand the response to the processing pipeline. Identify themselves with a clear user agent and contact URL.
• DNS cache (caching resolver): A local resolver per cluster that caches lookups for hours and answers most queries without leaving the network. Without it, DNS is the first bottleneck and public resolvers will throttle the crawler.
• Robots cache (per-host rules · 24 h TTL): Parsed robots.txt per host, refreshed daily. A host whose robots.txt fails to fetch is treated as disallowed for a while. Also stores crawl-delay and sitemap hints.
• Web servers: The hosts being crawled. They are slow, wrong, hostile, or down in every possible way, and the crawler must remain correct regardless.
• Parser / extractor (HTML parse · link extraction): Parses HTML, extracts and normalises links (resolve relative URLs, strip fragments and known tracking parameters, lowercase host), extracts canonical tags and metadata, and computes a content fingerprint for near-duplicate detection.
• URL-seen store (sharded KV of fingerprints): Tells whether a normalised URL is already known and when it was last fetched. Billions of 8-byte fingerprints in a sharded key-value store, fronted by a Bloom filter for fast negatives. This is what keeps the frontier from growing without bound.
• Content dedupe (SimHash index): Near-duplicate detection by SimHash: pages whose 64-bit fingerprints differ in at most 3 bits are treated as duplicates. Indexed in bit-permuted tables so lookup is a handful of exact-match queries, not a scan.
• Content store (object storage · WARC): Fetched pages with headers and fetch metadata, appended to compressed archive files (WARC) in object storage, with an index by URL fingerprint and fetch time. Downstream consumers read from here.
• Crawl log (Kafka · fetch events): One event per fetch outcome (status, size, change detected, timing). Feeds the freshness model, monitoring, and downstream indexers that want to know what changed.
• Freshness scheduler (change-rate model): Batch and streaming job that estimates each page's change rate from its fetch history and importance, and re-enqueues pages into the frontier at the right time with the right priority.
• Coordinator (ZooKeeper / etcd): Assigns host ranges to frontier partitions and fetcher groups, tracks membership, and rebalances when nodes join or fail. Small, strongly consistent, rarely on the hot path.

User flows

1. Fetch one page. The basic loop: lease a URL from a host that is allowed right now, check robots, fetch with limits, parse, discover links, store. Everything else in the design exists to keep this loop fast and polite.
   1. Fetcher asks the scheduler for work; the scheduler picks a host whose cooldown has expired and pops its next URL from the frontier. The scheduler holds a min-heap of hosts by next-allowed-time. It pops the earliest host, takes the head of that host's FIFO queue, marks the host in-flight, and leases the URL to the fetcher for 60 s. If no host is ready the fetcher waits; that is the politeness constraint biting, and the frontier must hold enough distinct hosts to avoid it.
   2. Fetcher resolves the host through the local DNS cache. Cached for the TTL or at least an hour; hosts recur constantly so the hit rate is above 95 %. The resolved IP is used for the connection; a host with many IPs is still one host for politeness.
   3. Fetcher checks the URL against the host's cached robots rules. Rules are cached per host for 24 h. On a miss the robots.txt fetch is itself a polite request against the host and counts toward its budget. Disallowed URLs are dropped and recorded as such so they are not re-discovered endlessly. Crawl-delay overrides the default per-host interval.
   4. Fetcher issues the request with conditional headers, a timeout, and a size cap. If-None-Match and If-Modified-Since from the last fetch let unchanged pages answer 304 and cost almost nothing. Hard limits: 10 s connect, 30 s total, 5 MB body, 5 redirects. Only fetch content types we process (HTML, some text); everything else is recorded as a hit and skipped.
   5. Fetcher releases the host with its next-allowed-time and hands the response to the parser. Next-allowed-time is now + max(crawl-delay, default 1 s, k × last response time): slow hosts get slower crawling automatically. Errors back off exponentially per host and a host with repeated failures is parked for hours.
   6. Parser extracts links, normalises them, and checks each against the URL-seen store. Normalisation: lowercase scheme and host, remove default ports and fragments, sort or strip known tracking parameters, resolve dot segments, honour rel=canonical. The fingerprint (64-bit hash of the normalised URL) is looked up in a Bloom filter first; a negative is definitely new. Positives go to the KV store, which returns last-fetch metadata.
   7. Parser writes the page to the content store, computes the SimHash, and enqueues new URLs into the frontier with an initial priority. Near-duplicates are stored (they were fetched) but flagged so downstream skips them and their outlinks get lower priority. New URLs enter the frontier partition for their host with a priority from the parent's importance and the link's position; the fetch event records outcome and timing for the freshness model.
2. Keeping the crawl polite at 400 pages per second. One request per host per second means the crawler needs hundreds of hosts ready at any instant and must never let two fetchers hit the same host. The scheduler and the frontier partitioning make that a local decision.
   1. Coordinator assigns each host to exactly one frontier partition by hashing the host name. All URLs for a host live in one partition, so that partition's scheduler is the single authority on when the host may be fetched next. No cross-node coordination is needed per request. Rebalancing moves whole hosts and pauses them briefly.
   2. Scheduler keeps a heap of hosts by next-allowed-time and hands out at most one in-flight URL per host. The heap is small (hosts with pending work, not all hosts). A host is removed while in flight and re-inserted on release with its new time. Fetchers request batches spanning many hosts so that a fetcher with 500 open connections is talking to 500 different hosts.
   3. A large site (Wikipedia, 100 M pages) is crawled at 1 page per second like everyone else, unless it says otherwise. At 1/s the site would take three years. Options that keep politeness: a higher rate for hosts that publish crawl-delay: 0 or that are verified to be large, an allow-list of hosts with negotiated rates, and prioritising within the host so its most valuable pages come first. Do not solve this by ignoring the limit.
   4. Many hosts share one IP (shared hosting); the crawler also limits per IP. A thousand blogs on one server would each get 1/s, which is 1 000/s against one machine. Track next-allowed-time per resolved IP as a second key in the scheduler, with a looser limit (say 10/s), and treat a 429 or 503 from any host on that IP as a signal to back off all of them.
   5. Host stats flow to the crawl log so operators can see which hosts are throttling, failing, or dominating. A dashboard of fetches per host, error rate per host, and the fraction of time fetchers are idle waiting for a ready host: the last one is the metric that says the frontier lacks host diversity.
3. Discovering a URL that is already known, or content that is. Most extracted links point to URLs the crawler already has, and a good fraction of pages are copies. Both checks must be cheap, because they run on every link and every page.
   1. Parser normalises the extracted URL to its canonical form. Without normalisation, http://Example.com/a/../b?utm_source=x#top and https://example.com/b are two URLs. Canonicalisation rules are the single biggest lever on duplicate rate and are a versioned, tested component.
   2. URL fingerprint is checked in the Bloom filter and then in the sharded URL-seen store. The Bloom filter (a few GB for 15 B keys at 1 % false positives) answers "definitely new" in memory. Positives are confirmed in the KV store, which also returns the last fetch time so the parser can decide whether to re-enqueue (only if due for re-crawl) or drop.
   3. New URLs are inserted into URL-seen and the frontier atomically enough: URL-seen first, then frontier. Order matters. Writing URL-seen first means a crash between the two loses the URL from the frontier but it is "seen", so it is never fetched; a periodic reconciliation re-enqueues seen-but-never-fetched URLs. The reverse order risks the same URL being enqueued twice, which is worse for politeness budgets.
   4. Fetched content is fingerprinted with SimHash and looked up in the near-duplicate index. SimHash over shingles of the visible text produces a 64-bit value where similar documents differ in few bits. The index stores permutations of the fingerprint so that "any fingerprint within Hamming distance 3" is answered by a few exact prefix lookups. Exact duplicates (same SHA-256) are the trivial case.
   5. A duplicate is recorded with a pointer to its original; its outlinks are deprioritised and its re-crawl interval lengthened. Mirrors and syndicated copies are still worth knowing about (the index may want to show the canonical one), but they must not consume fetch budget. The canonical is whichever was seen first or is declared by rel=canonical.
4. A host that never ends: crawler traps and abuse. Some sites generate infinite URLs: calendars, faceted search, session ids, symlink loops. Others are simply hostile. The crawler bounds work per host and detects patterns rather than trusting the web.
   1. A fetched page yields thousands of links to the same host with slightly varying query strings. Per-host frontier queues have a cap (say 100 k pending URLs); beyond it new URLs are sampled or dropped, keeping one host from consuming the frontier. Priority within a host favours shallow paths and few parameters.
   2. The scheduler notices the host's fetched count and duplicate ratio climbing without new content. Trap heuristics: URL depth over 15, more than 5 query parameters, repeated path segments (/a/b/a/b), a high share of near-duplicate content from the host, and a rising fraction of 404s. Any of these lowers the host's crawl budget for the cycle.
   3. Fetcher hits a page that streams forever or returns a 500 MB "HTML" file. The 30-second total timeout and 5 MB cap abort the transfer; the outcome is recorded as oversize and the URL is not retried for a long time. Response bodies are read with a bounded buffer, never into memory whole.
   4. Redirect chains loop or fan out across hosts. Follow at most 5 redirects, each treated as a new URL that goes through normalisation and URL-seen, so a loop is detected on the second visit. Cross-host redirects are re-scheduled under the target host's politeness budget rather than followed immediately.
   5. A host returns 429 or 503 with Retry-After, or its robots.txt changes to disallow. Honour Retry-After exactly; otherwise back off exponentially per host up to a day. Robots changes take effect on the next check; pending URLs for the host are filtered when leased, not purged, so a temporary robots error does not discard work.
5. Re-crawling for freshness. The web changes at wildly different rates. A news front page changes every few minutes; a 2009 blog post never will. The freshness model spends the fetch budget where change is likely and important.
   1. Fetch events with change detection flow into the crawl log. Each event says whether the content changed since the last fetch (by content hash, ignoring boilerplate) and by how much. 304 responses are "unchanged" for free.
   2. Freshness job updates each page's estimated change rate. A simple model works: treat changes as a Poisson process and estimate the rate from the history of changed/unchanged observations. A page observed changed 3 times in 4 weekly fetches has a rate near 0.75 per week; one unchanged in 20 fetches is nearly static. Newer pages start with the average for their host.
   3. Freshness job computes the next fetch time from change rate, importance, and budget, and re-enqueues the URL. Next fetch ≈ now + c / (rate × importance), clamped between one hour and three months. Importance comes from an offline link-graph score plus signals such as being linked from home pages. The result is that the same 400 fetches per second cover news every hour and the tail every quarter.
   4. Sitemaps and external hints short-circuit the model. A sitemap with lastmod, a publisher ping, or an RSS feed tells the crawler exactly what changed. Those URLs are enqueued at high priority; the model is for everything without such signals.
   5. Budget is enforced per host and globally: if the frontier's due set exceeds capacity, lower-value re-crawls slip. Due-but-not-fetched is the backlog metric. If it grows, either add fetchers (if hosts are available) or accept that the tail refreshes more slowly. Never let re-crawls starve discovery of new URLs; reserve a fraction of capacity for each.

Deep dives

Related

• Design a Matchmaking System
• Design an LLM Inference Service
• Design Dropbox
• Design a Distributed Job Scheduler
• Run this question for a friend: the interviewer kit