Design a Web Crawler

Fetch billions of pages a month politely, discover new links, skip duplicates, and keep the copy fresh without getting the crawler banned.

Difficulty: hard. Patterns: frontier, politeness, dedupe, scheduling, distributed-queues. Reported at Google, Microsoft, Amazon, OpenAI, Apple, Pinterest.

Study shows every answer; Practice hides them until you have produced your own.

Functional requirements

• Crawl from a seed set. Start from a list of URLs, fetch each page, extract links, and continue until a budget or scope limit is reached.
• Respect robots.txt and crawl politely. Honour disallow rules and crawl-delay, and never open more than a small number of concurrent connections to one host. A polite crawler is one that does not get blocked.
• Deduplicate URLs and content. Do not fetch the same URL twice in one cycle; detect near-duplicate pages (mirrors, tracking parameters, print versions) so the corpus is not inflated.
• Prioritise. Important or frequently changing pages are fetched first and more often. Priority comes from a score (link-based importance, change rate, freshness needs).
• Store the fetched content. Raw HTML with headers and fetch metadata, versioned, so downstream indexers and analysers can process it. Storage is append-only.
• Re-crawl for freshness. Pages are revisited on a schedule that adapts to how often they change.
• Handle traps and errors. Infinite calendars, session-id URLs, redirect loops, huge files, slow hosts. The crawler bounds its work per host and per page.
• Out of scope. Indexing and ranking, JavaScript rendering (mention the headless-browser lane), and the search product. We produce a corpus.

Non-functional requirements

• Throughput (1 B pages/month · ~400 pages/s). The number is modest per second; the difficulty is that it is spread over tens of millions of hosts with per-host limits.
• Politeness (≤ 1 request per host per second by default). The hardest constraint: it turns a throughput problem into a scheduling problem, because the frontier must always have a host ready that is not in cooldown.
• Freshness (news within hours · long tail within weeks). Drives the re-crawl scheduler and the priority model.
• Coverage and dedupe (< 5 % duplicate content). Duplicates waste fetch budget and storage and skew the corpus.
• Robustness (no single host can stall the crawl). Timeouts, per-host budgets and trap detection are mandatory, not optional.
• Durability (frontier survives crashes). The frontier is the crawler's state; losing it means restarting from seeds. It must be persisted.
• Scalability (horizontal on fetchers and frontier). Add fetchers to add throughput, as long as the frontier can partition hosts among them.

Back-of-envelope estimates

• Fetch rate: ~400 pages/s. 1 B pages/month ÷ 2.6 M s ≈ 385/s. Plan for 2× headroom (800/s) because retries, redirects and robots fetches add requests beyond page fetches.
• Concurrent connections: ~2 000. Average fetch takes ~2 s (DNS, connect, TLS, transfer, slow servers). 800/s × 2 s = 1 600 in flight; round to 2 000. A single fetcher machine handles thousands of idle connections, so a few dozen fetchers with headroom.
• Download bandwidth: ~1.6 Gbit/s. Average page with headers ~500 KB (HTML only; media not fetched). 400/s × 500 KB = 200 MB/s ≈ 1.6 Gbit/s. Bandwidth is not the bottleneck; politeness is.
• Storage per month: ~150 TB compressed. 1 B × 500 KB = 500 TB raw; HTML compresses ~3–4×, so ~150 TB/month, ~1.8 PB/year. Object storage with lifecycle rules; keep the latest version hot and older versions cold.
• URLs seen: ~10–20 B. Each page yields ~50 links; most point to already-known URLs. Over a year the seen-URL set reaches tens of billions. The URL-seen structure must hold ~15 B keys: as 8-byte fingerprints that is ~120 GB, feasible in a sharded key-value store or a large Bloom filter.
• Hosts in the frontier: ~50 M. The crawlable web is a few hundred million hosts; a 1 B-page crawl touches roughly 50 M of them. With a 1 req/s per-host limit, most hosts are idle at any moment: only ~400 need to be "ready" per second, so the scheduler must keep many hosts warm.
• DNS lookups: ~400/s → ~10/s with cache. One lookup per fetch without caching. Hosts repeat heavily, so a local DNS cache with a ~1 h TTL cuts external lookups by 95 %+. Run a caching resolver per fetcher cluster; public resolvers rate limit.

Components

• Seed / URL API: Where crawls start: seed lists, sitemaps, and an API for operators to inject or reprioritise URLs. Also receives external hints such as ping services and sitemap updates.
• URL frontier (per-host queues + priority): The heart of the crawler: the set of URLs to fetch, organised so that a fetcher can always get the next URL for a host that is not in cooldown. Two levels: priority queues that select which URLs are eligible, and per-host FIFO queues with a next-allowed-time that enforce politeness. Persisted, partitioned by host across frontier nodes.
• Host scheduler (min-heap of next-fetch times): Maintains, per frontier partition, a heap of hosts keyed by the earliest time they may be fetched again. Hands out (host, URL) leases to fetchers so that no host has more than one in-flight request and the per-host delay is respected.
• Fetcher fleet (async HTTP · many connections): Stateless workers that lease URLs, resolve DNS via a local cache, check robots rules, fetch with strict timeouts and size limits, and hand the response to the processing pipeline. Identify themselves with a clear user agent and contact URL.
• DNS cache (caching resolver): A local resolver per cluster that caches lookups for hours and answers most queries without leaving the network. Without it, DNS is the first bottleneck and public resolvers will throttle the crawler.
• Robots cache (per-host rules · 24 h TTL): Parsed robots.txt per host, refreshed daily. A host whose robots.txt fails to fetch is treated as disallowed for a while. Also stores crawl-delay and sitemap hints.
• Web servers: The hosts being crawled. They are slow, wrong, hostile, or down in every possible way, and the crawler must remain correct regardless.
• Parser / extractor (HTML parse · link extraction): Parses HTML, extracts and normalises links (resolve relative URLs, strip fragments and known tracking parameters, lowercase host), extracts canonical tags and metadata, and computes a content fingerprint for near-duplicate detection.
• URL-seen store (sharded KV of fingerprints): Tells whether a normalised URL is already known and when it was last fetched. Billions of 8-byte fingerprints in a sharded key-value store, fronted by a Bloom filter for fast negatives. This is what keeps the frontier from growing without bound.
• Content dedupe (SimHash index): Near-duplicate detection by SimHash: pages whose 64-bit fingerprints differ in at most 3 bits are treated as duplicates. Indexed in bit-permuted tables so lookup is a handful of exact-match queries, not a scan.
• Content store (object storage · WARC): Fetched pages with headers and fetch metadata, appended to compressed archive files (WARC) in object storage, with an index by URL fingerprint and fetch time. Downstream consumers read from here.
• Crawl log (Kafka · fetch events): One event per fetch outcome (status, size, change detected, timing). Feeds the freshness model, monitoring, and downstream indexers that want to know what changed.
• Freshness scheduler (change-rate model): Batch and streaming job that estimates each page's change rate from its fetch history and importance, and re-enqueues pages into the frontier at the right time with the right priority.
• Coordinator (ZooKeeper / etcd): Assigns host ranges to frontier partitions and fetcher groups, tracks membership, and rebalances when nodes join or fail. Small, strongly consistent, rarely on the hot path.

User flows

1. Fetch one page. The basic loop: lease a URL from a host that is allowed right now, check robots, fetch with limits, parse, discover links, store. Everything else in the design exists to keep this loop fast and polite.
2. Keeping the crawl polite at 400 pages per second. One request per host per second means the crawler needs hundreds of hosts ready at any instant and must never let two fetchers hit the same host. The scheduler and the frontier partitioning make that a local decision.
3. Discovering a URL that is already known, or content that is. Most extracted links point to URLs the crawler already has, and a good fraction of pages are copies. Both checks must be cheap, because they run on every link and every page.
4. A host that never ends: crawler traps and abuse. Some sites generate infinite URLs: calendars, faceted search, session ids, symlink loops. Others are simply hostile. The crawler bounds work per host and detects patterns rather than trusting the web.
5. Re-crawling for freshness. The web changes at wildly different rates. A news front page changes every few minutes; a 2009 blog post never will. The freshness model spends the fetch budget where change is likely and important.

Deep dives

1. URL frontier design. Why not one big priority queue of URLs? How do you combine priority with per-host politeness?
2. URL-seen at tens of billions. How do you check 20 000 extracted links per second against 15 billion known URLs cheaply?
3. Near-duplicate detection. Exact hashing catches mirrors byte for byte. How do you catch the 30 % of pages that are the same article with a different sidebar?
4. Politeness and identity. What exactly does "polite" mean in code, and what happens if you get it wrong?
5. Freshness scheduling. You can only fetch 400 pages a second. Which pages do you re-fetch, and when?