Engineering Atlas › System Design Interviews

Design a web crawler

21 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

Requirements

Functional:

Non-functional:

Out of scope: indexing and ranking, JavaScript rendering (an extension), and authenticated content.

Back-of-envelope

Rough numbers; the goal is orders of magnitude.

API

The crawler is an internal system, so its API is control-plane RPCs plus the frontier's internal interface.

POST /v1/crawls                    { seeds: [url], scope: {domains, max_depth}, priority }
  -> 202 { crawl_id }
GET  /v1/crawls/{crawl_id}/stats   -> { fetched, queued, errors_by_class, hosts_active }
PUT  /v1/hosts/{host}/policy       { min_delay_ms, max_parallel, paused }

// Frontier service (per shard)
Enqueue(urls: [ {url, priority, discovered_from, depth} ]) -> { accepted: int }
Lease(worker_id, max: int) -> [ {url, host, lease_id, deadline} ]
Complete(lease_id, status, http_code, content_hash, outlinks_count)

Lease and Complete make delivery at-least-once: if a worker dies, its lease expires and the URL is re-offered.

Data model

High-level design

flowchart LR
  S["Seeds / recrawl scheduler"] --> F["URL frontier (per-host queues)"]
  F --> W["Fetcher workers"]
  W --> D["DNS cache"]
  W --> R["robots.txt cache"]
  W --> WEB["Web servers"]
  W --> C["Content store (WARC)"]
  W --> P["Parser / link extractor"]
  P --> CD["Content dedup"]
  P --> N["URL normaliser + filters"]
  N --> US["URL-seen check"]
  US --> RT["Router by host hash"]
  RT --> F
  C --> IDX["Downstream indexer"]

Write path (discovering URLs): a parser extracts links from a fetched page, resolves them against the base URL, normalises them (lowercase scheme and host, drop fragments, remove default ports, strip known tracking parameters), and filters by scope and by a trap heuristic. Each surviving URL is hashed to its owning shard and sent there in batches. The owning shard checks the seen-set; new URLs are appended to the host's queue and the seen-set is updated.

Read path (fetching): fetcher workers on each shard ask the local frontier for work. The frontier pops the host whose next_fetch_at is earliest and already in the past, hands out one URL from that host's queue, and pushes the host back with next_fetch_at = now + delay. The worker resolves DNS from a local cache, checks robots.txt (fetching it first if stale), downloads with byte and time caps, writes the body to the content store, and passes it to the parser.

Deep dives

The polite frontier

Options: a single global priority queue (simple, but any worker may pick two URLs from the same host and hammer it); a queue per host with a global lock (correct, but a bottleneck); or the two-level design from Mercator: front queues ordered by priority, feeding back queues that each hold exactly one host, with a min-heap of back queues keyed by the earliest time each host may be contacted again. Mercator's authors ran about three times as many back queues as fetcher threads so every thread always had a host it was allowed to contact (Najork and Heydon, 2001). I choose the per-host back queues with a heap, sharded by host so each node owns its hosts outright.

The delay for a host is the larger of our default (say 1 second), the robots.txt crawl delay if we honour it, and an adaptive value of about 10 times the last response time, the rule Mercator used in production, so a slow server automatically gets fewer requests. Google's crawler documentation gives a similar ceiling from the site owner's side: Googlebot should not hit a site more than once every few seconds on average (Googlebot). On 429 or 503 responses the delay doubles, capped at several minutes, and resets slowly on success. Priority lives in the front queues: a page from a frequently changing, high-value site is refilled into its back queue before a deep page on a low-value site.

URL dedup and normalisation

Without dedup, link-heavy sites produce the same URL thousands of times and the frontier explodes. Options: an exact set of full URLs in a database (correct, but each lookup is a network round trip at tens of thousands of checks per second); a Bloom filter alone (fast and small, but false positives silently drop new URLs forever, around 1% of them); or a Bloom filter in front of an exact fingerprint store. I pick the layered approach: the filter answers most "new" checks from RAM, and only "maybe seen" results go to the local LSM store keyed by a 64-bit fingerprint. 64-bit fingerprints give a handful of collisions across 10B URLs (about n squared over 2 to the 65th), which a crawler can tolerate.

Normalisation matters as much as the set. HTTP://Example.com:80/a#top and http://example.com/a are the same resource. Session IDs in query strings are worse: they create a new URL on every visit. Per-host rules such as "parameter sid never changes content" can be learned by comparing content hashes of URLs that differ in one parameter.

Content dedup is the second line: an exact hash catches mirrors; SimHash with banded lookup catches near-duplicates that differ only in timestamps or ads. Google's crawl team showed that 64-bit SimHash fingerprints, treating pages within 3 differing bits as duplicates, work for a repository of 8 billion pages (Manku, Jain and Das Sarma, WWW 2007). A host whose pages are mostly duplicates gets its URL budget cut.

Recrawl and freshness

Options: uniform recrawl (simple, wastes budget on pages that never change), or per-URL scheduling from an estimated change rate. Treating changes as a Poisson process, each fetch tells us whether content changed since the last one; a simple estimator keeps the observed change rate per URL and sets the next interval proportional to its inverse, clamped between hours and months. Conditional GET with If-None-Match and If-Modified-Since makes the recheck of an unchanged page cost a 304 with no body (RFC 9110). The recrawl scheduler scans url_state by next_recrawl_at and enqueues into the same frontier, so politeness rules apply uniformly to new and old URLs.

Variant: a web search engine on top of the crawler

Asked to "design Google Search", the crawler becomes only the ingestion half. The problem splits into an offline pipeline (crawl, parse, deduplicate, score, index) and an online serving system with a latency budget near 300 ms at the p99. The two meet at the index, which the pipeline publishes as immutable segments and serving only reads.

The numbers change shape. Crawl volume is set by corpus size and refresh interval: 400B documents refreshed every 30 days is about 13B fetches a day, roughly 155K per second. Serving is sized by fan-out, not by user queries: 10B queries a day is about 230K QPS at peak, and if a replica has 1,000 leaf shards, each uncached query becomes 1,000 leaf requests.

The index is partitioned by document, not by term. Each shard holds a full inverted index (sorted, delta-encoded posting lists) for a slice of the web. Every query then touches every shard, but each does small, local, equal work, and a lost shard costs a sliver of recall instead of every result for a word. Jeff Dean's WSDM 2009 keynote describes Google weighing both layouts and sharding index and document servers by document ID (Dean, WSDM 2009).

Fan-out turns rare slowness into the norm. If each leaf is slow 1% of the time, a 1,000-leaf query almost always meets a slow one. The remedies are deadlines that travel with every RPC, partial results flagged for monitoring, hedged requests to a second replica after the 95th percentile latency, and a tree of mixers that bounds how many replies one host merges (Dean and Barroso, 2013).

Ranking runs as a cascade. Leaves score candidates cheaply, using early termination and precomputed per-document signals such as link-based authority, and return a few hundred or thousand hits. A heavier model reranks only the merged top few hundred, and doc servers build snippets for the final ten.

Freshness becomes a serving concern. Google's Caffeine replaced layers rebuilt in batches with continuous incremental indexing (Google, 2010). A small fresh tier carries news within minutes, the main tier merges new segments in the background like an LSM tree, and takedowns are tombstones checked at query time. The crawler's recrawl scheduler feeds both tiers by change rate and importance.

New failure modes follow: a query of death that crashes every leaf (send it to one or two canary leaves first), and a bad index push (roll out per replica and keep the previous generation for rollback).

Down the stack

Layer 1The contract

The frontier promises: every URL accepted by Enqueue is offered by Lease at least once unless it is filtered by robots.txt or budget; no host has more than max_parallel leases outstanding; and two leases for the same host are separated by at least that host's delay as measured on the shard's clock. The seen-set promises no false negatives (a seen URL is never re-enqueued) and accepts a small rate of false positives (a new URL occasionally dropped). Fetches are bounded: 30 seconds wall time and perhaps 10 MB body by default. Production crawlers cap far lower for HTML: Google's current Googlebot page (last updated February 2026) says it reads only the first 2 MB of a supported file type for Search (64 MB for a PDF), measured on the uncompressed bytes (Googlebot).

Layer 2The mechanism

Each shard keeps a hash map from host to a deque of URLs, plus a binary heap of (next_fetch_at, host). Lease pops the heap top if its time has passed, takes one URL from the deque, and reinserts the host only when the worker completes, so max_parallel = 1 falls out naturally. The on-disk frontier is a segmented append log, much like a Kafka partition: URLs are appended in batches, and a checkpoint records per-host read offsets. robots.txt is parsed into a longest-match rule list per user agent, with allow winning a tie, as RFC 9309 specifies, and cached for up to 24 hours (RFC 9309). The fetch result matters as much as the rules: a 4xx means no robots.txt, so the crawler may fetch anything; a 5xx or network error means assume complete disallow; after a long outage (the RFC suggests 30 days) the crawler may fall back to a cached copy or treat the file as absent. Parsers must read at least 500 KiB and follow up to five redirects. Google adds that it ignores crawl-delay, treats 429 as a server error rather than a 4xx, and pauses crawling for 12 hours when robots.txt returns 5xx (Google robots.txt spec). The seen-set is an LSM tree: a memtable absorbs inserts, flushes to sorted SSTables, and each SSTable carries its own Bloom filter so a negative lookup touches no disk.

Layer 3The machine

A fetch is DNS (a UDP round trip, avoided by caching), a TCP handshake (one RTT), a TLS 1.3 handshake (one more RTT), then the request and a response that arrives in roughly 10 segments for a 14 KB initial congestion window, more for bigger pages. With cross-continent RTTs of 100 to 200 ms, most of a fetch's 500 ms is waiting, which is why one process can drive thousands of sockets through epoll_wait with almost no CPU. The CPU goes to TLS handshakes, decompression and HTML parsing, perhaps a millisecond or two per page. On disk, appending WARC records is sequential writes at hundreds of MB/s; the seen-set's random reads hit NVMe at tens of microseconds per 4 KB page, so the Bloom filter in RAM (tens of nanoseconds per probe, one cache miss per hash) is what keeps the path cheap.

Why it's built this way

  • Remote servers are owned by other people and can be overloaded, so politeness must be enforced per host, which forces per-host queues.
  • Per-host state must be consistent, and cross-node locking is slow, so the frontier is sharded by host and each host has exactly one owner.
  • Most URLs a parser sees are already known, and network lookups are slow, so dedup must be answered from RAM, which makes a Bloom filter the first check.
  • Fetch latency is dominated by network round trips, not CPU, so the fetcher is an async event loop with thousands of sockets rather than a thread per request.
  • Bandwidth and storage cost money and most pages rarely change, so recrawl frequency follows measured change rates and uses conditional requests.

Failure modes

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Bloom filter only for URL-seenTiny, all in RAMFalse positives lose URLs for goodThrowaway or focused crawls
Bloom plus exact LSM storeNo silent loss, still fastMore moving parts, SSD neededLong-running web-scale crawl
Shard by host hashPoliteness is local, no locksSkewed load from giant hostsDefault for any polite crawler
Shard by URL hashEven loadPoliteness needs global coordinationCrawling one cooperative site
BFS orderingSimple, good early coverageWastes budget on low-value pagesSmall scoped crawls
Priority by estimated valueBest pages firstNeeds signals and tuningGeneral web crawl
Headless browser renderingSees JS-built content10 to 100 times the CPU per pageOnly for hosts that need it

Leadership lens

Check yourself

  1. Why is the frontier sharded by a hash of the registered domain rather than by URL hash?

    Show answer

    All of a host's queue, robots.txt cache, rate limit and DNS entry then live on one node, so politeness needs no cross-node coordination or locks. Sharding by URL hash spreads load evenly but forces global coordination to enforce politeness, at the cost of some load skew from giant hosts.

  2. Why not use a Bloom filter alone for the URL-seen set?

    Show answer

    Its false positives silently drop new URLs forever, around 1% of them at the chosen size. The design puts the Bloom filter in RAM in front of an exact 64-bit fingerprint store on SSD, so most "new" checks are answered from memory and "maybe seen" results are resolved without silent loss.

  3. At 1B pages a week and about 500 ms per fetch, how many fetches are in flight, and why does that favour an async event loop?

    Show answer

    1B per week is about 1,650 pages a second, and by Little's law 1,650 x 0.5 s is about 825 fetches in flight on average, a few thousand at peak. Most of each fetch is waiting on network round trips, so one process can drive thousands of sockets through epoll_wait with almost no CPU.

  4. An infinite calendar on one site keeps generating new URLs. What stops it from consuming the crawl?

    Show answer

    Trap heuristics limit URL length, path depth and repeated path segments, and each host has a URL budget per crawl cycle. Every fetch also has byte and time caps, and per-host queue length is capped so one site cannot consume all memory.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →