Capstone: Build a Parallel Web Crawler
In this capstone you'll build a concurrent web crawler that discovers and fetches pages in parallel. Threads, channels, shared state, and synchronisation, all stitched into one working program.
Architecture
The crawler has three main components:
- URL Frontier: A thread-safe queue of URLs to visit, with deduplication
- Worker Pool: Multiple threads that fetch pages and extract links
- Results Aggregator: Collects crawl results (page titles, link counts, errors)
The URL Frontier
use std::collections::{HashSet, VecDeque};use std::sync::{Arc, Mutex, Condvar};
// Everything the workers coordinate on lives behind ONE mutex,// so the condition variable always sees a consistent picture.struct FrontierState { queue: VecDeque<String>, seen: HashSet<String>, active_workers: usize, // workers currently processing a URL handed_out: usize, // pages given to workers so far}
struct UrlFrontier { state: Mutex<FrontierState>, condvar: Condvar, max_pages: usize,}
impl UrlFrontier { fn new(max_pages: usize) -> Self { UrlFrontier { state: Mutex::new(FrontierState { queue: VecDeque::new(), seen: HashSet::new(), active_workers: 0, handed_out: 0, }), condvar: Condvar::new(), max_pages, } }
fn add_url(&self, url: String) -> bool { let mut state = self.state.lock().unwrap(); if !state.seen.insert(url.clone()) { return false; // already queued or crawled } state.queue.push_back(url); self.condvar.notify_one(); true }
fn get_url(&self) -> Option<String> { let mut state = self.state.lock().unwrap(); loop { // Page budget spent: stop handing out work if state.handed_out >= self.max_pages { self.condvar.notify_all(); return None; }
if let Some(url) = state.queue.pop_front() { state.active_workers += 1; state.handed_out += 1; return Some(url); }
// Queue empty and nobody can add more: all work is done if state.active_workers == 0 { self.condvar.notify_all(); return None; }
// Wait for new URLs or completion. wait() releases the same // lock that add_url and mark_done take, so no wakeup is lost. state = self.condvar.wait(state).unwrap(); } }
fn mark_done(&self) { let mut state = self.state.lock().unwrap(); state.active_workers -= 1; if state.active_workers == 0 { self.condvar.notify_all(); } }}The frontier coordinates workers through a condition variable. When the queue empties but other workers are still chewing through pages, waiting threads sleep until a fresh URL lands or every worker has finished. No busy-waiting, and termination falls out cleanly.
The queue, the visited set and the active-worker count all sit behind one mutex, the same one the condition variable waits on. Split them across separate mutexes and you get a lost wakeup: a worker checks active_workers, another thread finishes and calls notify_all before the first one has started waiting, and the first one sleeps forever. A condition variable only works if every change to the condition happens under the mutex it waits with. max_pages is enforced here too, by counting pages handed out rather than URLs discovered.
Crawl Result
use std::time::Duration;
#[derive(Debug)]struct CrawlResult { url: String, status: CrawlStatus, links_found: usize, fetch_time: Duration,}
#[derive(Debug)]enum CrawlStatus { Success { title: Option<String>, content_length: usize }, HttpError(u16), FetchError(String),}Link Extraction (Simplified)
A real crawler would lean on a proper HTML parser, and on the url crate to resolve links (Url::join handles ../, query strings and the other cases this version ignores). Here's a stripped-down version that gets the idea across. Relative links resolve against the page's directory, root-relative ones against its origin, and only links on the same origin are kept:
// "https://example.com/docs/a.html" -> ("https://example.com", "https://example.com/docs/")fn origin_and_dir(page_url: &str) -> Option<(String, String)> { let host_start = page_url.find("://")? + 3; let path_start = page_url[host_start..] .find('/') .map_or(page_url.len(), |i| host_start + i); let origin = page_url[..path_start].to_string(); let dir = match page_url[path_start..].rfind('/') { Some(i) => page_url[..path_start + i + 1].to_string(), None => format!("{}/", origin), }; Some((origin, dir))}
fn extract_links(html: &str, page_url: &str) -> Vec<String> { let mut links = Vec::new(); let (origin, dir) = match origin_and_dir(page_url) { Some(parts) => parts, None => return links, };
// Simple regex-free link extraction let mut search_from = 0; while let Some(href_pos) = html[search_from..].find("href=\"") { let start = search_from + href_pos + 6; if let Some(end) = html[start..].find('"') { let link = &html[start..start + end]; search_from = start + end;
let full_url = if link.starts_with("http://") || link.starts_with("https://") { link.to_string() } else if link.starts_with('/') { format!("{}{}", origin, link) // root-relative } else if link.contains(':') || link.starts_with('#') { continue; // mailto:, javascript:, fragments } else { format!("{}{}", dir, link) // relative to the page's directory };
// Same origin only. Compare against "origin/" so that // https://example.com.evil.net doesn't sneak through. if full_url == origin || full_url.starts_with(&format!("{}/", origin)) { links.push(full_url); } } else { break; } }
links}
fn extract_title(html: &str) -> Option<String> { let start = html.find("<title>")? + 7; let end = html[start..].find("</title>")?; Some(html[start..start + end].trim().to_string())}Rate Limiter
A polite crawler caps its request rate:
use std::time::{Duration, Instant};use std::sync::Mutex;
struct RateLimiter { last_request: Mutex<Instant>, min_interval: Duration,}
impl RateLimiter { fn new(requests_per_second: f64) -> Self { RateLimiter { last_request: Mutex::new(Instant::now() - Duration::from_secs(1)), min_interval: Duration::from_secs_f64(1.0 / requests_per_second), } }
fn wait(&self) { let mut last = self.last_request.lock().unwrap(); let elapsed = last.elapsed(); if elapsed < self.min_interval { std::thread::sleep(self.min_interval - elapsed); } *last = Instant::now(); }}The Crawler
use std::sync::mpsc;use std::thread;use std::time::Instant;
struct Crawler { frontier: Arc<UrlFrontier>, rate_limiter: Arc<RateLimiter>, num_workers: usize,}
impl Crawler { fn new(num_workers: usize, max_pages: usize, rps: f64) -> Self { Crawler { frontier: Arc::new(UrlFrontier::new(max_pages)), rate_limiter: Arc::new(RateLimiter::new(rps)), num_workers, } }
fn crawl(&self, seed_url: &str) -> Vec<CrawlResult> { self.frontier.add_url(seed_url.to_string());
let (tx, rx) = mpsc::channel::<CrawlResult>(); let mut handles = Vec::new();
for _ in 0..self.num_workers { let frontier = Arc::clone(&self.frontier); let limiter = Arc::clone(&self.rate_limiter); let tx = tx.clone();
handles.push(thread::spawn(move || { // get_url returns None once the queue is drained // or max_pages pages have been handed out while let Some(url) = frontier.get_url() { limiter.wait(); let start = Instant::now(); let result = fetch_and_process(&url, &frontier, start); let _ = tx.send(result); frontier.mark_done(); } })); }
drop(tx); // Close sender so receiver iterator ends
let results: Vec<CrawlResult> = rx.into_iter().collect();
for h in handles { h.join().unwrap(); }
results }}
// A tiny fake website, so the crawler runs without touching the network.// Swap this for a real HTTP client (ureq, reqwest) when you're ready.fn simulate_fetch(url: &str) -> Result<(u16, String), String> { thread::sleep(Duration::from_millis(50)); // pretend network latency let body = match url { "https://example.com/" => { r#"<title>Home</title> <a href="/about.html">About</a> <a href="docs/index.html">Docs</a> <a href="https://example.com.evil.net/">Lookalike</a> <a href="mailto:hi@example.com">Mail</a>"# } "https://example.com/about.html" => r#"<title>About</title><a href="/">Home</a>"#, "https://example.com/docs/index.html" => { r#"<title>Docs</title><a href="intro.html">Intro</a><a href="/missing.html">Old link</a>"# } "https://example.com/docs/intro.html" => r#"<title>Intro</title><a href="index.html">Back</a>"#, "https://example.com/missing.html" => return Ok((404, String::new())), _ => return Err(format!("no route to {}", url)), }; Ok((200, body.to_string()))}
fn fetch_and_process(url: &str, frontier: &UrlFrontier, start: Instant) -> CrawlResult { match simulate_fetch(url) { Ok((200, body)) => { let links = extract_links(&body, url); let links_found = links.len();
for link in links { frontier.add_url(link); }
CrawlResult { url: url.to_string(), status: CrawlStatus::Success { title: extract_title(&body), content_length: body.len(), }, links_found, fetch_time: start.elapsed(), } } Ok((code, _)) => CrawlResult { url: url.to_string(), status: CrawlStatus::HttpError(code), links_found: 0, fetch_time: start.elapsed(), }, Err(e) => CrawlResult { url: url.to_string(), status: CrawlStatus::FetchError(e), links_found: 0, fetch_time: start.elapsed(), }, }}Main
fn main() { let crawler = Crawler::new( 4, // 4 worker threads 100, // max 100 pages 2.0, // 2 requests per second );
println!("Starting crawl..."); let results = crawler.crawl("https://example.com/");
println!("\nCrawl complete!"); println!("Pages crawled: {}", results.len());
let successes = results.iter() .filter(|r| matches!(r.status, CrawlStatus::Success { .. })) .count(); let errors = results.len() - successes;
println!("Successful: {}", successes); println!("Errors: {}", errors);
for result in &results { match &result.status { CrawlStatus::Success { title, .. } => { println!(" {} - {}", result.url, title.as_deref().unwrap_or("(no title)")); } CrawlStatus::FetchError(e) => { println!(" {} - ERROR: {}", result.url, e); } CrawlStatus::HttpError(code) => { println!(" {} - HTTP {}", result.url, code); } } }}What You Built
This crawler demonstrates:
- Arc<Mutex<T>> for shared mutable state (the URL frontier)
- Condition variables for efficient worker coordination
- Channels for collecting results from workers
- Rate limiting with shared state
- Graceful termination when all work is done
simulate_fetch serves a five-page fake site from memory, so the whole program runs offline and finishes in a couple of seconds at two requests per second.
Before you point it at the real web, two things a responsible crawler has to do. Fetch and obey each site's robots.txt (RFC 9309). And treat every discovered URL as untrusted input: a page can link to http://127.0.0.1:8080/admin or a cloud metadata address such as 169.254.169.254, and a crawler running inside your network will happily fetch it. Resolve the host and refuse private, loopback and link-local addresses before connecting.
Things to try next: swap simulate_fetch for a real HTTP client (ureq is simplest), swap in tokio for proper async I/O, persist state so crawls can resume, or dump results to JSON.