Skip to main content
Be Bitwise
RustConcurrency in Rust

Capstone: Build a Parallel Web Crawler

In this capstone you'll build a concurrent web crawler that discovers and fetches pages in parallel. Threads, channels, shared state, and synchronisation, all stitched into one working program.

Architecture

The crawler has three main components:

  1. URL Frontier: A thread-safe queue of URLs to visit, with deduplication
  2. Worker Pool: Multiple threads that fetch pages and extract links
  3. Results Aggregator: Collects crawl results (page titles, link counts, errors)

The URL Frontier

rust
use std::collections::{HashSet, VecDeque};
use std::sync::{Arc, Mutex, Condvar};
// Everything the workers coordinate on lives behind ONE mutex,
// so the condition variable always sees a consistent picture.
struct FrontierState {
queue: VecDeque<String>,
seen: HashSet<String>,
active_workers: usize, // workers currently processing a URL
handed_out: usize, // pages given to workers so far
}
struct UrlFrontier {
state: Mutex<FrontierState>,
condvar: Condvar,
max_pages: usize,
}
impl UrlFrontier {
fn new(max_pages: usize) -> Self {
UrlFrontier {
state: Mutex::new(FrontierState {
queue: VecDeque::new(),
seen: HashSet::new(),
active_workers: 0,
handed_out: 0,
}),
condvar: Condvar::new(),
max_pages,
}
}
fn add_url(&self, url: String) -> bool {
let mut state = self.state.lock().unwrap();
if !state.seen.insert(url.clone()) {
return false; // already queued or crawled
}
state.queue.push_back(url);
self.condvar.notify_one();
true
}
fn get_url(&self) -> Option<String> {
let mut state = self.state.lock().unwrap();
loop {
// Page budget spent: stop handing out work
if state.handed_out >= self.max_pages {
self.condvar.notify_all();
return None;
}
if let Some(url) = state.queue.pop_front() {
state.active_workers += 1;
state.handed_out += 1;
return Some(url);
}
// Queue empty and nobody can add more: all work is done
if state.active_workers == 0 {
self.condvar.notify_all();
return None;
}
// Wait for new URLs or completion. wait() releases the same
// lock that add_url and mark_done take, so no wakeup is lost.
state = self.condvar.wait(state).unwrap();
}
}
fn mark_done(&self) {
let mut state = self.state.lock().unwrap();
state.active_workers -= 1;
if state.active_workers == 0 {
self.condvar.notify_all();
}
}
}
Key Concept

The frontier coordinates workers through a condition variable. When the queue empties but other workers are still chewing through pages, waiting threads sleep until a fresh URL lands or every worker has finished. No busy-waiting, and termination falls out cleanly.

The queue, the visited set and the active-worker count all sit behind one mutex, the same one the condition variable waits on. Split them across separate mutexes and you get a lost wakeup: a worker checks active_workers, another thread finishes and calls notify_all before the first one has started waiting, and the first one sleeps forever. A condition variable only works if every change to the condition happens under the mutex it waits with. max_pages is enforced here too, by counting pages handed out rather than URLs discovered.

Crawl Result

rust
use std::time::Duration;
#[derive(Debug)]
struct CrawlResult {
url: String,
status: CrawlStatus,
links_found: usize,
fetch_time: Duration,
}
#[derive(Debug)]
enum CrawlStatus {
Success { title: Option<String>, content_length: usize },
HttpError(u16),
FetchError(String),
}

A real crawler would lean on a proper HTML parser, and on the url crate to resolve links (Url::join handles ../, query strings and the other cases this version ignores). Here's a stripped-down version that gets the idea across. Relative links resolve against the page's directory, root-relative ones against its origin, and only links on the same origin are kept:

rust
// "https://example.com/docs/a.html" -> ("https://example.com", "https://example.com/docs/")
fn origin_and_dir(page_url: &str) -> Option<(String, String)> {
let host_start = page_url.find("://")? + 3;
let path_start = page_url[host_start..]
.find('/')
.map_or(page_url.len(), |i| host_start + i);
let origin = page_url[..path_start].to_string();
let dir = match page_url[path_start..].rfind('/') {
Some(i) => page_url[..path_start + i + 1].to_string(),
None => format!("{}/", origin),
};
Some((origin, dir))
}
fn extract_links(html: &str, page_url: &str) -> Vec<String> {
let mut links = Vec::new();
let (origin, dir) = match origin_and_dir(page_url) {
Some(parts) => parts,
None => return links,
};
// Simple regex-free link extraction
let mut search_from = 0;
while let Some(href_pos) = html[search_from..].find("href=\"") {
let start = search_from + href_pos + 6;
if let Some(end) = html[start..].find('"') {
let link = &html[start..start + end];
search_from = start + end;
let full_url = if link.starts_with("http://") || link.starts_with("https://") {
link.to_string()
} else if link.starts_with('/') {
format!("{}{}", origin, link) // root-relative
} else if link.contains(':') || link.starts_with('#') {
continue; // mailto:, javascript:, fragments
} else {
format!("{}{}", dir, link) // relative to the page's directory
};
// Same origin only. Compare against "origin/" so that
// https://example.com.evil.net doesn't sneak through.
if full_url == origin || full_url.starts_with(&format!("{}/", origin)) {
links.push(full_url);
}
} else {
break;
}
}
links
}
fn extract_title(html: &str) -> Option<String> {
let start = html.find("<title>")? + 7;
let end = html[start..].find("</title>")?;
Some(html[start..start + end].trim().to_string())
}

Rate Limiter

A polite crawler caps its request rate:

rust
use std::time::{Duration, Instant};
use std::sync::Mutex;
struct RateLimiter {
last_request: Mutex<Instant>,
min_interval: Duration,
}
impl RateLimiter {
fn new(requests_per_second: f64) -> Self {
RateLimiter {
last_request: Mutex::new(Instant::now() - Duration::from_secs(1)),
min_interval: Duration::from_secs_f64(1.0 / requests_per_second),
}
}
fn wait(&self) {
let mut last = self.last_request.lock().unwrap();
let elapsed = last.elapsed();
if elapsed < self.min_interval {
std::thread::sleep(self.min_interval - elapsed);
}
*last = Instant::now();
}
}

The Crawler

rust
use std::sync::mpsc;
use std::thread;
use std::time::Instant;
struct Crawler {
frontier: Arc<UrlFrontier>,
rate_limiter: Arc<RateLimiter>,
num_workers: usize,
}
impl Crawler {
fn new(num_workers: usize, max_pages: usize, rps: f64) -> Self {
Crawler {
frontier: Arc::new(UrlFrontier::new(max_pages)),
rate_limiter: Arc::new(RateLimiter::new(rps)),
num_workers,
}
}
fn crawl(&self, seed_url: &str) -> Vec<CrawlResult> {
self.frontier.add_url(seed_url.to_string());
let (tx, rx) = mpsc::channel::<CrawlResult>();
let mut handles = Vec::new();
for _ in 0..self.num_workers {
let frontier = Arc::clone(&self.frontier);
let limiter = Arc::clone(&self.rate_limiter);
let tx = tx.clone();
handles.push(thread::spawn(move || {
// get_url returns None once the queue is drained
// or max_pages pages have been handed out
while let Some(url) = frontier.get_url() {
limiter.wait();
let start = Instant::now();
let result = fetch_and_process(&url, &frontier, start);
let _ = tx.send(result);
frontier.mark_done();
}
}));
}
drop(tx); // Close sender so receiver iterator ends
let results: Vec<CrawlResult> = rx.into_iter().collect();
for h in handles {
h.join().unwrap();
}
results
}
}
// A tiny fake website, so the crawler runs without touching the network.
// Swap this for a real HTTP client (ureq, reqwest) when you're ready.
fn simulate_fetch(url: &str) -> Result<(u16, String), String> {
thread::sleep(Duration::from_millis(50)); // pretend network latency
let body = match url {
"https://example.com/" => {
r#"<title>Home</title>
<a href="/about.html">About</a>
<a href="docs/index.html">Docs</a>
<a href="https://example.com.evil.net/">Lookalike</a>
<a href="mailto:hi@example.com">Mail</a>"#
}
"https://example.com/about.html" => r#"<title>About</title><a href="/">Home</a>"#,
"https://example.com/docs/index.html" => {
r#"<title>Docs</title><a href="intro.html">Intro</a><a href="/missing.html">Old link</a>"#
}
"https://example.com/docs/intro.html" => r#"<title>Intro</title><a href="index.html">Back</a>"#,
"https://example.com/missing.html" => return Ok((404, String::new())),
_ => return Err(format!("no route to {}", url)),
};
Ok((200, body.to_string()))
}
fn fetch_and_process(url: &str, frontier: &UrlFrontier, start: Instant) -> CrawlResult {
match simulate_fetch(url) {
Ok((200, body)) => {
let links = extract_links(&body, url);
let links_found = links.len();
for link in links {
frontier.add_url(link);
}
CrawlResult {
url: url.to_string(),
status: CrawlStatus::Success {
title: extract_title(&body),
content_length: body.len(),
},
links_found,
fetch_time: start.elapsed(),
}
}
Ok((code, _)) => CrawlResult {
url: url.to_string(),
status: CrawlStatus::HttpError(code),
links_found: 0,
fetch_time: start.elapsed(),
},
Err(e) => CrawlResult {
url: url.to_string(),
status: CrawlStatus::FetchError(e),
links_found: 0,
fetch_time: start.elapsed(),
},
}
}

Main

rust
fn main() {
let crawler = Crawler::new(
4, // 4 worker threads
100, // max 100 pages
2.0, // 2 requests per second
);
println!("Starting crawl...");
let results = crawler.crawl("https://example.com/");
println!("\nCrawl complete!");
println!("Pages crawled: {}", results.len());
let successes = results.iter()
.filter(|r| matches!(r.status, CrawlStatus::Success { .. }))
.count();
let errors = results.len() - successes;
println!("Successful: {}", successes);
println!("Errors: {}", errors);
for result in &results {
match &result.status {
CrawlStatus::Success { title, .. } => {
println!(" {} - {}", result.url,
title.as_deref().unwrap_or("(no title)"));
}
CrawlStatus::FetchError(e) => {
println!(" {} - ERROR: {}", result.url, e);
}
CrawlStatus::HttpError(code) => {
println!(" {} - HTTP {}", result.url, code);
}
}
}
}

What You Built

This crawler demonstrates:

  • Arc<Mutex<T>> for shared mutable state (the URL frontier)
  • Condition variables for efficient worker coordination
  • Channels for collecting results from workers
  • Rate limiting with shared state
  • Graceful termination when all work is done

simulate_fetch serves a five-page fake site from memory, so the whole program runs offline and finishes in a couple of seconds at two requests per second.

Before you point it at the real web, two things a responsible crawler has to do. Fetch and obey each site's robots.txt (RFC 9309). And treat every discovered URL as untrusted input: a page can link to http://127.0.0.1:8080/admin or a cloud metadata address such as 169.254.169.254, and a crawler running inside your network will happily fetch it. Resolve the host and refuse private, loopback and link-local addresses before connecting.

Things to try next: swap simulate_fetch for a real HTTP client (ureq is simplest), swap in tokio for proper async I/O, persist state so crawls can resume, or dump results to JSON.