Capstone: Build a Log Analyzer
In this capstone you'll build a command-line log analyser that chews through gigabyte-scale log files with streaming I/O, leaving memory mapping, parallelism and JSON output as extensions. It parses structured logs, applies filters, aggregates stats, and dumps the results.
Requirements
- Parse structured log lines (timestamp, level, source, message)
- Filter by log level, time range, and text patterns
- Aggregate statistics (counts by level, errors per source, events over time)
- Handle files larger than RAM using streaming
- Output results as text (JSON is one of the extensions)
Log Entry
use std::time::SystemTime;use chrono::{DateTime, Utc, NaiveDateTime};
#[derive(Debug, Clone)]pub struct LogEntry { pub timestamp: DateTime<Utc>, pub level: LogLevel, pub source: String, pub message: String,}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]pub enum LogLevel { Trace, Debug, Info, Warn, Error, Fatal,}
impl LogLevel { fn from_str(s: &str) -> Option<Self> { match s.to_uppercase().as_str() { "TRACE" => Some(LogLevel::Trace), "DEBUG" => Some(LogLevel::Debug), "INFO" => Some(LogLevel::Info), "WARN" | "WARNING" => Some(LogLevel::Warn), "ERROR" => Some(LogLevel::Error), "FATAL" => Some(LogLevel::Fatal), _ => None, } }}Streaming Parser
The parser handles one line at a time. The whole file never lives in memory at once:
use std::io::{BufRead, BufReader};use std::fs::File;
pub struct LogParser;
impl LogParser { pub fn parse_line(line: &str) -> Option<LogEntry> { // Format: "2024-01-15T10:30:00.123Z ERROR [server] Connection refused" let line = line.trim(); if line.is_empty() { return None; }
let (timestamp_str, rest) = line.split_once(' ')?; let (level_str, rest) = rest.split_once(' ')?; let (source, message) = if rest.starts_with('[') { let end = rest.find(']')?; (&rest[1..end], rest[end + 1..].trim()) } else { ("unknown", rest) };
let timestamp = DateTime::parse_from_rfc3339(timestamp_str).ok()? .with_timezone(&Utc); let level = LogLevel::from_str(level_str)?;
Some(LogEntry { timestamp, level, source: source.to_string(), message: message.to_string(), }) }
pub fn stream_file(path: &str) -> impl Iterator<Item = LogEntry> + '_ { let file = File::open(path).expect("Cannot open file"); let reader = BufReader::with_capacity(64 * 1024, file);
reader.lines() .filter_map(|line| line.ok()) .filter_map(|line| Self::parse_line(&line)) }}BufReader slurps the file in 64 KB gulps, and the iterator hands you one line at a time. Memory usage stays flat no matter how big the file gets. A 10 GB log uses the same RAM as a 10 KB one.
Filter Chain
pub struct Filter { pub min_level: Option<LogLevel>, pub source_pattern: Option<String>, pub message_pattern: Option<String>, pub after: Option<DateTime<Utc>>, pub before: Option<DateTime<Utc>>,}
impl Filter { pub fn new() -> Self { Filter { min_level: None, source_pattern: None, message_pattern: None, after: None, before: None, } }
pub fn matches(&self, entry: &LogEntry) -> bool { if let Some(min) = self.min_level { if (entry.level as u8) < (min as u8) { return false; } }
if let Some(ref pattern) = self.source_pattern { if !entry.source.contains(pattern.as_str()) { return false; } }
if let Some(ref pattern) = self.message_pattern { if !entry.message.contains(pattern.as_str()) { return false; } }
if let Some(after) = self.after { if entry.timestamp < after { return false; } }
if let Some(before) = self.before { if entry.timestamp > before { return false; } }
true }}Statistics Aggregator
use std::collections::HashMap;
#[derive(Debug, Default)]pub struct Statistics { pub total_entries: usize, pub by_level: HashMap<LogLevel, usize>, pub by_source: HashMap<String, usize>, pub errors_by_source: HashMap<String, usize>, pub first_timestamp: Option<DateTime<Utc>>, pub last_timestamp: Option<DateTime<Utc>>,}
impl Statistics { pub fn new() -> Self { Statistics::default() }
pub fn record(&mut self, entry: &LogEntry) { self.total_entries += 1; *self.by_level.entry(entry.level).or_insert(0) += 1; *self.by_source.entry(entry.source.clone()).or_insert(0) += 1;
if entry.level == LogLevel::Error || entry.level == LogLevel::Fatal { *self.errors_by_source.entry(entry.source.clone()).or_insert(0) += 1; }
match self.first_timestamp { None => self.first_timestamp = Some(entry.timestamp), Some(first) if entry.timestamp < first => { self.first_timestamp = Some(entry.timestamp); } _ => {} }
match self.last_timestamp { None => self.last_timestamp = Some(entry.timestamp), Some(last) if entry.timestamp > last => { self.last_timestamp = Some(entry.timestamp); } _ => {} } }
pub fn report(&self) { println!("=== Log Analysis Report ===\n"); println!("Total entries: {}", self.total_entries);
if let (Some(first), Some(last)) = (self.first_timestamp, self.last_timestamp) { println!("Time range: {} to {}", first, last); let duration = last - first; println!("Duration: {} seconds", duration.num_seconds()); }
println!("\nBy level:"); let mut levels: Vec<_> = self.by_level.iter().collect(); levels.sort_by_key(|(_, count)| std::cmp::Reverse(*count)); for (level, count) in levels { let pct = (*count as f64 / self.total_entries as f64) * 100.0; println!(" {:?}: {} ({:.1}%)", level, count, pct); }
println!("\nTop error sources:"); let mut sources: Vec<_> = self.errors_by_source.iter().collect(); sources.sort_by_key(|(_, count)| std::cmp::Reverse(*count)); for (source, count) in sources.iter().take(10) { println!(" {}: {} errors", source, count); } }}Main Application
use std::env;
fn main() { let args: Vec<String> = env::args().collect(); if args.len() < 2 { eprintln!("Usage: log-analyzer <file> [--level LEVEL] [--source PATTERN] [--grep PATTERN]"); std::process::exit(1); }
let file = &args[1]; let filter = parse_filter_args(&args[2..]); let mut stats = Statistics::new();
println!("Analyzing {}...", file);
let entries = LogParser::stream_file(file); let mut matched = 0;
for entry in entries { if filter.matches(&entry) { stats.record(&entry); matched += 1; } }
if matched == 0 { println!("No matching entries found."); } else { stats.report(); }}
fn parse_filter_args(args: &[String]) -> Filter { let mut filter = Filter::new(); let mut i = 0; while i < args.len() { match args[i].as_str() { "--level" => { if i + 1 < args.len() { // A typo must not silently switch the filter off match LogLevel::from_str(&args[i + 1]) { Some(level) => filter.min_level = Some(level), None => { eprintln!("unknown level: {}", args[i + 1]); std::process::exit(2); } } i += 1; } } "--source" => { if i + 1 < args.len() { filter.source_pattern = Some(args[i + 1].clone()); i += 1; } } "--grep" => { if i + 1 < args.len() { filter.message_pattern = Some(args[i + 1].clone()); i += 1; } } _ => {} } i += 1; } filter}Two quiet failure modes hide in that code. filter_map(|line| line.ok()) throws away any line that isn't valid UTF-8, and parse_line returning None drops a malformed line just as silently, which is a gift to anyone who wants a log entry to vanish. The runner version counts both instead. It also swaps chrono for plain string timestamps (RFC 3339 strings in the same zone and precision sort in time order) and reads from an in-memory buffer through the same BufRead::lines(), since there's no log file to open. A third trap, a typo in --level, is closed in parse_filter_args above; the runner keeps the old behaviour so you can see why that check exists.
Extensions
Once the basic version is humming along, try:
- Memory-mapped parsing: Use
memmap2and time it against the streaming reader on a large file - Parallel processing: Split the file into chunks and process with
rayon - JSON output: Add
--jsonflag using serde_json - Histogram: Show event distribution over time buckets
- Regex support: Use the
regexcrate for pattern matching - Multiple files: Process and merge logs from multiple sources
The takeaway: Rust's iterators and ownership rules together let you stream through files of any size with bounded memory.