Memory-Mapped Files
Memory mapping projects a file straight into your address space. You get back what looks like an ordinary byte slice. The memmap2 crate wraps the mmap syscall in a safe Rust API, opening the door to zero-copy file processing that's quick and pleasant to write.
Basic Memory Mapping
use memmap2::Mmap;use std::fs::File;
fn main() -> std::io::Result<()> { let file = File::open("data.txt")?; let mmap = unsafe { Mmap::map(&file)? };
// Access file contents as a byte slice println!("File size: {} bytes", mmap.len()); println!("First 100 bytes: {:?}", &mmap[..100.min(mmap.len())]);
// Process as string if let Ok(content) = std::str::from_utf8(&mmap) { let line_count = content.lines().count(); println!("Lines: {}", line_count); }
Ok(())}The unsafe on Mmap::map is honest about reality. Another process can mutate the mapped pages behind your back, and a file truncated while mapped will give you a SIGBUS. No Rust API can paper over those OS-level races. Both break the deal a &[u8] makes: Rust assumes the bytes behind a shared slice never change, so a concurrent writer (another process, or your own MmapMut on the same file) is a data race and undefined behaviour, and a truncation turns an ordinary slice read into a fatal signal. It's reasonable when you control the file and nothing else writes to or shrinks it while it's mapped. For files other processes can touch, copy the bytes out or use locking.
You can watch both sides of that bargain. memmap2 isn't available in the runner, so it calls libc's mmap directly. It maps a small log and counts the WARN lines, then changes four bytes through the ordinary file handle. The mapping and the file share the same page-cache pages, so the second count sees the edit without re-reading anything.
mmap itself barely does anything. It records a range of virtual addresses as backed by the file and returns. No page table entries are filled and nothing is read. The work happens on first touch, one page at a time:
Your slice and the kernel's page cache are the same physical pages, which is why the runner's second count saw the edit. If the file was already cached, the fault skips the disk and just fills the PTE (a minor fault). For a front-to-back scan, madvise(MADV_SEQUENTIAL), which memmap2 exposes as mmap.advise(Advice::Sequential), tells the kernel to read further ahead and lets it drop pages you've already passed.
Zero-Copy Parsing
Memory mapping really earns its keep on large files. Skip the buffer — parse straight out of the mapped bytes:
use memmap2::Mmap;use std::fs::File;
struct LogEntry<'a> { timestamp: &'a str, level: &'a str, message: &'a str,}
fn parse_log_line(line: &str) -> Option<LogEntry<'_>> { // Format: "2024-01-15T10:30:00 INFO Starting server" let mut parts = line.splitn(3, ' '); Some(LogEntry { timestamp: parts.next()?, level: parts.next()?, message: parts.next()?, })}
fn count_errors(path: &str) -> std::io::Result<usize> { let file = File::open(path)?; let mmap = unsafe { Mmap::map(&file)? }; let content = std::str::from_utf8(&mmap) .map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))?;
let count = content.lines() .filter_map(parse_log_line) .filter(|entry| entry.level == "ERROR") .count();
Ok(count)}Each LogEntry borrows straight out of the mapped pages. No copies, no per-line allocation. One caveat on laziness: from_utf8 validates the whole mapping up front, so it faults in every page before the first line is parsed. Validate line by line if you only want part of the file.
Mutable Memory Mapping
You can also flip a mapping to writable:
use memmap2::MmapMut;use std::fs::OpenOptions;
fn main() -> std::io::Result<()> { let file = OpenOptions::new() .read(true) .write(true) .create(true) .open("data.bin")?;
file.set_len(1024)?; // Ensure file is large enough
let mut mmap = unsafe { MmapMut::map_mut(&file)? };
// Write data directly mmap[0] = 0x42; mmap[1..5].copy_from_slice(&[1, 2, 3, 4]);
// Flush to disk mmap.flush()?;
Ok(())}Shared Memory IPC
Two Rust processes can swap data through a shared mapped file:
// Writer processuse memmap2::MmapMut;use std::fs::OpenOptions;
fn writer() -> std::io::Result<()> { let file = OpenOptions::new() .read(true) .write(true) .create(true) .open("/tmp/shared_data.bin")?;
file.set_len(4096)?; let mut mmap = unsafe { MmapMut::map_mut(&file)? };
let message = b"Hello from writer!"; mmap[..message.len()].copy_from_slice(message); mmap.flush()?;
Ok(())}
// Reader processfn reader() -> std::io::Result<()> { let file = std::fs::File::open("/tmp/shared_data.bin")?; let mmap = unsafe { memmap2::Mmap::map(&file)? };
let msg = std::str::from_utf8(&mmap[..18]).unwrap(); println!("Received: {}", msg);
Ok(())}Performance: mmap vs read
Which one wins depends on the access pattern, so measure on your own data (in a --release build). For a single front-to-back pass, plain read with the kernel's readahead is often just as fast or faster, since mmap pays a page fault per page and the TLB work that comes with it. mmap tends to win for random access, for files read repeatedly or by several processes sharing the page cache, and when you'd otherwise copy a huge file into a Vec:
use std::time::Instant;
fn benchmark_read(path: &str) -> std::io::Result<usize> { let start = Instant::now(); let data = std::fs::read(path)?; let count = data.iter().filter(|&&b| b == b'\n').count(); println!("read: {:?}, {} lines", start.elapsed(), count); Ok(count)}
fn benchmark_mmap(path: &str) -> std::io::Result<usize> { let start = Instant::now(); let file = std::fs::File::open(path)?; let mmap = unsafe { memmap2::Mmap::map(&file)? }; let count = mmap.iter().filter(|&&b| b == b'\n').count(); println!("mmap: {:?}, {} lines", start.elapsed(), count); Ok(count)}mmap advantages:
- No copy from kernel to user space
- Pages loaded on demand (good for sparse access)
- OS handles caching automatically
mmap disadvantages:
- Page fault overhead for first access
- Cannot handle truncated files gracefully
- Less predictable performance (depends on OS page cache)
Practical Pattern: Indexed Database
use std::collections::HashMap;
struct MappedDatabase { mmap: memmap2::Mmap, index: HashMap<String, (usize, usize)>, // key -> (offset, length)}
impl MappedDatabase { fn open(path: &str, index_path: &str) -> std::io::Result<Self> { let file = std::fs::File::open(path)?; let mmap = unsafe { memmap2::Mmap::map(&file)? }; let index = Self::load_index(index_path)?; Ok(MappedDatabase { mmap, index }) }
fn get(&self, key: &str) -> Option<&[u8]> { let (offset, length) = self.index.get(key)?; Some(&self.mmap[*offset..*offset + *length]) }
fn load_index(path: &str) -> std::io::Result<HashMap<String, (usize, usize)>> { // Load a pre-built index mapping keys to byte offsets let mut index = HashMap::new(); // ... parse index file ... Ok(index) }}Plenty of embedded databases ride on this pattern. The data file is memory-mapped for cheap random access, and a sidecar index translates keys to byte offsets.
Memory mapping is one of the sharper tools in the systems-programming kit. It collapses file I/O into a pointer dereference, which is exactly what you want when your input file is twenty gigabytes.