Skip to main content
Be Bitwise
RustRust Systems Programming

Memory-Mapped Files

Memory mapping projects a file straight into your address space. You get back what looks like an ordinary byte slice. The memmap2 crate wraps the mmap syscall in a safe Rust API, opening the door to zero-copy file processing that's quick and pleasant to write.

Basic Memory Mapping

rust
use memmap2::Mmap;
use std::fs::File;
fn main() -> std::io::Result<()> {
let file = File::open("data.txt")?;
let mmap = unsafe { Mmap::map(&file)? };
// Access file contents as a byte slice
println!("File size: {} bytes", mmap.len());
println!("First 100 bytes: {:?}", &mmap[..100.min(mmap.len())]);
// Process as string
if let Ok(content) = std::str::from_utf8(&mmap) {
let line_count = content.lines().count();
println!("Lines: {}", line_count);
}
Ok(())
}
Key Concept

The unsafe on Mmap::map is honest about reality. Another process can mutate the mapped pages behind your back, and a file truncated while mapped will give you a SIGBUS. No Rust API can paper over those OS-level races. Both break the deal a &[u8] makes: Rust assumes the bytes behind a shared slice never change, so a concurrent writer (another process, or your own MmapMut on the same file) is a data race and undefined behaviour, and a truncation turns an ordinary slice read into a fatal signal. It's reasonable when you control the file and nothing else writes to or shrinks it while it's mapped. For files other processes can touch, copy the bytes out or use locking.

You can watch both sides of that bargain. memmap2 isn't available in the runner, so it calls libc's mmap directly. It maps a small log and counts the WARN lines, then changes four bytes through the ordinary file handle. The mapping and the file share the same page-cache pages, so the second count sees the edit without re-reading anything.

Try itUncomment the set_len(0) line and run it: touching a mapped page that now lies past the end of the file gets the process killed with SIGBUS.
Rust (1.40.0)

mmap itself barely does anything. It records a range of virtual addresses as backed by the file and returns. No page table entries are filled and nothing is read. The work happens on first touch, one page at a time:

Your slice and the kernel's page cache are the same physical pages, which is why the runner's second count saw the edit. If the file was already cached, the fault skips the disk and just fills the PTE (a minor fault). For a front-to-back scan, madvise(MADV_SEQUENTIAL), which memmap2 exposes as mmap.advise(Advice::Sequential), tells the kernel to read further ahead and lets it drop pages you've already passed.

Zero-Copy Parsing

Memory mapping really earns its keep on large files. Skip the buffer — parse straight out of the mapped bytes:

rust
use memmap2::Mmap;
use std::fs::File;
struct LogEntry<'a> {
timestamp: &'a str,
level: &'a str,
message: &'a str,
}
fn parse_log_line(line: &str) -> Option<LogEntry<'_>> {
// Format: "2024-01-15T10:30:00 INFO Starting server"
let mut parts = line.splitn(3, ' ');
Some(LogEntry {
timestamp: parts.next()?,
level: parts.next()?,
message: parts.next()?,
})
}
fn count_errors(path: &str) -> std::io::Result<usize> {
let file = File::open(path)?;
let mmap = unsafe { Mmap::map(&file)? };
let content = std::str::from_utf8(&mmap)
.map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))?;
let count = content.lines()
.filter_map(parse_log_line)
.filter(|entry| entry.level == "ERROR")
.count();
Ok(count)
}

Each LogEntry borrows straight out of the mapped pages. No copies, no per-line allocation. One caveat on laziness: from_utf8 validates the whole mapping up front, so it faults in every page before the first line is parsed. Validate line by line if you only want part of the file.

Mutable Memory Mapping

You can also flip a mapping to writable:

rust
use memmap2::MmapMut;
use std::fs::OpenOptions;
fn main() -> std::io::Result<()> {
let file = OpenOptions::new()
.read(true)
.write(true)
.create(true)
.open("data.bin")?;
file.set_len(1024)?; // Ensure file is large enough
let mut mmap = unsafe { MmapMut::map_mut(&file)? };
// Write data directly
mmap[0] = 0x42;
mmap[1..5].copy_from_slice(&[1, 2, 3, 4]);
// Flush to disk
mmap.flush()?;
Ok(())
}

Shared Memory IPC

Two Rust processes can swap data through a shared mapped file:

rust
// Writer process
use memmap2::MmapMut;
use std::fs::OpenOptions;
fn writer() -> std::io::Result<()> {
let file = OpenOptions::new()
.read(true)
.write(true)
.create(true)
.open("/tmp/shared_data.bin")?;
file.set_len(4096)?;
let mut mmap = unsafe { MmapMut::map_mut(&file)? };
let message = b"Hello from writer!";
mmap[..message.len()].copy_from_slice(message);
mmap.flush()?;
Ok(())
}
// Reader process
fn reader() -> std::io::Result<()> {
let file = std::fs::File::open("/tmp/shared_data.bin")?;
let mmap = unsafe { memmap2::Mmap::map(&file)? };
let msg = std::str::from_utf8(&mmap[..18]).unwrap();
println!("Received: {}", msg);
Ok(())
}

Performance: mmap vs read

Which one wins depends on the access pattern, so measure on your own data (in a --release build). For a single front-to-back pass, plain read with the kernel's readahead is often just as fast or faster, since mmap pays a page fault per page and the TLB work that comes with it. mmap tends to win for random access, for files read repeatedly or by several processes sharing the page cache, and when you'd otherwise copy a huge file into a Vec:

rust
use std::time::Instant;
fn benchmark_read(path: &str) -> std::io::Result<usize> {
let start = Instant::now();
let data = std::fs::read(path)?;
let count = data.iter().filter(|&&b| b == b'\n').count();
println!("read: {:?}, {} lines", start.elapsed(), count);
Ok(count)
}
fn benchmark_mmap(path: &str) -> std::io::Result<usize> {
let start = Instant::now();
let file = std::fs::File::open(path)?;
let mmap = unsafe { memmap2::Mmap::map(&file)? };
let count = mmap.iter().filter(|&&b| b == b'\n').count();
println!("mmap: {:?}, {} lines", start.elapsed(), count);
Ok(count)
}

mmap advantages:

  • No copy from kernel to user space
  • Pages loaded on demand (good for sparse access)
  • OS handles caching automatically

mmap disadvantages:

  • Page fault overhead for first access
  • Cannot handle truncated files gracefully
  • Less predictable performance (depends on OS page cache)

Practical Pattern: Indexed Database

rust
use std::collections::HashMap;
struct MappedDatabase {
mmap: memmap2::Mmap,
index: HashMap<String, (usize, usize)>, // key -> (offset, length)
}
impl MappedDatabase {
fn open(path: &str, index_path: &str) -> std::io::Result<Self> {
let file = std::fs::File::open(path)?;
let mmap = unsafe { memmap2::Mmap::map(&file)? };
let index = Self::load_index(index_path)?;
Ok(MappedDatabase { mmap, index })
}
fn get(&self, key: &str) -> Option<&[u8]> {
let (offset, length) = self.index.get(key)?;
Some(&self.mmap[*offset..*offset + *length])
}
fn load_index(path: &str) -> std::io::Result<HashMap<String, (usize, usize)>> {
// Load a pre-built index mapping keys to byte offsets
let mut index = HashMap::new();
// ... parse index file ...
Ok(index)
}
}

Plenty of embedded databases ride on this pattern. The data file is memory-mapped for cheap random access, and a sidecar index translates keys to byte offsets.

Memory mapping is one of the sharper tools in the systems-programming kit. It collapses file I/O into a pointer dereference, which is exactly what you want when your input file is twenty gigabytes.