Memory Mapping & Shared Memory
Advanced Memory Management · Memory & Systems track
7 min read
In this module
Memory Mapping: Beyond malloc
malloc manages a heap inside the process's address space. mmap operates one floor down — at the virtual memory level. It creates new mappings in the process's page table, handing you direct control over how virtual addresses point at physical memory or files. One of the most potent system calls in Unix-like systems.
mmap lets you map files (or anonymous memory) directly into a process's address space. Reading those bytes triggers page faults that pull data in on demand, and multiple processes can map the same region for shared memory.
mmap Internals
The mmap system call creates a new virtual memory mapping:
void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset);When you call mmap, the kernel finds a suitable range of virtual addresses (or honours the one you suggested), builds a VMA (Virtual Memory Area) describing the mapping, slots it into the process's VMA tree, and returns the virtual address. No physical memory has been allocated yet — this is demand paging.
The first access into the mapped region triggers a page fault. The kernel's page fault handler then locates the VMA for the faulting address, allocates a physical page, initialises it (zeros for anonymous mappings, file data for file mappings), updates the page table to map the virtual page to the physical page, and returns to the faulting instruction, which now succeeds.
Copy-on-Write
Copy-on-write (COW) is a workhorse optimisation woven through the kernel. When fork() creates a child, the kernel doesn't copy the parent's memory. Instead, both processes share the same physical pages, with both page tables flipped to read-only.
When either process writes to a shared page:
- The CPU generates a page fault (write to read-only page)
- The kernel sees this is a COW page
- The kernel allocates a new physical page
- The kernel copies the old page's contents to the new page
- The kernel maps the new page as read-write in the writing process
- The kernel decrements the reference count on the old page
- If the old page's reference count reaches 1, it can be made read-write for the remaining process
For one page, the before and after look like this:
The parent's entry is still marked read-only afterwards. When the parent next writes, it faults too, but the kernel sees a reference count of 1 and simply flips the entry to read-write. No copy needed.
So fork() never copies the process's data up front, however big it is, and pages that only ever get read (code, constants) never get copied at all. (The kernel still has to duplicate the page tables, which is why a huge process forks a bit slower than a tiny one.)
You can catch COW in the act by counting minor page faults in the child. The parent fills a 32 MB buffer before forking; the child then reads it all, writes one page, and finally writes every page. Predict the three fault counts first.
Exact counts wobble by a few between runs. If the parent's buffer was backed by transparent huge pages, the child's first write may copy a whole 2 MB page and the later counts drop accordingly.
Large allocations in glibc's malloc use mmap. When you fork() and the child calls exec(), those mmap'd regions benefit from COW: the child process can discard them (during exec) without ever copying them. This is why fork+exec is efficient even for processes with large heaps. However, if the child writes to many heap pages before exec (or does not exec at all), COW causes a cascade of page faults and copies, potentially doubling memory usage temporarily.
Demand Paging
Demand paging reaches well beyond mmap. When a program starts, the OS maps its code and data sections from the executable, but pages only get pulled in from disk on first access. A function nobody calls is a function that never lives in memory.
The same goes for the stack. The kernel maps a big virtual address range for it but only commits physical pages as the stack grows. A process with a high stack limit and shallow call depth burns barely any physical memory on its stack.
Two stats the kernel tracks:
- Major page fault: page has to come off disk (file mapping, swap). Costs milliseconds.
- Minor page fault: only a page-table update is needed (anonymous page, COW copy, file page already in cache). Costs microseconds.
Shared Memory IPC
Shared memory is the fastest IPC mechanism going. Processes read and write shared pages directly, no kernel copy in the middle.
POSIX shared memory (shm_open + mmap):
// Process A: create shared memoryint fd = shm_open("/myshm", O_CREAT | O_RDWR, 0600);ftruncate(fd, 4096);void *ptr = mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);// Write data to *ptr
// Process B: open shared memoryint fd = shm_open("/myshm", O_RDWR, 0);void *ptr = mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);// Read data from *ptrBoth processes' virtual addresses point at the same physical pages. Writes by one show up to the other immediately (modulo cache coherence). You still need synchronisation (mutexes, semaphores) to keep data races at bay.
The runner below gets the same effect without a named object: an anonymous MAP_SHARED mapping created before fork() stays shared between parent and child. Both processes bump a shared counter a million times. The plain ++ is a read-modify-write, so updates from the two processes can overwrite each other.
How many updates go missing depends on how the two processes get scheduled, so the total changes from run to run. If the runner happens to give the job a single CPU core, the race fires less often and you may see 2,000,000 even without the fix.
System V shared memory (shmget/shmat) is the older API with the same gist, different naming and lifecycle semantics. Both are still seeing heavy use.
Memory-Mapped Files for Databases
Databases routinely use mmap to map data files directly into their address space. Instead of read() and write() syscalls — which copy data between kernel buffers and user buffers — the database touches file data through pointers.
Wins: the kernel's page cache handles caching automatically, no double buffering (kernel buffer plus app buffer), access patterns that only touch a slice of a huge file only pull those pages in, and the OS handles eviction under memory pressure.
Losses: the kernel owns eviction policy (which may not match the database's access pattern), dirty-page writeback timing is out of your hands, and error handling is awkward — I/O errors arrive as SIGBUS signals, not return codes.
Some databases lean on mmap heavily: LMDB is built around it, and MongoDB's original MMAPv1 storage engine was too, until its removal in MongoDB 4.2. SQLite uses it more selectively, for its WAL index and optionally for reads. Others (PostgreSQL, MySQL/InnoDB) run their own buffer pool, arguing that application-level control over caching and writeback is worth the complexity. The mmap-vs-custom-buffer-pool argument is an evergreen debate in database design.
Practical Applications
Memory mapping shows up everywhere in systems programming. Shared libraries are mmap'd so code pages get shared across processes. Log files can be appended efficiently through mmap. Ring buffers for IPC live in shared mmap regions. Virtual machine memory can be backed by huge-page mmap for performance. Any systems programmer wrestling with large data, high-performance I/O, or IPC needs to know mmap cold.