Memory Protection Mechanisms
Virtual Memory Advanced · Memory & Systems track
6 min read
In this module
Hardware-Enforced Memory Safety
Modern CPUs ship with several hardware mechanisms for guarding memory regions. They operate at the page table level — checked by the MMU on every memory access — and at the instruction level, checked during execution. Combined with software mitigations, they form a defence-in-depth strategy against memory corruption attacks.
Memory protections (NX, SMEP, SMAP, CET) are enforced by the CPU itself, mostly through the MMU and per-page bits, so a bug in your code can't simply switch them off. Each one closes off a specific category of attack (running data as code, dereferencing user pointers from kernel mode, jumping to non-call sites) at hardware speed.
NX / DEP: No-Execute Protection
The NX bit (No eXecute, called XD on Intel, DEP on Windows) is a single bit in every page table entry that marks a page as non-executable. When the CPU's instruction fetch unit tries to load an instruction from an NX-marked page, the hardware throws a fault.
Before NX existed, an attacker who could write arbitrary data to the stack or heap could plant machine code (shellcode) there and redirect execution to it. NX kills that move dead. Even when the attacker fully controls the data on a page, the CPU refuses to execute it.
NX is enabled by default on every modern operating system. Heap, stack, and data segments are marked NX. Only the text segment and library code sections get the executable bit. The fallout is that attackers have to fall back on code-reuse techniques (ROP, JOP) that chain together existing executable code rather than injecting fresh shellcode.
W^X: Write XOR Execute
W^X (Write XOR Execute, sometimes written W|X) is the policy that no page should be both writable and executable at the same time. Writable means not executable; executable means not writable. The result: attackers can't modify code pages, and they can't flip data pages into executable ones.
W^X policy: r-x code pages (readable, executable, not writable) rw- data pages (readable, writable, not executable) r-- rodata pages (readable only) NEVER: rwx (readable, writable, executable)JIT compilers (JavaScript engines, Java HotSpot) generate executable code at runtime, which sits in obvious tension with W^X. The standard workaround: allocate memory as RW, write the generated code, then flip permissions to RX with mprotect(). The memory is never simultaneously writable and executable.
Guard pages are pages with no permissions (---) placed at memory region boundaries. A guard page between the stack and the heap catches stack overflows before they corrupt heap data. A guard page after a heap allocation catches buffer overflows. Accessing a guard page generates an immediate fault, stopping corruption before it propagates. The kernel places guard pages at stack boundaries by default.
SMEP and SMAP
SMEP (Supervisor Mode Execution Prevention) stops the kernel from executing code that lives in user-space pages. Without SMEP, an attacker who lands kernel code execution through a vulnerability could redirect the kernel to run user-space shellcode. With SMEP, that move faults.
SMAP (Supervisor Mode Access Prevention) stops the kernel from reading or writing user-space memory unless explicitly permitted. Without SMAP, a kernel vulnerability could trick the kernel into trusting attacker-controlled data from user-space. With SMAP, the kernel has to explicitly drop the protection (using STAC/CLAC instructions) whenever it needs to access user memory through copy_from_user/copy_to_user.
Both are CPU features enabled via control register bits. Together they shut down a class of kernel exploitation techniques that used to rely on the kernel's unrestricted access to user-space memory.
Memory Protection Keys (PKU)
Memory Protection Keys (Intel PKU, also called MPK) provide a fast, fine-grained permission mechanism. Each page table entry carries a 4-bit protection key field (values 0-15). The PKRU register, a per-thread user-space register, holds 2 bits per key: access-disable (AD) and write-disable (WD).
Page table entry: [...| protection_key (4 bits) | ... | permissions |]PKRU register: [key15_WD|key15_AD| ... |key1_WD|key1_AD|key0_WD|key0_AD]Updating PKRU is blisteringly fast (~20 cycles, no syscall, no TLB flush). That speed unlocks:
- Sandboxing: park sensitive data (crypto keys, passwords) behind a protection key. Grant access only when truly needed, revoke the second you're done.
- Thread-local protection: different threads can hold different PKRU values, giving them different access to the same physical pages.
- Efficient mprotect alternative: mprotect demands a syscall and TLB flush (1000+ cycles); PKRU changes are plain user-space writes (~20 cycles).
One limit on the sandboxing story: protection keys govern data reads and writes only. Instruction fetches ignore PKRU, so a key can't stop code on a page from running, and any code in the process that can execute WRPKRU can grant itself access again.
Control-Flow Enforcement (CET)
Intel's Control-flow Enforcement Technology (AMD supports the shadow-stack half too) is the piece that handles the "jumping to non-call sites" from the Key Concept. It has two parts.
- Shadow stack:
CALLpushes the return address onto the normal stack and onto a second stack that ordinary stores can't write to (its pages carry a special shadow-stack type in the page tables).RETcompares the two copies and faults with a control-protection exception (#CP) if they differ, so overwriting a saved return address no longer redirects execution. - Indirect Branch Tracking (IBT): after an indirect
JMPorCALL, the next instruction must beENDBR64(ENDBR32in 32-bit code). Compilers emit it at the start of functions whose address can be taken (-fcf-protectionin GCC and Clang), and landing anywhere else faults. That shrinks the set of places a corrupted function pointer can send execution.
Operating system support arrived piecemeal: Linux enables user-space shadow stacks since kernel 6.6, and Windows calls the same feature hardware-enforced stack protection.
Pointer Authentication (PAC) on ARM
ARM's Pointer Authentication Codes (PAC) protect pointers from tampering. When a function is called, the CPU computes a cryptographic signature (the PAC) over the return address using a secret key and a context value, then tucks the PAC into the unused upper bits of the pointer. When the function returns, the CPU verifies it. If the pointer has been tampered with — say, by a stack buffer overflow — verification fails and the program crashes.
PAC protects forward-edge control flow (function pointers) and backward-edge control flow (return addresses). It doesn't prevent data corruption directly, but it makes hijacking control flow through corrupted pointers a much taller order.
Armv8.5 adds two neighbours: BTI (Branch Target Identification), the Arm counterpart of IBT, and MTE (Memory Tagging Extension), which tags memory and pointers so that most out-of-bounds and use-after-free accesses fault. Apple Silicon (M1/M2/M3) leans heavily on PAC. iOS and macOS sign return addresses, function pointers, and some data pointers, raising the bar for exploitation by a noticeable margin.
Defence in Depth
No single mechanism is enough on its own. ASLR randomizes addresses but folds in the face of information leaks. NX prevents code injection but does nothing against code reuse. W^X prevents code modification but complicates JIT. PAC protects pointers but not data. Each mechanism closes specific attack vectors. Their combination layers overlapping defences that make exploitation progressively harder. Modern exploit development has to slip past several of these protections at once, which is why the bar for exploitation has only kept climbing.