DEP / NX (Non-Executable Memory)
Exploit Mitigations · Exploitation Fundamentals track
11 min read
In this module
No More Running Data
In the classic stack overflow from Stack-Based Buffer Overflows, the attacker drops machine code (shellcode) into a stack buffer, then redirects execution to it. Data Execution Prevention (DEP), also called NX (No-eXecute), kills that pattern with one rule: writable memory can't be executable, and executable memory can't be writable.
The enforcement happens in hardware. Every page table entry on a modern CPU carries a bit marking the page non-executable. The moment the instruction pointer lands on an NX page, the CPU faults and the OS reaps the process.
The W^X Principle
The formal name for this policy is W^X (write XOR execute). Every memory page is either writable or executable, but never both:
- Code pages (.text segment) - executable but read-only. The program's instructions live here.
- Stack pages - writable but not executable. Local variables and return addresses live here.
- Heap pages - writable but not executable. Dynamically allocated data lives here.
An attacker can stuff a stack buffer with flawless shellcode and pivot the instruction pointer right at it; the CPU will still refuse to execute a single byte from a non-executable page.
DEP requires CPU support. Intel calls this the XD bit (eXecute Disable), and AMD calls it the NX bit (No eXecute). Both are part of the page table entry and enforced by the memory management unit (MMU) you learned about in Memory Fundamentals: Virtual Memory. Without hardware support, DEP cannot be enforced reliably.
How Attackers Adapt: Return-Oriented Programming
DEP killed off naive shellcode injection. Attackers shrugged and adapted. Why inject new code when the program is already full of it? Reuse what's there. The trick is Return-Oriented Programming (ROP), which Return-Oriented Programming covers in more detail.
A ROP chain stitches together short snippets of existing instructions (gadgets) that each end in a RET. The attacker rewrites the stack as a long string of return addresses, one per gadget. Each gadget runs, returns, pops the next address, and the chain marches on. None of it executes from writable memory; every instruction comes from a legitimate code page.
DEP in Practice
The simulator below shows the idea in miniature. Write instruction bytes to a stack-area address, then jump there. Our simplified simulator happily runs them. On a real system with DEP, the CPU would fault and the OS would put the process out of its misery.
For the best simulation experience, use a desktop browser.
Assembly
Registers
ALU
Memory
| Address | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | A | B | C | D | E | F |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 |
| 10 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 |
| 20 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 |
| 30 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 |
| 40 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 |
| 50 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 |
| 60 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 |
| 70 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 |
| 80 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 |
| 90 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 |
| A0 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 |
| B0 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 |
| C0 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 |
| D0 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 |
| E0 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 |
| F0 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 | 00 |
Pipeline
Cycle 0Worked Example: Reading Page Permissions
On Linux you can see W^X directly. Every process exposes its memory map at /proc/<pid>/maps, one line per mapping. Here's a trimmed map for a small PIE program called vuln, captured under gdb (which turns ASLR off, hence the tidy 0x5555... addresses; yours will differ):
555555554000-555555555000 r--p 00000000 08:01 1311 /home/user/vuln555555555000-555555556000 r-xp 00001000 08:01 1311 /home/user/vuln555555556000-555555557000 r--p 00002000 08:01 1311 /home/user/vuln555555557000-555555558000 r--p 00002000 08:01 1311 /home/user/vuln555555558000-555555559000 rw-p 00003000 08:01 1311 /home/user/vuln555555559000-55555557a000 rw-p 00000000 00:00 0 [heap]7ffff7d80000-7ffff7da8000 r--p 00000000 08:01 2042 /usr/lib/x86_64-linux-gnu/libc.so.67ffff7da8000-7ffff7f3d000 r-xp 00028000 08:01 2042 /usr/lib/x86_64-linux-gnu/libc.so.67ffffffde000-7ffffffff000 rw-p 00000000 00:00 0 [stack]The first column is the address range. The second is what matters here: four characters for read, write, execute and private (copy-on-write) or shared. Read it top to bottom:
r--pat the start ofvulnis the ELF header and read-only data. No writing, no running.r-xpis.text, the program's code. Executable, but you can't write to it.- The next two
r--plines are constants and, below them, the part of the data segment the loader made read-only after relocation. That second one is RELRO at work, which comes up again below. rw-pis.dataand.bss: global variables, writable, never executable.[heap]and[stack]are bothrw-p. Your shellcode in a stack buffer is sitting on a page with nox.
Not one line has both w and x. That's W^X, seen from user space. If you jump into the stack anyway, the CPU raises a page fault on the instruction fetch, and the kernel delivers SIGSEGV.
You can check a binary's intent without running it, too. The stack's permissions come from a program header called GNU_STACK:
$ readelf -lW ./vuln | grep GNU_STACK GNU_STACK 0x000000 0x0000000000000000 0x0000000000000000 0x000000 0x000000 RW 0x10RW means a non-executable stack. Link with -z execstack and the flags become RWE, and the [stack] line in the map turns into rwxp. Old CTF challenges do exactly this to make shellcode work again; in production it's a red flag.
Here's the same map as a widget. Click a row and try reading, writing and executing it, then press "Jump to the stack" and note which si_code comes back. Switch to -z execstack and jump again, and compare the heap jump under -z execstack and under "No NX". The mprotect buttons belong to the next section; come back to them once you've read it.
Page permissions in a process
A simulated /proc/<pid>/maps. The addresses are illustrative, from a run under gdb with ASLR off; yours will differ. Pick a row and try an access, or run one of the attacks.
The modern default. GNU_STACK is RW, the NX bit is set on every page without x, and no mapping is both writable and executable.
The lesson's trimmed listing leaves out [vdso]; it's added here.
Select a row to see its permissions and test an access.
prot 7 is PROT_READ|PROT_WRITE|PROT_EXEC. After the call, try the stack jump again.
No access tried yet.
Turning It Back On: mprotect
Page permissions aren't fixed for life. A process can change them with a system call:
int mprotect(void *addr, size_t len, int prot);addr must be page-aligned, and prot is a bit mask: PROT_READ is 1, PROT_WRITE is 2 and PROT_EXEC is 4. JIT compilers use this legitimately all the time, writing code into a buffer and then flipping it to executable.
Attackers use it too. A classic NX bypass is a short ROP chain whose only job is to call mprotect on the page holding the shellcode, then return into the shellcode. Following the System V calling convention, that means rdi = the page address (the buffer's address rounded down to a multiple of 0x1000), rsi = the length, and rdx = 7 for read, write and execute. ROP does the setup with pop gadgets and existing code; the shellcode does the rest. It's a hybrid: code reuse just long enough to switch injection back on.
Predict: the chain calls mprotect(0x7ffffffde000, 0x1000, 7) on a stack that looked like the map above. What does the stack look like in /proc/<pid>/maps afterwards?
The stack mapping splits in two. Permissions are tracked per mapping, so the kernel carves the changed page out of the old range. You'd see one rwxp line covering 7ffffffde000-7ffffffdf000, followed by the rest of the stack, 7ffffffdf000-7ffffffff000, still rw-p. Your shellcode only runs if it sits inside that first page. Get the rounding wrong, say by passing an address that isn't a multiple of 0x1000, and mprotect fails with EINVAL and changes nothing.
Reading checksec Output
Before you write a line of exploit, you want to know which defences are switched on. checksec reads the ELF file and reports them. The pwntools version (pwn checksec ./vuln) prints something like this:
[*] '/home/user/vuln' Arch: amd64-64-little RELRO: Partial RELRO Stack: No canary found NX: NX enabled PIE: No PIE (0x400000)Newer versions add lines such as SHSTK and IBT for the Intel CET features in CFI, Shadow Stacks & Safe Languages, and the standalone checksec script reports FORTIFY in its own column. Here's what each line is actually checking, and what it means for you.
NX. Read from the GNU_STACK header above. NX enabled means no shellcode on the stack, so plan on ROP, ret2libc or an mprotect chain. NX disabled means the stack is executable.
Stack. Whether the binary references __stack_chk_fail. Canary found means you need a leak or a non-sequential write before an overflow reaches the return address; Stack Canaries covers both, along with the ways this heuristic can mislead.
PIE. Whether the ELF type is ET_DYN (moves with ASLR) or ET_EXEC (fixed). No PIE (0x400000) gives you the binary's gadgets, PLT and GOT at known addresses. ASLR & PIE shows how that turns into a libc leak.
RELRO. This one deserves more than a sentence, because it decides whether the GOT is a target.
RELRO: Partial vs Full
When a program calls puts, it really calls a small stub in the PLT, which jumps through a pointer in the GOT. With lazy binding, that GOT slot starts out pointing back into the resolver, and the dynamic linker fills in the real puts address the first time it's called. Lazy binding has a cost: the GOT has to stay writable for the life of the process, because the linker writes to it on demand.
- No RELRO. Nothing is made read-only after loading. Rare today.
- Partial RELRO (
-z relro). The linker reorders sections so the GOT sits before.dataand.bss, which stops a linear overflow in a global array running forwards into it. After relocation, the loadermprotects a region (including.dynamic,.init_array,.fini_arrayand the non-PLT part of the GOT) to read-only. The PLT's own GOT slots, in.got.plt, stay writable so lazy binding still works. That leaves the classic GOT overwrite open: an arbitrary write can replaceputs@GOTwith the address ofsystem, and the program's nextputs(buf)becomessystem(buf). - Full RELRO (
-z relro -z now).-z nowasks for immediate binding, so the loader resolves every imported symbol at start-up, beforemainruns. Nothing needs writing later, so the whole GOT,.got.pltincluded, joins the read-only region. A write toputs@GOTnow faults. The cost is a slower start for programs that import a lot of symbols.
You can see the difference with readelf -d ./vuln: a fully RELRO binary has a BIND_NOW flag (or FLAGS_1 with NOW) in its dynamic section, and readelf -lW shows a GNU_RELRO program header for both partial and full. In the map from earlier, that second r--p line for vuln is the RELRO region.
FORTIFY
-D_FORTIFY_SOURCE=2, compiled with optimisation turned on, makes gcc swap calls such as memcpy, strcpy and sprintf for checked versions like __memcpy_chk whenever it can work out the destination's size at compile time. If a copy would overflow, the program aborts with *** buffer overflow detected *** instead of carrying on. Level 2 also rejects %n in format strings that live in writable memory, which takes away the format string write primitive in many cases. A newer level 3 extends the size checks to cases where the size is only known at runtime, on recent compilers and glibc. checksec spots FORTIFY by looking for those _chk symbols, so it can only tell you that some calls were fortified, never that all of them were.
Predict: checksec reports Partial RELRO, No canary found, NX enabled, No PIE. You have a stack overflow and also an arbitrary 8-byte write. Which of those four lines most changes your plan?
Partial RELRO, alongside No PIE. NX rules out shellcode and the missing canary makes the overflow straightforward, but the interesting part is the write. With no PIE, puts@GOT sits at a fixed address you can read straight out of the binary, and with only partial RELRO it's writable. Once you know where libc is, one write that replaces it with system turns the program's next puts(buf) into a shell command, with no ROP chain at all. Full RELRO would take that option off the table, and since many distributions (Fedora and Ubuntu among them) now link their packages with -z now, partial RELRO mostly turns up in older binaries and CTF challenges.
Labs
The ROP Chain lab runs with NX enabled, so the stack shellcode from earlier modules fails; you get past it by returning into code that already exists. Its follow-up, the Syscall ROP lab, has no helper function at all: you set rax, rdi, rsi and rdx with gadgets and call execve through the syscall instruction. Run checksec on each lab binary first and check it against what the lab description claims.
Defensive Thinking
DEP ships on by default in every modern OS, and it shuts down the simplest flavour of code injection cold. ROP shows the limit of that win: stopping injection isn't the same as stopping code reuse. Pair DEP with ASLR so gadget addresses become unpredictable, and layer on whatever else you've got. On its own, DEP is a speed bump.
Try the Simulations
Stack Canary Detection
Place a canary value between a buffer and return address. A simulated overflow corrupts the canary, and the check detects the corruption before the function returns.
DEP / NX Concept
Write the bytes of two instructions into a data area and jump there. Our simulator has no NX bit, so the injected code runs; a real CPU with DEP would fault and terminate the process.