DEP/NX Bypass
Exploit Mitigation Bypass Techniques · Exploit Development track
5 min read
In this module
Executing Without Execution
DEP (Data Execution Prevention) and NX (No-Execute) mark data pages as non-executable. Injected shellcode on the stack or heap can't run — the CPU refuses to fetch instructions from non-executable pages. It was a genuinely disruptive defence when first deployed. Attackers responded by finding ways to achieve arbitrary execution without injecting any new code at all.
ROP Revisited
Return-Oriented Programming is the canonical DEP bypass. Instead of injecting new code, ROP reuses existing code snippets (gadgets) from the program's executable pages. Each gadget is a short sequence of instructions ending with ret. The attacker chains gadgets by placing their addresses on the stack, and the ret at the end of each gadget pops the next address and transfers control.
A typical ROP chain to call mprotect() (making a page executable, then jumping to shellcode on that page):
ROP Chain Layout
The overflow has written the chain. Press Step to let the function return.
pop rdi; ret
pop rsi; ret
pop rdx; ret
mprotect syscall
shellcode (now executable)
Step through it and track rdi, rsi and rdx. Swap two of the pop gadgets in your head and the values land in the wrong registers. mprotect gets nonsense arguments and returns an error, and the final ret jumps into a page that's still non-executable.
Finding gadgets is handled automatically by tools like ROPgadget, ropper, and pwntools — they scan executable sections for sequences ending in ret and catalogue everything by function.
The number and variety of available gadgets depends on the size of the executable code. Large binaries and libc contain thousands of useful gadgets - enough to perform any computation. Small binaries may have limited gadgets, requiring more creative chaining or the use of library gadgets (which requires defeating ASLR first).
JIT Spraying
JIT (Just-In-Time) compilers in JavaScript engines, .NET, and Java convert interpreted code into native machine code at runtime. This native code lives on executable pages. JIT spraying exploits the fact that the JIT compiler embeds attacker-controlled constants directly into the generated code.
For example, a JavaScript expression like var x = 0x3c909090 ^ 0x3c909090 ^ 0x3c909090; compiles to native code containing the constant 0x3c909090 as an immediate operand. When the CPU executes these bytes starting at an offset of +1 from the intended entry point, 0x909090 decodes as three NOP instructions and 0x3c decodes as CMP AL, imm8.
By choosing constants carefully, the attacker controls the byte content of executable pages. This creates a writable-via-JIT, executable memory region containing attacker-chosen instruction sequences - effectively bypassing both DEP and ASLR (because the attacker can spray many copies and brute-force the location).
Modern JIT compilers fight back with constant blinding (XORing large immediates with a random key so the attacker's bytes never appear verbatim), random NOP padding, and randomised code layout, and most now keep JIT pages from being writable and executable at the same time. Classic JIT spraying, as used against Flash and early JavaScript engines around 2010, is now mostly a historical technique; JIT engines are still attacked, but through bugs in the compiler itself.
mprotect/VirtualProtect Gadgets
The most direct DEP bypass is to change the memory permissions of a data page to make it executable. On Linux, mprotect() changes page permissions. On Windows, VirtualProtect() does the same.
A ROP chain that calls mprotect(shellcode_page, size, PROT_READ|PROT_WRITE|PROT_EXEC) makes the shellcode page executable, and then the chain jumps to the shellcode.
The defence against this is restricting access to mprotect with W^X enforcement (no page can be both writable and executable simultaneously). Some security-hardened kernels (like grsecurity/PaX) enforce this at the kernel level, preventing any process from creating RWX pages.
Sigreturn-Oriented Programming (SROP)
SROP is an elegant technique that achieves complete control over all CPU registers with just one gadget. When the Linux kernel delivers a signal to a process, it pushes a signal frame onto the stack containing all register values. When the signal handler finishes, the kernel's sigreturn syscall restores all registers from this frame.
The attacker:
- Places a fake signal frame on the stack with all registers set to desired values
- Uses a gadget that calls the
sigreturnsyscall - The kernel reads the fake frame and sets all registers accordingly
SROP is powerful because it needs only a single sigreturn gadget. That's usually found in libc rather than the program: glibc's __restore_rt is exactly mov rax, 15; syscall, so once you know where libc is you have one. Without a libc leak you need a syscall gadget plus a way to set RAX to 15 (a read that returns 15 bytes works). It eliminates the need to find multiple pop rdi, pop rsi, etc. gadgets.
Stack Pivot
Sometimes the attacker controls only a small amount of stack space (not enough for a full ROP chain). A stack pivot redirects the stack pointer to a larger attacker-controlled buffer (like the heap) where a full ROP chain has been placed:
; Stack pivot gadget: set RSP to a controlled valuexchg rsp, rax ; If attacker controls RAX, RSP now points to their bufferret ; Continue ROP chain from the new stack locationOther pivot gadgets include leave; ret (which sets RSP from RBP), add rsp, large_value; ret, and mov rsp, [rbx]; ret.
DEP forced a paradigm shift in exploitation. Simple shellcode injection is dead on modern systems. Instead, exploits use code reuse (ROP, JOP, COP), JIT manipulation, or data-only attacks that corrupt security-critical data without executing new code. Each new defence spawns new bypass techniques, and each bypass drives the development of stronger defences. Understanding this arms race is essential for both offensive and defensive security work.