Defeating Non-Executable Memory
Module 10 had you writing shellcode — raw machine code injected into a process. Modern operating systems push back hard with DEP/NX (Data Execution Prevention / No-eXecute bit). With DEP active, stack and heap pages are non-executable at the hardware layer. The CPU flat-out refuses to fetch instructions from those regions, and your injected payload never executes a byte.
This lesson covers the attacker's evolutionary answer: rather than injecting new code, reuse the code already loaded in the process's executable memory. That single shift in mindset opened up a whole new family of techniques.
For the theoretical foundation, see Why ROP Exists.
The DEP/NX Wall
Recall the memory permission model:
Hover over each step to see details
The Insight: Code Reuse
The attacker can't smuggle in new executable code. But the process is already loaded with plenty of it — the program's own .text section, libc, the dynamic linker, every shared library on the side. All of it has to be mapped executable, because the program has to run.
The pivot: if you can't bring code to the instruction pointer, walk the instruction pointer to code that already exists.
ret2libc: The Precursor to ROP
The starter code-reuse attack is ret2libc (return-to-libc). Rather than returning into shellcode on the stack, you return into a libc function that already does something handy.
The most common target is system(), which takes a string argument and executes it as a shell command:
// libc function prototypeint system(const char *command);If you can land a system("/bin/sh") call, you've got a shell — without executing a single byte of injected code.
Setting Up ret2libc on x86-64
On x86-64, function arguments are passed in registers (System V ABI):
- First argument:
rdi - Return address: on the stack
So a ret2libc attack needs to:
- Load the address of the string
"/bin/sh"intordi - Transfer control to
system()
The stack layout for a basic ret2libc attack:
+----------------------------------+| address of "pop rdi; ret" gadget | <- overwritten return address+----------------------------------+| address of "/bin/sh" string | <- popped into rdi by the gadget+----------------------------------+| address of system() | <- ret from gadget jumps here+----------------------------------+In assembly terms, here is what happens:
; After the vulnerable function's RET:
; Step 1: RET pops address of "pop rdi; ret" gadget into rip; rip now points to: pop rdi ; pops next stack value ("/bin/sh" address) into rdi ret ; pops next stack value (system() address) into rip
; Step 2: rip now points to system(); rdi contains address of "/bin/sh"; system("/bin/sh") executes, spawning a shellDEP/NX was built to kill shellcode injection, and on its own terms it won. The catch: it pushed attackers into code-reuse attacks, which turned out to be more capable and harder to defend against. Same pattern keeps repeating in security — every defensive win spawns a new offensive technique, which spawns the next defence. DEP bred ret2libc and then full ROP; ASLR (developed in parallel, not as a reply to ROP) made gadget and function addresses unpredictable, which drove info leaks; CFI arrived to constrain where a ret or indirect call may land. The chronology isn't a tidy single line, but the pressure is real, and it's why modern security demands defence in depth.
A Concrete ret2libc Example
Suppose we know the following addresses, lifted from a binary built without ASLR:
system() = 0x7ffff7a52390"/bin/sh" in libc = 0x7ffff7b99e1apop rdi; ret = 0x0000000000401233The string "/bin/sh" already lives inside libc — system() uses it internally — so we just point at it.
The exploit payload:
import struct
offset = 72 # buffer + saved rbp
payload = b'A' * offset # paddingpayload += struct.pack('<Q', 0x401233) # pop rdi; ret gadgetpayload += struct.pack('<Q', 0x7ffff7b99e1a) # address of "/bin/sh" in libcpayload += struct.pack('<Q', 0x7ffff7a52390) # address of system()The Exploit Flow
Hover over each step to see details
The execution sequence:
1. Vulnerable function returns ret pops 0x401233 into rip
2. CPU executes instruction at 0x401233: pop rdi -> rdi = 0x7ffff7b99e1a (address of "/bin/sh") ret -> pops 0x7ffff7a52390 into rip
3. CPU executes system() at 0x7ffff7a52390 rdi = pointer to "/bin/sh" system("/bin/sh") spawns a shell
4. Attacker has a shell, without executing any injected codeHere's the same chain as it sits on the stack, frozen at step 2: the vulnerable function's ret has just jumped to the gadget, so rsp points at the second slot. The middle slot is the one to study. It isn't code at all. pop rdi consumes it as data, and the gadget's own ret then takes the third slot as the next place to run.
ROP Chain Layout
pop rdi ; rdi = next slot ret ; jump to the slot after that
never executed: popped into rdi as system's argument
runs with rdi -> "/bin/sh"
The pop rdi; ret Gadget
The sequence pop rdi; ret is what we call a gadget. A short snippet of existing code that ends with ret. We didn't inject it — it's already sitting somewhere in the binary or one of its loaded libraries. The pop rdi loads a value from the stack into rdi (the first argument register), and ret hands control to whatever address comes next on the stack.
That's the bridge: from stack control (which the overflow gave us) to register control (which calling a function needs). The gadget converts stack bytes into register state.
Hunting gadgets gets its own lesson next.
Why ret2libc Is Limited
ret2libc lets you call one function with arguments. Plenty powerful (especially with system("/bin/sh")) but it doesn't go far:
- What if the chain needs to call multiple functions in sequence?
- What if your target function takes arguments across several registers?
- What if
system()isn't on offer, or it's been stripped out?
Those limits drove the generalisation into Return-Oriented Programming (ROP) — chaining together many gadgets to do arbitrary computation. Lessons 2 and 3 of this module dig in.
Stack Alignment Considerations
On x86-64, the System V ABI insists the stack be 16-byte aligned at the moment a call happens. Hand system() (or any libc function) a misaligned stack and you'll catch a SIGSEGV inside something like movaps that wants aligned operands.
If your ret2libc explodes inside the library function, slip in an extra ret gadget for alignment:
payload = b'A' * offsetpayload += struct.pack('<Q', 0x40101a) # ret gadget (for alignment)payload += struct.pack('<Q', 0x401233) # pop rdi; retpayload += struct.pack('<Q', 0x7ffff7b99e1a) # "/bin/sh"payload += struct.pack('<Q', 0x7ffff7a52390) # system()The lonely ret gadget does nothing but pop and jump, and the side effect — bumping rsp by 8 — is exactly what corrects the alignment.
The Bigger Picture
ret2libc made the point clearly: DEP/NX doesn't stop exploitation, it just shifts the technique. The attacker's core primitive — owning rip via a buffer overflow — is unchanged. DEP forced a swap from "inject and execute" to "reuse and redirect," but the threat is still on the table.
That's why modern defences pile up: DEP + ASLR + canaries + CFI. Each layer pushes back on a different stretch of the kill chain. Beating all of them at the same time is exactly what makes modern exploitation so much harder than the cheery "overwrite and jump" attacks of the 90s.
What is Next
The ret2libc attack used one gadget (pop rdi; ret) to load one argument. The next lesson goes deep on gadgets: what they are, how to dig them out of a binary, and why x86's variable-length encoding hands attackers a vast catalogue of unintended gadgets that no developer ever set out to create.