Skip to main content
Be Bitwise
x86 AssemblyROP & Advanced Techniques

Defeating Non-Executable Memory

Module 10 had you writing shellcode — raw machine code injected into a process. Modern operating systems push back hard with DEP/NX (Data Execution Prevention / No-eXecute bit). With DEP active, stack and heap pages are non-executable at the hardware layer. The CPU flat-out refuses to fetch instructions from those regions, and your injected payload never executes a byte.

This lesson covers the attacker's evolutionary answer: rather than injecting new code, reuse the code already loaded in the process's executable memory. That single shift in mindset opened up a whole new family of techniques.

For the theoretical foundation, see Why ROP Exists.

The DEP/NX Wall

Recall the memory permission model:

Overflow
Fake Stack
Gadget 1
Gadget 2
Gadget N
Goal Achieved

Hover over each step to see details

The Insight: Code Reuse

The attacker can't smuggle in new executable code. But the process is already loaded with plenty of it — the program's own .text section, libc, the dynamic linker, every shared library on the side. All of it has to be mapped executable, because the program has to run.

The pivot: if you can't bring code to the instruction pointer, walk the instruction pointer to code that already exists.

ret2libc: The Precursor to ROP

The starter code-reuse attack is ret2libc (return-to-libc). Rather than returning into shellcode on the stack, you return into a libc function that already does something handy.

The most common target is system(), which takes a string argument and executes it as a shell command:

c
// libc function prototype
int system(const char *command);

If you can land a system("/bin/sh") call, you've got a shell — without executing a single byte of injected code.

Setting Up ret2libc on x86-64

On x86-64, function arguments are passed in registers (System V ABI):

  • First argument: rdi
  • Return address: on the stack

So a ret2libc attack needs to:

  1. Load the address of the string "/bin/sh" into rdi
  2. Transfer control to system()

The stack layout for a basic ret2libc attack:

c
+----------------------------------+
| address of "pop rdi; ret" gadget | <- overwritten return address
+----------------------------------+
| address of "/bin/sh" string | <- popped into rdi by the gadget
+----------------------------------+
| address of system() | <- ret from gadget jumps here
+----------------------------------+

In assembly terms, here is what happens:

nasm
; After the vulnerable function's RET:
; Step 1: RET pops address of "pop rdi; ret" gadget into rip
; rip now points to:
pop rdi ; pops next stack value ("/bin/sh" address) into rdi
ret ; pops next stack value (system() address) into rip
; Step 2: rip now points to system()
; rdi contains address of "/bin/sh"
; system("/bin/sh") executes, spawning a shell
The Evolutionary Arms Race

DEP/NX was built to kill shellcode injection, and on its own terms it won. The catch: it pushed attackers into code-reuse attacks, which turned out to be more capable and harder to defend against. Same pattern keeps repeating in security — every defensive win spawns a new offensive technique, which spawns the next defence. DEP bred ret2libc and then full ROP; ASLR (developed in parallel, not as a reply to ROP) made gadget and function addresses unpredictable, which drove info leaks; CFI arrived to constrain where a ret or indirect call may land. The chronology isn't a tidy single line, but the pressure is real, and it's why modern security demands defence in depth.

A Concrete ret2libc Example

Suppose we know the following addresses, lifted from a binary built without ASLR:

c
system() = 0x7ffff7a52390
"/bin/sh" in libc = 0x7ffff7b99e1a
pop rdi; ret = 0x0000000000401233

The string "/bin/sh" already lives inside libc — system() uses it internally — so we just point at it.

The exploit payload:

python
import struct
offset = 72 # buffer + saved rbp
payload = b'A' * offset # padding
payload += struct.pack('<Q', 0x401233) # pop rdi; ret gadget
payload += struct.pack('<Q', 0x7ffff7b99e1a) # address of "/bin/sh" in libc
payload += struct.pack('<Q', 0x7ffff7a52390) # address of system()

The Exploit Flow

Attacker Input
Buffer
Overflow
Return Addr
RET Executes
Attacker Code

Hover over each step to see details

The execution sequence:

c
1. Vulnerable function returns
ret pops 0x401233 into rip
2. CPU executes instruction at 0x401233:
pop rdi -> rdi = 0x7ffff7b99e1a (address of "/bin/sh")
ret -> pops 0x7ffff7a52390 into rip
3. CPU executes system() at 0x7ffff7a52390
rdi = pointer to "/bin/sh"
system("/bin/sh") spawns a shell
4. Attacker has a shell, without executing any injected code

Here's the same chain as it sits on the stack, frozen at step 2: the vulnerable function's ret has just jumped to the gadget, so rsp points at the second slot. The middle slot is the one to study. It isn't code at all. pop rdi consumes it as data, and the gadget's own ret then takes the third slot as the next place to run.

ROP Chain Layout

Stack (SP →)
1.0x401233→pop rdi; ret◄ active
2.0x7FFFF7B99E1A→"/bin/sh" in libc (data)
3.0x7FFFF7A52390→system()
Code (Gadgets)
0x401233pop rdi; ret
pop rdi   ; rdi = next slot
ret       ; jump to the slot after that
0x7FFFF7B99E1A"/bin/sh" in libc (data)
never executed:
popped into rdi as system's argument
0x7FFFF7A52390system()
runs with rdi -> "/bin/sh"
Each RET pops the next address from the stack and jumps to it, chaining gadgets together.

The pop rdi; ret Gadget

The sequence pop rdi; ret is what we call a gadget. A short snippet of existing code that ends with ret. We didn't inject it — it's already sitting somewhere in the binary or one of its loaded libraries. The pop rdi loads a value from the stack into rdi (the first argument register), and ret hands control to whatever address comes next on the stack.

That's the bridge: from stack control (which the overflow gave us) to register control (which calling a function needs). The gadget converts stack bytes into register state.

Hunting gadgets gets its own lesson next.

Why ret2libc Is Limited

ret2libc lets you call one function with arguments. Plenty powerful (especially with system("/bin/sh")) but it doesn't go far:

  • What if the chain needs to call multiple functions in sequence?
  • What if your target function takes arguments across several registers?
  • What if system() isn't on offer, or it's been stripped out?

Those limits drove the generalisation into Return-Oriented Programming (ROP) — chaining together many gadgets to do arbitrary computation. Lessons 2 and 3 of this module dig in.

Stack Alignment Considerations

On x86-64, the System V ABI insists the stack be 16-byte aligned at the moment a call happens. Hand system() (or any libc function) a misaligned stack and you'll catch a SIGSEGV inside something like movaps that wants aligned operands.

If your ret2libc explodes inside the library function, slip in an extra ret gadget for alignment:

python
payload = b'A' * offset
payload += struct.pack('<Q', 0x40101a) # ret gadget (for alignment)
payload += struct.pack('<Q', 0x401233) # pop rdi; ret
payload += struct.pack('<Q', 0x7ffff7b99e1a) # "/bin/sh"
payload += struct.pack('<Q', 0x7ffff7a52390) # system()

The lonely ret gadget does nothing but pop and jump, and the side effect — bumping rsp by 8 — is exactly what corrects the alignment.

The Bigger Picture

ret2libc made the point clearly: DEP/NX doesn't stop exploitation, it just shifts the technique. The attacker's core primitive — owning rip via a buffer overflow — is unchanged. DEP forced a swap from "inject and execute" to "reuse and redirect," but the threat is still on the table.

That's why modern defences pile up: DEP + ASLR + canaries + CFI. Each layer pushes back on a different stretch of the kill chain. Beating all of them at the same time is exactly what makes modern exploitation so much harder than the cheery "overwrite and jump" attacks of the 90s.

What is Next

The ret2libc attack used one gadget (pop rdi; ret) to load one argument. The next lesson goes deep on gadgets: what they are, how to dig them out of a binary, and why x86's variable-length encoding hands attackers a vast catalogue of unintended gadgets that no developer ever set out to create.