Skip to main content
Be Bitwise

Why ROP Exists

Return-Oriented Programming · Exploitation Fundamentals track

8 min read

In this module

The End of Simple Shellcode

In Stack-Based Buffer Overflows, you saw how a buffer overflow lets an attacker inject shellcode into a stack buffer and redirect execution to it. This worked because the CPU was willing to execute instructions stored anywhere in memory - including the stack. Defenders eventually closed this door.

DEP and the NX Bit

Modern processors support DEP (Data Execution Prevention) or the NX bit (No-Execute). Pages get permission flags: readable, writable, or executable. The stack and heap are marked writable but not executable. If the CPU's program counter points to a non-executable page, the processor raises a hardware exception and the program terminates.

This one defence killed traditional shellcode injection. The attacker can still overflow a buffer and corrupt the return address — but any shellcode they've placed in that buffer can't execute.

DEP / NX

Data Execution Prevention (DEP), also called NX (No-Execute), is a hardware-enforced security feature that prevents the CPU from executing instructions in memory regions marked as data-only. This blocks the classic shellcode technique where attackers place instructions in a stack buffer and jump to them.

What NX Looks Like on x86-64 Linux

On x86-64 the switch is a single bit. Every page-table entry has an execute-disable bit (Intel calls it XD, AMD calls it NX) in bit 63. When it's set and the CPU tries to fetch an instruction from that page, it raises a page fault, and Linux turns that into a SIGSEGV. The kernel decides which pages get the bit from the binary's program headers: a PT_GNU_STACK header without the execute flag, which every modern toolchain emits by default, means a non-executable stack. Linking with -z execstack asks for the old behaviour, which is how practice binaries for shellcode get built.

You can see the result in any process's memory map. An abridged, illustrative /proc/<pid>/maps for a small program:

text
555555554000-555555555000 r--p /home/user/vuln
555555555000-555555556000 r-xp /home/user/vuln
7ffff7dc8000-7ffff7f5d000 r-xp /usr/lib/x86_64-linux-gnu/libc.so.6
7ffffffde000-7ffffffff000 rw-p [stack]

The permissions column is the whole story. The stack is rw-p: readable, writable, private, no x. The program's own code and libc's code are r-xp: executable, not writable. Writable or executable, never both. An attacker who overflows a stack buffer can write anything they like into rw-p memory and run none of it. The r-xp regions are the opposite: runnable, but read-only. ROP is the answer to "how do I get useful behaviour out of memory I can run but can't change?"

Try it on that map. Press "Jump to the stack" with NX on and watch it fault, then switch to -z execstack and jump again. "Return into libc's code" works either way, and that's the gap ROP climbs through.

Page permissions in a process

A simulated /proc/<pid>/maps. The addresses are illustrative, from a run under gdb with ASLR off; yours will differ. Pick a row and try an access, or run one of the attacks.

The modern default. GNU_STACK is RW, the NX bit is set on every page without x, and no mapping is both writable and executable.

Select a row to see its permissions and test an access.

Attacks and accidents

No access tried yet.

What Happens When You Try Shellcode Anyway

It's worth seeing the failure once, because it looks different from a broken payload. Take the Stack Smash setup, put shellcode at the start of the buffer, point the return address at the buffer, and run it under gdb on a system with NX:

text
Program received signal SIGSEGV, Segmentation fault.
0x00007fffffffe080 in ?? ()
(gdb) x/i $rip
=> 0x7fffffffe080: xor esi,esi

Look at where rip is. The return worked perfectly: execution arrived at the first byte of the shellcode, and gdb can even disassemble it. The fault fired on the instruction fetch, before a single instruction ran. A buggy shellcode crashes partway through, somewhere inside it, usually with odd register values. An NX fault always stops exactly on the first byte of the stack you jumped to. If you see that pattern, stop debugging the shellcode; the page permissions are the problem, and no amount of tweaking the bytes will change them.

Before ROP: Return-to-libc

The first answer was simpler than full ROP. In 1997 Solar Designer published a return-into-libc exploit on the Bugtraq mailing list, written against a patch that made the stack non-executable. Instead of jumping into shellcode, the overflow returned straight into system() in the C library, which was already mapped executable in every process.

On 32-bit x86 that was beautifully direct, because the old cdecl convention passes arguments on the stack. The payload after the padding was three words:

Stack slotContentsWhy
Return addressaddress of systemret jumps into system
Next wordany address, often exitWhere system will return to; it reads this as its own return address
Word after thataddress of a "/bin/sh" stringsystem reads its first argument from here

system starts running and finds a perfectly normal-looking frame: a return address on top, its argument just above. It has no way to know nobody called it.

Predict: you take that three-word payload, widen each word to 8 bytes, and use it against a 64-bit Linux binary. Does system run /bin/sh?

No. system still gets entered, but it ignores the stack. Under the System V AMD64 ABI the first integer or pointer argument is passed in rdi, so system runs whatever command string rdi happens to point at when the overflowed function returned. Usually that's a leftover pointer to something unrelated, or junk, and you get a crash or a confusing error from sh. The "/bin/sh" pointer sits on the stack where nothing reads it.

Why x86-64 Needs Gadgets

That prediction is the reason ret2libc on 64-bit Linux is already a small ROP chain. The System V AMD64 ABI passes the first six integer arguments in registers, in this order:

Argument123456
Registerrdirsirdxrcxr8r9

To call system("/bin/sh") you have to get a pointer into rdi first, and an overflow only gives you control of stack memory. You need a piece of existing code that moves a value from the stack into rdi and then keeps going. The instruction pair pop rdi; ret does precisely that, and it encodes as just two bytes, 5f c3. Find those two bytes in any executable page and you have the bridge from "I control the stack" to "I control the first argument".

Syscalls use the same registers with one swap and one addition: the syscall number goes in rax, and the fourth argument moves from rcx to r10 because the syscall instruction itself overwrites rcx. So execve("/bin/sh", NULL, NULL) needs rax = 59, rdi pointing at the string, and rsi and rdx both zero. Four registers, four gadgets, one syscall. The ROP Chain: Syscall lab asks for exactly that.

You might hope the right value is already sitting in rdi when the overflowed function returns. Occasionally it is, and exploit writers do check. But rdi is a caller-saved scratch register under System V, so by the time a function reaches its ret, rdi holds whatever the last call inside it left behind: often a pointer to your own input, sometimes junk. Relying on that is a gamble. A pop rdi; ret turns it into a certainty, which is why it's the first gadget almost every x86-64 exploit goes looking for.

The Attacker Adapts

With shellcode injection blocked, attackers went a different direction: instead of bringing new code to the program, they reuse code already there. Every program's code section — always marked executable — contains thousands of instruction sequences. Chain enough small pieces together and you can do arbitrary computation without executing a single byte from the stack. The stack still matters enormously; it just holds addresses and data now, never instructions.

This is Return-Oriented Programming (ROP). The core idea: any instruction sequence ending in RET can be a building block.

The Building Block: Gadgets

A gadget is a short sequence of instructions ending with RET. These sequences already exist in the program's legitimate code - they are just fragments of normal functions. The attacker does not create gadgets; they discover them by scanning the program's executable code.

Each gadget performs a small operation: load a value into a register, add two registers, store a value to memory. Individually they are trivial. But chained together through the RET instruction, they can perform arbitrary computation - all using code the program already trusts.

In the next lesson, you will see exactly what gadgets look like and how attackers catalogue them.

From a Trick to a Theory

Return-to-libc was a clever hack. Return-oriented programming made it general. In 2007 Hovav Shacham's paper The Geometry of Innocent Flesh on the Bone showed that the gadgets found in a standard x86 libc were enough to build a Turing-complete set of operations: loads, stores, arithmetic, conditional branches. Defenders could no longer argue that removing a few dangerous functions such as system would close the hole, because the attacker no longer needed any particular function. Any big enough pile of executable bytes would do.

x86 makes the pile unusually rich. Instructions vary from 1 to 15 bytes and can start at any byte offset, and ret is the single byte c3. Any c3 anywhere in executable memory, including one buried in the middle of a longer instruction or a constant, ends a potential gadget. The next lesson shows how to find them.

Try It

The ROP Chain lab gives you an NX-enabled binary, so the shellcode habits from the Stack Smash labs stop working. Before you build anything, check the [stack] line in /proc/<pid>/maps for yourself while the program runs, and confirm there's no x in it.