Skip to main content
Be Bitwise
x86 AssemblyBuffer Overflows

Anatomy of a Stack Overflow

Few bug classes have shaped computer security as much as the buffer overflow. They've fuelled decades of exploits and driven most of the defences you'll meet later in this course. The Morris Worm in 1988, Code Red, Slammer, and a long line of modern attacks all share one ancestor: local buffers and control-flow data living together on the same stack.

This lesson takes you through how that plays out at the assembly level.

For the theoretical foundation on stack frame structure, see Anatomy of a Stack Frame.

A Vulnerable C Function

Consider this deceptively simple function:

c
#include <string.h>
#include <stdio.h>
void vulnerable(char *input) {
char buffer[64];
strcpy(buffer, input); // no bounds checking
printf("You entered: %s\n", buffer);
}
int main(int argc, char **argv) {
if (argc > 1) {
vulnerable(argv[1]);
}
return 0;
}

The vulnerable function carves out a 64-byte buffer on the stack and pours user input into it with strcpy. There is no length check anywhere. Hand it more than 64 bytes and strcpy keeps writing right past the end of buffer into whatever happens to live next door.

The Disassembly

Compile with gcc -fno-stack-protector -no-pie -O0 -o vuln vuln.c and disassemble:

nasm
vulnerable:
push rbp ; save caller's frame pointer
mov rbp, rsp ; set up our frame pointer
sub rsp, 80 ; allocate stack space (64 + alignment)
mov qword [rbp-72], rdi ; save the input pointer (argument 1)
; strcpy(buffer, input)
mov rdx, qword [rbp-72] ; rdx = input pointer
lea rax, [rbp-64] ; rax = &buffer (starts at rbp-64)
mov rsi, rdx ; second arg: source string
mov rdi, rax ; first arg: destination buffer
call strcpy ; copy with NO bounds check
; printf("You entered: %s\n", buffer)
lea rax, [rbp-64] ; rax = &buffer
mov rsi, rax ; second arg: the buffer
lea rdi, [rel fmt_str] ; first arg: format string
mov eax, 0
call printf
nop
leave ; mov rsp, rbp; pop rbp
ret ; pop return address into rip
The Fundamental Insight

Local buffers and control-flow data sit in the same contiguous slab of stack. buffer lives at [rbp-64], the saved rbp at [rbp+0], the return address at [rbp+8]. Nothing in between — no fence, no guard, no hardware-enforced boundary. An unchecked write into buffer will roll right through the saved frame pointer and then the return address.

The Stack Frame Layout

While vulnerable runs, buffer sits at rbp-64, the saved rbp at rbp, and the return address at rbp+8. The visualiser draws that frame with high addresses at the top; the addresses and saved values are illustrative. Drag the slider to hand strcpy a longer and longer argument and watch the bytes land.

vulnerable's frame (no stack protector)

Every byte stays inside the buffer.

0x7fffffffe0b00000000000000000Buffer
0x7fffffffe0b80000000000000000Buffer
0x7fffffffe0c00000000000000000Buffer
0x7fffffffe0c80000000000000000Buffer
0x7fffffffe0d00000000000000000Buffer
0x7fffffffe0d80000000000000000Buffer
0x7fffffffe0e00000000000000000Buffer
0x7fffffffe0e80000000000000000Buffer
0x7fffffffe0f010E1FFFFFF7F0000Saved rbp
0x7fffffffe0f8A511400000000000Return addr

Each row is 8 bytes, lowest address on the left. Read it right to left for the little-endian value.

Regions
Buffer64 B
Saved rbp8 B
0x00007fffffffe110
Return addr8 B
0x00000000004011a5
Overflow direction
Buffer→Saved rbp→Return addr

Sixty-four characters fill buffer exactly, yet the terminating 0x00 already lands on the low byte of the saved rbp. At 72 the saved rbp is all As and the terminator reaches the return address. Stop at 78 and look at what ret would load: six 0x41 bytes followed by two zeros, 0x0000414141414141. User-space addresses on x86-64 have their top two bytes clear, so an attacker only needs six bytes of their own plus the 0x00 that strcpy adds for free.

Why strcpy Is Dangerous

The C function strcpy has this signature:

c
char *strcpy(char *dest, const char *src);

It copies bytes from src into dest until it hits a null byte (\0). It has zero awareness of the destination's size. If src carries 200 bytes before its terminator, strcpy writes all 200 starting at dest — never mind how much space was actually allocated.

In our disassembly dest is [rbp-64], the bottom of our 64-byte buffer. Writes march upward through memory toward higher addresses, plowing through:

  1. Bytes 0-63: The legitimate buffer space
  2. Bytes 64-71: The saved rbp value (frame pointer)
  3. Bytes 72-79: The return address
  4. Bytes 80+: The caller's stack frame

Seeing It Side by Side

Set the vulnerable C against its assembly output. Notice how the friendly abstraction of a "local variable" boils down to a raw offset from rbp with no bounds enforcement at all:

c
// C: a named buffer with a size
char buffer[64];
strcpy(buffer, input);
nasm
; Assembly: just a pointer offset from rbp
lea rax, [rbp-64] ; buffer is simply rbp minus 64
mov rdi, rax ; pass it to strcpy
call strcpy ; strcpy knows nothing about 64

The C compiler is fully aware that buffer is 64 bytes. The emitted assembly carries no fingerprint of that fact. lea computes an address; strcpy writes to it without limit. The size information evaporates during compilation.

Security Context: The Scale of This Problem

By raw count, web bugs such as cross-site scripting and SQL injection report more CVEs, but memory-safety bugs cause a disproportionate share of the serious ones. In MITRE's 2025 CWE Top 25, CWE-787 (Out-of-bounds Write, the parent of stack overflows) ranks 5th, with CWE-120 (Buffer Copy without Checking Size of Input) at 11th and CWE-121 (Stack-based Buffer Overflow) at 14th. The cause is structural — C and C++ don't enforce bounds at runtime, and the stack layout parks attacker-controlled data right next to control-flow metadata.

The Critical Distance

The number of bytes from the start of buffer to the return address is the most important number in any stack-based exploit:

c
Distance = size_of_buffer + size_of_saved_rbp
= 64 + 8
= 72 bytes

Write exactly 72 bytes and you've filled the buffer and clobbered the saved rbp. Write 73 to 80 and the return address starts to take damage. By byte 80 the return address is fully replaced with whatever the attacker chose.

This distance — buffer start to return address — is the offset. Finding it is step one of every stack-based exploit.

Why the Compiler Does Not Help

With -fno-stack-protector the compiler emits no guard at all. Even with optimisations off (-O0), the generated code translates the C source faithfully into instructions that trust their inputs. sub rsp, 80 reserves the space; nothing in the rest of the function ever asks whether a write stays inside it.

This isn't a compiler bug. It's the design philosophy C inherited — trust the programmer. From a security angle, the compiler is not your ally unless you specifically ask for protection (we'll get to that in Module 12).

Try It Yourself

Compile the vulnerable program in the sandbox below and dump the disassembly with objdump -d -M intel. Find the buffer offset and the slot that holds the saved return address.

Practise on your own machine

You need Linux on x86-64: a virtual machine, WSL 2 on Windows, or a container. Then install the tools with sudo apt install nasm gcc gdb binutils.

What is Next

With the stack layout under your belt — and a feel for why buffers sit so dangerously close to control data — the next lesson walks through what happens, byte by byte, when you write past the buffer boundary, and how an attacker turns that overrun into hijacked execution.