Skip to main content
Be Bitwise
x86 AssemblyStack & Calling Conventions

Stack Mechanics

The stack is a region of memory that breathes — it expands and contracts as your program runs. Function calls, local variables, control-flow bookkeeping all live there. It's also the single most important memory structure to understand for security work, because most classic exploit techniques aim straight at it.

For background on how the CPU uses the stack, see CPU Basics.

How the Stack Works

The stack is a LIFO data structure tracked by RSP, the stack pointer. On x86-64 Linux:

  1. It grows downward. Pushing data decreases RSP. Popping increases it.
  2. RSP points to the top of the stack — the most recently pushed value.
  3. The stack starts up at a high address and crawls toward lower ones.

Stack Frame View

0048Buffer← SP
0169Buffer
0221Buffer
030ABuffer
0400Buffer
0500Buffer
0600Buffer
0700Buffer
0810Saved rbp
09E4Saved rbp
0AFFSaved rbp
0BFFSaved rbp
0CFFSaved rbp
0D7FSaved rbp
0E00Saved rbp
0F00Saved rbp
1025Return addr
1110Return addr
1240Return addr
1300Return addr
1400Return addr
1500Return addr
1600Return addr
1700Return addr
Regions
Buffer8 B
Saved rbp8 B
0x00007fffffffe410
Return addr8 B
0x0000000000401025
Overflow direction
Buffer→Saved rbp→Return addr

See how the stack fits into the overall process memory layout:

Process Memory Layout

0xFFFF...0x0000...StackgrowsFree SpaceHeapgrowsBSSDataText

PUSH - Put Data on the Stack

PUSH drops RSP by 8 and stores the operand at the freshly-vacated address:

nasm
push rax ; RSP = RSP - 8, then [RSP] = RAX
push rbx ; RSP = RSP - 8, then [RSP] = RBX
push qword 42 ; RSP = RSP - 8, then [RSP] = 42
push qword [rdi] ; RSP = RSP - 8, then [RSP] = value at [rdi]

After pushing RAX then RBX:

c
RSP before push: 0x7FFF_0100
After push rax:
RSP = 0x7FFF_00F8
[0x7FFF_00F8] = value of RAX
After push rbx:
RSP = 0x7FFF_00F0
[0x7FFF_00F0] = value of RBX
[0x7FFF_00F8] = value of RAX (still there)

POP - Take Data off the Stack

POP reads the value at RSP and bumps RSP up by 8:

nasm
pop rbx ; RBX = [RSP], then RSP = RSP + 8
pop rax ; RAX = [RSP], then RSP = RSP + 8
Critical: PUSH and POP Order

LIFO means you pop in reverse order. Push RAX then RBX, pop RBX first, then RAX. Get the order wrong and you've quietly swapped your register values — a subtle bug that lurks happily until something downstream breaks.

Manipulating RSP Directly

You don't always need PUSH and POP. Stack space can be carved out and released directly:

nasm
; Allocate 32 bytes on the stack (for local variables)
sub rsp, 32 ; RSP moves down by 32 bytes
; Access the allocated space
mov qword [rsp], 0 ; first local variable
mov qword [rsp + 8], 0 ; second local variable
mov qword [rsp + 16], 0 ; third local variable
mov qword [rsp + 24], 0 ; fourth local variable
; Deallocate when done
add rsp, 32 ; RSP moves back up

Cheaper than a string of PUSH/POPs when you want a slab of space for locals. Compilers almost always allocate locals this way.

Stack Alignment

The System V AMD64 ABI (Linux's calling convention) requires that RSP be 16-byte aligned at the point of a CALL. In other words, RSP must be a multiple of 16 when CALL executes. The CALL then pushes an 8-byte return address, so on entry to any function RSP is 8 more than a multiple of 16.

_start is the exception, because nothing called it. The ABI guarantees RSP is 16-byte aligned at process entry (it points at argc), so _start can call straight away. Inside an ordinary function you have to fix up the 8 bytes yourself:

nasm
_start:
; RSP is 16-byte aligned here: the kernel set it up that way
call my_function ; fine, no adjustment needed
my_function:
; RSP is 8 mod 16 here: CALL just pushed the return address
push rbp ; one more 8-byte push: RSP is aligned again
mov rbp, rsp
call other_function ; safe: RSP is a multiple of 16
pop rbp
ret
no_frame_pointer:
; no frame pointer this time, so realign by hand
sub rsp, 24 ; 8 (entry offset) + 24 = 32, a multiple of 16
call other_function
add rsp, 24
ret

Why bother? SSE instructions — used liberally by libc and the system functions you'll call — demand 16-byte aligned stack access. Violate it and you can segfault inside an innocent-looking printf.

Security Concept: The Stack is Exploit Target #1

Three valuable things sit elbow-to-elbow on the stack: local variables (often holding user input), saved register values, and return addresses (where execution resumes when a function returns). A classic buffer overflow scribbles past the end of a local buffer and overwrites the saved return address with one the attacker chose. When the function returns, control jumps wherever they pointed it. That's stack smashing — the foundational technique that drove every modern stack defense: stack canaries, ASLR, DEP/NX, and shadow stacks.

What CALL and RET Actually Do

CALL and RET are the levers behind every function invocation:

nasm
; CALL is equivalent to:
; push rip (the address of the instruction AFTER the call)
; jmp target
call my_function
; The return address (address of next instruction) is now on the stack
; RET is equivalent to:
; pop rip (pop the return address and jump to it)
my_function:
; do work...
ret ; pop return address, jump there

That return address sitting on the stack is what makes stack smashing possible. Overwrite it, and RET happily redirects execution wherever the attacker pointed.

Here's the round trip with numbers (the addresses are illustrative). A call with a 32-bit relative target is 5 bytes long, so the return address is the call's own address plus 5. Watch rsp drop by 8 and come back, and notice that the 16-byte alignment the ABI wants at the call means rsp is 8 past a multiple of 16 on entry to the callee.

Practical Example: Saving and Restoring Registers

One of the workhorse uses of the stack is preserving register values across calls:

nasm
my_function:
; Save registers we want to preserve
push rbx
push r12
push r13
; Use rbx, r12, r13 freely
mov rbx, rdi
mov r12, rsi
mov r13, rdx
; ... do work ...
; Restore registers (reverse order!)
pop r13
pop r12
pop rbx
ret

Practical Example: Stack-Allocated Buffer

nasm
; Read user input into a stack-allocated buffer
read_input:
sub rsp, 128 ; allocate 128 bytes on the stack
; Read from stdin into the stack buffer
mov rax, 0 ; sys_read
mov rdi, 0 ; stdin
mov rsi, rsp ; buffer = top of stack
mov rdx, 128 ; max bytes to read
syscall
; rax = number of bytes read
; Process the data at [rsp]...
add rsp, 128 ; deallocate buffer
ret
Security Note: Stack Buffer Overflow

The snippet above reads up to 128 bytes into a 128-byte buffer. Correct. Now imagine the same code with mov rdx, 256 — 256 bytes into 128. Writes march toward higher addresses, so the overflow tramples everything sitting above the buffer: other locals, saved registers, the return address. Classic stack buffer overflow. Modern defenses pile on layers: stack canaries (a random value tucked between locals and the return address, checked on the way out), and ASLR (which scrambles the address space so the attacker can't predict where to point a hijacked return).

Visualizing the Stack

Trace through what happens to the stack as a function runs:

Stack Frame View

0048Buffer← SP
0169Buffer
0221Buffer
030ABuffer
0400Buffer
0500Buffer
0600Buffer
0700Buffer
0810Saved rbp
09E4Saved rbp
0AFFSaved rbp
0BFFSaved rbp
0CFFSaved rbp
0D7FSaved rbp
0E00Saved rbp
0F00Saved rbp
1025Return addr
1110Return addr
1240Return addr
1300Return addr
1400Return addr
1500Return addr
1600Return addr
1700Return addr
Regions
Buffer8 B
Saved rbp8 B
0x00007fffffffe410
Return addr8 B
0x0000000000401025
Overflow direction
Buffer→Saved rbp→Return addr

What is Next

With the raw mechanics under your belt, the next lesson tackles the System V calling convention — the rulebook for argument passing, return values, and register etiquette. Those rules are what make calling libc from assembly (and vice versa) work at all.