Skip to main content
Be Bitwise
x86 AssemblyStack & Calling Conventions

System V Calling Convention

A calling convention is the contract between caller and callee. It pins down how arguments are passed, where the return value lives, and who's on the hook for preserving each register. Strip the contract away and separately-compiled functions stop being able to talk to each other.

On Linux x86-64, the contract is the System V AMD64 ABI. Every C compiler on Linux speaks it. Your assembly code has to speak it too if it wants to talk to C in either direction.

For a broader view of calling conventions across platforms, see x86 Calling Conventions.

Argument Passing

The first six integer (or pointer) arguments are passed in registers, in this exact order:

ArgumentRegister
1strdi
2ndrsi
3rdrdx
4thrcx
5thr8
6thr9

More than six arguments? Anything past the sixth gets pushed onto the stack, in reverse order — right-to-left.

Floating-point arguments use xmm0 through xmm7. This track sticks with integer arguments throughout.

nasm
; Calling a function with 3 arguments: func(10, 20, 30)
mov rdi, 10 ; first argument
mov rsi, 20 ; second argument
mov rdx, 30 ; third argument
call func
; Return value is now in rax
nasm
; Calling a function with 8 arguments: func(1,2,3,4,5,6,7,8)
push 8 ; 8th argument (pushed first -- right to left)
push 7 ; 7th argument
mov rdi, 1 ; 1st argument
mov rsi, 2 ; 2nd argument
mov rdx, 3 ; 3rd argument
mov rcx, 4 ; 4th argument
mov r8, 5 ; 5th argument
mov r9, 6 ; 6th argument
call func
add rsp, 16 ; clean up the 2 pushed arguments (2 * 8 bytes)

Return Values

Return values land in rax for integers and pointers, xmm0 for floats. 128-bit values use rdx:rax — high half in rdx, low half in rax.

nasm
; A function that returns a value
compute_something:
; ... do work ...
mov rax, 42 ; return 42
ret
; Calling it
call compute_something
; rax now holds 42

Caller-Saved vs Callee-Saved Registers

This is the part of the calling convention that bites people. Registers split into two camps:

Caller-saved (volatile) — the callee can stomp on these freely. If you need them preserved across a call, save them first.

RegisterPurpose
raxReturn value
rcx4th argument
rdx3rd argument
rdi1st argument
rsi2nd argument
r85th argument
r96th argument
r10Temporary
r11Temporary

Callee-saved (non-volatile) — the callee owes the caller their original values back. Want to use one? Push it on entry, pop it on exit.

RegisterPurpose
rbxGeneral purpose
rbpFrame pointer
r12General purpose
r13General purpose
r14General purpose
r15General purpose
rspStack pointer (must always be restored)
Key Concept: Why Two Groups?

The split is a performance hack. Make every register callee-saved and every function pays to push/pop sixteen registers, most of which the caller couldn't care less about. Make every register caller-saved and every call site does the saving instead. Splitting them lets each side preserve only the registers it actually cares about, which slashes the total push/pop traffic across the whole program.

The Red Zone

The System V ABI carves out a red zone: 128 bytes below RSP that signal handlers and interrupts promise not to touch. Leaf functions — those that don't call anything else — can scribble there without ever bumping RSP:

nasm
; Leaf function using the red zone
leaf_func:
; No need to sub rsp -- use space below RSP
mov qword [rsp - 8], rdi ; store in the red zone
mov qword [rsp - 16], rsi ; also in the red zone
; ... compute using those values ...
ret ; red zone is automatically reclaimed

The red zone trick spares small functions the overhead of touching RSP. But the moment a function calls another, the callee's frame overwrites the red zone. Leaf functions only.

Security Note: No Red Zone in Kernel Code

The Linux kernel turns the red zone off — built with -mno-red-zone. The reason: interrupts can fire at any moment and start writing below RSP immediately. Anything stashed in the red zone would be flattened. If you're writing kernel modules or exploit payloads that run in kernel context, the red zone is off the table. One of many ways the kernel ABI parts company with userspace.

Calling C Library Functions

Once the calling convention clicks, calling libc from assembly stops being mysterious:

nasm
; Call printf from assembly
section .data
fmt db "Result: %d", 10, 0 ; format string (null-terminated for C)
section .text
extern printf
global main ; use 'main' when linking with gcc
main:
; Align stack (main is called by crt0, RSP is 8 mod 16 after its CALL)
sub rsp, 8
; printf("Result: %d\n", 42)
lea rdi, [rel fmt] ; first arg: format string
mov rsi, 42 ; second arg: the value to print
xor eax, eax ; AL = 0 (no floating-point args in XMM regs)
call printf
; Return 0 from main
xor eax, eax
add rsp, 8
ret

Build this with gcc (which links the C runtime):

bash
nasm -f elf64 program.asm -o program.o
gcc program.o -o program -no-pie
./program

Notice the xor eax, eax right before printf. The ABI requires AL to hold the count of XMM registers used for floating-point arguments. We're passing none, so AL lands at zero.

Practical Example: Wrapping a Syscall

Here's a function wrapping sys_write, written to obey the calling convention so C can call it directly:

nasm
; ssize_t my_write(int fd, const void *buf, size_t count)
; Arguments arrive in: rdi=fd, rsi=buf, rdx=count
; Return: rax = bytes written (or negative errno)
global my_write
my_write:
mov rax, 1 ; sys_write syscall number
; rdi, rsi, rdx already hold the correct arguments!
syscall
; rax holds the return value
ret

It works because C's first three argument registers happen to be the syscall ABI's first three: rdi, rsi, rdx. That's by design, though the direction is the other way round: the Linux kernel's syscall convention (written up in an appendix of the x86-64 psABI) reuses the user-level function-call argument order. There's one difference. The fourth argument goes in r10 instead of rcx, because the syscall instruction overwrites rcx with the return address.

Security Note: Calling Convention Knowledge for Exploits

Calling conventions are core knowledge for exploit work. Building a ROP (return-oriented programming) chain means setting registers to specific values before any of your "calls" fire. Knowing rdi carries the first argument is what tells you to hunt for a pop rdi; ret gadget. Same story for shellcode that hand-rolls syscalls — get the register layout wrong and the kernel rejects the call (or worse, calls something else entirely).

Common Mistakes

  1. Forgetting stack alignment. If RSP isn't 16-byte aligned at a CALL, SSE inside the callee segfaults. Track your adjustments.

  2. Trusting caller-saved registers across calls. Stash something in rcx, call a function, and rcx is gone. Use callee-saved registers (rbx, r12-r15) or stack-spill before the call.

  3. Skipping AL for variadic functions. printf and friends read AL to learn how many XMM arguments they got. Forget it and you'll see crashes or scrambled output.

  4. Not cleaning up stack arguments. Push args before a call? You owe a matching add rsp, N after the call. RSP isn't going to fix itself.

What is Next

Next up: stack frames — the structured slab each function carves on the stack for its locals, saved registers, and housekeeping. You'll meet the prologue/epilogue pattern and see how debuggers use frame pointers to walk back up the call chain.