System V Calling Convention
A calling convention is the contract between caller and callee. It pins down how arguments are passed, where the return value lives, and who's on the hook for preserving each register. Strip the contract away and separately-compiled functions stop being able to talk to each other.
On Linux x86-64, the contract is the System V AMD64 ABI. Every C compiler on Linux speaks it. Your assembly code has to speak it too if it wants to talk to C in either direction.
For a broader view of calling conventions across platforms, see x86 Calling Conventions.
Argument Passing
The first six integer (or pointer) arguments are passed in registers, in this exact order:
| Argument | Register |
|---|---|
| 1st | rdi |
| 2nd | rsi |
| 3rd | rdx |
| 4th | rcx |
| 5th | r8 |
| 6th | r9 |
More than six arguments? Anything past the sixth gets pushed onto the stack, in reverse order — right-to-left.
Floating-point arguments use xmm0 through xmm7. This track sticks with integer arguments throughout.
; Calling a function with 3 arguments: func(10, 20, 30)mov rdi, 10 ; first argumentmov rsi, 20 ; second argumentmov rdx, 30 ; third argumentcall func; Return value is now in rax; Calling a function with 8 arguments: func(1,2,3,4,5,6,7,8)push 8 ; 8th argument (pushed first -- right to left)push 7 ; 7th argumentmov rdi, 1 ; 1st argumentmov rsi, 2 ; 2nd argumentmov rdx, 3 ; 3rd argumentmov rcx, 4 ; 4th argumentmov r8, 5 ; 5th argumentmov r9, 6 ; 6th argumentcall funcadd rsp, 16 ; clean up the 2 pushed arguments (2 * 8 bytes)Return Values
Return values land in rax for integers and pointers, xmm0 for floats. 128-bit values use rdx:rax — high half in rdx, low half in rax.
; A function that returns a valuecompute_something: ; ... do work ... mov rax, 42 ; return 42 ret
; Calling it call compute_something ; rax now holds 42Caller-Saved vs Callee-Saved Registers
This is the part of the calling convention that bites people. Registers split into two camps:
Caller-saved (volatile) — the callee can stomp on these freely. If you need them preserved across a call, save them first.
| Register | Purpose |
|---|---|
rax | Return value |
rcx | 4th argument |
rdx | 3rd argument |
rdi | 1st argument |
rsi | 2nd argument |
r8 | 5th argument |
r9 | 6th argument |
r10 | Temporary |
r11 | Temporary |
Callee-saved (non-volatile) — the callee owes the caller their original values back. Want to use one? Push it on entry, pop it on exit.
| Register | Purpose |
|---|---|
rbx | General purpose |
rbp | Frame pointer |
r12 | General purpose |
r13 | General purpose |
r14 | General purpose |
r15 | General purpose |
rsp | Stack pointer (must always be restored) |
The split is a performance hack. Make every register callee-saved and every function pays to push/pop sixteen registers, most of which the caller couldn't care less about. Make every register caller-saved and every call site does the saving instead. Splitting them lets each side preserve only the registers it actually cares about, which slashes the total push/pop traffic across the whole program.
The Red Zone
The System V ABI carves out a red zone: 128 bytes below RSP that signal handlers and interrupts promise not to touch. Leaf functions — those that don't call anything else — can scribble there without ever bumping RSP:
; Leaf function using the red zoneleaf_func: ; No need to sub rsp -- use space below RSP mov qword [rsp - 8], rdi ; store in the red zone mov qword [rsp - 16], rsi ; also in the red zone ; ... compute using those values ... ret ; red zone is automatically reclaimedThe red zone trick spares small functions the overhead of touching RSP. But the moment a function calls another, the callee's frame overwrites the red zone. Leaf functions only.
The Linux kernel turns the red zone off — built with -mno-red-zone. The reason: interrupts can fire at any moment and start writing below RSP immediately. Anything stashed in the red zone would be flattened. If you're writing kernel modules or exploit payloads that run in kernel context, the red zone is off the table. One of many ways the kernel ABI parts company with userspace.
Calling C Library Functions
Once the calling convention clicks, calling libc from assembly stops being mysterious:
; Call printf from assemblysection .data fmt db "Result: %d", 10, 0 ; format string (null-terminated for C)
section .text extern printf global main ; use 'main' when linking with gcc
main: ; Align stack (main is called by crt0, RSP is 8 mod 16 after its CALL) sub rsp, 8
; printf("Result: %d\n", 42) lea rdi, [rel fmt] ; first arg: format string mov rsi, 42 ; second arg: the value to print xor eax, eax ; AL = 0 (no floating-point args in XMM regs) call printf
; Return 0 from main xor eax, eax add rsp, 8 retBuild this with gcc (which links the C runtime):
nasm -f elf64 program.asm -o program.ogcc program.o -o program -no-pie./programNotice the xor eax, eax right before printf. The ABI requires AL to hold the count of XMM registers used for floating-point arguments. We're passing none, so AL lands at zero.
Practical Example: Wrapping a Syscall
Here's a function wrapping sys_write, written to obey the calling convention so C can call it directly:
; ssize_t my_write(int fd, const void *buf, size_t count); Arguments arrive in: rdi=fd, rsi=buf, rdx=count; Return: rax = bytes written (or negative errno)global my_writemy_write: mov rax, 1 ; sys_write syscall number ; rdi, rsi, rdx already hold the correct arguments! syscall ; rax holds the return value retIt works because C's first three argument registers happen to be the syscall ABI's first three: rdi, rsi, rdx. That's by design, though the direction is the other way round: the Linux kernel's syscall convention (written up in an appendix of the x86-64 psABI) reuses the user-level function-call argument order. There's one difference. The fourth argument goes in r10 instead of rcx, because the syscall instruction overwrites rcx with the return address.
Calling conventions are core knowledge for exploit work. Building a ROP (return-oriented programming) chain means setting registers to specific values before any of your "calls" fire. Knowing rdi carries the first argument is what tells you to hunt for a pop rdi; ret gadget. Same story for shellcode that hand-rolls syscalls — get the register layout wrong and the kernel rejects the call (or worse, calls something else entirely).
Common Mistakes
-
Forgetting stack alignment. If RSP isn't 16-byte aligned at a
CALL, SSE inside the callee segfaults. Track your adjustments. -
Trusting caller-saved registers across calls. Stash something in
rcx, call a function, andrcxis gone. Use callee-saved registers (rbx,r12-r15) or stack-spill before the call. -
Skipping AL for variadic functions.
printfand friends readALto learn how many XMM arguments they got. Forget it and you'll see crashes or scrambled output. -
Not cleaning up stack arguments. Push args before a call? You owe a matching
add rsp, Nafter the call. RSP isn't going to fix itself.
What is Next
Next up: stack frames — the structured slab each function carves on the stack for its locals, saved registers, and housekeeping. You'll meet the prologue/epilogue pattern and see how debuggers use frame pointers to walk back up the call chain.