Part ofSystems Programming Companion, week 4
Function Calls Under the Hood
Every C function call kicks off a small choreography: pushing data onto the stack, jumping to fresh code, eventually unwinding the lot to come back. This lesson traces every beat of it, from call to return.
How main() Gets Called
Reasonable assumption: main() is where your program starts. Wrong. The real entry point is _start, supplied by the C runtime (crt0). Here's the sequence of events before main ever sees a register:
- The OS loader maps your executable into memory
- Control jumps to
_start _startcalls__libc_start_main- The runtime initializes the heap, sets up stdio buffers, and runs global constructors
- Finally,
main(argc, argv)is called
Pop open the disassembly of any compiled C program and you'll see it:
objdump -d program | grep -A 20 "<_start>"_start: xor ebp, ebp ; Clear the frame pointer: marks the outermost frame mov r9, rdx ; Loader's cleanup function (rtld_fini), 6th argument pop rsi ; argc, straight off the stack, 2nd argument mov rdx, rsp ; argv now sits at the top of the stack, 3rd argument and rsp, -16 ; Realign the stack to 16 bytes before the call push rax ; Padding, keeps the alignment push rsp ; Stack end, 7th argument xor r8d, r8d ; fini (unused since glibc 2.34) xor ecx, ecx ; init (unused since glibc 2.34) mov rdi, QWORD PTR [rip+0x2fd8] ; Address of main, 1st argument call QWORD PTR [rip+0x2fd2] ; __libc_start_main(main, argc, argv, ...) hlt ; Should never reach hereThis is glibc's _start for x86-64 (sysdeps/x86_64/start.S), and the offsets in the last two loads will differ in your binary. The first argument to __libc_start_main is main itself, a function pointer, which is how the runtime knows what to call once setup is done. At entry the kernel left argc at the top of the stack with the argv pointers right above it, so a single pop both fetches argc and leaves rsp pointing at argv. It's a neat trick, and a good first reminder that the calling convention below applies to calls between functions. The kernel's handover to _start follows its own rules.
The System V AMD64 Calling Convention
On x86-64 Linux (and macOS), function calls obey the System V AMD64 ABI — a shared contract every piece of compiled code on the system signs up to. The rules:
Integer/pointer arguments are passed in registers, in this order:
rdi- first argumentrsi- second argumentrdx- third argumentrcx- fourth argumentr8- fifth argumentr9- sixth argument
More than six arguments? Anything past the sixth spills onto the stack.
Return values come back in rax (or rax:rdx for 128-bit values).
Caller-saved registers (the caller must save these if it needs them after the call): rax, rcx, rdx, rsi, rdi, r8, r9, r10, r11.
Callee-saved registers (the called function must restore these before returning): rbx, rbp, r12, r13, r14, r15.
Watch it play out:
int compute(int a, int b, int c) { return a * b + c;}
int main() { int result = compute(3, 4, 5); return result;}Compiled with gcc -S -masm=intel -O0:
compute: push rbp ; Save caller's base pointer mov rbp, rsp ; Establish our stack frame mov DWORD PTR [rbp-4], edi ; a (from rdi) -> stack mov DWORD PTR [rbp-8], esi ; b (from rsi) -> stack mov DWORD PTR [rbp-12], edx ; c (from rdx) -> stack mov eax, DWORD PTR [rbp-4] ; Load a imul eax, DWORD PTR [rbp-8] ; eax = a * b add eax, DWORD PTR [rbp-12] ; eax = a * b + c pop rbp ; Restore caller's base pointer ret ; Return with result in eax
main: push rbp mov rbp, rsp sub rsp, 16 ; Reserve stack space for locals mov edx, 5 ; Third arg (c) -> edx mov esi, 4 ; Second arg (b) -> esi mov edi, 3 ; First arg (a) -> edi call compute ; Push return address, jump to compute mov DWORD PTR [rbp-4], eax ; Store result from eax mov eax, DWORD PTR [rbp-4] ; Load result for return leave ; Equivalent to: mov rsp, rbp; pop rbp retAnatomy of a Stack Frame
Here's compute's frame just after the three movs that spill its arguments. Row numbers are byte offsets counted up from rbp-16, so rbp itself sits at 10. The blue band (the visualiser calls it a buffer) is the 16 bytes below rbp: c = 5 at rbp-12, b = 4 at rbp-8, a = 3 at rbp-4, each stored little-endian, with four unused bytes at the bottom. Above it sit the 8 bytes of saved rbp and then the return address that call pushed. Only its lowest byte is drawn, and both saved values are illustrative because they change from run to run.
Stack Frame View
rsp points at the saved rbp, the same place as rbp, because compute never runs sub rsp. It calls nothing, so the ABI's 128-byte red zone below rsp is safe scratch space for its locals.
The function prologue (push rbp; mov rbp, rsp) builds the frame. The epilogue (pop rbp; ret) demolishes it.
call does two things in one shot:
- Pushes the address of the next instruction (the return address) onto the stack
- Jumps to the target function
ret runs the same trick in reverse:
- Pops the return address off the stack
- Jumps there
More Than Six Arguments
So what about more than six integer arguments?
int many_args(int a, int b, int c, int d, int e, int f, int g, int h) { return a + b + c + d + e + f + g + h;}
int main() { return many_args(1, 2, 3, 4, 5, 6, 7, 8);}main: push rbp mov rbp, rsp push 8 ; 8th arg goes on stack (pushed first!) push 7 ; 7th arg goes on stack mov r9d, 6 ; 6th arg -> r9 mov r8d, 5 ; 5th arg -> r8 mov ecx, 4 ; 4th arg -> ecx mov edx, 3 ; 3rd arg -> edx mov esi, 2 ; 2nd arg -> esi mov edi, 1 ; 1st arg -> edi call many_args add rsp, 16 ; Clean up the 2 pushed args (8 bytes each) leave retArguments 7 and 8 hit the stack backwards — 8 first, then 7. That ordering puts them in the right slot when the callee reads memory low-to-high. The first six rides registers as before.
With nine arguments, which ones does the callee find above its saved rbp (in the caller's frame) and which below it? Make a guess, then check:
g, h and i show up at rbp+16, +24 and +32: past the saved rbp at +0 and the return address at +8, one 8-byte slot each, even for an int. The first six have negative offsets because at -O0 the function copies each register into its own frame so it has an address to give you. The exact negative numbers depend on the compiler.
The cdecl Convention (32-bit)
Back on 32-bit x86, the older cdecl convention parked every argument on the stack:
; 32-bit cdecl: calling add(3, 4)push 4 ; Second argument pushed firstpush 3 ; First argument pushed secondcall addadd esp, 8 ; Caller cleans up the stackEvery argument costs a memory write on the way in (push) and a memory read once inside the function. System V keeps the first six in registers instead. On a modern CPU the gap is smaller than it sounds, because store-to-load forwarding hands a freshly pushed value straight to the load that reads it, but it's still extra instructions and extra memory traffic on every call.
Return Values
Simple types come back through rax:
long get_value() { return 42;}get_value: mov eax, 42 ; Return value in eax (zero-extends to rax) retFor 128-bit values — say, a struct holding two 64-bit fields — the compiler splits it across rax and rdx:
struct pair { long a; long b; };
struct pair make_pair() { struct pair p = {10, 20}; return p;}make_pair: mov rax, 10 ; First 8 bytes of return value mov rdx, 20 ; Second 8 bytes of return value retBigger structs trigger a quiet sleight of hand: the caller slips in a pointer as an invisible first argument, and the function fills the return value through it.
Tail Calls
A tail call is just any call sitting in the very last slot of a function's body:
int factorial_helper(int n, int acc) { if (n <= 1) return acc; return factorial_helper(n - 1, acc * n); // Tail call}Turn optimisation on and the compiler swaps the recursive call for a jmp, recycling the current stack frame:
factorial_helper: cmp edi, 1 jle .return_acc imul esi, edi ; acc = acc * n dec edi ; n = n - 1 jmp factorial_helper ; Jump, not call! No new stack frame.return_acc: mov eax, esi retThat's tail call optimisation (TCO). Recursion turns into a loop and stack usage stays flat no matter how deep the call chain pretends to go. Not every compiler promises TCO, but GCC and Clang deliver it reliably at -O2 or higher.
What You Should Take Away
Every function call costs you something measurable:
- Pushing and popping the stack frame (2-4 instructions)
- Saving and restoring callee-saved registers (when they get touched)
- The
callandretinstructions themselves - Possible cache misses if the callee's code isn't already sitting in the instruction cache
Inside a hot loop those costs add up fast. Hence why compilers love inlining small functions — drop the call entirely, paste the function body straight into the caller, done.
Calling conventions also explain a lot of seemingly unrelated mysteries: why variadic functions like printf work, why function pointers carry the signatures they do, and how debuggers reconstruct a stack trace. With frame pointers that's just walking the chain of saved rbp values back through every frame. Optimised x86-64 code usually omits them (-O2 implies -fomit-frame-pointer), so debuggers, profilers and C++ exception handling fall back on the unwind tables the compiler writes into .eh_frame (DWARF call-frame information), which describe how to find each caller's frame from any instruction address.