Writing Functions
Functions are how raw assembly turns into reusable, testable code. In this lesson you'll write complete functions that obey the System V calling convention — from minimal leaf functions all the way up to recursion. You'll also see how leaf and non-leaf functions differ in practice.
Leaf Functions
A leaf function calls nothing. That earns it a lighter touch with the calling convention — it only has to preserve callee-saved registers it actually touches.
; long add_three(long a, long b, long c); Returns a + b + c; This is a leaf function: no calls, minimal overheadglobal add_threeadd_three: mov rax, rdi ; rax = a add rax, rsi ; rax = a + b add rax, rdx ; rax = a + b + c retNo stack frame needed here. Everything lives in caller-saved registers — rax for the return, plus the argument registers. No prologue, no epilogue. Compute, return.
A slightly meatier leaf function:
; long max_of_three(long a, long b, long c); Returns the maximum of three signed valuesglobal max_of_threemax_of_three: mov rax, rdi ; assume a is the max cmp rsi, rax cmovg rax, rsi ; if b > current max, take b cmp rdx, rax cmovg rax, rdx ; if c > current max, take c retNon-Leaf Functions
A non-leaf function calls other functions. That obligates it to:
- Set up a proper stack frame
- Hit 16-byte stack alignment before every
CALL - Preserve anything it needs across calls (callee-saved registers, or stack spill)
; long process_pair(long x, long y); Returns add_three(x, y, x+y)global process_pairprocess_pair: push rbp mov rbp, rsp push rbx ; save callee-saved registers WE will use push r12 ; Stack is now: [ret addr][saved rbp][saved rbx][saved r12] ; That is 4 pushes (including the implicit push from CALL) = 32 bytes ; 32 is a multiple of 16, so we are aligned!
mov rbx, rdi ; save x mov r12, rsi ; save y
; Compute x + y for the third argument lea rdx, [rdi + rsi] ; rdx = x + y (third arg) ; rdi already holds x (first arg) ; rsi already holds y (second arg) call add_three
; rax now holds the result ; rbx and r12 still hold x and y (callee-saved!)
pop r12 ; restore callee-saved registers pop rbx pop rbp retBefore every CALL, tally up everything pushed onto the stack — including the return address from the CALL that brought you here. The total has to be a multiple of 16. If it isn't, sub rsp, 8 for an 8-byte pad. The arithmetic: the return address weighs 8 bytes, each push 8, each sub rsp, N adds N. Sum it all and aim for 0 mod 16 at the moment your outgoing CALL fires.
Recursive Functions
Recursion in assembly is just a non-leaf function whose target happens to be itself. The stack grows on each call down and shrinks on each return up.
The textbook example, factorial:
; long factorial(long n); Returns n! (n factorial); factorial(0) = 1; factorial(n) = n * factorial(n-1)global factorialfactorial: push rbp mov rbp, rsp push rbx ; we need a callee-saved register to hold n ; (24 bytes pushed, so rsp is 8 off alignment at the call ; below; harmless here because factorial only calls itself ; and never touches SSE, but add sub rsp, 8 before calling libc)
mov rbx, rdi ; save n in rbx
; Base case: if n <= 1, return 1 cmp rdi, 1 jle .base_case
; Recursive case: return n * factorial(n-1) lea rdi, [rbx - 1] ; argument: n - 1 call factorial ; rax = factorial(n-1) imul rax, rbx ; rax = n * factorial(n-1) jmp .done
.base_case: mov rax, 1 ; return 1
.done: pop rbx ; restore rbx pop rbp retLet's trace the stack for factorial(4):
Call Stack Animator
Each recursive call burns 24 bytes of stack — return address, saved rbp, saved rbx. factorial(1000) chews through 24 KB.
The stack isn't the first thing to give out, though. imul keeps only the low 64 bits, so factorial(20) = 2,432,902,008,176,640,000 is the last one that fits in a signed 64-bit long. From factorial(21) onwards the result silently wraps into garbage. Check OF after imul (as in the arithmetic lesson of module 2) if the input isn't under your control.
Every recursive call eats stack. If an attacker controls the recursion depth — say, by feeding a parser deeply nested input — they can blow the stack and crash the process. Classic denial of service. Linux's default stack ceiling sits around 8 MB. Defenses: switch to iterative algorithms where you can, enforce explicit recursion depth limits, or bump the stack with ulimit -s. For security-critical parsers (JSON, XML, HTML), depth limits aren't optional.
Converting Recursion to Iteration
Factorial rewrites cleanly as a loop, with zero extra stack:
; long factorial_iter(long n); Iterative factorial -- O(1) stack usageglobal factorial_iterfactorial_iter: mov rax, 1 ; result = 1 cmp rdi, 1 jle .done ; if n <= 1, return 1
.loop: imul rax, rdi ; result *= n dec rdi ; n-- cmp rdi, 1 jg .loop ; while n > 1
.done: retThis is a leaf function with no stack frame at all. Faster, constant stack usage, immune to exhaustion attacks.
Practical Example: Naive Recursive Fibonacci
A meatier recursive case, with two self-calls per invocation and no memo table, so it recomputes the same values over and over:
; long fibonacci(long n); Returns the nth Fibonacci number; fib(0) = 0, fib(1) = 1, fib(n) = fib(n-1) + fib(n-2)global fibonaccifibonacci: push rbp mov rbp, rsp push rbx ; save n push r12 ; save intermediate result
mov rbx, rdi ; rbx = n
; Base cases cmp rdi, 0 je .return_zero cmp rdi, 1 je .return_one
; Recursive case lea rdi, [rbx - 1] ; arg: n-1 call fibonacci ; rax = fib(n-1) mov r12, rax ; save fib(n-1) in r12
lea rdi, [rbx - 2] ; arg: n-2 call fibonacci ; rax = fib(n-2)
add rax, r12 ; rax = fib(n-1) + fib(n-2) jmp .done
.return_zero: xor rax, rax jmp .done
.return_one: mov rax, 1
.done: pop r12 pop rbx pop rbp retNaive Fibonacci runs in exponential time — each call spawns two more, while stack depth grows linearly with n. For big n, you're looking at glacial speed and a real exhaustion risk.
Draw the calls for fibonacci(4) and both halves of that sentence show up. There are nine calls, and fib(2) alone is computed twice. Yet the stack never holds more than four frames at once (one path from the root to a leaf), each 32 bytes here: the return address plus the three pushes. Add one to n and the tree roughly multiplies by 1.6, while the depth grows by one.
Function Pointers and Indirect Calls
Functions can be called through pointers held in registers or memory:
; Call a function whose address is in rax mov rdi, 42 ; set up argument call rax ; indirect call
; Call a function whose address is stored in memory call [function_table + rcx*8] ; index into a function pointer arrayIndirect calls drive C++ vtables, jump tables, and any kind of polymorphic dispatch. On the security side, control-flow integrity (CFI) clamps down on the set of valid targets for an indirect call, blocking attackers from redirecting hijacked function pointers anywhere they please.
Complete Program: A Mini Calculator
A small program that strings some of these pieces together. To keep it short it hardcodes the operands instead of reading input, and print_string and prompt are left in place for you to wire up when you extend it:
section .data prompt db "Enter operation (1=add, 2=sub, 3=mul): ", 0 prompt_len equ $ - prompt - 1 result_msg db "Result: " result_msg_len equ $ - result_msg
section .bss input resb 4
section .text global _start
; Helper: print a string; rdi = pointer, rsi = lengthprint_string: push rbp mov rbp, rsp mov rdx, rsi ; length mov rsi, rdi ; buffer pointer mov rdi, 1 ; stdout mov rax, 1 ; sys_write syscall pop rbp ret
; Arithmetic functionsdo_add: lea rax, [rdi + rsi] ret
do_sub: mov rax, rdi sub rax, rsi ret
do_mul: mov rax, rdi imul rax, rsi ret
_start: ; For simplicity, we will hardcode operands and operation ; In a full program, you would read these from input mov r12, 15 ; operand a = 15 mov r13, 7 ; operand b = 7
; Call do_add(15, 7) mov rdi, r12 mov rsi, r13 call do_add ; rax = 22
; Exit with the result as the status code mov rdi, rax mov rax, 60 syscallBuild and test:
nasm -f elf64 calc.asm -o calc.old calc.o -o calc./calcecho $? # prints 22The ret instruction pops an address off the stack and jumps to it. Overwrite that address via a buffer overflow and execution lands wherever the attacker pointed. Modern defenses pile on: stack canaries (catch tampering before ret runs), ASLR (scrambles addresses so the attacker can't predict targets), NX/DEP (refuses to execute data on the stack), and shadow stacks (a hardware-backed second copy of return addresses, out of reach of memory writes). Once ret clicks, every one of those defenses suddenly makes sense.
What is Next
That's the foundational four modules of x86-64 assembly behind you. You can now read and write register operations, do arithmetic, steer control flow, and build functions with proper stack management. From here we build outward into memory addressing, string operations, C interop, and security-focused territory — buffer overflows and shellcode development.
Challenges
Write a recursive factorial function in x86-64 assembly. Compute factorial(5) and print the result (120) followed by a newline.