Your First Syscall
A syscall (system call) is how your program asks the kernel to do something for it. Without syscalls, all your assembly can manage is shuffling bits between registers and memory. No printing, no reading input, no files, no allocations, nothing that touches the outside world.
Syscalls are the boundary between user-mode and kernel-mode code. Every conversation your program has with the OS passes through that one gate.
How Syscalls Work
On Linux x86-64, the syscall interface obeys a fixed register convention:
| Register | Purpose |
|---|---|
rax | Syscall number (which operation to perform) |
rdi | First argument |
rsi | Second argument |
rdx | Third argument |
r10 | Fourth argument |
r8 | Fifth argument |
r9 | Sixth argument |
rax (after) | Return value (negative values indicate errors) |
syscall hands control over to the kernel. It reads rax to figure out which operation you asked for, runs it, drops the result back into rax, and bounces you home.
Here's one write(1, msg, len) crossing that boundary and coming back. Two details tend to surprise people. The CPU uses rcx and r11 to remember where you were, so both are trashed by every syscall. And the kernel's entry address comes from a model-specific register the kernel set up at boot, so user code can't pick where it lands.
The CPU runs in two privilege levels. Your assembly code runs in user mode (ring 3), where it cannot directly access hardware or other processes' memory. The kernel runs in kernel mode (ring 0), where it has full access to everything. The syscall instruction is the controlled gateway between these modes. The CPU switches privilege levels, executes the kernel handler, then switches back. This boundary is fundamental to OS security - if user code could bypass it, all process isolation would collapse.
Essential Syscalls
The handful you'll reach for constantly:
sys_write (number 1) - Write to a File Descriptor
; ssize_t write(int fd, const void *buf, size_t count)mov rax, 1 ; sys_writemov rdi, 1 ; fd = 1 (stdout), use 2 for stderrmov rsi, msg ; pointer to buffermov rdx, msg_len ; number of bytes to writesyscall ; returns bytes written in rax, or negative on errorsys_read (number 0) - Read from a File Descriptor
; ssize_t read(int fd, void *buf, size_t count)mov rax, 0 ; sys_readmov rdi, 0 ; fd = 0 (stdin)mov rsi, buffer ; pointer to destination buffermov rdx, 256 ; maximum bytes to readsyscall ; returns bytes read in rax, or negative on errorsys_exit (number 60) - Terminate the Process
; void exit(int status)mov rax, 60 ; sys_exitmov rdi, 0 ; exit status (0 = success)syscall ; does not returnA Complete Read-Write Program
A program that reads a line from the user and echoes it straight back:
SYS_READ equ 0SYS_WRITE equ 1SYS_EXIT equ 60STDIN equ 0STDOUT equ 1
section .data prompt db "Enter text: " prompt_len equ $ - prompt reply db "You said: " reply_len equ $ - reply
section .bss input resb 256 ; buffer for user input
section .text global _start
_start: ; Print the prompt mov rax, SYS_WRITE mov rdi, STDOUT mov rsi, prompt mov rdx, prompt_len syscall
; Read user input mov rax, SYS_READ mov rdi, STDIN mov rsi, input mov rdx, 256 syscall ; Save the number of bytes read mov r12, rax ; r12 is callee-saved, safe to keep here
; Print "You said: " prefix mov rax, SYS_WRITE mov rdi, STDOUT mov rsi, reply mov rdx, reply_len syscall
; Print the user's input mov rax, SYS_WRITE mov rdi, STDOUT mov rsi, input mov rdx, r12 ; number of bytes the user typed syscall
; Exit mov rax, SYS_EXIT xor rdi, rdi syscallBuilding and Running
Two steps build any standalone assembly program:
nasm -f elf64 echo.asm -o echo.o # step 1: assembleld echo.o -o echo # step 2: link./echo # run itnasm -f elf64 translates assembly source into a 64-bit ELF object file (.o). Machine code, symbols, no executable bit yet.
ld is the GNU linker. It resolves symbols (_start, friends), pins down memory addresses, and produces the final ELF executable. For a standalone program with no C library in the picture, ld is the whole show.
gcc can't assemble this file: its assembler is GNU as, which expects AT&T or GAS-flavoured Intel syntax, not NASM. You can still let gcc drive the link step, though: nasm -f elf64 echo.asm -o echo.o followed by gcc -nostdlib -no-pie echo.o -o echo. Running nasm and ld by hand gives you more control and shows what each step actually does. Once you start linking against the C library in later modules, gcc as the linker driver becomes the convenient choice.
Syscall Return Values and Error Handling
Syscalls hand the result back in rax. On success the meaning depends on the syscall (sys_read returns the bytes read, for example). On failure rax carries a negative errno value.
; Read from stdin mov rax, SYS_READ mov rdi, STDIN mov rsi, input mov rdx, 256 syscall
; Check for error test rax, rax ; sets SF if rax is negative js read_error ; jump if sign flag is set (negative return) ; rax now holds the number of bytes read jmp read_success
read_error: neg rax ; convert -errno to positive errno value ; handle the error...
read_success: ; process the input...Common error codes worth memorising: -1 (EPERM, operation not permitted), -13 (EACCES, permission denied), -2 (ENOENT, file not found), -9 (EBADF, bad file descriptor), -14 (EFAULT, bad address).
Finding Syscall Numbers
Syscall numbers live in the kernel headers. The path depends on the distro:
cat /usr/include/x86_64-linux-gnu/asm/unistd_64.h | head -30 # Debian, Ubuntucat /usr/include/asm/unistd_64.h | head -30 # Fedora, ArchOr just keep a reference table at hand. The greatest hits:
| Number | Name | Purpose |
|---|---|---|
| 0 | sys_read | Read from file descriptor |
| 1 | sys_write | Write to file descriptor |
| 2 | sys_open | Open a file |
| 3 | sys_close | Close a file descriptor |
| 9 | sys_mmap | Map memory |
| 12 | sys_brk | Change data segment size |
| 57 | sys_fork | Create a child process |
| 59 | sys_execve | Execute a program |
| 60 | sys_exit | Terminate process |
Syscalls are the primary attack surface between user programs and the kernel. Every syscall is a potential entry point for kernel exploitation. Techniques like seccomp (secure computing mode) restrict which syscalls a process can make, dramatically reducing the attack surface. In sandboxed environments (like containers or browser processes), syscall filtering is a critical defense layer. Tools like strace let you observe every syscall a program makes - an essential technique in both debugging and malware analysis.
Tracing Syscalls
Watch every syscall a program issues with strace:
strace ./echoEach call shows up with its arguments and return value. Brilliant for debugging assembly, and just as good for unmasking what a compiled binary really does at the OS layer. When an unknown binary lands on your desk, strace is usually the first move.
What is Next
Registers, memory sections, syscalls — the foundations are in place. The next module covers the core instructions for moving data and doing arithmetic, the raw atoms every other operation will be built from.
Challenges
Write an x86-64 NASM program that prints "Hello, x86!" followed by a newline to stdout using the write syscall, then exits cleanly with exit code 0.