The Linux Syscall Interface
System calls sit on the boundary between your program and the kernel. Every interaction with the outside world — reading files, writing to the screen, allocating memory, spawning processes, opening sockets — funnels through a syscall. Userspace has no other way in.
Reading the syscall interface at the assembly level is foundational for security work — syscalls are the program's attack surface. Every one of them is a doorway, used by legitimate code and exploit payloads alike.
For background on how the kernel manages these transitions, see Sandboxing and Isolation.
The syscall Instruction
On x86-64 Linux, syscall punts control from userspace (ring 3) into the kernel (ring 0). The kernel reads your registers, does the work, and hands control back at the instruction following syscall.
; Every syscall follows this pattern: mov rax, SYSCALL_NUMBER ; which operation to perform mov rdi, arg1 ; first argument mov rsi, arg2 ; second argument mov rdx, arg3 ; third argument mov r10, arg4 ; fourth argument (NOT rcx!) mov r8, arg5 ; fifth argument mov r9, arg6 ; sixth argument syscall ; enter the kernel ; return value is in raxUserspace can't reach hardware, files, network interfaces, or other processes directly. The CPU enforces that through privilege rings. The syscall instruction is the controlled chokepoint that briefly switches to ring 0 so the kernel can do the work on your behalf. A printf ends in a write, a network send in sendto, and a malloc that runs out of heap asks for more with brk or mmap. Two caveats keep this honest: malloc usually hands out memory from arenas it already owns without entering the kernel at all, and a few read-only calls such as clock_gettime and gettimeofday are served from the vDSO, a small page of kernel-provided code mapped into every process so it can answer them without a syscall.
Register Convention
The syscall calling convention is a near-twin of System V's function calling convention, but with one telling difference:
| Register | Syscall Role | Function Call Role |
|---|---|---|
RAX | Syscall number (in), return value (out) | Return value |
RDI | Argument 1 | Argument 1 |
RSI | Argument 2 | Argument 2 |
RDX | Argument 3 | Argument 3 |
R10 | Argument 4 | (not used - RCX is arg 4 in functions) |
R8 | Argument 5 | Argument 5 |
R9 | Argument 6 | Argument 6 |
Argument 4 is where they part ways: functions use RCX, syscalls use R10. The reason is mechanical — syscall overwrites RCX with the return address, so the kernel can't read arguments out of it.
syscall also stomps on R11, where it stashes the saved RFLAGS. Treat both RCX and R11 as clobbered after any syscall.
Return Values and Error Handling
On success, RAX holds the return value — bytes read, a file descriptor, whatever the call yields. On failure, RAX comes back negative, somewhere in the range -1 to -4095. That value is the negated errno.
; Check for syscall error mov rax, 0 ; sys_read mov rdi, 0 ; stdin lea rsi, [buffer] mov rdx, 256 syscall
test rax, rax ; check return value js .error ; negative = error (SF=1) jz .eof ; zero = end of file / connection closed ; positive = number of bytes read jmp .success
.error: neg rax ; negate to get errno value ; rax now contains the errno (e.g., 11 = EAGAIN)Essential Syscall Reference
The syscalls you'll lean on most often. The number lives in RAX.
| # | Name | Arguments | Returns |
|---|---|---|---|
| 0 | read | fd, buf, count | bytes read |
| 1 | write | fd, buf, count | bytes written |
| 2 | open | filename, flags, mode | file descriptor |
| 3 | close | fd | 0 on success |
| 9 | mmap | addr, len, prot, flags, fd, offset | mapped address |
| 10 | mprotect | addr, len, prot | 0 on success |
| 41 | socket | domain, type, protocol | socket fd |
| 42 | connect | sockfd, addr, addrlen | 0 on success |
| 43 | accept | sockfd, addr, addrlen | new socket fd |
| 44 | sendto | sockfd, buf, len, flags, dest, addrlen | bytes sent |
| 49 | bind | sockfd, addr, addrlen | 0 on success |
| 50 | listen | sockfd, backlog | 0 on success |
| 60 | exit | status | does not return (ends the calling thread) |
| 231 | exit_group | status | does not return (ends every thread in the process) |
Example: Write to stdout
section .data msg db "Syscall complete", 10 msg_len equ $ - msg
section .text global _start
_start: mov rax, 1 ; sys_write mov rdi, 1 ; fd = stdout lea rsi, [msg] ; buffer mov rdx, msg_len ; count syscall
; Check that all bytes were written cmp rax, msg_len jne .write_error ; partial write or error
mov rax, 60 ; sys_exit xor rdi, rdi ; status = 0 syscall
.write_error: mov rax, 60 mov rdi, 1 ; exit with error status syscallExample: mprotect - Changing Memory Permissions
The mprotect syscall changes permissions on memory pages. From a security angle, it's load-bearing — it's the call that can make memory executable.
; Make a page of memory executable (dangerous!) ; PROT_READ | PROT_WRITE | PROT_EXEC = 7 mov rax, 10 ; sys_mprotect mov rdi, page_addr ; must be page-aligned mov rsi, 4096 ; one page mov rdx, 7 ; rwx permissions syscallW^X (write xor execute) is the policy that a page should never be writable and executable at once, since that single rule blocks a huge class of code-injection attacks. Stock Linux doesn't enforce it here: this mprotect succeeds and you get an RWX page, unless SELinux (its execmem checks), PaX or a similar policy denies it. What Linux does by default is map the stack, heap and data non-executable, so an attacker has to get a call like this made before injected code can run. The DEP/NX lesson in module 8 comes back to this.
Security Perspective: Syscall Filtering with seccomp
Because syscalls are the only path into the kernel, filtering them is a serious security lever. Linux's seccomp (secure computing) lets a program restrict the syscalls it's allowed to make.
; A sandboxed program might only be allowed: ; read (0), write (1), exit (60), exit_group (231) ; exit ends one thread; exit_group (what libc's exit() calls) ends ; the whole process, so a filter needs both ; Any other syscall causes the process to be killedIt's how containers, browsers, and sandboxed apps shrink their attack surface. An attacker who lands code execution in a process locked down to read, write, and exit can't open new files, dial out over the network, or spawn anything.
The principle is plain: every syscall you don't need is one you should strip. When we build our echo server in this module, ask yourself which syscalls it genuinely needs, and which a seccomp filter could lock out entirely.
What is Next
With the syscall mechanism in hand, the next lesson focuses on file descriptors — the kernel's universal abstraction for I/O — and the read/write/open/close syscalls that drive them.