Skip to main content
Be Bitwise
x86 AssemblyCapstone: System Calls & I/O

The Linux Syscall Interface

System calls sit on the boundary between your program and the kernel. Every interaction with the outside world — reading files, writing to the screen, allocating memory, spawning processes, opening sockets — funnels through a syscall. Userspace has no other way in.

Reading the syscall interface at the assembly level is foundational for security work — syscalls are the program's attack surface. Every one of them is a doorway, used by legitimate code and exploit payloads alike.

For background on how the kernel manages these transitions, see Sandboxing and Isolation.

The syscall Instruction

On x86-64 Linux, syscall punts control from userspace (ring 3) into the kernel (ring 0). The kernel reads your registers, does the work, and hands control back at the instruction following syscall.

nasm
; Every syscall follows this pattern:
mov rax, SYSCALL_NUMBER ; which operation to perform
mov rdi, arg1 ; first argument
mov rsi, arg2 ; second argument
mov rdx, arg3 ; third argument
mov r10, arg4 ; fourth argument (NOT rcx!)
mov r8, arg5 ; fifth argument
mov r9, arg6 ; sixth argument
syscall ; enter the kernel
; return value is in rax
Syscalls Are the ONLY Way to Talk to the Kernel

Userspace can't reach hardware, files, network interfaces, or other processes directly. The CPU enforces that through privilege rings. The syscall instruction is the controlled chokepoint that briefly switches to ring 0 so the kernel can do the work on your behalf. A printf ends in a write, a network send in sendto, and a malloc that runs out of heap asks for more with brk or mmap. Two caveats keep this honest: malloc usually hands out memory from arenas it already owns without entering the kernel at all, and a few read-only calls such as clock_gettime and gettimeofday are served from the vDSO, a small page of kernel-provided code mapped into every process so it can answer them without a syscall.

Register Convention

The syscall calling convention is a near-twin of System V's function calling convention, but with one telling difference:

RegisterSyscall RoleFunction Call Role
RAXSyscall number (in), return value (out)Return value
RDIArgument 1Argument 1
RSIArgument 2Argument 2
RDXArgument 3Argument 3
R10Argument 4(not used - RCX is arg 4 in functions)
R8Argument 5Argument 5
R9Argument 6Argument 6

Argument 4 is where they part ways: functions use RCX, syscalls use R10. The reason is mechanical — syscall overwrites RCX with the return address, so the kernel can't read arguments out of it.

syscall also stomps on R11, where it stashes the saved RFLAGS. Treat both RCX and R11 as clobbered after any syscall.

Return Values and Error Handling

On success, RAX holds the return value — bytes read, a file descriptor, whatever the call yields. On failure, RAX comes back negative, somewhere in the range -1 to -4095. That value is the negated errno.

nasm
; Check for syscall error
mov rax, 0 ; sys_read
mov rdi, 0 ; stdin
lea rsi, [buffer]
mov rdx, 256
syscall
test rax, rax ; check return value
js .error ; negative = error (SF=1)
jz .eof ; zero = end of file / connection closed
; positive = number of bytes read
jmp .success
.error:
neg rax ; negate to get errno value
; rax now contains the errno (e.g., 11 = EAGAIN)

Essential Syscall Reference

The syscalls you'll lean on most often. The number lives in RAX.

#NameArgumentsReturns
0readfd, buf, countbytes read
1writefd, buf, countbytes written
2openfilename, flags, modefile descriptor
3closefd0 on success
9mmapaddr, len, prot, flags, fd, offsetmapped address
10mprotectaddr, len, prot0 on success
41socketdomain, type, protocolsocket fd
42connectsockfd, addr, addrlen0 on success
43acceptsockfd, addr, addrlennew socket fd
44sendtosockfd, buf, len, flags, dest, addrlenbytes sent
49bindsockfd, addr, addrlen0 on success
50listensockfd, backlog0 on success
60exitstatusdoes not return (ends the calling thread)
231exit_groupstatusdoes not return (ends every thread in the process)

Example: Write to stdout

nasm
section .data
msg db "Syscall complete", 10
msg_len equ $ - msg
section .text
global _start
_start:
mov rax, 1 ; sys_write
mov rdi, 1 ; fd = stdout
lea rsi, [msg] ; buffer
mov rdx, msg_len ; count
syscall
; Check that all bytes were written
cmp rax, msg_len
jne .write_error ; partial write or error
mov rax, 60 ; sys_exit
xor rdi, rdi ; status = 0
syscall
.write_error:
mov rax, 60
mov rdi, 1 ; exit with error status
syscall

Example: mprotect - Changing Memory Permissions

The mprotect syscall changes permissions on memory pages. From a security angle, it's load-bearing — it's the call that can make memory executable.

nasm
; Make a page of memory executable (dangerous!)
; PROT_READ | PROT_WRITE | PROT_EXEC = 7
mov rax, 10 ; sys_mprotect
mov rdi, page_addr ; must be page-aligned
mov rsi, 4096 ; one page
mov rdx, 7 ; rwx permissions
syscall

W^X (write xor execute) is the policy that a page should never be writable and executable at once, since that single rule blocks a huge class of code-injection attacks. Stock Linux doesn't enforce it here: this mprotect succeeds and you get an RWX page, unless SELinux (its execmem checks), PaX or a similar policy denies it. What Linux does by default is map the stack, heap and data non-executable, so an attacker has to get a call like this made before injected code can run. The DEP/NX lesson in module 8 comes back to this.

Security Perspective: Syscall Filtering with seccomp

Because syscalls are the only path into the kernel, filtering them is a serious security lever. Linux's seccomp (secure computing) lets a program restrict the syscalls it's allowed to make.

nasm
; A sandboxed program might only be allowed:
; read (0), write (1), exit (60), exit_group (231)
; exit ends one thread; exit_group (what libc's exit() calls) ends
; the whole process, so a filter needs both
; Any other syscall causes the process to be killed

It's how containers, browsers, and sandboxed apps shrink their attack surface. An attacker who lands code execution in a process locked down to read, write, and exit can't open new files, dial out over the network, or spawn anything.

The principle is plain: every syscall you don't need is one you should strip. When we build our echo server in this module, ask yourself which syscalls it genuinely needs, and which a seccomp filter could lock out entirely.

What is Next

With the syscall mechanism in hand, the next lesson focuses on file descriptors — the kernel's universal abstraction for I/O — and the read/write/open/close syscalls that drive them.