Skip to main content
Be Bitwise

Linux Syscall Interface

Syscall Interfaces & the Kernel Boundary · OS Core Internals track

5 min read

In this module

Making a System Call on Linux

When a Linux programme calls something like write(), the C library quietly translates that pleasant function call into a raw system call. On x86-64, that means stuffing the syscall number and arguments into specific registers, then firing off the syscall instruction. Get familiar with this interface and assembly-level binary analysis stops feeling like archaeology — you can see exactly where user code hands off to the kernel.

The Register Convention

Linux on x86-64 uses a clean, well-defined register convention for system calls:

RegisterPurpose
RAXSystem call number (input) / return value (output)
RDIFirst argument
RSISecond argument
RDXThird argument
R10Fourth argument
R8Fifth argument
R9Sixth argument

Two registers are clobbered by the syscall instruction itself. RCX is overwritten with the saved RIP, and R11 with the saved RFLAGS. This is hardware behaviour baked into the instruction, so you can't preserve them across a syscall no matter how careful you are.

Key Concept

Linux x86-64 system calls use the syscall instruction with the call number in RAX and up to six arguments in RDI, RSI, RDX, R10, R8, and R9. The return value comes back in RAX. RCX and R11 are destroyed by the instruction itself, as the hardware uses them to save RIP and RFLAGS.

A Concrete Example

Here is a write(1, "hello\n", 6) system call in assembly - writing to stdout:

nasm
section .data
msg db "hello", 10 ; "hello\n"
section .text
global _start
_start:
mov rax, 1 ; syscall number for write
mov rdi, 1 ; file descriptor 1 (stdout)
lea rsi, [rel msg] ; pointer to the string
mov rdx, 6 ; length of the string
syscall ; enter kernel
mov rax, 60 ; syscall number for exit
xor rdi, rdi ; exit code 0
syscall ; enter kernel - does not return

RAX comes back holding the number of bytes written on success, or a negative error code on failure. Those errors are just negated errno values — -EACCES for permission denied, and so on.

This hand-written _start uses exit (60), which ends only the calling thread. When a C program returns from main or calls exit(), glibc uses exit_group (231) instead, which ends every thread in the process. You'll see 231 far more often in strace output.

The System Call Table

Inside the kernel, a table maps syscall numbers to handler functions. On x86-64 Linux that's the sys_call_table, an array of function pointers indexed by syscall number. Nothing fancier than that.

When the kernel receives a syscall:

  1. It reads the number from RAX
  2. Validates that the number is within range
  3. Indexes into sys_call_table to find the handler
  4. Calls the handler with the arguments from the user-mode registers
  5. Places the return value in RAX and executes sysret
bash
# List all system calls on your system
ausyscall --dump
# Trace system calls made by a programme
strace -f ./my_program

The vDSO: Skipping the Kernel

Some syscalls are so frequent and so simple that crossing into the kernel for each one is just wasteful. The classic example is clock_gettime(). Programmes ask for the current time constantly.

Linux provides the vDSO (Virtual Dynamic Shared Object), a tiny shared library the kernel maps into every process's address space. It contains user-space implementations of a handful of syscalls. When a programme calls clock_gettime(), the C library actually hits the vDSO version, which reads a kernel-maintained memory page directly. No privilege transition needed.

For the functions it covers, the vDSO is typically several times to an order of magnitude faster than a real syscall; the exact ratio depends on the CPU and on which mitigations (such as KPTI) the kernel has enabled. It skips the mode switch, the register save/restore, and the entire kernel entry/exit path.

bash
# See the vDSO mapped into a process
cat /proc/self/maps | grep vdso
# List vDSO functions
objdump -T /usr/lib/x86_64-linux-gnu/vdso/vdso64.so

Seccomp: Filtering System Calls

Seccomp (Secure Computing Mode) lets a process voluntarily restrict which syscalls it can make. A seccomp filter is a BPF programme the kernel evaluates on every syscall, deciding whether to allow it, kill the process, or hand back an error.

This is how sandboxing works in Chrome, Docker, and plenty of other security-conscious applications. A sandboxed process might only be allowed read, write, exit, and a handful of other calls. Everything else gets blocked at the boundary.

Why It Matters

The syscall interface is where user-space programmes shake hands with the kernel. Every interaction your application has with the OS passes through it. Reverse engineers learn to spot syscall patterns in assembly the way birdwatchers spot finches. Exploit developers rely on syscalls as the mechanism by which shellcode does anything useful at all. And on the defensive side, seccomp filtering at this layer is one of the strongest sandboxing tools you'll find.