Skip to main content
Be Bitwise

What Are System Calls?

System Calls & Processes · Assembly & OS track

4 min read

In this module

The Gateway to the Kernel

In the previous module, you learned that user programs can't directly touch hardware. They run in user space with restricted privileges. So how does a program read a file, write to the screen, or create a new process? It makes a system call — a syscall.

A syscall is a formal request from user code to the kernel. It's the API between your program and the operating system. Every meaningful interaction with the outside world goes through this chokepoint.

How a Syscall Works

When your program calls a function like read() in C, here is what actually happens under the hood:

  1. The C library function places the syscall number into a specific CPU register (on x86-64 Linux, this is rax)
  2. Arguments (file descriptor, buffer address, byte count) go into other designated registers
  3. The program executes a special trap instruction (syscall on x86-64)
  4. The CPU switches from user mode to kernel mode
  5. The kernel looks up the syscall number in a table and calls the corresponding kernel function
  6. The kernel performs the operation and places the return value in rax (on failure, a negative error code such as -2 for ENOENT; the C library wrapper turns that into a return of -1 and stores the code in errno)
  7. Control returns to user space

Here are the same seven steps for one concrete call, read(3, buf, 100) on x86-64 Linux, with the register contents and the CPU's privilege level (CPL) at each point. Ring 3 is user mode and ring 0 is kernel mode.

The Trap Instruction

The trap instruction is what makes syscalls possible. It is a CPU instruction that intentionally triggers a controlled exception, causing the processor to switch to kernel mode and jump to a predefined handler. Unlike a regular function call, it crosses the user/kernel boundary - changing the CPU's privilege level in the process.

Worked Example: write(1, "hi\n", 3)

Take the smallest useful program: print hi and a newline. In C you'd write write(1, "hi\n", 3), meaning "send 3 bytes from this string to file descriptor 1", which is standard output. Here's what the libc wrapper loads before it executes syscall on x86-64 Linux:

RegisterValueMeaning
rax1syscall number for write
rdi1first argument: file descriptor 1
rsi0x402004second argument: address of the string (illustrative)
rdx3third argument: byte count

The kernel checks that descriptor 1 is open for writing and that the 3 bytes at 0x402004 really belong to the process, copies them to the terminal and puts 3 in rax, the number of bytes written. If you run the program under strace, which prints every syscall a process makes, you'll see that exact call decoded:

c
write(1, "hi\n", 3) = 3

Now change the descriptor to 99, which the program never opened. The kernel returns -9 in rax, because 9 is the error number EBADF ("bad file descriptor") on Linux. The libc wrapper spots the negative value, stores 9 in errno and returns -1 to your code. strace shows both halves of that at once: write(99, "hi\n", 3) = -1 EBADF (Bad file descriptor).

Predict what each call prints before you run it, including the errno value for the second one.

Try itRun it, then try a byte count of 2 in the first call, or a negative descriptor such as -1 in the second.
C (GCC 9.2.0)

printf is a syscall user too. It collects its text in a libc buffer and, when the output isn't a terminal, may not call write until the program exits. Redirect the output to a file under strace and you'd see the hi written by itself, then both printf lines arrive together in one final write.

Common System Calls

Most operating systems provide hundreds of syscalls, but a handful come up constantly:

  • read - read bytes from a file descriptor
  • write - write bytes to a file descriptor
  • open - open a file and get a file descriptor
  • close - release a file descriptor
  • fork - create a new process by duplicating the current one
  • execve - replace the current process image with a new program (the C library's exec* family, such as execl and execvp, all end up here)
  • exit - terminate the current process

Why Not Just Call the Kernel Directly?

The trap instruction means the kernel is always entered through a single controlled gate. On x86-64 the address syscall jumps to sits in a model-specific register (IA32_LSTAR) that only ring 0 can write, so user code has no say in where kernel execution starts. The kernel validates every argument before acting on it. No user program can jump to an arbitrary kernel address — the hardware won't permit it.

Even a malicious program can't sidestep these protections, because the only way in is a door the kernel built and guards.