Skip to main content
Be Bitwise

Syscall Mechanisms

System Calls Across Operating Systems · Operating Systems track

10 min read

In this module

The Gateway Between Worlds

A system call is how user-space code asks the kernel to do something. Open a file, allocate memory, fork a process, send a packet — every one of those routes through a syscall. The conceptual idea is universal. The exact way the CPU vaults from user mode into kernel mode? That bit changes across OSes, and across CPU architectures within the same OS.

The Fundamental Problem

User-space code lives at a low privilege ring (ring 3 on x86). It can't directly run privileged instructions or peer at kernel memory. So the CPU has to provide a controlled doorway. That doorway has to:

  1. Promote the CPU from ring 3 to ring 0
  2. Land execution at a known kernel entry point (and only that one — no jumping into arbitrary kernel code)
  3. Stash the user-space context (registers, stack pointer)
  4. Run whatever kernel function was requested
  5. Restore user-space context and drop back to ring 3

It also has to do all that quickly. Syscalls happen millions of times a second on a busy system. And it has to do it securely. Sloppy gating here is exactly how privilege escalations get born.

Linux Syscall Mechanism (x86-64)

Modern x86-64 Linux uses the SYSCALL instruction. Here's what actually happens, including two steps that are easy to miss. SYSCALL doesn't switch stacks, so the kernel's first job is to stop using the user's stack. And on CPUs that need KPTI (the Meltdown mitigation), the kernel isn't even mapped until it swaps page tables, so that happens before anything else touches kernel memory:

The two CR3 writes are the KPTI tax mentioned in the previous module's memory-management lesson. On CPUs that aren't vulnerable to Meltdown, Linux skips both.

In raw assembly it looks like this:

asm
; Linux x86-64: write(1, "Hello\n", 6)
section .data
msg db "Hello", 10 ; 10 = newline
section .text
global _start
_start:
mov rax, 1 ; syscall number: sys_write
mov rdi, 1 ; arg1: file descriptor (stdout)
lea rsi, [rel msg] ; arg2: buffer address
mov rdx, 6 ; arg3: byte count
syscall ; transition to kernel
mov rax, 60 ; syscall number: sys_exit
xor rdi, rdi ; arg1: exit code 0
syscall

Here's that program with one extra call: a write to file descriptor 99, which isn't open. Before you run it, guess what lands in rax. The raw kernel interface doesn't use errno; it hands back the negated error number, and it's libc's write() wrapper that turns that into -1 plus errno.

Try itRun it, then change the second write's fd from 99 back to 1 and see the error line turn into a second Hello.
Assembly (NASM 2.14.02)

The Syscall Table

Linux keeps a flat array of function pointers, indexed by syscall number:

c
// Simplified view of the syscall table
const sys_call_ptr_t sys_call_table[__NR_syscall_max + 1] = {
[0] = sys_read,
[1] = sys_write,
[2] = sys_open,
[3] = sys_close,
[9] = sys_mmap,
[56] = sys_clone,
[57] = sys_fork,
[59] = sys_execve,
[60] = sys_exit,
// ... hundreds more
};

As of Linux 6.x, there are over 450 system calls on x86-64. New syscalls are added conservatively - once added, they can never be removed without breaking backward compatibility.

Syscall Number Stability

Linux syscall numbers are part of the kernel's stable ABI. Once assigned, a syscall number never changes. This is why programs compiled decades ago still run on modern kernels. Windows takes a different approach - Nt* syscall numbers can change between Windows versions, which is why user-space code calls through ntdll.dll rather than using raw syscall numbers.

The vDSO: Avoiding the Transition

Some "system calls" do not actually need to enter the kernel. Linux maps a small shared library called the vDSO (virtual Dynamic Shared Object) into every process. It contains user-space implementations of frequently called functions:

text
// These are handled in user space via vDSO, no kernel transition needed:
gettimeofday() // Reads kernel-mapped time data
clock_gettime() // Same mechanism
getcpu() // Reads CPU/NUMA node from userspace-mapped data

The vDSO reads kernel data that is mapped read-only into user space, avoiding the overhead of a full syscall for these high-frequency operations.

Windows Syscall Mechanism

Windows syscalls follow a different path. User-space applications never call the kernel directly. Instead, they call through a chain of DLLs:

The System Service Descriptor Table (SSDT)

Windows uses the SSDT (System Service Descriptor Table) instead of a simple array. The SSDT contains a table of offsets to kernel functions:

text
SSDT (KeServiceDescriptorTable):
┌──────┬───────────────────────────┐
│ # │ Function │
├──────┼───────────────────────────┤
│ 0x0 │ NtAccessCheck │
│ 0x1 │ NtWorkerFactoryWorkerReady│
│ ... │
│ 0x55│ NtCreateFile │
│ ... │
│ 0x?? │ NtWriteFile │
└──────┴───────────────────────────┘

Critical difference: Windows syscall numbers are not stable between versions. The 0x55 in the stub above is only an example: the number assigned to a given Nt* function can move between Windows builds and feature releases, and community tables such as j00ru's Windows syscall table track the values per build. You can check yours by disassembling ntdll!NtCreateFile on your own machine. This is why Microsoft requires applications to call through ntdll.dll rather than making raw syscalls - ntdll.dll is updated with each Windows version to contain the correct syscall numbers.

Key Concept
Unstable Syscall Numbers

Unlike Linux, Windows syscall numbers can change between builds and feature releases. Applications must call through ntdll.dll, which contains the correct stubs for the running kernel version. Direct syscall invocation (used by some malware and security tools) must dynamically resolve the correct numbers.

WoW64: 32-bit on 64-bit Windows

When a 32-bit application runs on 64-bit Windows, the WoW64 (Windows on Windows 64) subsystem handles the translation. 32-bit syscalls are intercepted and converted:

text
32-bit App → wow64.dll → wow64cpu.dll → Transition to 64-bit mode
│
▼
ntdll.dll (64-bit)
│
▼
64-bit kernel

This involves switching the CPU from 32-bit compatibility mode to 64-bit mode, translating pointer sizes, and adapting structure layouts.

macOS Syscall Mechanism

macOS has a unique dual-syscall architecture because XNU combines the Mach microkernel with a BSD layer:

BSD Syscalls

POSIX-compatible syscalls (open, read, write, fork) go through the BSD syscall path. On x86-64, these use the SYSCALL instruction with positive syscall numbers:

asm
; macOS x86-64: write(1, "Hi\n", 3) via BSD syscall
mov rax, 0x2000004 ; BSD syscall class (0x2000000) + write (4)
mov rdi, 1 ; fd = stdout
lea rsi, [rel msg] ; buffer
mov rdx, 3 ; count
syscall

The 0x2000000 prefix indicates the BSD syscall class. macOS uses different class prefixes to distinguish syscall types:

text
Class 0: Invalid
Class 1: Mach traps (0x1000000 + trap_number)
Class 2: BSD syscalls (0x2000000 + syscall_number)
Class 3: Machine-dependent

Mach Traps

Mach-specific operations (IPC, task/thread management, VM operations) use Mach traps. Each has a small positive index in the kernel's Mach trap table; how you ask for one depends on the architecture. On x86-64 you put 0x1000000 + index in rax (class 1 above); on ARM64 you put the negated index in x16, and the kernel treats negative numbers as Mach traps and positive ones as BSD syscalls:

text
// Mach trap examples (as passed in x16 on ARM64; x86-64 uses 0x1000000 + 31 and so on)
mach_msg_trap() // -31: Send/receive Mach messages
mach_reply_port() // -26: Create a reply port
task_self_trap() // -28: Get current task port
thread_self_trap() // -27: Get current thread port
mach_vm_allocate_trap() // -10: Allocate virtual memory

Most user applications use the BSD syscall interface through the standard C library (libSystem.dylib on macOS). Mach traps are used by system frameworks and the kernel's own services.

Apple's official position is that libSystem is the ABI: raw syscall numbers aren't a supported interface and can change between releases. That's why Go, which used to make raw syscalls on macOS, switched to calling through libSystem in Go 1.11.

ARM64 (Apple Silicon)

On Apple Silicon Macs, the syscall mechanism uses the SVC (Supervisor Call) instruction:

asm
; Apple Silicon: write(1, buf, len)
mov x16, #4 ; syscall number for write
mov x0, #1 ; fd = stdout
adr x1, msg ; buffer
mov x2, #6 ; length
svc #0x80 ; supervisor call

Apple Silicon uses x16 for the syscall number and x0-x7 for arguments, following the ARM64 calling convention.

Predict: how would you write exit(42) as raw assembly on Linux ARM64 and on macOS ARM64?

On Linux ARM64 the syscall number goes in x8 and the trap is svc #0. exit is 93 there (the generic table, not x86-64's 60):

asm
mov x8, #93 ; exit
mov x0, #42 ; status
svc #0

On macOS ARM64 the number goes in x16 and the trap is svc #0x80. BSD exit is syscall 1:

asm
mov x16, #1 ; exit
mov x0, #42 ; status
svc #0x80

Run either and echo $? prints 42. The x86-64 Linux version in the runner above uses rax = 60 and syscall. Three platforms, three numbers, three registers: that's why portable code goes through libc.

Comparing the Mechanisms

AspectLinuxWindowsmacOS
InstructionSYSCALLSYSCALL (via ntdll stub)SYSCALL (x86-64) / SVC (ARM64)
Syscall # registerRAXEAXRAX (x86-64) / X16 (ARM64)
Numbers stable?Yes (ABI guarantee)No (change between versions)Not guaranteed (only libSystem is supported)
Tablesys_call_tableSSDTMach trap table + BSD sysent
User-space fast pathvDSOSharedUserData pagecommpage
Dual interfacesNoNoYes (Mach traps + BSD syscalls)

Understanding the syscall mechanism is fundamental to systems programming, reverse engineering, and security research. It is the lowest-level interface between your code and the operating system, and every higher-level API ultimately reduces to these raw transitions between user mode and kernel mode.