Syscall Mechanisms
System Calls Across Operating Systems · Operating Systems track
10 min read
In this module
The Gateway Between Worlds
A system call is how user-space code asks the kernel to do something. Open a file, allocate memory, fork a process, send a packet — every one of those routes through a syscall. The conceptual idea is universal. The exact way the CPU vaults from user mode into kernel mode? That bit changes across OSes, and across CPU architectures within the same OS.
The Fundamental Problem
User-space code lives at a low privilege ring (ring 3 on x86). It can't directly run privileged instructions or peer at kernel memory. So the CPU has to provide a controlled doorway. That doorway has to:
- Promote the CPU from ring 3 to ring 0
- Land execution at a known kernel entry point (and only that one — no jumping into arbitrary kernel code)
- Stash the user-space context (registers, stack pointer)
- Run whatever kernel function was requested
- Restore user-space context and drop back to ring 3
It also has to do all that quickly. Syscalls happen millions of times a second on a busy system. And it has to do it securely. Sloppy gating here is exactly how privilege escalations get born.
Linux Syscall Mechanism (x86-64)
Modern x86-64 Linux uses the SYSCALL instruction. Here's what actually happens, including two steps that are easy to miss. SYSCALL doesn't switch stacks, so the kernel's first job is to stop using the user's stack. And on CPUs that need KPTI (the Meltdown mitigation), the kernel isn't even mapped until it swaps page tables, so that happens before anything else touches kernel memory:
The two CR3 writes are the KPTI tax mentioned in the previous module's memory-management lesson. On CPUs that aren't vulnerable to Meltdown, Linux skips both.
In raw assembly it looks like this:
; Linux x86-64: write(1, "Hello\n", 6)section .data msg db "Hello", 10 ; 10 = newline
section .textglobal _start_start: mov rax, 1 ; syscall number: sys_write mov rdi, 1 ; arg1: file descriptor (stdout) lea rsi, [rel msg] ; arg2: buffer address mov rdx, 6 ; arg3: byte count syscall ; transition to kernel
mov rax, 60 ; syscall number: sys_exit xor rdi, rdi ; arg1: exit code 0 syscallHere's that program with one extra call: a write to file descriptor 99, which isn't open. Before you run it, guess what lands in rax. The raw kernel interface doesn't use errno; it hands back the negated error number, and it's libc's write() wrapper that turns that into -1 plus errno.
The Syscall Table
Linux keeps a flat array of function pointers, indexed by syscall number:
// Simplified view of the syscall tableconst sys_call_ptr_t sys_call_table[__NR_syscall_max + 1] = { [0] = sys_read, [1] = sys_write, [2] = sys_open, [3] = sys_close, [9] = sys_mmap, [56] = sys_clone, [57] = sys_fork, [59] = sys_execve, [60] = sys_exit, // ... hundreds more};As of Linux 6.x, there are over 450 system calls on x86-64. New syscalls are added conservatively - once added, they can never be removed without breaking backward compatibility.
Linux syscall numbers are part of the kernel's stable ABI. Once assigned, a syscall number never changes. This is why programs compiled decades ago still run on modern kernels. Windows takes a different approach - Nt* syscall numbers can change between Windows versions, which is why user-space code calls through ntdll.dll rather than using raw syscall numbers.
The vDSO: Avoiding the Transition
Some "system calls" do not actually need to enter the kernel. Linux maps a small shared library called the vDSO (virtual Dynamic Shared Object) into every process. It contains user-space implementations of frequently called functions:
// These are handled in user space via vDSO, no kernel transition needed:gettimeofday() // Reads kernel-mapped time dataclock_gettime() // Same mechanismgetcpu() // Reads CPU/NUMA node from userspace-mapped dataThe vDSO reads kernel data that is mapped read-only into user space, avoiding the overhead of a full syscall for these high-frequency operations.
Windows Syscall Mechanism
Windows syscalls follow a different path. User-space applications never call the kernel directly. Instead, they call through a chain of DLLs:
The System Service Descriptor Table (SSDT)
Windows uses the SSDT (System Service Descriptor Table) instead of a simple array. The SSDT contains a table of offsets to kernel functions:
SSDT (KeServiceDescriptorTable):┌──────┬───────────────────────────┐│ # │ Function │├──────┼───────────────────────────┤│ 0x0 │ NtAccessCheck ││ 0x1 │ NtWorkerFactoryWorkerReady││ ... ││ 0x55│ NtCreateFile ││ ... ││ 0x?? │ NtWriteFile │└──────┴───────────────────────────┘Critical difference: Windows syscall numbers are not stable between versions. The 0x55 in the stub above is only an example: the number assigned to a given Nt* function can move between Windows builds and feature releases, and community tables such as j00ru's Windows syscall table track the values per build. You can check yours by disassembling ntdll!NtCreateFile on your own machine. This is why Microsoft requires applications to call through ntdll.dll rather than making raw syscalls - ntdll.dll is updated with each Windows version to contain the correct syscall numbers.
Unlike Linux, Windows syscall numbers can change between builds and feature releases. Applications must call through ntdll.dll, which contains the correct stubs for the running kernel version. Direct syscall invocation (used by some malware and security tools) must dynamically resolve the correct numbers.
WoW64: 32-bit on 64-bit Windows
When a 32-bit application runs on 64-bit Windows, the WoW64 (Windows on Windows 64) subsystem handles the translation. 32-bit syscalls are intercepted and converted:
32-bit App → wow64.dll → wow64cpu.dll → Transition to 64-bit mode │ ▼ ntdll.dll (64-bit) │ ▼ 64-bit kernelThis involves switching the CPU from 32-bit compatibility mode to 64-bit mode, translating pointer sizes, and adapting structure layouts.
macOS Syscall Mechanism
macOS has a unique dual-syscall architecture because XNU combines the Mach microkernel with a BSD layer:
BSD Syscalls
POSIX-compatible syscalls (open, read, write, fork) go through the BSD syscall path. On x86-64, these use the SYSCALL instruction with positive syscall numbers:
; macOS x86-64: write(1, "Hi\n", 3) via BSD syscallmov rax, 0x2000004 ; BSD syscall class (0x2000000) + write (4)mov rdi, 1 ; fd = stdoutlea rsi, [rel msg] ; buffermov rdx, 3 ; countsyscallThe 0x2000000 prefix indicates the BSD syscall class. macOS uses different class prefixes to distinguish syscall types:
Class 0: InvalidClass 1: Mach traps (0x1000000 + trap_number)Class 2: BSD syscalls (0x2000000 + syscall_number)Class 3: Machine-dependentMach Traps
Mach-specific operations (IPC, task/thread management, VM operations) use Mach traps. Each has a small positive index in the kernel's Mach trap table; how you ask for one depends on the architecture. On x86-64 you put 0x1000000 + index in rax (class 1 above); on ARM64 you put the negated index in x16, and the kernel treats negative numbers as Mach traps and positive ones as BSD syscalls:
// Mach trap examples (as passed in x16 on ARM64; x86-64 uses 0x1000000 + 31 and so on)mach_msg_trap() // -31: Send/receive Mach messagesmach_reply_port() // -26: Create a reply porttask_self_trap() // -28: Get current task portthread_self_trap() // -27: Get current thread portmach_vm_allocate_trap() // -10: Allocate virtual memoryMost user applications use the BSD syscall interface through the standard C library (libSystem.dylib on macOS). Mach traps are used by system frameworks and the kernel's own services.
Apple's official position is that libSystem is the ABI: raw syscall numbers aren't a supported interface and can change between releases. That's why Go, which used to make raw syscalls on macOS, switched to calling through libSystem in Go 1.11.
ARM64 (Apple Silicon)
On Apple Silicon Macs, the syscall mechanism uses the SVC (Supervisor Call) instruction:
; Apple Silicon: write(1, buf, len)mov x16, #4 ; syscall number for writemov x0, #1 ; fd = stdoutadr x1, msg ; buffermov x2, #6 ; lengthsvc #0x80 ; supervisor callApple Silicon uses x16 for the syscall number and x0-x7 for arguments, following the ARM64 calling convention.
Predict: how would you write exit(42) as raw assembly on Linux ARM64 and on macOS ARM64?
On Linux ARM64 the syscall number goes in x8 and the trap is svc #0. exit is 93 there (the generic table, not x86-64's 60):
mov x8, #93 ; exitmov x0, #42 ; statussvc #0On macOS ARM64 the number goes in x16 and the trap is svc #0x80. BSD exit is syscall 1:
mov x16, #1 ; exitmov x0, #42 ; statussvc #0x80Run either and echo $? prints 42. The x86-64 Linux version in the runner above uses rax = 60 and syscall. Three platforms, three numbers, three registers: that's why portable code goes through libc.
Comparing the Mechanisms
| Aspect | Linux | Windows | macOS |
|---|---|---|---|
| Instruction | SYSCALL | SYSCALL (via ntdll stub) | SYSCALL (x86-64) / SVC (ARM64) |
| Syscall # register | RAX | EAX | RAX (x86-64) / X16 (ARM64) |
| Numbers stable? | Yes (ABI guarantee) | No (change between versions) | Not guaranteed (only libSystem is supported) |
| Table | sys_call_table | SSDT | Mach trap table + BSD sysent |
| User-space fast path | vDSO | SharedUserData page | commpage |
| Dual interfaces | No | No | Yes (Mach traps + BSD syscalls) |
Understanding the syscall mechanism is fundamental to systems programming, reverse engineering, and security research. It is the lowest-level interface between your code and the operating system, and every higher-level API ultimately reduces to these raw transitions between user mode and kernel mode.