Comparing Syscall ABIs Across OSes
Syscall Interfaces & the Kernel Boundary · OS Core Internals track
4 min read
In this module
Three Operating Systems, Three Approaches
Linux, Windows, and macOS all face the same basic problem: how does user-mode code ask the kernel to do something? Each one picks a different answer, and those answers reveal a lot about each project's priorities — stability, compatibility, security. Lining the three up side by side surfaces trade-offs that bite hard in cross-platform security work.
Invoking a System Call
All three platforms use the syscall instruction on x86-64, but the register conventions differ:
| Linux | Windows | macOS | |
|---|---|---|---|
| Instruction | syscall | syscall (via ntdll) | syscall |
| Number register | RAX | EAX | RAX (with class bits) |
| Arg 1 | RDI | RCX → R10 | RDI |
| Arg 2 | RSI | RDX | RSI |
| Arg 3 | RDX | R8 | RDX |
| Arg 4 | R10 | R9 | R10 |
| Arg 5 | R8 | stack | R8 |
| Arg 6 | R9 | stack | R9 |
| Return | RAX | RAX | RAX |
| Clobbered | RCX, R11 | RCX, R11 | RCX, R11 |
Linux and macOS share the same argument register order, following the System V AMD64 ABI, while Windows goes its own way. All three clobber RCX and R11, because that's a hardware behaviour of the syscall instruction and nothing to do with OS design.
All three major operating systems use the same syscall instruction on x86-64, but they differ in register conventions, number stability, and architectural philosophy. Linux exposes a stable, documented ABI. Windows hides its numbers behind ntdll. macOS uses class-encoded numbers that distinguish BSD syscalls from Mach traps.
Syscall Number Stability
This is where the three platforms diverge most dramatically:
Linux: syscall numbers are part of the stable kernel ABI. Once a number is assigned, that's it — it never changes. write is syscall 1 on x86-64 today, and it'll be syscall 1 forever. The stability is a deliberate promise from the kernel maintainers.
Windows: syscall numbers are internal implementation details that drift between Windows versions, service packs, and even individual patches. Microsoft's stability contract is at the Win32 API level. Calling syscalls directly is unsupported, and it breaks.
macOS: Apple treats the syscall interface as private and unsupported. The blessed path is via libSystem. Apple reserves the right to change syscall numbers and semantics at any point, and they've made good on that across macOS versions.
macOS: A Dual-Personality Kernel
macOS has a unique syscall architecture because its kernel, XNU, is a hybrid of two systems:
- BSD layer - provides POSIX-compatible system calls (file I/O, process management, networking)
- Mach layer - provides microkernel-style primitives (ports, messages, tasks, memory objects)
The syscall number encodes which subsystem to dispatch to using class bits in the upper portion of the number:
- Class 1 (Mach traps) - numbers are
0x1000000 + trap_number - Class 2 (BSD syscalls) - numbers are
0x2000000 + syscall_number
So a BSD write call on macOS uses syscall number 0x2000004 (class 2, BSD syscall 4), not simply 4.
Error Reporting
Error handling conventions also differ:
- Linux: returns a negative value in RAX on error (e.g.,
-ENOENTfor "file not found"). The C library converts that into settingerrnoand returning -1. - Windows: returns an
NTSTATUScode in RAX. Success is typically 0 (STATUS_SUCCESS); errors carry the high bit. The Win32 layer translates these intoGetLastError()codes. - macOS: hijacks the carry flag (CF) for error signalling. CF set means RAX holds the error number. CF clear means RAX holds the return value.
Explore the Differences
The first viewer below takes one operation, writing to stdout, and shows the syscall number, register layout and calling convention on each platform. Switch on side-by-side mode and the rows that differ light up.
Syscall: write()
Linux write()
Stable syscall number. Arguments passed in registers following the System V AMD64 ABI. The syscall instruction saves RIP to RCX and RFLAGS to R11.
mov rax, 1 ; syscall number (stable)
mov rdi, 1 ; fd = stdout
mov rsi, msg ; buffer
mov rdx, len ; count
syscallThe second does the same for allocating memory with mmap (or its Windows counterpart), and adds the arm64 rows the table above leaves out. Compare where the number lives on each: Linux uses x8, macOS uses x16 with no class bits, and Windows puts it in the svc instruction itself.
Syscall: mmap() and its Windows counterpart
Linux mmap()
Six arguments, all in registers on both architectures. The fourth argument (flags) goes in R10 on x86-64 because syscall overwrites RCX. The arm64 number comes from the generic syscall table, so it differs from x86-64.
; x86-64
mov rax, 9 ; mmap
syscall
; arm64
mov x8, #222 ; mmap
svc #0Security Implications
These architectural differences have direct security consequences:
Shellcode portability: syscall-based shellcode is inherently platform-specific. Different numbers, different registers, different error conventions.
Hooking and monitoring: on Linux, seccomp filters operate on stable syscall numbers. On Windows, EDR products hook ntdll because the syscall numbers are unstable - direct syscalls bypass these hooks.
Analysis complexity: macOS binaries may use both Mach traps and BSD syscalls, requiring analysts to understand two dispatch mechanisms within a single binary.
Why It Matters
Cross-platform security work demands an ABI-level understanding of these differences. Writing portable exploit code, building a cross-platform sandbox, analysing malware that targets multiple operating systems — wherever you start, the syscall interface is where the analysis grounds out. The comparison here gives you a mental map for translating the same problem across three very different solutions.