Skip to main content
Be Bitwise

Comparing Syscall ABIs Across OSes

Syscall Interfaces & the Kernel Boundary · OS Core Internals track

4 min read

In this module

Three Operating Systems, Three Approaches

Linux, Windows, and macOS all face the same basic problem: how does user-mode code ask the kernel to do something? Each one picks a different answer, and those answers reveal a lot about each project's priorities — stability, compatibility, security. Lining the three up side by side surfaces trade-offs that bite hard in cross-platform security work.

Invoking a System Call

All three platforms use the syscall instruction on x86-64, but the register conventions differ:

LinuxWindowsmacOS
Instructionsyscallsyscall (via ntdll)syscall
Number registerRAXEAXRAX (with class bits)
Arg 1RDIRCX → R10RDI
Arg 2RSIRDXRSI
Arg 3RDXR8RDX
Arg 4R10R9R10
Arg 5R8stackR8
Arg 6R9stackR9
ReturnRAXRAXRAX
ClobberedRCX, R11RCX, R11RCX, R11

Linux and macOS share the same argument register order, following the System V AMD64 ABI, while Windows goes its own way. All three clobber RCX and R11, because that's a hardware behaviour of the syscall instruction and nothing to do with OS design.

Key Concept

All three major operating systems use the same syscall instruction on x86-64, but they differ in register conventions, number stability, and architectural philosophy. Linux exposes a stable, documented ABI. Windows hides its numbers behind ntdll. macOS uses class-encoded numbers that distinguish BSD syscalls from Mach traps.

Syscall Number Stability

This is where the three platforms diverge most dramatically:

Linux: syscall numbers are part of the stable kernel ABI. Once a number is assigned, that's it — it never changes. write is syscall 1 on x86-64 today, and it'll be syscall 1 forever. The stability is a deliberate promise from the kernel maintainers.

Windows: syscall numbers are internal implementation details that drift between Windows versions, service packs, and even individual patches. Microsoft's stability contract is at the Win32 API level. Calling syscalls directly is unsupported, and it breaks.

macOS: Apple treats the syscall interface as private and unsupported. The blessed path is via libSystem. Apple reserves the right to change syscall numbers and semantics at any point, and they've made good on that across macOS versions.

macOS: A Dual-Personality Kernel

macOS has a unique syscall architecture because its kernel, XNU, is a hybrid of two systems:

  • BSD layer - provides POSIX-compatible system calls (file I/O, process management, networking)
  • Mach layer - provides microkernel-style primitives (ports, messages, tasks, memory objects)

The syscall number encodes which subsystem to dispatch to using class bits in the upper portion of the number:

  • Class 1 (Mach traps) - numbers are 0x1000000 + trap_number
  • Class 2 (BSD syscalls) - numbers are 0x2000000 + syscall_number

So a BSD write call on macOS uses syscall number 0x2000004 (class 2, BSD syscall 4), not simply 4.

Error Reporting

Error handling conventions also differ:

  • Linux: returns a negative value in RAX on error (e.g., -ENOENT for "file not found"). The C library converts that into setting errno and returning -1.
  • Windows: returns an NTSTATUS code in RAX. Success is typically 0 (STATUS_SUCCESS); errors carry the high bit. The Win32 layer translates these into GetLastError() codes.
  • macOS: hijacks the carry flag (CF) for error signalling. CF set means RAX holds the error number. CF clear means RAX holds the return value.

Explore the Differences

The first viewer below takes one operation, writing to stdout, and shows the syscall number, register layout and calling convention on each platform. Switch on side-by-side mode and the rows that differ light up.

Syscall: write()

Lx

Linux write()

Stable syscall number. Arguments passed in registers following the System V AMD64 ABI. The syscall instruction saves RIP to RCX and RFLAGS to R11.

Linux
mov rax, 1        ; syscall number (stable)
mov rdi, 1        ; fd = stdout
mov rsi, msg      ; buffer
mov rdx, len      ; count
syscall
Syscall #1 (stable across all versions)
Instructionsyscall
Number regRAX
Arg registersRDI, RSI, RDX, R10, R8, R9
ReturnRAX (negative errno on error)
ClobberedRCX, R11

The second does the same for allocating memory with mmap (or its Windows counterpart), and adds the arm64 rows the table above leaves out. Compare where the number lives on each: Linux uses x8, macOS uses x16 with no class bits, and Windows puts it in the svc instruction itself.

Syscall: mmap() and its Windows counterpart

Lx

Linux mmap()

Six arguments, all in registers on both architectures. The fourth argument (flags) goes in R10 on x86-64 because syscall overwrites RCX. The arm64 number comes from the generic syscall table, so it differs from x86-64.

Linux
; x86-64
mov rax, 9      ; mmap
syscall

; arm64
mov x8, #222    ; mmap
svc #0
x86-64 number9 (RAX)
x86-64 argsRDI, RSI, RDX, R10, R8, R9
arm64 instructionsvc #0
arm64 number222 in x8
arm64 argsx0 to x5
Error signalNegative errno in RAX / x0

Security Implications

These architectural differences have direct security consequences:

Shellcode portability: syscall-based shellcode is inherently platform-specific. Different numbers, different registers, different error conventions.

Hooking and monitoring: on Linux, seccomp filters operate on stable syscall numbers. On Windows, EDR products hook ntdll because the syscall numbers are unstable - direct syscalls bypass these hooks.

Analysis complexity: macOS binaries may use both Mach traps and BSD syscalls, requiring analysts to understand two dispatch mechanisms within a single binary.

Why It Matters

Cross-platform security work demands an ABI-level understanding of these differences. Writing portable exploit code, building a cross-platform sandbox, analysing malware that targets multiple operating systems — wherever you start, the syscall interface is where the analysis grounds out. The comparison here gives you a mental map for translating the same problem across three very different solutions.