Skip to main content
Be Bitwise
x86 Assemblyx86 Basics & Environment

Your First Syscall

A syscall (system call) is how your program asks the kernel to do something for it. Without syscalls, all your assembly can manage is shuffling bits between registers and memory. No printing, no reading input, no files, no allocations, nothing that touches the outside world.

Syscalls are the boundary between user-mode and kernel-mode code. Every conversation your program has with the OS passes through that one gate.

How Syscalls Work

On Linux x86-64, the syscall interface obeys a fixed register convention:

RegisterPurpose
raxSyscall number (which operation to perform)
rdiFirst argument
rsiSecond argument
rdxThird argument
r10Fourth argument
r8Fifth argument
r9Sixth argument
rax (after)Return value (negative values indicate errors)

syscall hands control over to the kernel. It reads rax to figure out which operation you asked for, runs it, drops the result back into rax, and bounces you home.

Here's one write(1, msg, len) crossing that boundary and coming back. Two details tend to surprise people. The CPU uses rcx and r11 to remember where you were, so both are trashed by every syscall. And the kernel's entry address comes from a model-specific register the kernel set up at boot, so user code can't pick where it lands.

User Mode vs Kernel Mode

The CPU runs in two privilege levels. Your assembly code runs in user mode (ring 3), where it cannot directly access hardware or other processes' memory. The kernel runs in kernel mode (ring 0), where it has full access to everything. The syscall instruction is the controlled gateway between these modes. The CPU switches privilege levels, executes the kernel handler, then switches back. This boundary is fundamental to OS security - if user code could bypass it, all process isolation would collapse.

Essential Syscalls

The handful you'll reach for constantly:

sys_write (number 1) - Write to a File Descriptor

nasm
; ssize_t write(int fd, const void *buf, size_t count)
mov rax, 1 ; sys_write
mov rdi, 1 ; fd = 1 (stdout), use 2 for stderr
mov rsi, msg ; pointer to buffer
mov rdx, msg_len ; number of bytes to write
syscall ; returns bytes written in rax, or negative on error

sys_read (number 0) - Read from a File Descriptor

nasm
; ssize_t read(int fd, void *buf, size_t count)
mov rax, 0 ; sys_read
mov rdi, 0 ; fd = 0 (stdin)
mov rsi, buffer ; pointer to destination buffer
mov rdx, 256 ; maximum bytes to read
syscall ; returns bytes read in rax, or negative on error

sys_exit (number 60) - Terminate the Process

nasm
; void exit(int status)
mov rax, 60 ; sys_exit
mov rdi, 0 ; exit status (0 = success)
syscall ; does not return

A Complete Read-Write Program

A program that reads a line from the user and echoes it straight back:

nasm
SYS_READ equ 0
SYS_WRITE equ 1
SYS_EXIT equ 60
STDIN equ 0
STDOUT equ 1
section .data
prompt db "Enter text: "
prompt_len equ $ - prompt
reply db "You said: "
reply_len equ $ - reply
section .bss
input resb 256 ; buffer for user input
section .text
global _start
_start:
; Print the prompt
mov rax, SYS_WRITE
mov rdi, STDOUT
mov rsi, prompt
mov rdx, prompt_len
syscall
; Read user input
mov rax, SYS_READ
mov rdi, STDIN
mov rsi, input
mov rdx, 256
syscall
; Save the number of bytes read
mov r12, rax ; r12 is callee-saved, safe to keep here
; Print "You said: " prefix
mov rax, SYS_WRITE
mov rdi, STDOUT
mov rsi, reply
mov rdx, reply_len
syscall
; Print the user's input
mov rax, SYS_WRITE
mov rdi, STDOUT
mov rsi, input
mov rdx, r12 ; number of bytes the user typed
syscall
; Exit
mov rax, SYS_EXIT
xor rdi, rdi
syscall

Building and Running

Two steps build any standalone assembly program:

bash
nasm -f elf64 echo.asm -o echo.o # step 1: assemble
ld echo.o -o echo # step 2: link
./echo # run it

nasm -f elf64 translates assembly source into a 64-bit ELF object file (.o). Machine code, symbols, no executable bit yet.

ld is the GNU linker. It resolves symbols (_start, friends), pins down memory addresses, and produces the final ELF executable. For a standalone program with no C library in the picture, ld is the whole show.

Why Not GCC?

gcc can't assemble this file: its assembler is GNU as, which expects AT&T or GAS-flavoured Intel syntax, not NASM. You can still let gcc drive the link step, though: nasm -f elf64 echo.asm -o echo.o followed by gcc -nostdlib -no-pie echo.o -o echo. Running nasm and ld by hand gives you more control and shows what each step actually does. Once you start linking against the C library in later modules, gcc as the linker driver becomes the convenient choice.

Syscall Return Values and Error Handling

Syscalls hand the result back in rax. On success the meaning depends on the syscall (sys_read returns the bytes read, for example). On failure rax carries a negative errno value.

nasm
; Read from stdin
mov rax, SYS_READ
mov rdi, STDIN
mov rsi, input
mov rdx, 256
syscall
; Check for error
test rax, rax ; sets SF if rax is negative
js read_error ; jump if sign flag is set (negative return)
; rax now holds the number of bytes read
jmp read_success
read_error:
neg rax ; convert -errno to positive errno value
; handle the error...
read_success:
; process the input...

Common error codes worth memorising: -1 (EPERM, operation not permitted), -13 (EACCES, permission denied), -2 (ENOENT, file not found), -9 (EBADF, bad file descriptor), -14 (EFAULT, bad address).

Finding Syscall Numbers

Syscall numbers live in the kernel headers. The path depends on the distro:

bash
cat /usr/include/x86_64-linux-gnu/asm/unistd_64.h | head -30 # Debian, Ubuntu
cat /usr/include/asm/unistd_64.h | head -30 # Fedora, Arch

Or just keep a reference table at hand. The greatest hits:

NumberNamePurpose
0sys_readRead from file descriptor
1sys_writeWrite to file descriptor
2sys_openOpen a file
3sys_closeClose a file descriptor
9sys_mmapMap memory
12sys_brkChange data segment size
57sys_forkCreate a child process
59sys_execveExecute a program
60sys_exitTerminate process
Security Note: Syscalls as the Attack Surface

Syscalls are the primary attack surface between user programs and the kernel. Every syscall is a potential entry point for kernel exploitation. Techniques like seccomp (secure computing mode) restrict which syscalls a process can make, dramatically reducing the attack surface. In sandboxed environments (like containers or browser processes), syscall filtering is a critical defense layer. Tools like strace let you observe every syscall a program makes - an essential technique in both debugging and malware analysis.

Tracing Syscalls

Watch every syscall a program issues with strace:

bash
strace ./echo

Each call shows up with its arguments and return value. Brilliant for debugging assembly, and just as good for unmasking what a compiled binary really does at the OS layer. When an unknown binary lands on your desk, strace is usually the first move.

What is Next

Registers, memory sections, syscalls — the foundations are in place. The next module covers the core instructions for moving data and doing arithmetic, the raw atoms every other operation will be built from.

Challenges

Write an x86-64 NASM program that prints "Hello, x86!" followed by a newline to stdout using the write syscall, then exits cleanly with exit code 0.

Assembly (NASM 2.14.02)