User-Kernel Transition Mechanics
Syscall Interfaces & the Kernel Boundary · OS Core Internals track
5 min read
In this module
Two Worlds
A modern CPU operates in (at least) two privilege levels. User mode is where your applications run, with restricted access to hardware and memory. Kernel mode is where the operating system runs, with full access to everything. The boundary between them is the most important security boundary on the system.
When a programme needs to do something privileged (reading a file, allocating memory, sending a network packet), it can't do it directly. It has to ask the kernel. The mechanism for making that request is a system call, and the transition from user mode to kernel mode is one of the most tightly choreographed operations in the entire system.
Privilege Rings
The x86 architecture defines four protection rings, numbered 0 through 3. Ring 0 has the highest privilege (full hardware access), and Ring 3 has the least.
In practice, modern operating systems use only two:
- Ring 0 - kernel mode. The kernel, drivers, and core OS code run here.
- Ring 3 - user mode. All applications run here.
The CPU enforces this separation in hardware. Code running in Ring 3 can't execute privileged instructions (like modifying page tables or disabling interrupts), can't access kernel memory (the page table entries for kernel pages have the User/Supervisor bit cleared), and can't talk directly to hardware I/O ports.
The CPU enforces a hardware boundary between user mode (Ring 3) and kernel mode (Ring 0). Applications can't reach hardware, kernel memory, or privileged instructions directly. They have to cross this boundary via a controlled system call interface, which validates every request before acting on it.
The Mode Switch
When a programme executes a syscall instruction (on x86-64), the CPU runs through a precise sequence:
- Saves the return address - the current value of RIP is saved into RCX
- Saves the flags - the current RFLAGS is saved into R11
- Loads the kernel entry point - from the
LSTARmodel-specific register (MSR), which the kernel configured at boot - Changes privilege level - the CPL (Current Privilege Level) flips from 3 to 0
- Masks interrupts - specific RFLAGS bits are cleared via the
SFMASKMSR
Notice what's missing from that list: the stack. syscall leaves RSP pointing at the user's stack, and the kernel's entry code has to deal with that itself (next section).
The kernel is now running. It saves the remaining user-mode registers, looks up the requested system call, validates the arguments, performs the operation, and finally executes sysret to return to user mode.
The Kernel Stack
Each thread has two stacks: a user stack (in user-mode memory) and a kernel stack (in kernel memory). The kernel must never run on the user stack: another user thread could rewrite it mid-system-call, or the programme could simply point RSP at garbage before calling in.
How the switch happens depends on how the kernel was entered. For interrupts and exceptions, the hardware switches stacks itself, loading a kernel stack pointer from the Task State Segment (TSS). syscall was designed to be fast and does no such thing, so Linux's entry stub (entry_SYSCALL_64) does it by hand in its first few instructions:
swapgsswaps the user's GS base for the kernel's, which points at this CPU's private data area.- The user RSP is stashed in that per-CPU area.
- With page table isolation (KPTI, the Meltdown mitigation) enabled, CR3 is switched from the trimmed user page tables to the full kernel ones.
- RSP is loaded with this thread's kernel stack.
- Only then are the user registers pushed onto the kernel stack.
Until step 4 completes, the kernel can't safely touch the stack at all, which is why this stretch is hand-written assembly. Two more hardware features guard the boundary once inside: SMEP stops the kernel executing user pages, and SMAP stops it reading or writing them except in code that explicitly asks to, such as copy_from_user.
The kernel stack is small (16 KB on x86-64 Linux) because kernel code paths are designed to stay shallow. Each thread's task_struct holds a pointer to its kernel stack.
Why Not Just a Function Call?
You might wonder why system calls need such elaborate machinery. Why not just call a kernel function directly?
The answer is privilege isolation. A regular function call doesn't change the CPU's privilege level. If user-mode code could call kernel functions directly, it would have to run at kernel privilege, which defeats the whole point of protection rings. The syscall instruction is the controlled gate the hardware provides, and the kernel decides exactly which entry point it opens onto.
Historical Context
Before syscall existed, x86 systems used int 0x80 (a software interrupt) to enter the kernel. Slower, because it went through the full interrupt descriptor table (IDT) lookup and saved more state. The syscall instruction, introduced with AMD's x86-64 architecture, is a purpose-built mechanism that skips the interrupt path's overhead.
On 32-bit x86, Linux used int 0x80. Some 32-bit systems also used sysenter/sysexit (an Intel optimisation). On 64-bit x86-64, syscall/sysret is the standard.
Why It Matters
The user-kernel boundary is the attack surface for privilege escalation. Every system call is a potential entry point where user-controlled data crosses into kernel space. Fail to validate arguments properly and an attacker can corrupt kernel memory, escalate privileges, or break out of a sandbox. Understanding how this transition works at the hardware level is step one for both kernel exploitation and kernel hardening.