Process & Memory Syscalls Compared
System Calls Across Operating Systems · Operating Systems track
12 min read
In this module
Creating Processes and Managing Memory at the Kernel Level
Last lesson covered how the CPU flips between user and kernel mode. Now for the workhorses: the actual syscalls that spawn processes, fork threads, and carve up virtual memory on Linux and Windows. Every program you've ever run sits on top of these.
Process Creation: fork/exec vs NtCreateProcess
Linux: fork() and execve()
Linux splits process creation into two steps. fork() (which actually wraps clone() under the hood) makes a child, and execve() then swaps in a new program image:
// fork() signature (simplified - actually calls clone internally)pid_t fork(void);// Returns: child PID to parent, 0 to child, -1 on error
// execve() signatureint execve(const char *pathname, char *const argv[], char *const envp[]);// Replaces current process image with new program// Only returns on error (-1)The kernel implementation of fork() via clone():
// What the kernel does during fork()/clone():// 1. Allocate new task_struct// 2. Copy/share parent's memory descriptor (mm_struct)// - With COW: mark all writable pages as read-only// - Set up page fault handler to copy on first write// 3. Copy/share file descriptor table// 4. Copy/share signal handlers// 5. Allocate new kernel stack// 6. Set child's return value to 0// 7. Add child to scheduler run queue// 8. Return child PID to parentStep 2 is easy to see from user space. After fork(), parent and child print the same virtual address for counter, yet a write in the child doesn't show up in the parent: the first write faults, and the kernel gives the child its own copy of that page.
Splitting fork and exec is what makes Unix shells tractable. In the gap between the two calls, the child gets a chance to mess with itself:
pid_t pid = fork();if (pid == 0) { // Child process - modify environment before exec close(STDIN_FILENO); // Close stdin open("input.txt", O_RDONLY); // Redirect stdin from file dup2(pipe_fd[1], STDOUT_FILENO); // Redirect stdout to pipe chroot("/sandbox"); // Change root directory (needs root) chdir("/"); // Don't keep a cwd outside the jail setgroups(0, NULL); // Drop supplementary groups (needs root too) setgid(nobody_gid); // Then the primary group... setuid(nobody_uid); // ...then the user, last of all execve("/bin/program", args, env); // Now load the program _exit(1); // Only reached if execve fails}That fork-tweak-exec dance is precisely how every shell pulls off pipes, redirections, and privilege drops.
By separating process creation (fork) from program loading (exec), Linux allows the child process to modify its execution environment (file descriptors, working directory, credentials, namespace, resource limits) before the new program starts. This is fundamentally different from Windows, where these modifications must be specified as parameters to CreateProcess.
posix_spawn(): The Modern Alternative
Because fork() + exec() can be inefficient (especially on systems without efficient COW), POSIX introduced posix_spawn():
#include <spawn.h>
posix_spawn_file_actions_t actions;posix_spawn_file_actions_init(&actions);posix_spawn_file_actions_addopen(&actions, STDIN_FILENO, "input.txt", O_RDONLY, 0);
posix_spawnattr_t attr;posix_spawnattr_init(&attr);
pid_t pid;int status = posix_spawn(&pid, "/bin/program", &actions, &attr, argv, envp);posix_spawn() collapses fork and exec into one library call, much like Windows CreateProcess(). It isn't a syscall on Linux: glibc builds it on clone() with CLONE_VM | CLONE_VFORK followed by exec, which skips the wasted page-table copy.
The order in the fork example above matters: chroot() needs root, so it has to happen before setuid() throws root away, and the supplementary groups and group ID have to be dropped before the user ID for the same reason.
Windows: NtCreateProcess / NtCreateUserProcess
At the native API level, Windows process creation is handled by NtCreateUserProcess() (modern) or the older NtCreateProcess() + NtCreateThread():
// Simplified view of what NtCreateUserProcess does:// 1. Create a new EPROCESS and KPROCESS structure// 2. Create a new address space (page directory)// 3. Map ntdll.dll into the new address space// 4. Create the process's PEB (Process Environment Block)// 5. Map the executable image (from a section object)// 6. Create the initial thread (ETHREAD + TEB)// 7. Set up the initial thread's stack and context// 8. The thread starts executing in LdrInitializeThunk (ntdll)// which loads dependent DLLs and calls the entry pointThe Win32 API wrapper CreateProcess() handles the common case:
BOOL CreateProcess( LPCSTR lpApplicationName, // EXE path (optional) LPSTR lpCommandLine, // Command line string LPSECURITY_ATTRIBUTES lpProcessAttributes, // Process security LPSECURITY_ATTRIBUTES lpThreadAttributes, // Thread security BOOL bInheritHandles, // Handle inheritance DWORD dwCreationFlags, // CREATE_SUSPENDED, etc. LPVOID lpEnvironment, // Environment block LPCSTR lpCurrentDirectory, // Working directory LPSTARTUPINFO lpStartupInfo, // Window/IO settings LPPROCESS_INFORMATION lpProcessInformation // Output: handles);Everything Linux happily does in the gap between fork and exec — I/O redirection, working directory, environment block — has to be funnelled through CreateProcess's parameter list up front.
Thread Creation Syscalls
Linux: clone() with CLONE_THREAD
Linux threads are created with clone() using specific flags:
// POSIX threads (pthreads) call clone() with these flags:clone(thread_func, stack_top, // New stack for the thread CLONE_VM | // Share virtual memory CLONE_FS | // Share filesystem info CLONE_FILES | // Share file descriptors CLONE_SIGHAND | // Share signal handlers CLONE_THREAD | // Same thread group (same PID) CLONE_SYSVSEM | // Share SysV semaphore undo values CLONE_SETTLS | // Set thread-local storage CLONE_PARENT_SETTID | // Write TID to parent CLONE_CHILD_CLEARTID, // Clear TID on exit (for futex wake) arg);Here's the trick: Linux threads and processes are literally the same kernel object (task_struct), just born with different sharing flags. A thread is a task that shares everything with its creator. That's it.
Windows: NtCreateThreadEx
Since Vista, CreateThread() goes through NtCreateThreadEx(), which takes an attribute list and lets the kernel build the stack and TEB itself. The older NtCreateThread() below, where the caller supplies the initial register context and stack, still exists and shows what a thread is made of:
// Native API, legacy form (simplified)NTSTATUS NtCreateThread( PHANDLE ThreadHandle, // Output handle ACCESS_MASK DesiredAccess, // Access rights POBJECT_ATTRIBUTES ObjectAttributes, HANDLE ProcessHandle, // Which process to create thread in PCLIENT_ID ClientId, // Output: thread/process IDs PCONTEXT ThreadContext, // Initial register state PINITIAL_TEB InitialTeb, // Stack configuration BOOLEAN CreateSuspended // Start suspended?);Windows treats threads as first-class objects, distinct from processes. Each one carries its own ETHREAD in the kernel and a TEB (Thread Environment Block) in user space.
Memory Syscalls
Linux: mmap, mprotect, munmap
Linux's primary memory management syscall is mmap():
void *mmap( void *addr, // Preferred address (hint, or NULL) size_t length, // Size to map int prot, // PROT_READ | PROT_WRITE | PROT_EXEC int flags, // MAP_PRIVATE | MAP_ANONYMOUS | MAP_SHARED int fd, // File descriptor (or -1 for anonymous) off_t offset // Offset into file);
// Change protection on existing mappingint mprotect(void *addr, size_t len, int prot);
// Remove mappingint munmap(void *addr, size_t length);Common usage patterns:
// Allocate anonymous private memory (like malloc, but at page granularity)void *mem = mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
// Map a file into memoryint fd = open("data.bin", O_RDONLY);void *file = mmap(NULL, file_size, PROT_READ, MAP_PRIVATE, fd, 0);
// Allocate executable memory (for JIT compilation)void *code = mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);// Write machine code to 'code'...mprotect(code, 4096, PROT_READ | PROT_EXEC); // Make executableHere's that JIT pattern end to end. The six bytes are x86-64 machine code for mov eax, 42 then ret. The page is writable while you copy them in, executable afterwards, and never both at once. That's W^X, the same rule NX enforces against shellcode in later modules.
Windows: NtAllocateVirtualMemory, NtProtectVirtualMemory
Windows's native memory syscalls work directly on a process's virtual address space:
// Allocate virtual memoryNTSTATUS NtAllocateVirtualMemory( HANDLE ProcessHandle, // Can allocate in OTHER processes PVOID *BaseAddress, // In/out: requested/actual address ULONG_PTR ZeroBits, // Address space restriction PSIZE_T RegionSize, // In/out: requested/actual size ULONG AllocationType, // MEM_COMMIT | MEM_RESERVE ULONG Protect // PAGE_READWRITE, PAGE_EXECUTE_READ, etc.);
// Change memory protectionNTSTATUS NtProtectVirtualMemory( HANDLE ProcessHandle, PVOID *BaseAddress, PSIZE_T RegionSize, ULONG NewProtect, PULONG OldProtect // Output: previous protection);
// Free virtual memoryNTSTATUS NtFreeVirtualMemory( HANDLE ProcessHandle, PVOID *BaseAddress, PSIZE_T RegionSize, ULONG FreeType // MEM_DECOMMIT or MEM_RELEASE);Windows makes a distinction between reserving address space (MEM_RESERVE) and committing physical storage (MEM_COMMIT). You can reserve a large range of addresses without consuming physical memory, then commit pages within that range as needed. Linux's mmap with MAP_NORESERVE provides similar lazy allocation, but the two-phase model is not as explicit.
Here's one address range's life on each OS. On Windows, commit is a promise from the commit charge, and the physical page still only turns up when you first touch it. Linux folds reserve and commit into a single mmap call, so the interesting transition there is the first-touch page fault.
Three things Windows does differently from Linux mmap:
- Two-phase allocation: Windows splits reserving an address range (MEM_RESERVE) from committing actual physical pages (MEM_COMMIT). mmap does both in one shot.
- Cross-process: Hand Windows another process's handle and you can allocate inside its address space. Debuggers love this. So do DLL injectors.
- Granularity: when you reserve, Windows rounds the base address down to the 64 KB allocation granularity, and sizes are rounded up to whole 4 KB pages. mmap just aligns to the page (4 KB).
The Win32 Wrappers
In practice, most Windows programmers never touch the Nt* layer directly — they go through Win32:
// Win32 VirtualAlloc wraps NtAllocateVirtualMemoryLPVOID VirtualAlloc( LPVOID lpAddress, // Requested address or NULL SIZE_T dwSize, // Size in bytes DWORD flAllocationType, // MEM_COMMIT | MEM_RESERVE DWORD flProtect // PAGE_READWRITE, etc.);
// Win32 VirtualProtect wraps NtProtectVirtualMemoryBOOL VirtualProtect( LPVOID lpAddress, SIZE_T dwSize, DWORD flNewProtect, PDWORD lpflOldProtect);
// Allocate in another process (used for DLL injection)LPVOID VirtualAllocEx( HANDLE hProcess, // Target process handle LPVOID lpAddress, SIZE_T dwSize, DWORD flAllocationType, DWORD flProtect);Side-by-Side Comparison
| Operation | Linux Syscall | Windows Native API | Win32 API |
|---|---|---|---|
| Create process | clone/fork + execve | NtCreateUserProcess | CreateProcess |
| Create thread | clone (CLONE_THREAD) | NtCreateThreadEx | CreateThread |
| Wait for exit | waitpid | NtWaitForSingleObject | WaitForSingleObject |
| Allocate memory | mmap | NtAllocateVirtualMemory | VirtualAlloc |
| Change protection | mprotect | NtProtectVirtualMemory | VirtualProtect |
| Free memory | munmap | NtFreeVirtualMemory | VirtualFree |
| Read other process | process_vm_readv | NtReadVirtualMemory | ReadProcessMemory |
Get fluent with the native layer and you'll see how the higher abstractions — malloc, C++ new, the Go runtime, V8's JIT — actually graft onto the kernel underneath. Reverse engineers and exploit writers basically take up residence here.