Skip to main content
Be Bitwise

Process & Memory Syscalls Compared

System Calls Across Operating Systems · Operating Systems track

12 min read

In this module

Creating Processes and Managing Memory at the Kernel Level

Last lesson covered how the CPU flips between user and kernel mode. Now for the workhorses: the actual syscalls that spawn processes, fork threads, and carve up virtual memory on Linux and Windows. Every program you've ever run sits on top of these.

Process Creation: fork/exec vs NtCreateProcess

Linux: fork() and execve()

Linux splits process creation into two steps. fork() (which actually wraps clone() under the hood) makes a child, and execve() then swaps in a new program image:

c
// fork() signature (simplified - actually calls clone internally)
pid_t fork(void);
// Returns: child PID to parent, 0 to child, -1 on error
// execve() signature
int execve(const char *pathname, char *const argv[],
char *const envp[]);
// Replaces current process image with new program
// Only returns on error (-1)

The kernel implementation of fork() via clone():

text
// What the kernel does during fork()/clone():
// 1. Allocate new task_struct
// 2. Copy/share parent's memory descriptor (mm_struct)
// - With COW: mark all writable pages as read-only
// - Set up page fault handler to copy on first write
// 3. Copy/share file descriptor table
// 4. Copy/share signal handlers
// 5. Allocate new kernel stack
// 6. Set child's return value to 0
// 7. Add child to scheduler run queue
// 8. Return child PID to parent

Step 2 is easy to see from user space. After fork(), parent and child print the same virtual address for counter, yet a write in the child doesn't show up in the parent: the first write faults, and the kernel gives the child its own copy of that page.

Try itPredict the parent's value before you run it, then move the counter += 1 above the fork() and run it again.
C (GCC 9.2.0)

Splitting fork and exec is what makes Unix shells tractable. In the gap between the two calls, the child gets a chance to mess with itself:

c
pid_t pid = fork();
if (pid == 0) {
// Child process - modify environment before exec
close(STDIN_FILENO); // Close stdin
open("input.txt", O_RDONLY); // Redirect stdin from file
dup2(pipe_fd[1], STDOUT_FILENO); // Redirect stdout to pipe
chroot("/sandbox"); // Change root directory (needs root)
chdir("/"); // Don't keep a cwd outside the jail
setgroups(0, NULL); // Drop supplementary groups (needs root too)
setgid(nobody_gid); // Then the primary group...
setuid(nobody_uid); // ...then the user, last of all
execve("/bin/program", args, env); // Now load the program
_exit(1); // Only reached if execve fails
}

That fork-tweak-exec dance is precisely how every shell pulls off pipes, redirections, and privilege drops.

The fork/exec Split

By separating process creation (fork) from program loading (exec), Linux allows the child process to modify its execution environment (file descriptors, working directory, credentials, namespace, resource limits) before the new program starts. This is fundamentally different from Windows, where these modifications must be specified as parameters to CreateProcess.

posix_spawn(): The Modern Alternative

Because fork() + exec() can be inefficient (especially on systems without efficient COW), POSIX introduced posix_spawn():

c
#include <spawn.h>
posix_spawn_file_actions_t actions;
posix_spawn_file_actions_init(&actions);
posix_spawn_file_actions_addopen(&actions, STDIN_FILENO,
"input.txt", O_RDONLY, 0);
posix_spawnattr_t attr;
posix_spawnattr_init(&attr);
pid_t pid;
int status = posix_spawn(&pid, "/bin/program",
&actions, &attr, argv, envp);

posix_spawn() collapses fork and exec into one library call, much like Windows CreateProcess(). It isn't a syscall on Linux: glibc builds it on clone() with CLONE_VM | CLONE_VFORK followed by exec, which skips the wasted page-table copy.

The order in the fork example above matters: chroot() needs root, so it has to happen before setuid() throws root away, and the supplementary groups and group ID have to be dropped before the user ID for the same reason.

Windows: NtCreateProcess / NtCreateUserProcess

At the native API level, Windows process creation is handled by NtCreateUserProcess() (modern) or the older NtCreateProcess() + NtCreateThread():

text
// Simplified view of what NtCreateUserProcess does:
// 1. Create a new EPROCESS and KPROCESS structure
// 2. Create a new address space (page directory)
// 3. Map ntdll.dll into the new address space
// 4. Create the process's PEB (Process Environment Block)
// 5. Map the executable image (from a section object)
// 6. Create the initial thread (ETHREAD + TEB)
// 7. Set up the initial thread's stack and context
// 8. The thread starts executing in LdrInitializeThunk (ntdll)
// which loads dependent DLLs and calls the entry point

The Win32 API wrapper CreateProcess() handles the common case:

c
BOOL CreateProcess(
LPCSTR lpApplicationName, // EXE path (optional)
LPSTR lpCommandLine, // Command line string
LPSECURITY_ATTRIBUTES lpProcessAttributes, // Process security
LPSECURITY_ATTRIBUTES lpThreadAttributes, // Thread security
BOOL bInheritHandles, // Handle inheritance
DWORD dwCreationFlags, // CREATE_SUSPENDED, etc.
LPVOID lpEnvironment, // Environment block
LPCSTR lpCurrentDirectory, // Working directory
LPSTARTUPINFO lpStartupInfo, // Window/IO settings
LPPROCESS_INFORMATION lpProcessInformation // Output: handles
);

Everything Linux happily does in the gap between fork and exec — I/O redirection, working directory, environment block — has to be funnelled through CreateProcess's parameter list up front.

Thread Creation Syscalls

Linux: clone() with CLONE_THREAD

Linux threads are created with clone() using specific flags:

c
// POSIX threads (pthreads) call clone() with these flags:
clone(thread_func,
stack_top, // New stack for the thread
CLONE_VM | // Share virtual memory
CLONE_FS | // Share filesystem info
CLONE_FILES | // Share file descriptors
CLONE_SIGHAND | // Share signal handlers
CLONE_THREAD | // Same thread group (same PID)
CLONE_SYSVSEM | // Share SysV semaphore undo values
CLONE_SETTLS | // Set thread-local storage
CLONE_PARENT_SETTID | // Write TID to parent
CLONE_CHILD_CLEARTID, // Clear TID on exit (for futex wake)
arg);

Here's the trick: Linux threads and processes are literally the same kernel object (task_struct), just born with different sharing flags. A thread is a task that shares everything with its creator. That's it.

Windows: NtCreateThreadEx

Since Vista, CreateThread() goes through NtCreateThreadEx(), which takes an attribute list and lets the kernel build the stack and TEB itself. The older NtCreateThread() below, where the caller supplies the initial register context and stack, still exists and shows what a thread is made of:

c
// Native API, legacy form (simplified)
NTSTATUS NtCreateThread(
PHANDLE ThreadHandle, // Output handle
ACCESS_MASK DesiredAccess, // Access rights
POBJECT_ATTRIBUTES ObjectAttributes,
HANDLE ProcessHandle, // Which process to create thread in
PCLIENT_ID ClientId, // Output: thread/process IDs
PCONTEXT ThreadContext, // Initial register state
PINITIAL_TEB InitialTeb, // Stack configuration
BOOLEAN CreateSuspended // Start suspended?
);

Windows treats threads as first-class objects, distinct from processes. Each one carries its own ETHREAD in the kernel and a TEB (Thread Environment Block) in user space.

Memory Syscalls

Linux: mmap, mprotect, munmap

Linux's primary memory management syscall is mmap():

c
void *mmap(
void *addr, // Preferred address (hint, or NULL)
size_t length, // Size to map
int prot, // PROT_READ | PROT_WRITE | PROT_EXEC
int flags, // MAP_PRIVATE | MAP_ANONYMOUS | MAP_SHARED
int fd, // File descriptor (or -1 for anonymous)
off_t offset // Offset into file
);
// Change protection on existing mapping
int mprotect(void *addr, size_t len, int prot);
// Remove mapping
int munmap(void *addr, size_t length);

Common usage patterns:

c
// Allocate anonymous private memory (like malloc, but at page granularity)
void *mem = mmap(NULL, 4096, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
// Map a file into memory
int fd = open("data.bin", O_RDONLY);
void *file = mmap(NULL, file_size, PROT_READ,
MAP_PRIVATE, fd, 0);
// Allocate executable memory (for JIT compilation)
void *code = mmap(NULL, 4096, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
// Write machine code to 'code'...
mprotect(code, 4096, PROT_READ | PROT_EXEC); // Make executable

Here's that JIT pattern end to end. The six bytes are x86-64 machine code for mov eax, 42 then ret. The page is writable while you copy them in, executable afterwards, and never both at once. That's W^X, the same rule NX enforces against shellcode in later modules.

Try itChange the 0x2a byte to make the function return something else, then uncomment the write after mprotect and see the now read-only page fault.
C (GCC 9.2.0)

Windows: NtAllocateVirtualMemory, NtProtectVirtualMemory

Windows's native memory syscalls work directly on a process's virtual address space:

c
// Allocate virtual memory
NTSTATUS NtAllocateVirtualMemory(
HANDLE ProcessHandle, // Can allocate in OTHER processes
PVOID *BaseAddress, // In/out: requested/actual address
ULONG_PTR ZeroBits, // Address space restriction
PSIZE_T RegionSize, // In/out: requested/actual size
ULONG AllocationType, // MEM_COMMIT | MEM_RESERVE
ULONG Protect // PAGE_READWRITE, PAGE_EXECUTE_READ, etc.
);
// Change memory protection
NTSTATUS NtProtectVirtualMemory(
HANDLE ProcessHandle,
PVOID *BaseAddress,
PSIZE_T RegionSize,
ULONG NewProtect,
PULONG OldProtect // Output: previous protection
);
// Free virtual memory
NTSTATUS NtFreeVirtualMemory(
HANDLE ProcessHandle,
PVOID *BaseAddress,
PSIZE_T RegionSize,
ULONG FreeType // MEM_DECOMMIT or MEM_RELEASE
);
Key Concept
Reserve vs Commit

Windows makes a distinction between reserving address space (MEM_RESERVE) and committing physical storage (MEM_COMMIT). You can reserve a large range of addresses without consuming physical memory, then commit pages within that range as needed. Linux's mmap with MAP_NORESERVE provides similar lazy allocation, but the two-phase model is not as explicit.

Here's one address range's life on each OS. On Windows, commit is a promise from the commit charge, and the physical page still only turns up when you first touch it. Linux folds reserve and commit into a single mmap call, so the interesting transition there is the first-touch page fault.

Lifecycle of one address range: Windows reserve/commit vs Linux anonymous mmap

Three things Windows does differently from Linux mmap:

  1. Two-phase allocation: Windows splits reserving an address range (MEM_RESERVE) from committing actual physical pages (MEM_COMMIT). mmap does both in one shot.
  2. Cross-process: Hand Windows another process's handle and you can allocate inside its address space. Debuggers love this. So do DLL injectors.
  3. Granularity: when you reserve, Windows rounds the base address down to the 64 KB allocation granularity, and sizes are rounded up to whole 4 KB pages. mmap just aligns to the page (4 KB).

The Win32 Wrappers

In practice, most Windows programmers never touch the Nt* layer directly — they go through Win32:

c
// Win32 VirtualAlloc wraps NtAllocateVirtualMemory
LPVOID VirtualAlloc(
LPVOID lpAddress, // Requested address or NULL
SIZE_T dwSize, // Size in bytes
DWORD flAllocationType, // MEM_COMMIT | MEM_RESERVE
DWORD flProtect // PAGE_READWRITE, etc.
);
// Win32 VirtualProtect wraps NtProtectVirtualMemory
BOOL VirtualProtect(
LPVOID lpAddress,
SIZE_T dwSize,
DWORD flNewProtect,
PDWORD lpflOldProtect
);
// Allocate in another process (used for DLL injection)
LPVOID VirtualAllocEx(
HANDLE hProcess, // Target process handle
LPVOID lpAddress,
SIZE_T dwSize,
DWORD flAllocationType,
DWORD flProtect
);

Side-by-Side Comparison

OperationLinux SyscallWindows Native APIWin32 API
Create processclone/fork + execveNtCreateUserProcessCreateProcess
Create threadclone (CLONE_THREAD)NtCreateThreadExCreateThread
Wait for exitwaitpidNtWaitForSingleObjectWaitForSingleObject
Allocate memorymmapNtAllocateVirtualMemoryVirtualAlloc
Change protectionmprotectNtProtectVirtualMemoryVirtualProtect
Free memorymunmapNtFreeVirtualMemoryVirtualFree
Read other processprocess_vm_readvNtReadVirtualMemoryReadProcessMemory

Get fluent with the native layer and you'll see how the higher abstractions — malloc, C++ new, the Go runtime, V8's JIT — actually graft onto the kernel underneath. Reverse engineers and exploit writers basically take up residence here.