Stack Protection Mechanisms
Binary Hardening & Compiler Defenses · Security Fundamentals track
6 min read
In this module
Defending the Stack
The stack is one of the most-attacked regions of memory. It packs return addresses, saved frame pointers, and local variables right next to each other. One buffer overflow can clobber a return address and hand control to whoever wrote the input. Over the decades, compilers and hardware engineers have piled on layered defences to make stack exploitation progressively miserable for attackers.
Stack Canaries
Stack canaries (Windows calls them stack cookies) are the oldest and most widely deployed stack protection mechanism. The idea is dirt simple: drop a known value between the local variables and the saved return address. Before the function returns, check the value hasn't changed.
The compiler inserts canary-checking code at compile time. In the function prologue, it loads a random value from a fixed location (on x86-64 Linux, thread-local storage at fs:0x28; on some other architectures, the global __stack_chk_guard) and stashes it on the stack. MSVC's /GS does the same with the global __security_cookie, XORed with the stack or frame pointer before it's stored. In the epilogue, it reloads the reference value and compares it against the stack copy. Any mismatch and the program calls __stack_chk_fail and aborts on the spot.
; Function prologue with stack canary (x86-64)push rbpmov rbp, rspsub rsp, 0x80 ; Allocate local variablesmov rax, qword [fs:0x28] ; Load canary from TLSmov qword [rbp-0x8], rax ; Store canary on stack
; ... function body ...
; Function epilogue with canary checkmov rax, qword [rbp-0x8] ; Load canary from stackxor rax, qword [fs:0x28] ; Compare with originaljne .stack_chk_fail ; Abort if modifiedleaveretModern glibc canaries are random, with the lowest byte forced to zero. That null byte stops string functions such as strcpy from copying a canary through, and stops printf("%s")-style leaks from reading past it. The older StackGuard design used a fixed terminator canary made of 0x00, 0x0a (newline), 0x0d (carriage return) and 0xff for the same reason, but a fixed value can simply be written back by any overflow that isn't string-based.
Stack canaries only catch linear overwrites that have to pass through the canary on the way to the return address. They miss non-linear writes entirely — anything that jumps over the canary using a format string bug or an indexed write — and they don't protect the local variables sandwiched between the buffer and the canary.
The two frames below make that limitation concrete. Each shows a 16-byte buffer squeezed to one row per byte, with the canary and return address each shown as a single row (their lowest byte). In the first, a strcpy of 26 As (0x41) has run straight up the stack, so the canary byte is no longer 00 and the epilogue check aborts.
Stack Frame View
In the second, the bug is an indexed write, buf[i] = value, with i under the attacker's control. Setting i to 25 lands one byte on the return address and never touches anything in between. The canary still reads 00, the check passes, and ret jumps wherever the attacker pointed it.
Stack Frame View
SafeStack
SafeStack is a compiler-based defence (Clang/LLVM) that swings harder than canaries. Rather than detecting corruption after the fact, it stops the corruption from reaching anything important to begin with.
The trick is to split the stack in two. The safe stack holds return addresses, register spills, and anything else that absolutely cannot be allowed to get clobbered. The unsafe stack holds local variables, especially the arrays and buffers liable to get overflowed. An overflow on the unsafe stack can't touch the return addresses because they live in a completely separate region of memory.
SafeStack carries low overhead — usually under 1% performance hit — and the application doesn't notice it. The catch is that it doesn't protect the unsafe stack itself: local variables that aren't return addresses or register spills can still get scribbled over.
Hardware Shadow Stacks
Any software defence will eventually fall to an attacker with the right primitives. Hardware shadow stacks shove the defence into the CPU itself, raising the bar for bypass considerably.
A shadow stack is a separate, hardware-protected stack that stores nothing but return addresses. Every CALL makes the CPU push the return address to both the regular stack and the shadow stack. On RET, the CPU compares the return address from the regular stack against the shadow copy. Any mismatch and the CPU faults.
Intel CET (Control-flow Enforcement Technology) lands shadow stacks in hardware starting with 11th-generation (Tiger Lake) Intel CPUs; AMD added them with Zen 3. The shadow stack lives in special pages tagged with a shadow stack attribute in the page tables. Only CALL, RET and a handful of dedicated shadow-stack instructions (such as RSTORSSP, or WRSS if the OS enables it) can write them. A regular MOV to a shadow stack page faults.
ARM goes a different way with Pointer Authentication Codes (PAC). Instead of a separate stack, PAC cryptographically signs the return address using a key held in a special register. The signed return address lives on the regular stack and gets verified before use. Corrupt the return address and the signature no longer checks out — the CPU faults.
; ARM PAC: function prologuepaciasp ; Sign LR (return address) with SP and key Astp x29, x30, [sp, #-16]! ; Store signed LR on stack
; ARM PAC: function epilogueldp x29, x30, [sp], #16 ; Load signed LR from stackautiasp ; Verify signature - corrupt LR fails to authenticateretA failed autiasp leaves LR as an invalid pointer, so the ret faults; cores with the FEAT_FPAC extension fault on the authentication itself.
Software can build a shadow stack too. Clang's ShadowCallStack (-fsanitize=shadow-call-stack) on AArch64 saves each return address to a separate stack pointed to by register x18 and returns through that copy. It isn't hardware-protected, so it relies on the shadow stack's address staying secret; Android uses it in parts of its kernel and system components.
Compiler Heuristics for Protection
Not every function gets a canary. The compiler uses heuristics to decide which functions are worth protecting — typically the ones that declare local arrays or use alloca. The behaviour is controlled by the -fstack-protector family of flags:
-fstack-protector: Protects functions with local buffers larger than 8 bytes-fstack-protector-strong: Protects functions that use local arrays, take addresses of local variables, or usealloca-fstack-protector-all: Protects every function (highest overhead)
It's the usual performance versus coverage dance. The strong variant is the default across most Linux distributions because it catches the overwhelming majority of vulnerable functions with negligible overhead.
These mechanisms are meant to stack on top of each other. A hardened binary might combine stack canaries (catching linear overwrites), SafeStack (isolating return addresses), and hardware shadow stacks (locking down return address modification entirely). Each layer catches the attacks that wriggle past the others.