Skip to main content
Be Bitwise

Stack Protection Mechanisms

Binary Hardening & Compiler Defenses · Security Fundamentals track

6 min read

In this module

Defending the Stack

The stack is one of the most-attacked regions of memory. It packs return addresses, saved frame pointers, and local variables right next to each other. One buffer overflow can clobber a return address and hand control to whoever wrote the input. Over the decades, compilers and hardware engineers have piled on layered defences to make stack exploitation progressively miserable for attackers.

Stack Canaries

Stack canaries (Windows calls them stack cookies) are the oldest and most widely deployed stack protection mechanism. The idea is dirt simple: drop a known value between the local variables and the saved return address. Before the function returns, check the value hasn't changed.

The compiler inserts canary-checking code at compile time. In the function prologue, it loads a random value from a fixed location (on x86-64 Linux, thread-local storage at fs:0x28; on some other architectures, the global __stack_chk_guard) and stashes it on the stack. MSVC's /GS does the same with the global __security_cookie, XORed with the stack or frame pointer before it's stored. In the epilogue, it reloads the reference value and compares it against the stack copy. Any mismatch and the program calls __stack_chk_fail and aborts on the spot.

nasm
; Function prologue with stack canary (x86-64)
push rbp
mov rbp, rsp
sub rsp, 0x80 ; Allocate local variables
mov rax, qword [fs:0x28] ; Load canary from TLS
mov qword [rbp-0x8], rax ; Store canary on stack
; ... function body ...
; Function epilogue with canary check
mov rax, qword [rbp-0x8] ; Load canary from stack
xor rax, qword [fs:0x28] ; Compare with original
jne .stack_chk_fail ; Abort if modified
leave
ret

Modern glibc canaries are random, with the lowest byte forced to zero. That null byte stops string functions such as strcpy from copying a canary through, and stops printf("%s")-style leaks from reading past it. The older StackGuard design used a fixed terminator canary made of 0x00, 0x0a (newline), 0x0d (carriage return) and 0xff for the same reason, but a fixed value can simply be written back by any overflow that isn't string-based.

Canary Limitations

Stack canaries only catch linear overwrites that have to pass through the canary on the way to the return address. They miss non-linear writes entirely — anything that jumps over the canary using a format string bug or an indexed write — and they don't protect the local variables sandwiched between the buffer and the canary.

The two frames below make that limitation concrete. Each shows a 16-byte buffer squeezed to one row per byte, with the canary and return address each shown as a single row (their lowest byte). In the first, a strcpy of 26 As (0x41) has run straight up the stack, so the canary byte is no longer 00 and the epilogue check aborts.

Stack Frame View

0041Buffer← SP
0141Buffer
0241Buffer
0341Buffer
0441Buffer
0541Buffer
0641Buffer
0741Buffer
0841Buffer
0941Buffer
0A41Buffer
0B41Buffer
0C41Buffer
0D41Buffer
0E41Buffer
0F41Buffer
1041Canary
1141Saved rbp
1241Saved rbp
1341Saved rbp
1441Saved rbp
1541Saved rbp
1641Saved rbp
1741Saved rbp
1841Saved rbp
1941Return addr
1A00Return addr
1B00Return addr
1C00Return addr
1D00Return addr
1E00Return addr
1F00Return addr
2000Return addr
Regions
Buffer16 B
Canary1 B
Saved rbp8 B
0x4141414141414141
Return addr8 B
0x0000000000000041
Overflow direction
Buffer→Canary→Saved rbp→Return addr

In the second, the bug is an indexed write, buf[i] = value, with i under the attacker's control. Setting i to 25 lands one byte on the return address and never touches anything in between. The canary still reads 00, the check passes, and ret jumps wherever the attacker pointed it.

Stack Frame View

0042Buffer← SP
016FBuffer
0262Buffer
0300Buffer
0400Buffer
0500Buffer
0600Buffer
0700Buffer
0800Buffer
0900Buffer
0A00Buffer
0B00Buffer
0C00Buffer
0D00Buffer
0E00Buffer
0F00Buffer
1000Canary
11D0Saved rbp
12E0Saved rbp
13FFSaved rbp
14FFSaved rbp
15FFSaved rbp
167FSaved rbp
1700Saved rbp
1800Saved rbp
1941Return addr
1A00Return addr
1B00Return addr
1C00Return addr
1D00Return addr
1E00Return addr
1F00Return addr
2000Return addr
Regions
Buffer16 B
Canary1 B
Saved rbp8 B
0x00007fffffffe0d0
Return addr8 B
0x0000000000000041
Overflow direction
Buffer→Canary→Saved rbp→Return addr

SafeStack

SafeStack is a compiler-based defence (Clang/LLVM) that swings harder than canaries. Rather than detecting corruption after the fact, it stops the corruption from reaching anything important to begin with.

The trick is to split the stack in two. The safe stack holds return addresses, register spills, and anything else that absolutely cannot be allowed to get clobbered. The unsafe stack holds local variables, especially the arrays and buffers liable to get overflowed. An overflow on the unsafe stack can't touch the return addresses because they live in a completely separate region of memory.

SafeStack carries low overhead — usually under 1% performance hit — and the application doesn't notice it. The catch is that it doesn't protect the unsafe stack itself: local variables that aren't return addresses or register spills can still get scribbled over.

Hardware Shadow Stacks

Any software defence will eventually fall to an attacker with the right primitives. Hardware shadow stacks shove the defence into the CPU itself, raising the bar for bypass considerably.

A shadow stack is a separate, hardware-protected stack that stores nothing but return addresses. Every CALL makes the CPU push the return address to both the regular stack and the shadow stack. On RET, the CPU compares the return address from the regular stack against the shadow copy. Any mismatch and the CPU faults.

Intel CET (Control-flow Enforcement Technology) lands shadow stacks in hardware starting with 11th-generation (Tiger Lake) Intel CPUs; AMD added them with Zen 3. The shadow stack lives in special pages tagged with a shadow stack attribute in the page tables. Only CALL, RET and a handful of dedicated shadow-stack instructions (such as RSTORSSP, or WRSS if the OS enables it) can write them. A regular MOV to a shadow stack page faults.

ARM goes a different way with Pointer Authentication Codes (PAC). Instead of a separate stack, PAC cryptographically signs the return address using a key held in a special register. The signed return address lives on the regular stack and gets verified before use. Corrupt the return address and the signature no longer checks out — the CPU faults.

nasm
; ARM PAC: function prologue
paciasp ; Sign LR (return address) with SP and key A
stp x29, x30, [sp, #-16]! ; Store signed LR on stack
; ARM PAC: function epilogue
ldp x29, x30, [sp], #16 ; Load signed LR from stack
autiasp ; Verify signature - corrupt LR fails to authenticate
ret

A failed autiasp leaves LR as an invalid pointer, so the ret faults; cores with the FEAT_FPAC extension fault on the authentication itself.

Software can build a shadow stack too. Clang's ShadowCallStack (-fsanitize=shadow-call-stack) on AArch64 saves each return address to a separate stack pointed to by register x18 and returns through that copy. It isn't hardware-protected, so it relies on the shadow stack's address staying secret; Android uses it in parts of its kernel and system components.

Compiler Heuristics for Protection

Not every function gets a canary. The compiler uses heuristics to decide which functions are worth protecting — typically the ones that declare local arrays or use alloca. The behaviour is controlled by the -fstack-protector family of flags:

  • -fstack-protector: Protects functions with local buffers larger than 8 bytes
  • -fstack-protector-strong: Protects functions that use local arrays, take addresses of local variables, or use alloca
  • -fstack-protector-all: Protects every function (highest overhead)

It's the usual performance versus coverage dance. The strong variant is the default across most Linux distributions because it catches the overwhelming majority of vulnerable functions with negligible overhead.

Defence in Depth

These mechanisms are meant to stack on top of each other. A hardened binary might combine stack canaries (catching linear overwrites), SafeStack (isolating return addresses), and hardware shadow stacks (locking down return address modification entirely). Each layer catches the attacks that wriggle past the others.