Skip to main content
Be Bitwise
x86 AssemblyReverse Engineering & Hardening

Capstone: Analyze and Harden

This is the capstone for the entire security-focused arc of the x86-64 track. You'll bring together every skill from the last four modules: reading disassembly, identifying vulnerability classes, understanding exploitation techniques, and applying binary hardening. The brief is straightforward — dissect a deliberately vulnerable binary, catalogue every weakness, and prescribe the full defensive treatment.

The Journey from Hello World to Security Professional

You started this track printing "Hello, x86!" through raw syscalls. Since then you've learned to read and write assembly, understand the stack and calling conventions, exploit buffer overflows, develop shellcode, build ROP chains, bypass ASLR, read compiler output, and spot vulnerabilities in disassembly. This capstone braids it all together. Picking up a binary, finding its weaknesses, and hardening it — that's the working skill set of a professional security engineer.

The Target Binary

Take a network-service binary with the following disassembly, compiled without any hardening. Walk each function and catalogue every vulnerability:

Function: handle_request

nasm
handle_request:
push rbp
mov rbp, rsp
sub rsp, 448 ; large stack frame
; No fs:[0x28] access -- NO STACK CANARY
; Frame layout (offsets from rbp, all negative):
; [rbp-4] bytes_read (int)
; [rbp-8] socket fd (int)
; [rbp-12] length (int)
; [rbp-24] resp pointer (8 bytes)
; [rbp-296] request buffer (272 bytes) -> rbp-296 .. rbp-25
; [rbp-448] path buffer (144 bytes) -> rbp-448 .. rbp-305
mov dword [rbp-8], edi ; save socket fd
; Read request from network
mov edx, 1024 ; read up to 1024 bytes
lea rsi, [rbp-296] ; request buffer (only 272 bytes!)
mov edi, dword [rbp-8] ; socket fd
call read ; read(fd, buffer, 1024)
; BUG: reads 1024 into 272-byte buffer!
mov dword [rbp-4], eax ; save bytes_read
; BUG: no buffer[n] = 0 -- not NUL-terminated
; Log the request
lea rdi, [rbp-296] ; user-controlled buffer
mov eax, 0
call printf ; BUG: printf(user_input) -- format string!
; Parse command
lea rdi, [rbp-296]
lea rsi, [rel .cmd_get] ; "GET"
mov edx, 3
call strncmp
test eax, eax
jne .not_get
; Handle GET: extract path
lea rdi, [rbp-296]
add rdi, 4 ; skip "GET "
lea rsi, [rbp-448] ; path buffer (144 bytes)
call strcpy ; BUG: strcpy, no bounds check
; Response size comes from a length field in the request header
mov eax, dword [rbp-292] ; length field at request+4 (attacker-controlled)
mov dword [rbp-12], eax ; length as 32-bit int
; Allocate response buffer
mov eax, dword [rbp-12]
imul eax, eax, 4 ; BUG: integer overflow (32-bit multiply)
mov edi, eax
call malloc ; may allocate too little
mov qword [rbp-24], rax ; save malloc result
; No NULL check on malloc return! ; BUG: missing NULL check
; Copy data to response
mov rdi, qword [rbp-24]
lea rsi, [rbp-448]
mov edx, dword [rbp-12]
call memcpy ; BUG: copies `length` bytes, ignores alloc size
; Send response
mov edx, dword [rbp-12]
mov rsi, qword [rbp-24]
mov edi, dword [rbp-8]
call write
; Free response buffer
mov rdi, qword [rbp-24]
call free ; pointer not zeroed, but the function
; returns next, so the slot dies with it
.not_get:
leave
ret ; returns with corrupted stack if overflowed

Vulnerability Catalogue

Pull the function apart and tag every vulnerability in turn:

Vulnerability 1: Stack Buffer Overflow (CWE-121)

nasm
mov edx, 1024 ; read up to 1024 bytes
lea rsi, [rbp-296] ; into a 272-byte buffer
call read

The read accepts up to 1024 bytes into the 272-byte request buffer at [rbp-296]. The overflow runs up through the other locals, then clobbers:

  • The saved rbp at [rbp+0]
  • The return address at [rbp+8]

Offset to the return address: 296 (buffer start to saved rbp) + 8 = 304 bytes. With read happy to take 1024 bytes, the attacker has over 700 bytes of controlled data beyond the return address — well more than enough for a ROP chain.

Exploitation path: overwrite the return address with a ROP chain and ride it to code execution. No canary in the way, so no extra bypass needed.

Vulnerability 2: Missing NUL Termination (CWE-170)

nasm
call read
mov dword [rbp-4], eax ; bytes_read saved, but no terminator written

read doesn't NUL-terminate. The code never writes buffer[bytes_read] = 0, yet every call after it — printf, strncmp, strcpy, strlen-style scans — treats the buffer as a C string. They run off the end until they happen to hit a zero byte further down the stack, leaking adjacent frame data or walking into unmapped memory. It also feeds the two overflows below: a request with no embedded NUL makes strcpy copy well past the path buffer.

Vulnerability 3: Format String (CWE-134)

nasm
lea rdi, [rbp-296] ; user data as format string
call printf

User-controlled data flows straight into printf as the format argument. An attacker can:

  • Read the stack: send %p.%p.%p... to leak stack values — canary (if one existed), saved return addresses, libc pointers
  • Write to memory: use %n to write to arbitrary addresses
  • Defeat ASLR: pull runtime addresses out via leaks

That single bug hands over both read and write primitives. Even with full hardening, it's enough for end-to-end exploitation.

Vulnerability 4: Second Stack Buffer Overflow via strcpy (CWE-120)

nasm
lea rsi, [rbp-448] ; 144-byte path buffer
call strcpy ; copies from user-controlled data

The request's path gets copied via strcpy into the 144-byte path buffer at [rbp-448] with no length check anywhere. Push a path past 144 bytes and the overflow spills into whatever stack data sits next door.

Vulnerability 5: Integer Overflow (CWE-190)

nasm
mov eax, dword [rbp-12] ; 32-bit length (from request header)
imul eax, eax, 4 ; length * 4, 32-bit multiply
mov edi, eax
call malloc

The length comes straight from a 32-bit field in the request, so the attacker sets it to anything. Multiplying by 4 in 32-bit arithmetic, feed it 0x40000001 and the product 0x100000004 wraps down to 0x4. malloc(4) runs, then the memcpy copies 0x40000001 bytes out of the path buffer into a 4-byte allocation. Because the size is read from the input rather than derived from strlen, the overflow is directly reachable.

Vulnerability 6: Missing NULL Check (CWE-476)

nasm
call malloc
mov qword [rbp-24], rax ; save result -- might be NULL!
; ... later ...
mov rdi, qword [rbp-24] ; use without checking
call memcpy ; writes to NULL or near-NULL address

If malloc fails and returns NULL, the code keeps marching and uses the NULL pointer anyway. Depending on the OS, writing to address 0 either segfaults or — on systems where page 0 is mappable — hands the attacker a tidy little exploitation primitive.

Not a bug here: stale pointer after free

nasm
call free ; free the buffer
; rbp-24 still holds the freed pointer, but...
.not_get:
leave ; ... the function returns immediately
ret ; so the slot is gone -- no later use

A quick scan flags the un-zeroed pointer as a use-after-free, but trace the control flow: the function returns right after the free, so the stack slot holding the stale pointer dies with the frame. Zeroing it would be tidy hygiene, not a fix. This is the kind of false positive that separates reading disassembly from pattern-matching it — the pointer is only dangerous if something dereferences it after the free, and nothing does. Keep the habit (ptr = NULL) for longer-lived pointers, where it genuinely matters.

Missing Mitigations Analysis

Run a mental checksec over this binary:

c
RELRO: No RELRO (GOT is writable -- format string can overwrite it)
Stack: No canary (no fs:[0x28] access in any function)
NX: Unknown (check executable stack flag)
PIE: No PIE (absolute addresses used -- ROP gadgets at fixed addresses)
FORTIFY: Not enabled (strcpy used instead of __strcpy_chk)

Nothing is on. This is as bad as a hardening checklist gets.

The Hardening Prescription

Switch on every available compiler and linker protection:

bash
# A backslash must end the line, so the notes go above each flag, not after it.
gcc -fstack-protector-strong \
-D_FORTIFY_SOURCE=2 \
-fPIE -pie \
-Wl,-z,relro,-z,now \
-fcf-protection \
-fstack-clash-protection \
-O2 \
-Wformat -Wformat-security \
-Werror=format-security \
-o hardened server.c
# -fstack-protector-strong stack canaries on vulnerable functions
# -D_FORTIFY_SOURCE=2 checked versions of string functions (needs -O1+)
# -fPIE -pie position-independent executable (ASLR for code)
# -Wl,-z,relro,-z,now full RELRO (read-only GOT)
# -fcf-protection emit CET endbr64 markers and notes
# -fstack-clash-protection probe the stack to stop stack-clash jumps
# -O2 optimisation, required for FORTIFY to kick in
# -Wformat -Wformat-security warn on dangerous printf-family calls
# -Werror=format-security turn those warnings into build errors

Hardening flags alone won't save you. The source has to get fixed too:

c
// Fix 1: Bounded read, and check it before using n as an index
ssize_t n = read(fd, buffer, sizeof(buffer) - 1);
if (n < 0) { close(fd); return; } // read failed: buffer[-1] would write out of bounds
buffer[n] = '\0';
// Fix 2: Fix format string
printf("%s", buffer); // NOT printf(buffer)
// Fix 3: Use strncpy or strlcpy
strncpy(path, src, sizeof(path) - 1);
path[sizeof(path) - 1] = '\0';
// Fix 4: Check for integer overflow
if (length > SIZE_MAX / 4) { /* error */ }
size_t alloc_size = (size_t)length * 4;
// Fix 5: Check malloc return
char *resp = malloc(alloc_size);
if (resp == NULL) { /* handle error */ }
// Fix 6: Zero pointer after free
free(resp);
resp = NULL;

Interactive Analysis

Use the exploit builder below to draft a proof of concept for each vulnerability, then confirm your hardening holds up in the sandbox:

Exploit Builder

basic-overflow

Fill the buffer, then overwrite the return address with the target address.

NOP
Shellcode
Address
Padding
Offset 0x00Buffer (272B)RET @ 0x130
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
??
??
??
??
Stack Layout
Buffer [272B]
SFP
RET [4B]
Target: 0x00401000

Practise on your own machine

You need Linux on x86-64: a virtual machine, WSL 2 on Windows, or a container. Then install the tools with sudo apt install nasm gcc gdb binutils.

How Hardening Changes the Disassembly

Once hardening lands, the same function's disassembly is barely recognisable:

nasm
handle_request:
endbr64 ; CET: valid indirect call target marker
push rbp
mov rbp, rsp
sub rsp, 464 ; a touch larger: canary sits at rbp-8,
; buffers moved down to leave room for it
; CANARY INSERTION (buffers now start below rbp-8)
mov rax, qword fs:[0x28]
mov qword [rbp-8], rax
xor eax, eax
; Bounded read (source fix)
mov edx, 271 ; sizeof(buffer) - 1
lea rsi, [rbp-304] ; request buffer, below the canary slot
mov edi, dword [rbp-20] ; socket fd
call read
mov eax, dword [rbp-12] ; bytes_read
mov byte [rbp-304+rax], 0 ; NUL-terminate (source fix)
; Safe printf (source fix)
lea rsi, [rbp-304] ; user data as ARGUMENT
lea rdi, [rip + .LC_fmt] ; "%s" as format string
call printf
; FORTIFY: __strncpy_chk instead of strcpy
mov edx, 143 ; max length
mov ecx, 144 ; buffer size for check
call __strncpy_chk
; ... rest of fixed code ...
; CANARY VERIFICATION
mov rax, qword [rbp-8]
xor rax, qword fs:[0x28]
jne .canary_fail
leave
ret
.canary_fail:
call __stack_chk_fail

Final Assessment Framework

For any binary you're sizing up, run the systematic loop:

  1. Enumerate the attack surface: what inputs does the program take? (network, files, CLI args, environment variables)
  2. Map the vulnerability classes: for every input path, sweep for overflows, format strings, integer issues, and use-after-free
  3. Assess the mitigations: run checksec and verify canaries, ASLR, DEP, RELRO, PIE, FORTIFY, and CFI
  4. Evaluate exploitability: which vulnerabilities are reachable from external input, and what primitives do they hand over — read, write, code execution?
  5. Prescribe defences: spell out both the source-level fixes and the binary hardening flags
  6. Verify: rebuild with hardening on and confirm the mitigations actually show up in the disassembly

The same framework holds whether you're auditing your own code, running a pentest engagement, working a malware sample, or grinding a CTF.

The Broader Context

The security-focused modules of the x86-64 track are now behind you. You can speak fluently about:

  • How the stack works and why buffer overflows exist (Module 9)
  • How shellcode gets written, why it has to be position-independent, and how to test it (Module 10)
  • How ROP chains reuse existing code to step around DEP, and how info leaks blow past ASLR (Module 11)
  • How to read compiler output, spot vulnerabilities in disassembly, and harden binaries (Module 12)

This is the substrate of careers in vulnerability research, penetration testing, malware analysis, incident response, and secure software development. The principles are stable even as the specific techniques churn: the offence-defence arms race keeps driving innovation on both sides of the line.

For continued study, explore:

Challenges

Print the essential GCC hardening flags, one per line: -fstack-protector-strong, -D_FORTIFY_SOURCE=2, -Wl,-z,relro,-z,now, -fPIE, -pie. These are the minimum flags for compiling security-critical C code.

Assembly (NASM 2.14.02)