Skip to main content
Be Bitwise
x86 AssemblyReverse Engineering & Hardening

Identifying Vulnerabilities in Disassembly

Security researchers, malware analysts, and CTF competitors find vulnerabilities by reading disassembly all the time. When source isn't available — closed-source software, firmware, malware — disassembly is what you have to work with. This lesson trains you to recognise the assembly-level fingerprints of exploitable bugs: missing bounds checks, dangerous function calls, integer overflows, format-string sinks, and absent stack protections.

For the theoretical foundation on reading disassembly, see Reading Disassembly.

Pattern 1: Missing Bounds Checks

The classic vulnerability shape: data copied or accessed without anyone vetting the index or length.

Vulnerable pattern (no bounds check before array access):

nasm
; User-controlled index in rdi, array base in rsi
; Vulnerable: no CMP/JB before the access
mov rax, qword [rsi + rdi*8] ; array[user_index] -- no check!

Safe pattern (bounds check present):

nasm
; Safe: checks index before access
cmp rdi, 100 ; is index < 100?
jae .out_of_bounds ; if not, handle error
mov rax, qword [rsi + rdi*8] ; array[user_index] -- checked

When you see an array access (base + index * scale) without a cmp and conditional jump in front of it, the access could well be exploitable if an attacker controls the index.

What to scan for:

  • Memory access with a register-driven index: [base + reg*scale]
  • No cmp or test on that register in the preceding 5–10 instructions
  • The indexing register pulled from user input (a function arg, a buffer read from a file or network)

Pattern 2: Dangerous Function Calls

A handful of C library functions are dangerous by design — they perform unbounded operations. Spot them by their PLT entries or resolved addresses:

nasm
; DANGEROUS: call to gets() -- no length limit, always vulnerable
call gets
; DANGEROUS: call to strcpy() -- copies until null, no size check
lea rdi, [rbp-64] ; destination buffer (64 bytes from rbp)
call strcpy
; DANGEROUS: call to sprintf() -- unbounded format output
call sprintf
; DANGEROUS: call to scanf with %s -- unbounded string read
lea rdi, [rel .format_s] ; "%s" format
lea rsi, [rbp-32] ; 32-byte buffer
call scanf

Safer alternatives you should see instead:

nasm
; SAFER: fgets with explicit length
mov edx, 64 ; max bytes (third arg)
call fgets
; BOUNDED, BUT CHECK THE TERMINATOR: strncpy with length parameter
mov rdx, 64 ; max length (third arg)
call strncpy ; no NUL written if the source is 64+ bytes
; SAFER: snprintf with buffer size
mov esi, 64 ; buffer size (second arg)
call snprintf

strncpy deserves a second look whenever you meet it. If the source fills the limit it doesn't write a terminator, so check for a following store of 0 to dest[n-1]. snprintf and an explicit length check plus memcpy are the cleaner choices.

If you see call gets, call strcpy, or call sprintf in a disassembly, congratulations — you've probably just found a vulnerability. These functions are so notoriously bad that compilers now warn on use, and gets was outright removed from the C11 standard.

Pattern 3: Missing Stack Canary Checks

With -fstack-protector enabled, the compiler slips a canary — a random value — between the buffer and the saved rbp, and re-checks it before returning:

Protected function (canary present):

nasm
function_with_canary:
push rbp
mov rbp, rsp
sub rsp, 80
; CANARY WRITE -- load canary from fs:[0x28]
mov rax, qword fs:[0x28] ; read the canary value
mov qword [rbp-8], rax ; store it on the stack
xor eax, eax ; clear rax (prevent leaking canary)
; ... function body ...
; CANARY CHECK -- before return
mov rax, qword [rbp-8] ; reload stored canary
xor rax, qword fs:[0x28] ; compare with original
jne .canary_fail ; if different, stack was corrupted!
leave
ret
.canary_fail:
call __stack_chk_fail ; abort the process

Unprotected function (no canary):

nasm
function_without_canary:
push rbp
mov rbp, rsp
sub rsp, 80
; No fs:[0x28] access anywhere
; ... function body ...
leave
ret ; no canary check before return!

A function that holds a local buffer yet never touches fs:[0x28] was compiled without stack protection. That's a loud signal that a buffer overflow inside it can overwrite the return address with nobody noticing.

Static Analysis Hunts the Same Patterns

The patterns in this lesson are what automated static-analysis tools hunt for. Coverity, CodeQL and Semgrep do it on source code; on a binary you're left with disassembler-based tools such as Ghidra's analyzers and scripts, plus checksec-style scans for missing hardening. Learning to recognise the same patterns by hand makes you sharper at triaging tool output and catches bugs the tools miss. This is daily-bread skill for vulnerability researchers, malware reverse engineers, and CTF players.

Pattern 4: Integer Overflow Points

Integer overflows happen when arithmetic on a size or length wraps into a value the code wasn't expecting. Tells to watch for:

nasm
; Potential integer overflow: user-controlled multiplication
mov eax, dword [rbp-4] ; user-controlled size
imul eax, eax, 8 ; size * 8 (can overflow 32-bit!)
mov edi, eax ; pass to malloc
call malloc ; allocates LESS than expected
; If size = 0x20000001, then size * 8 = 0x100000008
; But in 32-bit arithmetic: 0x100000008 truncated to 0x00000008
; malloc(8) is called, but the caller thinks it allocated size*8 bytes

What to scan for:

  • Arithmetic on values loaded from user input (imul, add, shl)
  • The result handed straight to malloc, calloc, or mmap
  • No overflow check — no jo (jump on overflow) after the arithmetic
  • 32-bit and 64-bit arithmetic mixed in the same flow (mov eax next to mov rax)

Pattern 5: Format String Sinks

A format-string bug happens when user input is handed in as the format argument to anything in the printf family:

nasm
; VULNERABLE: user input as format string
lea rdi, [rbp-256] ; rdi = user-controlled buffer
mov eax, 0
call printf ; printf(user_buffer) -- vulnerable!
; SAFE: format string is a constant
lea rdi, [rel .LC0] ; rdi = constant "%s\n"
lea rsi, [rbp-256] ; rsi = user data (as argument, not format)
mov eax, 0
call printf ; printf("%s\n", user_buffer) -- safe

The whole question collapses to one detail: is the first argument to printf a known constant (a lea from .rodata) or a pointer to attacker-controlled data on the stack or heap?

Pattern 6: Use-After-Free Indicators

nasm
; Potential use-after-free:
call free ; free the object
; ... some code ...
mov rax, qword [rbx] ; access the freed object through rbx
; rbx still holds the old pointer!
call qword [rax + 16] ; virtual function call on freed object

Tells to scan for:

  • A call free (or an indirect jump into free)
  • The pointer register (here, rbx) NOT cleared after the free
  • Later code dereferencing that same register

Pattern 7: Uninitialized Stack Variables

nasm
; Potential info leak via uninitialized stack variable
sub rsp, 32 ; allocate 32 bytes
; No write to [rsp+0] through [rsp+31] before reading
mov edi, 1 ; fd (first arg), e.g. stdout or a socket
mov rsi, rsp ; buffer (second arg)
mov edx, 32 ; length (third arg)
call write ; writes uninitialized stack contents to fd!

Stack memory that's read or sent down a wire before it's written holds whatever the previous function call left behind — pointers, canaries, possibly keys.

Vulnerability Audit Checklist

Attacker Input
Buffer
Overflow
Return Addr
RET Executes
Attacker Code

Hover over each step to see details

When you're auditing a disassembled function, run the checklist top to bottom:

  1. Buffer operations: any call strcpy/gets/sprintf? Can you reason about the buffer size?
  2. Bounds checks: every array access preceded by cmp / jae / jbe?
  3. Stack protection: does the function read fs:[0x28] for a canary?
  4. Integer arithmetic: are size calculations checked for overflow?
  5. Format strings: is user data ever the first argument to a printf-family function?
  6. Memory management: after call free, is the pointer zeroed?
  7. Return-value checks: after call malloc / call open, does the code actually test the result (test rax, rax; jz error)?

Real-World Application

Real-world auditing stacks these patterns up. A function that:

  • has no stack canary (no fs:[0x28] access)
  • calls strcpy into a stack buffer
  • takes user-controlled input as the source
  • is compiled without PIE

…is a textbook exploitable buffer overflow. Each missing mitigation makes exploitation that much easier.

Flip the dials, and a function that:

  • has a stack canary
  • uses snprintf or a length-checked memcpy
  • lives in a PIE binary with full RELRO
  • pulls in _FORTIFY_SOURCE wrappers (calls __strcpy_chk instead of strcpy)

…hides a stack of defences the attacker has to peel apart layer by layer.

Security Context: Daily Application

These vuln-identification skills are bread-and-butter across multiple security roles. Vulnerability researchers audit closed-source software straight from disassembly. Malware analysts map out the capabilities and infection vectors of samples. CTF players chase down the intended bug in challenge binaries. Security auditors confirm that the hardening flags somebody promised actually shipped in the production build. Looking at a disassembly and clocking a missing bounds check on sight is, frankly, one of the most valuable instincts in computer security.

What is Next

Spotting vulnerabilities in disassembly is one half of the picture. The next lesson flips to the defender's side — binary hardening. You'll meet the compiler and linker flags that add stack canaries, enable ASLR, enforce read-only relocations, and more, and you'll see exactly what each flag changes in the emitted assembly.