Identifying Vulnerabilities in Disassembly
Security researchers, malware analysts, and CTF competitors find vulnerabilities by reading disassembly all the time. When source isn't available — closed-source software, firmware, malware — disassembly is what you have to work with. This lesson trains you to recognise the assembly-level fingerprints of exploitable bugs: missing bounds checks, dangerous function calls, integer overflows, format-string sinks, and absent stack protections.
For the theoretical foundation on reading disassembly, see Reading Disassembly.
Pattern 1: Missing Bounds Checks
The classic vulnerability shape: data copied or accessed without anyone vetting the index or length.
Vulnerable pattern (no bounds check before array access):
; User-controlled index in rdi, array base in rsi; Vulnerable: no CMP/JB before the accessmov rax, qword [rsi + rdi*8] ; array[user_index] -- no check!Safe pattern (bounds check present):
; Safe: checks index before accesscmp rdi, 100 ; is index < 100?jae .out_of_bounds ; if not, handle errormov rax, qword [rsi + rdi*8] ; array[user_index] -- checkedWhen you see an array access (base + index * scale) without a cmp and conditional jump in front of it, the access could well be exploitable if an attacker controls the index.
What to scan for:
- Memory access with a register-driven index:
[base + reg*scale] - No
cmporteston that register in the preceding 5–10 instructions - The indexing register pulled from user input (a function arg, a buffer read from a file or network)
Pattern 2: Dangerous Function Calls
A handful of C library functions are dangerous by design — they perform unbounded operations. Spot them by their PLT entries or resolved addresses:
; DANGEROUS: call to gets() -- no length limit, always vulnerablecall gets
; DANGEROUS: call to strcpy() -- copies until null, no size checklea rdi, [rbp-64] ; destination buffer (64 bytes from rbp)call strcpy
; DANGEROUS: call to sprintf() -- unbounded format outputcall sprintf
; DANGEROUS: call to scanf with %s -- unbounded string readlea rdi, [rel .format_s] ; "%s" formatlea rsi, [rbp-32] ; 32-byte buffercall scanfSafer alternatives you should see instead:
; SAFER: fgets with explicit lengthmov edx, 64 ; max bytes (third arg)call fgets
; BOUNDED, BUT CHECK THE TERMINATOR: strncpy with length parametermov rdx, 64 ; max length (third arg)call strncpy ; no NUL written if the source is 64+ bytes
; SAFER: snprintf with buffer sizemov esi, 64 ; buffer size (second arg)call snprintfstrncpy deserves a second look whenever you meet it. If the source fills the limit it doesn't write a terminator, so check for a following store of 0 to dest[n-1]. snprintf and an explicit length check plus memcpy are the cleaner choices.
If you see call gets, call strcpy, or call sprintf in a disassembly, congratulations — you've probably just found a vulnerability. These functions are so notoriously bad that compilers now warn on use, and gets was outright removed from the C11 standard.
Pattern 3: Missing Stack Canary Checks
With -fstack-protector enabled, the compiler slips a canary — a random value — between the buffer and the saved rbp, and re-checks it before returning:
Protected function (canary present):
function_with_canary: push rbp mov rbp, rsp sub rsp, 80
; CANARY WRITE -- load canary from fs:[0x28] mov rax, qword fs:[0x28] ; read the canary value mov qword [rbp-8], rax ; store it on the stack xor eax, eax ; clear rax (prevent leaking canary)
; ... function body ...
; CANARY CHECK -- before return mov rax, qword [rbp-8] ; reload stored canary xor rax, qword fs:[0x28] ; compare with original jne .canary_fail ; if different, stack was corrupted!
leave ret
.canary_fail: call __stack_chk_fail ; abort the processUnprotected function (no canary):
function_without_canary: push rbp mov rbp, rsp sub rsp, 80
; No fs:[0x28] access anywhere ; ... function body ...
leave ret ; no canary check before return!A function that holds a local buffer yet never touches fs:[0x28] was compiled without stack protection. That's a loud signal that a buffer overflow inside it can overwrite the return address with nobody noticing.
The patterns in this lesson are what automated static-analysis tools hunt for. Coverity, CodeQL and Semgrep do it on source code; on a binary you're left with disassembler-based tools such as Ghidra's analyzers and scripts, plus checksec-style scans for missing hardening. Learning to recognise the same patterns by hand makes you sharper at triaging tool output and catches bugs the tools miss. This is daily-bread skill for vulnerability researchers, malware reverse engineers, and CTF players.
Pattern 4: Integer Overflow Points
Integer overflows happen when arithmetic on a size or length wraps into a value the code wasn't expecting. Tells to watch for:
; Potential integer overflow: user-controlled multiplicationmov eax, dword [rbp-4] ; user-controlled sizeimul eax, eax, 8 ; size * 8 (can overflow 32-bit!)mov edi, eax ; pass to malloccall malloc ; allocates LESS than expected
; If size = 0x20000001, then size * 8 = 0x100000008; But in 32-bit arithmetic: 0x100000008 truncated to 0x00000008; malloc(8) is called, but the caller thinks it allocated size*8 bytesWhat to scan for:
- Arithmetic on values loaded from user input (
imul,add,shl) - The result handed straight to
malloc,calloc, ormmap - No overflow check — no
jo(jump on overflow) after the arithmetic - 32-bit and 64-bit arithmetic mixed in the same flow (
mov eaxnext tomov rax)
Pattern 5: Format String Sinks
A format-string bug happens when user input is handed in as the format argument to anything in the printf family:
; VULNERABLE: user input as format stringlea rdi, [rbp-256] ; rdi = user-controlled buffermov eax, 0call printf ; printf(user_buffer) -- vulnerable!
; SAFE: format string is a constantlea rdi, [rel .LC0] ; rdi = constant "%s\n"lea rsi, [rbp-256] ; rsi = user data (as argument, not format)mov eax, 0call printf ; printf("%s\n", user_buffer) -- safeThe whole question collapses to one detail: is the first argument to printf a known constant (a lea from .rodata) or a pointer to attacker-controlled data on the stack or heap?
Pattern 6: Use-After-Free Indicators
; Potential use-after-free:call free ; free the object
; ... some code ...
mov rax, qword [rbx] ; access the freed object through rbx ; rbx still holds the old pointer!call qword [rax + 16] ; virtual function call on freed objectTells to scan for:
- A
call free(or an indirect jump into free) - The pointer register (here,
rbx) NOT cleared after the free - Later code dereferencing that same register
Pattern 7: Uninitialized Stack Variables
; Potential info leak via uninitialized stack variablesub rsp, 32 ; allocate 32 bytes; No write to [rsp+0] through [rsp+31] before readingmov edi, 1 ; fd (first arg), e.g. stdout or a socketmov rsi, rsp ; buffer (second arg)mov edx, 32 ; length (third arg)call write ; writes uninitialized stack contents to fd!Stack memory that's read or sent down a wire before it's written holds whatever the previous function call left behind — pointers, canaries, possibly keys.
Vulnerability Audit Checklist
Hover over each step to see details
When you're auditing a disassembled function, run the checklist top to bottom:
- Buffer operations: any
call strcpy/gets/sprintf? Can you reason about the buffer size? - Bounds checks: every array access preceded by
cmp/jae/jbe? - Stack protection: does the function read
fs:[0x28]for a canary? - Integer arithmetic: are size calculations checked for overflow?
- Format strings: is user data ever the first argument to a
printf-family function? - Memory management: after
call free, is the pointer zeroed? - Return-value checks: after
call malloc/call open, does the code actually test the result (test rax, rax; jz error)?
Real-World Application
Real-world auditing stacks these patterns up. A function that:
- has no stack canary (no
fs:[0x28]access) - calls
strcpyinto a stack buffer - takes user-controlled input as the source
- is compiled without PIE
…is a textbook exploitable buffer overflow. Each missing mitigation makes exploitation that much easier.
Flip the dials, and a function that:
- has a stack canary
- uses
snprintfor a length-checkedmemcpy - lives in a PIE binary with full RELRO
- pulls in
_FORTIFY_SOURCEwrappers (calls__strcpy_chkinstead ofstrcpy)
…hides a stack of defences the attacker has to peel apart layer by layer.
These vuln-identification skills are bread-and-butter across multiple security roles. Vulnerability researchers audit closed-source software straight from disassembly. Malware analysts map out the capabilities and infection vectors of samples. CTF players chase down the intended bug in challenge binaries. Security auditors confirm that the hardening flags somebody promised actually shipped in the production build. Looking at a disassembly and clocking a missing bounds check on sight is, frankly, one of the most valuable instincts in computer security.
What is Next
Spotting vulnerabilities in disassembly is one half of the picture. The next lesson flips to the defender's side — binary hardening. You'll meet the compiler and linker flags that add stack canaries, enable ASLR, enforce read-only relocations, and more, and you'll see exactly what each flag changes in the emitted assembly.