From Crash to Root Cause
Zero-Day Discovery Workflow · Zero-Day Research track
6 min read
In this module
The Crash Is Just the Beginning
A fuzzer produces thousands of crashes. A manual audit surfaces suspicious code paths. But a crash on its own tells you almost nothing. It could be catastrophic heap corruption enabling remote code execution — or it could be a benign assertion failure with zero security consequence. Crash triage is the process of sorting them out: classify each crash, determine root cause, and judge whether it's exploitable.
This is where the automated part ends and the skilled analysis begins. The tools surface the crashes. You figure out what they mean.
Crash Triage Methodology
When you encounter a crash, follow a systematic process:
Step 1: Reproduce the Crash
Before any analysis, confirm that the crash is reproducible. Same input, same conditions. If it's not deterministic, note what it seems to depend on — timing (likely a race), memory layout (heap corruption that behaves differently under ASLR), or environment quirks.
Step 2: Run Under a Debugger
Load the crashing input in a debugger to capture the full state at the moment of the crash.
GDB (Linux):
$ gdb --args ./target_app crash_input.dat(gdb) run# Programme crashes(gdb) bt # Full backtrace (call stack)(gdb) info registers # Register state at crash(gdb) x/10i $rip # Disassemble at crash instruction(gdb) x/32xw $rsp # Examine stack contentsWinDbg (Windows):
0:000> !analyze -v # Automated crash analysis0:000> kb # Stack backtrace with parameters0:000> r # Register state0:000> u @rip L5 # Disassemble at crash point0:000> dd @rsp L20 # Examine stackStep 3: Examine the Crash Site
At the crash site, answer these questions:
- What instruction caused the crash? Is it a read, a write, a call, or a return?
- What address was accessed? Is it a null pointer (near 0x0), an unmapped page, a freed heap chunk, or a stack address?
- What are the register values? Do any registers contain attacker-controlled data (recognisable patterns from the input file)?
- What does the call stack show? Which function was the programme executing, and what was the call chain that led to this point?
Step 4: Classify the Crash Type
The crash characteristics tell you the vulnerability class:
| Crash Signature | Likely Vulnerability |
|---|---|
| Write to an address slightly past a heap allocation | Heap buffer overflow |
| Access to a freed heap chunk (identifiable by freed metadata patterns) | Use-after-free |
Abort in __stack_chk_fail ("stack smashing detected"), or SIGSEGV at ret with a corrupted return address | Stack buffer overflow (the write runs upwards, towards the saved return address) |
| Fault just below the stack's lowest mapped page, often after very deep recursion | Stack exhaustion (unbounded recursion or a huge stack allocation) |
| Dereferencing address 0x0 or a very small value | Null pointer dereference |
| Instruction pointer contains data from the input | Control flow hijack (highly exploitable) |
| Integer value causing a too-small allocation followed by overflow | Integer overflow leading to buffer overflow |
Compile the target with AddressSanitizer (ASan), MemorySanitizer (MSan), or UndefinedBehaviourSanitizer (UBSan) to get precise diagnostic output. ASan detects heap overflows, use-after-free, and stack overflows with exact allocation and deallocation backtraces. This transforms a cryptic crash into an immediately classifiable vulnerability. Always triage with sanitisers enabled when possible.
# Compile with AddressSanitizer for precise diagnostics$ gcc -fsanitize=address -g -o target_asan target.c$ ./target_asan crash_input.dat
# ASan output clearly identifies the vulnerability:# ==12345==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000000050# WRITE of size 4 at 0x602000000050 thread T0# #0 0x4011a3 in parse_ifd_entry src/tiff_decode.c:142# #1 0x401067 in process_tiff src/tiff_decode.c:89# 0x602000000050 is located 0 bytes after 16-byte region [0x602000000040, 0x602000000050)# allocated by thread T0 here:# #0 0x7f... in malloc# #1 0x401034 in parse_ifd_entry src/tiff_decode.c:138Step 5: Deduplicate
Fuzzing campaigns often produce hundreds or thousands of crashes, many of which are different inputs triggering the same underlying bug. Deduplication groups these crashes so you analyse each unique vulnerability only once.
Common deduplication strategies:
- Stack hash: Hash the top N frames of the call stack at the crash point. Crashes with the same stack trace are likely the same bug.
- Crash address bucketing: Group crashes that occur at the same instruction address.
- Coverage-based: Compare code coverage profiles of crashing inputs. Inputs that exercise the same code paths likely trigger the same bug.
No single strategy is perfect. Stack hashing can merge distinct bugs that happen to crash in the same function, and it can split a single bug that manifests at different depths depending on the input. Use multiple signals and manual review for important targets.
Step 6: Assess Exploitability
Not all crashes are exploitable. Exploitability assessment determines whether an attacker can turn the crash into a security-relevant outcome:
High exploitability indicators:
- Attacker controls the instruction pointer (RIP/EIP)
- Attacker controls the destination of a write operation
- Use-after-free where the freed object's type and allocation size are controllable
- Heap overflow into an adjacent object with function pointers or vtable pointers
Low exploitability indicators:
- Null pointer dereference (typically a denial of service only, though kernel null derefs can be more serious)
- Read-only out-of-bounds access (information leak, not code execution, though this can be a building block)
- Assertion failures or controlled programme exits
A vulnerability that appears unexploitable in isolation may become exploitable when chained with other bugs. A read-only out-of-bounds access can defeat ASLR, enabling a subsequent write primitive to achieve code execution. Always consider the vulnerability in the context of the full attack surface, not in isolation.
Microsoft's !exploitable WinDbg extension automates a basic assessment, classifying crashes as EXPLOITABLE, PROBABLY_EXPLOITABLE, PROBABLY_NOT_EXPLOITABLE, or UNKNOWN. On Linux, crashwalk does similar work. Both are useful for initial sorting — but they're a first pass, not a verdict. Promising crashes need manual eyes.