Skip to main content
Be Bitwise

Crash Analysis & Triage

Dynamic Analysis & Debugging for VR · VR Analysis & Discovery track

8 min read

In this module

Making Sense of Crashes

A fuzzer ran overnight and produced 500 crash files. A manual test caused a segfault. Production threw a core dump. Three situations, same three questions: What happened? Why did it happen? Can it be exploited? Crash analysis and triage is the systematic process of answering those questions without drowning.

Analysing Crash Dumps

When a program crashes, the operating system can save its state as a core dump (Linux/macOS) or minidump/crash dump (Windows). This snapshot contains register values, memory contents, and stack state at the moment of the crash.

bash
# Ensure core dumps are generated (Linux)
ulimit -c unlimited
echo '/tmp/cores/core.%e.%p' | sudo tee /proc/sys/kernel/core_pattern
# Analyse a core dump with GDB
gdb ./target_program /tmp/cores/core.target.12345
(gdb) bt # Backtrace: what was the call chain?
(gdb) bt full # Backtrace with local variables
(gdb) info registers # Register state at crash
(gdb) x/i $rip # What instruction crashed?
(gdb) x/32xg $rsp # Stack contents at crash

The first thing to examine is the faulting instruction and the register state:

text
Program received signal SIGSEGV, Segmentation fault.
0x0000000000401234 in process_chunk (data=0x7fff50000000, len=0x41414141)
at parser.c:142
142 memcpy(output, data + offset, chunk_len);
(gdb) info registers
rdi 0x602000000010 # destination: output buffer (heap)
rsi 0x7fff50000008 # source address
rdx 0x41414141 # length - this is 'AAAA'!
rip 0x401234 # instruction that crashed

The 0x41414141 in RDX (the length argument to memcpy) is a clear sign of attacker-controlled data reaching a dangerous operation. The ASCII pattern 'AAAA' is the classic buffer fill used in exploit development.

Determining Exploitability

Not all crashes are exploitable. A NULL pointer dereference is mostly noise. A heap overflow that corrupts allocator metadata is a completely different situation. Systematic triage classifies crashes by exploitability:

EXPLOITABLE indicators:

  • Program counter (RIP/EIP) contains attacker-controlled data - direct control-flow hijack
  • Write to an attacker-controlled address - at least a write-where, and a write-what-where if the value written is controlled too
  • Call or jump through an attacker-controlled function pointer - vtable corruption, GOT overwrite
  • Heap metadata corruption detected by the allocator - potential heap exploitation

PROBABLY EXPLOITABLE indicators:

  • Write access violation near a heap chunk boundary - possible heap overflow
  • Stack buffer overflow that reaches but does not fully control the return address
  • Use-after-free detected by ASan - the freed object might be reclaimable

UNKNOWN indicators:

  • Read access violation from an attacker-influenced address - information disclosure, might be a stepping stone
  • Division by zero with attacker-controlled divisor - usually just DoS
  • Assertion failure in security-relevant code - logic error, needs investigation

PROBABLY NOT EXPLOITABLE indicators:

  • NULL pointer dereference at a fixed offset - typically a missing NULL check
  • Read access violation at a non-mappable address - usually just a crash
  • Stack overflow from deep recursion - typically DoS only
Automated Exploitability Assessment

Microsoft's WinDbg extension !exploitable and the GDB plugin exploitable automate this classification. They examine the crash state and apply heuristics to categorise the crash. These tools are useful for bulk triage but are not infallible - some crashes they classify as "unknown" turn out to be highly exploitable, and some "exploitable" crashes may have mitigations that prevent practical exploitation. Always verify manually for high-value targets.

Classifying Crashes

Crash symptoms map to specific vulnerability classes:

Read Access Violation (read AV) - the program tried to read from an unmapped or protected address. Check what address it tried to read:

  • Near-NULL read (0x0 to 0xFFFF): likely NULL pointer dereference
  • Pattern-like address (0x41414141): attacker-controlled data used as a pointer
  • Just past a heap allocation: out-of-bounds read (information disclosure)

Write Access Violation (write AV) - the program tried to write to an unmapped or protected address. Write AVs are generally more exploitable than read AVs:

  • Pattern-like target address: the attacker influences where the write goes. That's write-what-where only if they also control the value written, so check the source operand before you call it highly exploitable
  • Just past a heap allocation: heap buffer overflow
  • Stack address past the guard page: stack overflow

Use-After-Free indicators:

  • Crash accessing a freed heap chunk (ASan explicitly reports this)
  • Crash with a pointer to freed memory visible in registers
  • Non-deterministic crashes at the same code location (the freed memory is sometimes reallocated)

Double-Free indicators:

  • Allocator aborts with "double free detected"
  • Corruption in allocator metadata
  • Crash inside free() or operator delete

Root Cause Analysis

Once you have classified the crash, trace backward to find the root cause - the actual bug that allows the corruption:

  1. Start at the crash point - what operation failed?
  2. Examine the corrupted data - where did the bad values come from?
  3. Set watchpoints - if a memory location is corrupted, set a watchpoint on it and re-run to catch the corruption
  4. Trace backward through the call chain - the backtrace shows which function called which, but the root cause may be several frames up
  5. Bisect the input - if the crash comes from a large input, try progressively smaller inputs to isolate which part triggers the bug
bash
# Use ASan for precise root cause identification
ASAN_OPTIONS=detect_leaks=0 ./target_asan input_crash
# ASan provides the exact location of the bug:
# ==12345==ERROR: AddressSanitizer: heap-buffer-overflow
# WRITE of size 4 at 0x602000000044
# #0 0x401234 in parse_chunk parser.c:142
# #1 0x401567 in process_file main.c:87
#
# 0x602000000044 is located 4 bytes after 64-byte region
# [0x602000000000,0x602000000040)
# allocated by thread T0 here:
# #0 0x... in malloc
# #1 0x4011ff in parse_chunk parser.c:130

ASan tells you: a 64-byte allocation at parser.c:130 was overflowed by 4 bytes at parser.c:142. This is the root cause - now examine line 130 (the allocation) and line 142 (the write) to understand why the size was wrong.

Minimizing Test Cases

A fuzzer might produce a 10MB crash file, but the actual vulnerability might be triggered by just 50 bytes. Test case minimisation reduces the crash input to the smallest version that still triggers the same bug:

bash
# AFL++'s automatic minimiser
afl-tmin -i crash_input -o minimised_crash -- ./target @@
# Manual binary search approach
# 1. Try first half of input
head -c $(($(wc -c < crash_input) / 2)) crash_input > half.bin
./target half.bin # Does it still crash?
# 2. If yes, continue halving
# 3. If no, try second half, or first 3/4, etc.

Minimised test cases matter for everything downstream: fewer bytes means less noise in root cause analysis, vendors actually engage with minimal reproducers, and comparing minimised crashes reveals whether different inputs are hitting the same underlying bug.

Crash Deduplication

When a fuzzer produces hundreds of crashes, many will be duplicate triggers of the same underlying bug. Deduplication groups crashes by root cause:

Stack hash deduplication - compute a hash of the crashing function and its callers. Crashes with the same stack hash are likely the same bug:

bash
# Extract crash location from ASan output
grep "^ #0" crash_*.log | sort | uniq -c | sort -rn
# Groups crashes by the immediate crash function
# More sophisticated: hash the top 3 stack frames
for log in crash_*.log; do
hash=$(grep "^ #[0-2]" "$log" | md5sum | cut -d' ' -f1)
echo "$hash $log"
done | sort | uniq -w32 -c | sort -rn

Coverage-based deduplication - crashes that exercise the same code paths (same coverage bitmap) are likely the same bug. AFL++ does this automatically with its crash grouping.

Address-based deduplication - crashes at the same instruction address with the same fault type are often the same bug, though different inputs might trigger them through different paths.

After deduplication, 500 crash files typically collapse to 5–20 unique bugs. Work them in exploitability order: EXPLOITABLE first, then PROBABLY_EXPLOITABLE, and so on down the list.

The Triage Pipeline

Professional vulnerability research teams build automated triage pipelines: fuzzer produces crashes -> crashes are minimised -> minimised crashes are deduplicated -> unique crashes are run through exploitability analysis -> results are sorted by severity -> humans investigate the top findings. This pipeline can process thousands of crashes per day, surfacing only the most promising bugs for manual analysis.