Tracing & Instrumentation
Dynamic Analysis & Debugging for VR · VR Analysis & Discovery track
9 min read
In this module
Observing Program Behaviour at Every Level
Debuggers let you pause and inspect at specific points. Tracing tools give you the full film: every system call, every library function, every executed instruction. That continuous record reveals behaviour patterns that point-in-time debugging misses, and it's particularly useful when you don't yet know where to look.
System Call Tracing with strace/ltrace
strace intercepts every system call a program makes, showing you the program's complete interaction with the operating system:
# Basic strace: trace all system callsstrace ./target_program input_file
# Filter to specific system call categoriesstrace -e trace=network ./server # Only network callsstrace -e trace=file ./parser input # Only file operationsstrace -e trace=memory ./allocator # Only memory management
# Follow forked child processesstrace -f ./daemon
# Attach to a running processstrace -p $(pidof target_process)
# Show timestamps and call durationsstrace -T -t ./target_programA typical strace output for a network server processing a request:
accept(3, {sa_family=AF_INET, sin_port=htons(12345)}, [16]) = 4read(4, "\x01\x00\x00\x08AAAAAAAA", 4096) = 12brk(0x5562a3c21000) = 0x5562a3c21000write(4, "OK\n", 3) = 3close(4) = 0For vulnerability research, strace reveals:
- What files the program opens (potential TOCTOU targets)
- What network operations it performs (protocol behaviour)
- Memory allocation patterns (mmap, brk calls)
- Process creation (fork/exec for potential privilege escalation)
ltrace is similar but traces library function calls rather than system calls:
# Trace library callsltrace ./target_program input_file
# Output shows function calls with arguments:# malloc(256) = 0x55a3b4c00010# strcpy(0x55a3b4c00010, "AAAA...") = 0x55a3b4c00010# strlen("AAAA...") = 300# free(0x55a3b4c00010)ltrace shows you exactly which library functions are called with which arguments. Seeing strcpy fed a 300-byte string destined for a 256-byte buffer makes the problem obvious immediately.
The runner fakes ltrace the way an LD_PRELOAD shim would: the parser's calls go through wrappers that print each call and its return value. Read the trace first and find the bug from the log alone, the way you would with a binary you have no source for. Only then read parse_record.
API Monitor on Windows
On Windows, API Monitor provides the equivalent of strace/ltrace with a rich GUI. It intercepts calls to Windows API functions and displays them with full argument decoding:
- Win32 API calls (CreateFile, ReadFile, WriteFile)
- Socket operations (WSARecv, connect, send)
- Registry operations (RegOpenKeyEx, RegSetValueEx)
- Cryptographic operations (CryptEncrypt, CryptDecrypt)
API Monitor's filtering and breakpoint capabilities make it practical for monitoring complex Windows applications that make thousands of API calls per second.
All tracing tools add overhead to the target program. strace uses ptrace, which requires a context switch for every system call - expect 2-10x slowdown. ltrace uses similar mechanisms for library calls. Intel PIN's instrumentation adds 5-50x overhead depending on the analysis complexity. For VR, this overhead is usually acceptable because you are analysing specific behaviours, not running production workloads.
Intel PIN for Custom Instrumentation
When strace and ltrace are not enough, Intel PIN lets you build completely custom instrumentation. You write a "pintool" - a shared library that PIN loads alongside the target program:
// coverage_tool.cpp - Record which basic blocks execute#include "pin.H"#include <fstream>#include <set>
std::set<ADDRINT> executed_blocks; // not thread-safe: wrap inserts in a PIN_LOCK for multithreaded targetsstd::ofstream outfile;
VOID RecordBlock(ADDRINT addr) { executed_blocks.insert(addr);}
VOID Trace(TRACE trace, VOID *v) { for (BBL bbl = TRACE_BblHead(trace); BBL_Valid(bbl); bbl = BBL_Next(bbl)) { BBL_InsertCall(bbl, IPOINT_BEFORE, (AFUNPTR)RecordBlock, IARG_ADDRINT, BBL_Address(bbl), IARG_END); }}
VOID Fini(INT32 code, VOID *v) { outfile.open("coverage.log"); for (ADDRINT addr : executed_blocks) { outfile << std::hex << addr << std::endl; } outfile.close();}
int main(int argc, char *argv[]) { PIN_Init(argc, argv); TRACE_AddInstrumentFunction(Trace, 0); PIN_AddFiniFunction(Fini, 0); PIN_StartProgram(); return 0;}This pintool records every basic block the program executes, producing a coverage map. Compare coverage maps between different inputs to find code paths that only specific inputs reach - potential areas where edge-case handling might contain vulnerabilities.
Recording Execution Traces
Full execution traces record every instruction executed, enabling post-hoc analysis:
# Record a trace with Intel PINpin -t trace_tool.so -- ./target_program input_file
# Record with QEMU (full system trace)qemu-system-x86_64 -d exec,in_asm -D trace.log ...Traces balloon fast — gigabytes for seconds of execution — but they enable analysis you can't do any other way:
Diff-based analysis — record traces for two similar inputs where one crashes and one doesn't. Diff them to find where behaviour diverges. The divergence point often is the vulnerability:
# Record trace with normal inputpin -t trace.so -- ./parser normal_input > trace_normal.log
# Record trace with crashing inputpin -t trace.so -- ./parser crash_input > trace_crash.log
# Find where they divergediff trace_normal.log trace_crash.log | head -50The first difference shows where the crashing input causes the program to take a different code path - this is often the point where insufficient validation allows corrupted data through. One catch: with ASLR on, every address in the two traces differs, so a raw diff is all noise. Run both under setarch -R to turn ASLR off for that process, or rewrite addresses as module-plus-offset before diffing.
PIN isn't the only way to get a trace, and on Linux it's often not the first thing people reach for now. rr records a run once and replays it deterministically under GDB, reverse execution included. Intel Processor Trace, captured with perf record -e intel_pt// on CPUs that support it, logs control flow in hardware at a fraction of PIN's overhead. Frida injects JavaScript hooks into a running process, and bpftrace attaches eBPF probes to kernel and user functions without touching the target binary.
Coverage Measurement
Code coverage measures which parts of the program execute for a given input. For VR, coverage serves two purposes:
- Fuzzing guidance - coverage-guided fuzzers use coverage to detect when an input explores new code
- Audit completeness - after manual testing, coverage shows which code paths you have not exercised
# Compile with coverage instrumentationgcc -fprofile-arcs -ftest-coverage -g target.c -o target
# Run with test inputs./target input1.bin./target input2.bin
# Generate coverage reportgcov target.clcov --capture --directory . --output-file coverage.infogenhtml coverage.info --output-directory coverage_htmlThe coverage report shows exactly which lines and branches were executed. Lines and branches in security-critical code (parsers, validators) that are never executed by your test inputs are potential hiding spots for untested, vulnerable code.
QEMU for Full-System Tracing
When you need to trace an entire system (including kernel code, drivers, and all processes), QEMU provides full-system tracing capabilities:
# Log all executed translation blocks (TCG only: no -enable-kvm)qemu-system-x86_64 -d exec -D /tmp/qemu_trace.log \ -drive file=research.qcow2
# Log each block's guest instructions and TCG ops when it's translated# (once per translation, not every time it runs)qemu-system-x86_64 -d in_asm,op -D /tmp/qemu_detailed.log \ -drive file=research.qcow2These -d logs come from QEMU's TCG translator, so they're empty under KVM, where guest code runs natively on the CPU. Expect TCG to be much slower. QEMU tracing captures everything: user-space code, system calls, kernel functions, interrupt handlers, and device driver code. This is essential for kernel vulnerability research where the bug spans the user-kernel boundary.
The tradeoff is volume - full-system traces generate enormous amounts of data. Use QEMU's filtering options to focus on specific address ranges or translation blocks to keep traces manageable.