Skip to main content
Be Bitwise

Tracing & Instrumentation

Dynamic Analysis & Debugging for VR · VR Analysis & Discovery track

9 min read

In this module

Observing Program Behaviour at Every Level

Debuggers let you pause and inspect at specific points. Tracing tools give you the full film: every system call, every library function, every executed instruction. That continuous record reveals behaviour patterns that point-in-time debugging misses, and it's particularly useful when you don't yet know where to look.

System Call Tracing with strace/ltrace

strace intercepts every system call a program makes, showing you the program's complete interaction with the operating system:

bash
# Basic strace: trace all system calls
strace ./target_program input_file
# Filter to specific system call categories
strace -e trace=network ./server # Only network calls
strace -e trace=file ./parser input # Only file operations
strace -e trace=memory ./allocator # Only memory management
# Follow forked child processes
strace -f ./daemon
# Attach to a running process
strace -p $(pidof target_process)
# Show timestamps and call durations
strace -T -t ./target_program

A typical strace output for a network server processing a request:

text
accept(3, {sa_family=AF_INET, sin_port=htons(12345)}, [16]) = 4
read(4, "\x01\x00\x00\x08AAAAAAAA", 4096) = 12
brk(0x5562a3c21000) = 0x5562a3c21000
write(4, "OK\n", 3) = 3
close(4) = 0

For vulnerability research, strace reveals:

  • What files the program opens (potential TOCTOU targets)
  • What network operations it performs (protocol behaviour)
  • Memory allocation patterns (mmap, brk calls)
  • Process creation (fork/exec for potential privilege escalation)

ltrace is similar but traces library function calls rather than system calls:

bash
# Trace library calls
ltrace ./target_program input_file
# Output shows function calls with arguments:
# malloc(256) = 0x55a3b4c00010
# strcpy(0x55a3b4c00010, "AAAA...") = 0x55a3b4c00010
# strlen("AAAA...") = 300
# free(0x55a3b4c00010)

ltrace shows you exactly which library functions are called with which arguments. Seeing strcpy fed a 300-byte string destined for a 256-byte buffer makes the problem obvious immediately.

The runner fakes ltrace the way an LD_PRELOAD shim would: the parser's calls go through wrappers that print each call and its return value. Read the trace first and find the bug from the log alone, the way you would with a binary you have no source for. Only then read parse_record.

Try itFind the overflow from the trace alone, then change the name length from 300 to 255 and to 256 and work out which is the first length that overflows.
C (GCC 9.2.0)

API Monitor on Windows

On Windows, API Monitor provides the equivalent of strace/ltrace with a rich GUI. It intercepts calls to Windows API functions and displays them with full argument decoding:

  • Win32 API calls (CreateFile, ReadFile, WriteFile)
  • Socket operations (WSARecv, connect, send)
  • Registry operations (RegOpenKeyEx, RegSetValueEx)
  • Cryptographic operations (CryptEncrypt, CryptDecrypt)

API Monitor's filtering and breakpoint capabilities make it practical for monitoring complex Windows applications that make thousands of API calls per second.

Tracing Overhead

All tracing tools add overhead to the target program. strace uses ptrace, which requires a context switch for every system call - expect 2-10x slowdown. ltrace uses similar mechanisms for library calls. Intel PIN's instrumentation adds 5-50x overhead depending on the analysis complexity. For VR, this overhead is usually acceptable because you are analysing specific behaviours, not running production workloads.

Intel PIN for Custom Instrumentation

When strace and ltrace are not enough, Intel PIN lets you build completely custom instrumentation. You write a "pintool" - a shared library that PIN loads alongside the target program:

cpp
// coverage_tool.cpp - Record which basic blocks execute
#include "pin.H"
#include <fstream>
#include <set>
std::set<ADDRINT> executed_blocks; // not thread-safe: wrap inserts in a PIN_LOCK for multithreaded targets
std::ofstream outfile;
VOID RecordBlock(ADDRINT addr) {
executed_blocks.insert(addr);
}
VOID Trace(TRACE trace, VOID *v) {
for (BBL bbl = TRACE_BblHead(trace); BBL_Valid(bbl); bbl = BBL_Next(bbl)) {
BBL_InsertCall(bbl, IPOINT_BEFORE, (AFUNPTR)RecordBlock,
IARG_ADDRINT, BBL_Address(bbl), IARG_END);
}
}
VOID Fini(INT32 code, VOID *v) {
outfile.open("coverage.log");
for (ADDRINT addr : executed_blocks) {
outfile << std::hex << addr << std::endl;
}
outfile.close();
}
int main(int argc, char *argv[]) {
PIN_Init(argc, argv);
TRACE_AddInstrumentFunction(Trace, 0);
PIN_AddFiniFunction(Fini, 0);
PIN_StartProgram();
return 0;
}

This pintool records every basic block the program executes, producing a coverage map. Compare coverage maps between different inputs to find code paths that only specific inputs reach - potential areas where edge-case handling might contain vulnerabilities.

Recording Execution Traces

Full execution traces record every instruction executed, enabling post-hoc analysis:

bash
# Record a trace with Intel PIN
pin -t trace_tool.so -- ./target_program input_file
# Record with QEMU (full system trace)
qemu-system-x86_64 -d exec,in_asm -D trace.log ...

Traces balloon fast — gigabytes for seconds of execution — but they enable analysis you can't do any other way:

Diff-based analysis — record traces for two similar inputs where one crashes and one doesn't. Diff them to find where behaviour diverges. The divergence point often is the vulnerability:

bash
# Record trace with normal input
pin -t trace.so -- ./parser normal_input > trace_normal.log
# Record trace with crashing input
pin -t trace.so -- ./parser crash_input > trace_crash.log
# Find where they diverge
diff trace_normal.log trace_crash.log | head -50

The first difference shows where the crashing input causes the program to take a different code path - this is often the point where insufficient validation allows corrupted data through. One catch: with ASLR on, every address in the two traces differs, so a raw diff is all noise. Run both under setarch -R to turn ASLR off for that process, or rewrite addresses as module-plus-offset before diffing.

PIN isn't the only way to get a trace, and on Linux it's often not the first thing people reach for now. rr records a run once and replays it deterministically under GDB, reverse execution included. Intel Processor Trace, captured with perf record -e intel_pt// on CPUs that support it, logs control flow in hardware at a fraction of PIN's overhead. Frida injects JavaScript hooks into a running process, and bpftrace attaches eBPF probes to kernel and user functions without touching the target binary.

Coverage Measurement

Code coverage measures which parts of the program execute for a given input. For VR, coverage serves two purposes:

  1. Fuzzing guidance - coverage-guided fuzzers use coverage to detect when an input explores new code
  2. Audit completeness - after manual testing, coverage shows which code paths you have not exercised
bash
# Compile with coverage instrumentation
gcc -fprofile-arcs -ftest-coverage -g target.c -o target
# Run with test inputs
./target input1.bin
./target input2.bin
# Generate coverage report
gcov target.c
lcov --capture --directory . --output-file coverage.info
genhtml coverage.info --output-directory coverage_html

The coverage report shows exactly which lines and branches were executed. Lines and branches in security-critical code (parsers, validators) that are never executed by your test inputs are potential hiding spots for untested, vulnerable code.

QEMU for Full-System Tracing

When you need to trace an entire system (including kernel code, drivers, and all processes), QEMU provides full-system tracing capabilities:

bash
# Log all executed translation blocks (TCG only: no -enable-kvm)
qemu-system-x86_64 -d exec -D /tmp/qemu_trace.log \
-drive file=research.qcow2
# Log each block's guest instructions and TCG ops when it's translated
# (once per translation, not every time it runs)
qemu-system-x86_64 -d in_asm,op -D /tmp/qemu_detailed.log \
-drive file=research.qcow2

These -d logs come from QEMU's TCG translator, so they're empty under KVM, where guest code runs natively on the CPU. Expect TCG to be much slower. QEMU tracing captures everything: user-space code, system calls, kernel functions, interrupt handlers, and device driver code. This is essential for kernel vulnerability research where the bug spans the user-kernel boundary.

The tradeoff is volume - full-system traces generate enormous amounts of data. Use QEMU's filtering options to focus on specific address ranges or translation blocks to keep traces manageable.