Skip to main content
Be Bitwise

Practical Instrumentation

Function Hooking & Instrumentation · Reverse Engineering track

7 min read

In this module

Putting It All Together

Theory only carries you so far. Hooking is a craft, learned by sitting in front of real binaries with a real script open. The scenarios below are the kind of jobs that land on a security researcher's desk in any given week. Each one trains a different reflex.

Scenario 1: API Call Tracing

You've got a suspicious binary and you'd rather not run it on metal you care about. Hook a few choice APIs, sandbox the thing, and watch what it does:

javascript
// Frida script: trace file and network operations
var apis = ['open', 'read', 'write', 'connect', 'send', 'recv'];
apis.forEach(function(name) {
var addr = Module.findGlobalExportByName(name); // null if not found
if (addr) {
Interceptor.attach(addr, {
onEnter: function(args) {
console.log('[' + name + '] called from ' +
Thread.backtrace(this.context, Backtracer.ACCURATE)
.map(DebugSymbol.fromAddress).join('\n '));
}
});
}
});

Files opened, data read and written, network connections opened — all of it surfaces, with the binary unchanged. The backtrace is the part that earns its keep. Knowing which API got called isn't enough; you need to know who in the program reached for it. A connect originating from a buried library the program ships looks a lot more suspect than the same call from a well-known networking framework.

Key Concept

Practical instrumentation starts with identifying the right functions to hook. For behaviour analysis, focus on system API calls (file I/O, network, process creation). For protocol analysis, hook encryption/decryption functions. For vulnerability research, hook memory allocation and string operations. The key is to match your hooking strategy to your analytical goal.

Scenario 2: Decrypting Network Traffic

A program speaks TLS to its server and you want plaintext. Setting up a MITM proxy means wrestling with certificate pinning, system trust stores, and validation logic. Skip the whole song and dance — hook the encryption routines inside the program:

javascript
// Hook SSL_write to capture outgoing plaintext
Interceptor.attach(Process.getModuleByName('libssl.so').getExportByName('SSL_write'), {
onEnter: function(args) {
var buf = args[1];
var len = args[2].toInt32();
console.log('SSL_write: ' + buf.readUtf8String(len));
}
});

You catch the data after the application has assembled it, before libssl ever sees it. Plaintext, no broken crypto. Mirror the trick on SSL_read for the inbound side, and the two hooks together give you the whole conversation, whatever cipher suite is in play.

Mobile researchers live by this technique. Modern apps tend to pin their server certs, which kills traditional MITM proxying stone dead. Hook a layer up at SSL_write/read and the app does the decryption work for you, willingly.

Scenario 3: Bypassing Authentication

During a security assessment, you might need to bypass a login check to test the rest of an application:

javascript
// Hook the authentication function and force success.
// 0x1234 is the function's offset from the module base, taken from your
// disassembler (rebase the program to 0 or subtract its image base).
var authCheck = Process.getModuleByName('target_app').base.add(0x1234);
Interceptor.attach(authCheck, {
onLeave: function(retval) {
console.log('Original auth result: ' + retval);
retval.replace(1); // force return true
console.log('Patched to: success');
}
});

Why base plus offset rather than the address your disassembler shows? With PIE and ASLR the module loads somewhere different on every run, so a hard-coded ptr('0x401234') only works for a non-PIE binary. The offset within the module stays the same.

The auth function's return value gets rewritten in flight, with nothing touched on disk. Temporary, non-destructive, and gone the moment you detach. Standard caveat: do this on systems you've been hired to test. Doing it elsewhere is a crime in most jurisdictions.

Scenario 4: Code Coverage for Fuzzing

When you're fuzzing — bombing a target with malformed inputs hoping something breaks — coverage tells you which parts of the program your inputs actually reach. DBI frameworks collect that for free:

text
// DynamoRIO code coverage client (conceptual)
for each basic_block executed:
record basic_block.address
record basic_block.size
// After fuzzing:
// Analyse which basic blocks were reached
// Generate new inputs targeting unreached blocks

AFL and LibFuzzer do something similar, usually via compiler instrumentation rather than DBI (though AFL++'s QEMU and Frida modes get the same coverage from closed-source binaries with DBI), to nudge their input generators towards unexplored code paths. The idea is almost embarrassingly simple. If an input reaches a basic block no prior input reached, that input is "interesting" and worth mutating further. The resulting feedback loop finds bugs orders of magnitude faster than blind random testing.

Scenario 5: Performance Profiling

Hooks turn into a profiler when you point them at timing. How long does a function run? How often does it fire? Who's calling it?

javascript
var callCount = {};
var totalTime = {};
Interceptor.attach(Process.getModuleByName('target_app').base.add(0x2000), { // process_data
onEnter: function() {
this.start = Date.now();
},
onLeave: function() {
var elapsed = Date.now() - this.start;
var name = 'process_data';
callCount[name] = (callCount[name] || 0) + 1;
totalTime[name] = (totalTime[name] || 0) + elapsed;
}
});

For closed-source binaries, where firing up perf or gprof isn't an option, this is your profiler. Hook at whatever granularity suits the question — single functions, whole modules — and the bottlenecks, hot paths, and call frequencies map themselves.

Combining Techniques

A real session almost never sticks to one technique. The usual rhythm goes something like this. Start with wide-net API tracing to see what the program is up to. Trace results point you at interesting functions. Hook those, capture arguments and return values, build a behavioural map. Then zoom in for the deep dive — maybe rewrite arguments to fish for error-handling bugs, maybe flip return values to surface hidden code paths.

Best Practices

Start broad, then narrow down. Hook a wide net of APIs first, get a feel for what the program does, then zero in on the functions that matter for your specific question.

Use backtraces. When a hook trips, grab the call stack. Often the who of a call is more interesting than the what — arguments tell you the call shape, the stack tells you the intent.

Watch for reentrancy. Your hook might call something you've also hooked, and now you're in an infinite loop. Set a thread-local flag and bail early on the second entry.

Mind the performance. Logging every malloc will grind the program to a halt and fill your terminal with megabytes of garbage. Filter aggressively, or sample.

Take notes. Instrumentation is exploratory by nature. Write down what you hooked, what came back, what surprised you. Today's frantic experiment is tomorrow's reproducible analysis — but only if you wrote it down.