Code Obfuscation
Anti-Reversing Techniques · Advanced Reverse Engineering track
6 min read
In this module
Making Code Harder to Read
Code obfuscation transforms a program to resist analysis while preserving its behaviour. It's the primary weapon software vendors use to protect IP, licensing logic, and proprietary algorithms from reverse engineering.
Unlike encryption, obfuscation offers no hard guarantee: a determined analyst can always get through it, given enough time. The goal is to make analysis expensive enough that it stops being worth the effort.
Dead Code Insertion
The simplest obfuscation technique adds instructions that do nothing useful. They execute but have no effect on the program's actual output.
// Originalx = a + b
// Obfuscated with dead codetemp1 = a * 7temp2 = b ^ 0xFFx = a + b // the real computationtemp3 = temp1 - temp2 // meaninglesstemp4 = x ^ temp3 ^ temp3 // XOR cancels outThe analyst must determine which computations are real and which are noise. Automated dead code elimination can sometimes remove these, but more sophisticated techniques make it harder to distinguish live from dead code.
Opaque Predicates
An opaque predicate is a conditional branch whose outcome is always the same (always true or always false) but is difficult for analysis tools to determine statically.
// This is always true, but proving it requires number theoryif ((x * x + x) % 2 == 0) { // real code goes here} else { // dead code (never executes) // can contain anything, including crash-inducing code}The expression x*x + x = x(x+1) is always even because consecutive integers always include an even number. But a decompiler sees a conditional branch and must analyse both paths, potentially including the dead path that contains misleading code.
Opaque predicates create fake branches that always go one way but are hard to analyse statically. Combined with dead code in the unreachable path, they confuse decompilers into presenting false complexity. Removing them requires proving the predicate's outcome, which can be arbitrarily difficult.
Control Flow Flattening
Control flow flattening destroys the natural nesting structure of a program by converting all blocks into a flat switch-based dispatcher.
// Original structured codeif (x > 0) { a();} else { b();}c();
// After flatteningstate = 1while (state != 0) { switch (state) { case 1: if (x > 0) { state = 2; } else { state = 3; } break; case 2: a(); state = 4; break; case 3: b(); state = 4; break; case 4: c(); state = 0; break; }}Here are the two control flow graphs. On the left, the diamond you'd spot in a second. On the right, every block has the same parent and the same child, the dispatcher, so the graph's shape says nothing about which block follows which. That information now lives in the values written to state:
The decompiler sees a single large switch inside a loop. The original structure (a simple if/else) is no longer visible. For complex functions with many blocks, this produces an enormous, incomprehensible dispatcher.
String Encryption
Strings in binaries are goldmines for reverse engineers - error messages, API names, URLs, and format strings reveal what the code does. String encryption encodes strings at compile time and decrypts them at runtime just before use.
// Instead of a readable string in the binary:char* url = "https://api.example.com/license";
// The string is encrypted in the binary, decoded at runtime:char encrypted[] = {0x4a, 0x7b, 0x3c, ...};char* url = decrypt(encrypted, key);The analyst must find and understand the decryption routine before they can read any strings. Tools like FLOSS (the FLARE Obfuscated String Solver, maintained by Mandiant's FLARE team) attempt to automatically extract encrypted strings by emulating the decryption code.
The weakest and most common scheme is a single-byte XOR. The runner holds the URL above encoded with one key, decodes it the way the malware would, and then plays analyst: it tries all 255 keys and keeps any that turn the bytes into something starting with a guessed prefix (a "crib").
FLOSS goes a step further and doesn't need a crib: it finds the decoding function and emulates it, collecting whatever strings come out.
Instruction Substitution
Replace standard instructions with equivalent but more complex sequences:
// OriginalADD EAX, 5
// SubstitutedSUB EAX, -5 // equivalent: subtracting negative is adding
// Or even more complexLEA EAX, [EAX + 5] // also adds 5 to EAX, but leaves the flags aloneThe substitutions are only equivalent where nothing reads the flags afterwards: ADD and SUB set them, LEA doesn't, and SUB EAX, -5 sets the carry flag differently from ADD EAX, 5. An obfuscator has to check that before it swaps one for another.
At scale, with many substitution patterns, this makes the assembly harder to read and defeats simple pattern matching in analysis tools.
Mixed Boolean-Arithmetic (MBA)
MBA expressions combine arithmetic and boolean operations into expressions that are equivalent to simpler ones but very difficult to simplify:
// Original: x + y// MBA equivalent:(x ^ y) + 2 * (x & y)Both expressions compute x + y, but the MBA version is much harder for a decompiler to simplify back to x + y. More complex MBA expressions can have dozens of terms and resist automated simplification.
The Arms Race
Obfuscation is an arms race. Control flow flattening gets countered by symbolic execution. String encryption gets countered by emulation. Opaque predicates get countered by SMT solvers. Each defensive improvement motivates a stronger attack, which motivates a stronger defence. The cycle continues.
Practical Impact on Analysis
Obfuscated code means your analysis takes longer and requires heavier tooling. Start by identifying which techniques are in play — each has known countermeasures. Automated tools handle common patterns, but heavily obfuscated code often demands manual work: you need to understand the obfuscation scheme well enough to think past it. This is where a thorough grounding in assembly, compiler internals, and binary analysis actually earns its keep.