Skip to main content
Be Bitwise

Packing & Runtime Decryption

Anti-Reversing Techniques · Advanced Reverse Engineering track

6 min read

In this module

Hiding Code Until Runtime

Packing compresses or encrypts an executable's code and data, replacing everything with a small unpacking stub. When the binary runs, the stub restores the original code into memory and jumps to it. Static analysis tools see only the stub and an opaque blob — the actual logic is invisible until runtime.

The technique has legitimate uses (reducing binary size, protecting IP) and malicious ones (evading antivirus, resisting analysis). Understanding it is necessary for both sides: malware analysts need to unpack; malware authors need to know what analysts will try.

How Packers Work

The packing process:

  1. Take the original compiled binary
  2. Compress and/or encrypt its code and data sections
  3. Create a new binary containing the compressed data and an unpacking stub
  4. The stub becomes the new entry point

At runtime:

  1. The OS loads the packed binary
  2. The unpacking stub executes
  3. It allocates memory, decompresses the original code, fixes up imports and relocations
  4. Transfers execution to the original entry point (OEP)
text
Packed binary on disk:
+------------------+
| Unpacking stub | <-- entry point
+------------------+
| Compressed/ |
| encrypted |
| original code |
+------------------+
At runtime after unpacking:
+------------------+
| Original code | <-- execution transferred here
| (decompressed) |
+------------------+

The import table fixup is a critical step. The original binary references functions from DLLs (like CreateFile from kernel32.dll). A packed binary usually keeps only a minimal import table, often little more than LoadLibraryA and GetProcAddress (UPX keeps one import from each original DLL so the loader maps them), because the full list would reveal the program's capabilities. The unpacking stub resolves the rest at runtime, looking up each function by name or by hash and writing the addresses into the reconstructed import table.

Key Concept

Packers hide the real program code behind compression or encryption. The unpacking stub restores the code at runtime. To analyse packed software, you must either unpack it (find the original entry point after decompression) or dump the process memory after unpacking completes. The transfer from stub to original code is the critical moment in analysis.

Common Packers

UPX (Ultimate Packer for eXecutables): The most common open-source packer. Primarily used for compression rather than protection. Easy to unpack with upx -d because UPX is designed to be reversible. Many malware samples use UPX as a first layer of packing. It packs Linux ELF binaries as well as Windows PE files, and UPX-packed ELF malware (IoT botnets especially) is common; there the stub maps memory with mmap instead of rebuilding an import table.

Themida / WinLicense: Commercial protectors that combine packing with anti-debug, anti-dump, virtualisation, and licence checking. Widely used by commercial software and also by malware. Themida is significantly harder to unpack than UPX.

VMProtect: A commercial protector focused on code virtualisation (converting native code to custom bytecode). One of the strongest available protections. Analysing VMProtect-protected code requires understanding its custom virtual machine.

Custom packers: Malware authors frequently write their own packers with unique decompression routines. Since the packer is custom, automated unpacking tools do not work, and the analyst must reverse-engineer each packer individually. This is time-consuming but provides no security against a determined analyst.

Unpacking Strategies

Manual unpacking: Run the packed binary in a debugger, set breakpoints after the decompression routine, and dump the memory when the original code is revealed. The key challenge is finding the Original Entry Point (OEP) - the address where the unpacker transfers control to the real program.

Common OEP detection techniques:

  • Set a hardware breakpoint on the stack (the unpacker restores ESP to a known value before jumping to OEP)
  • Watch for a large JMP to a different section, which typically signals the transition from stub to original code
  • Trace execution until the unpacker loop ends and a distinctly different code pattern begins
  • Use memory breakpoints on the code section that are triggered when execution reaches newly unpacked code

Here's how those tells fall out of a run of 32-bit UPX, the packer you'll meet most. UPX ships an empty section (UPX0) with no bytes on disk and a second section (UPX1) holding the stub and the compressed code:

The pushad at the top is why the stack hardware breakpoint works: break on access to the bytes it pushed, and the popad that reads them back sits a few instructions before the tail jump. That jump is also the "large JMP to a different section", since it leaves UPX1 for UPX0.

Automated unpacking: Tools like Unipacker, built on the Unicorn emulator, run the unpacking stub in emulation and dump the result. These work for known packers but fail on custom ones.

Memory dumping: Run the binary and dump the process memory after it has unpacked itself. Tools like Scylla can reconstruct the import table from a memory dump, producing a working unpacked executable. This is often the simplest approach when manual debugging is too time-consuming.

Anti-Dumping Techniques

Sophisticated packers defend against memory dumping:

  • Import table destruction: Erase the IAT after resolving imports, replacing pointers with stubs that resolve functions on demand
  • Section header wiping: Remove PE section headers from memory so dump tools cannot reconstruct the file structure
  • Guard pages: Use memory protection tricks that cause exceptions when dump tools access certain memory regions
  • Incremental decryption: Only decrypt small portions of code at a time, re-encrypting them after use. The entire unpacked binary is never present in memory simultaneously

Self-Modifying Code

Some protectors take packing further with self-modifying code - code that modifies itself as it runs. A function might decrypt its own body, execute, then re-encrypt itself. This makes it impossible to dump a fully unpacked binary because different parts are encrypted at different times.

Self-modifying code also breaks static analysis tools, which assume code bytes do not change. The only reliable way to analyse self-modifying code is dynamic analysis - executing it and observing the modifications in real time.

Packing vs. Obfuscation

Packing and obfuscation serve different purposes:

  • Packing hides code from static analysis. Once unpacked, the code is readable.
  • Obfuscation makes code hard to understand even when visible.
  • Strong protection combines both: pack the binary to prevent static analysis, and obfuscate the unpacked code to resist dynamic analysis.

In practice, malware layers these protections: a custom packer wrapping obfuscated code that also uses anti-debug checks. The analyst has to defeat each layer in order before the payload becomes visible. Onion-peeling, but the onion bites back.