Skip to main content
Be Bitwise

Shellcode Development

Red Team Tooling Internals · Advanced Exploitation track

6 min read

In this module

Writing Position-Independent Code

Shellcode is raw machine code that has to run wherever it happens to land. No imports, no relocations, no fixed addresses — the OS loader isn't involved at all. Understanding how shellcode is constructed is what lets you spot it: in memory dumps, on the wire, and embedded in exploit payloads.

Position Independence

The fundamental constraint of shellcode is that it cannot contain absolute addresses. It does not know where in memory it will be placed. Every reference must be relative.

On x86-64, RIP-relative addressing makes this natural - most instructions can reference data relative to the current instruction pointer. On 32-bit x86, shellcode uses the classic GetPC (get program counter) technique:

nasm
; 64-bit: RIP-relative addressing is built into the architecture
lea rdi, [rel shell_string] ; Load address of "/bin/sh" relative to RIP
; 32-bit: GetPC technique
call next
next:
pop ebx ; EBX now contains the address of 'next'
; All data references are relative to EBX
lea eax, [ebx + (shell_string - next)]

Resolving API Functions at Runtime

On Windows, shellcode cannot call APIs directly because it does not know where DLLs are loaded. It must find the addresses at runtime by walking the Process Environment Block (PEB):

  1. Access the PEB through the Thread Environment Block (TEB) at gs:[0x60] (64-bit)
  2. Walk the PEB's InMemoryOrderModuleList to find kernel32.dll
  3. Parse kernel32.dll's export table to find GetProcAddress and LoadLibraryA
  4. Use GetProcAddress to resolve any other needed functions
nasm
; Windows x64 shellcode: find kernel32.dll base address
mov rax, gs:[0x60] ; PEB
mov rax, [rax + 0x18] ; PEB->Ldr
mov rax, [rax + 0x20] ; InMemoryOrderModuleList
mov rax, [rax] ; Second entry (ntdll.dll)
mov rax, [rax] ; Third entry (kernel32.dll)
mov rax, [rax + 0x20] ; DllBase of kernel32.dll

On Linux, shellcode typically makes system calls directly using the syscall instruction, avoiding the need to resolve library functions:

nasm
; Linux x64: execve("/bin/sh", NULL, NULL)
xor rsi, rsi ; argv = NULL
xor rdx, rdx ; envp = NULL
lea rdi, [rel binsh] ; filename = "/bin/sh"
mov al, 59 ; syscall number for execve (assumes rax is already 0)
syscall
binsh: db "/bin/sh", 0

That version is easy to read but has three problems as real shellcode. mov al, 59 only sets the low byte, so it works only if RAX happened to be zero already. The lea encodes its offset as a 32-bit displacement (48 8d 3d 04 00 00 00), which is mostly zero bytes. And the string's terminator is a zero byte too. If the payload travels through strcpy or similar, the copy stops at the first of them. A null-free version builds the string on the stack instead:

nasm
; Linux x64: execve("//bin/sh", NULL, NULL), no zero bytes
xor eax, eax ; rax = 0
push rax ; NUL terminator for the string
mov rbx, 0x68732f6e69622f2f ; "//bin/sh" (the extra / pads it to 8 bytes)
push rbx
mov rdi, rsp ; filename = pointer to the string
xor esi, esi ; argv = NULL
xor edx, edx ; envp = NULL
mov al, 59 ; execve, safe now that rax was zeroed
syscall

Which bytes count as "bad" depends on how the payload is delivered. Zero bytes break C string functions; a payload sent in an HTTP header can't contain \r or \n; one read by scanf("%s") can't contain whitespace. Work out the bad bytes for your delivery path first, then check the assembled payload against them.

Syscall-Based Shellcode

On Windows, modern shellcode increasingly uses direct syscalls (executing the syscall instruction directly) instead of calling ntdll functions. This bypasses EDR hooks on ntdll, making the shellcode harder to detect. The shellcode includes a table of syscall numbers for the target Windows version, or resolves them dynamically by parsing ntdll's code.

Staged vs Stageless Shellcode

Stageless shellcode contains the complete payload in one block — self-contained, but potentially megabytes for a full C2 implant.

Staged shellcode is a compact first-stage loader (often under 500 bytes) that fetches and executes a larger second stage at runtime:

text
Stage 1 (small, fits in exploit buffer):
1. Create a TCP connection to the C2 server
2. Read the second stage into allocated memory
3. Jump to the second stage
Stage 2 (large, downloaded from C2):
1. Full C2 beacon with all capabilities
2. Process migration, file operations, etc.

Staging keeps the initial exploit buffer small, but the download is a detection opportunity. Stageless payloads skip the network fetch but must fit the exploitation context — that trade-off drives a lot of payload engineering decisions.

Encoding and Encryption

Raw shellcode contains null bytes and other characters that may be terminated by string-handling functions. Encoding and encryption address this:

XOR encoding: Each byte is XORed with a key. A small decoder stub at the beginning reverses the encoding before execution. The decoder itself must be null-free.

The encoder below is loaded with a 12-byte x86-64 exit(0) payload (mov rax, 60; xor edi, edi; syscall). The 32-bit immediate in mov rax, 60 brings three null bytes with it, which the Analyze tab flags. XOR-encode it with the default key 0x41 and the nulls vanish. Then try a key equal to one of the payload's own bytes, such as 0x48, and watch a new null appear: that's why encoders search for a key that doesn't occur anywhere in the payload.

Shellcode Analyzer & Encoder

Byte Analysis (12 bytes, 3 null bytes):

48
c7
c0
3c
00
00
00
48
31
ff
0f
05

Found 3 null byte(s) — shellcode will be truncated by strcpy/string functions

12

Total bytes

3

Null bytes

9

Unique bytes

Custom encoders: More complex schemes like Shikata Ga Nai (from Metasploit) use polymorphic encoding that changes the decoder stub on each generation, which once defeated static signatures. Shikata Ga Nai is now so widely studied that antivirus and EDR products detect its decoder loop and its distinctive use of FPU instructions to find its own address, so on its own it buys little evasion today.

AES encryption: The shellcode is AES-encrypted and a small decryptor stub (using hardcoded AES implementation or resolved API calls) decrypts it at runtime.

Detecting Shellcode

Defenders detect shellcode through multiple methods:

  • Signature-based: Known shellcode patterns (PEB walking sequences, GetPC code, specific syscall sequences)
  • Heuristic-based: Characteristics like small executable code blocks with no imports, NOP sleds, or decoder stub patterns
  • Behavioural: Code that walks the PEB, resolves API addresses, or performs direct syscalls from unusual locations
  • Memory scanning: Executable pages that are not backed by files on disk (indicating injected code)
  • Network detection: Shellcode patterns in network payloads (NOP sleds, decoder stubs, known encoder signatures)
The Detection Challenge

Encrypted shellcode looks like random data until it is decrypted. The only detectable component is the decoder stub, which can be made polymorphic (different every time). This is why behavioural detection (monitoring what code does) is more reliable than content detection (scanning what code looks like). A decoder stub that writes to executable memory and then transfers control to it is suspicious regardless of what the decoded payload contains.