Skip to main content
Be Bitwise

Network-Level Binary Analysis

Low-Level Detection & Instrumentation · Defense & Forensics track

5 min read

In this module

Detecting Threats on the Wire

Network-level binary analysis pulls apart raw traffic to spot malicious payloads, shellcode, covert channels, and protocol behaviour that smells off. Endpoint detection catches threats on the host. Network analysis sees lateral movement, exfiltration, and C2 traffic — things that never sit still on one machine long enough for the host agent to catch.

Deep Packet Inspection

Deep packet inspection (DPI) reaches past the packet headers — source and destination IP, ports — and looks at what's inside. For security work, that's where the malicious patterns hide.

Protocol identification: ports lie. A C2 channel might sit on port 443 and speak a custom protocol instead of HTTPS. DPI fingerprints protocols by content rather than port number. Tools like nDPI and Zeek (formerly Bro) classify traffic by matching protocol signatures and behavioural patterns.

Payload analysis: DPI engines scan packet payloads for known indicators of compromise (IoCs), shellcode patterns, exploit signatures, and data exfiltration markers. A buffer overflow exploit sent over the network carries the overflow payload in the packet data — DPI spots it by recognising NOP sleds, ROP gadgets, or shellcode decoder stubs.

text
Packet Analysis Layers:
Layer 3 (IP): Source/destination, geolocation
Layer 4 (TCP/UDP): Ports, session tracking
Layer 7 (Application): Protocol identification, content inspection
Payload: Shellcode detection, malware signatures, anomaly detection
Encrypted Traffic Challenge

TLS encryption blinds traditional DPI. The payload is ciphertext, and there's nothing to inspect. Network defenders work around this in three main ways: TLS interception proxies that decrypt, inspect, and re-encrypt; JA3/JA4 fingerprinting that identifies clients from the unencrypted TLS handshake metadata; and traffic analysis that finds anomalies in packet sizes, timing, and flow patterns without ever seeing what's inside.

Protocol Dissection

Protocol dissectors parse network traffic according to protocol specifications, extracting structured fields from raw bytes. Wireshark includes dissectors for hundreds of protocols, and Zeek provides programmable protocol analysis.

For security analysis, custom protocol dissection becomes necessary in a few specific situations:

  • Proprietary protocols: industrial control systems (SCADA), IoT devices, and malware often speak custom dialects that off-the-shelf tools can't parse
  • Tunneled protocols: data smuggled inside legitimate carriers — DNS tunneling, HTTP tunneling, ICMP tunneling
  • Modified protocols: standard protocols with non-standard fields that act as covert channels
text
# Zeek script to detect DNS tunneling by monitoring query lengths
event dns_request(c: connection, msg: dns_msg, query: string, qtype: count, qclass: count) {
# DNS tunneling typically uses very long domain names
if (|query| > 60) {
print fmt("Possible DNS tunnel: %s -> %s query=%s len=%d",
c$id$orig_h, c$id$resp_h, query, |query|);
}
}

Detecting Shellcode in Network Traffic

Shellcode crossing the wire has fingerprints:

NOP sleds: long runs of 0x90 (NOP) bytes — or any equivalent no-op — that act as a landing zone for the exploit. Plain sleds light up trivial signatures; polymorphic NOP equivalents like xchg eax, eax or inc ecx; dec ecx need cleverer detection. Sleds belong mostly to older exploits, though. With ASLR and NX, modern exploits jump to precise addresses through ROP chains, so there's often no sled to find.

Decoder stubs: encoded shellcode begins with a decoder stub, a short sequence of instructions that decodes the rest of the payload. Decoder stubs often contain telltale patterns like GetPC (get program counter) sequences.

Syscall patterns: shellcode making direct system calls includes instructions like int 0x80 (Linux 32-bit), syscall (Linux 64-bit), or sysenter (Windows). Those opcodes turning up inside what should be data is a red flag.

Network intrusion detection systems (IDS) like Snort and Suricata use rules to detect these patterns:

text
# Snort rule to detect potential shellcode (NOP sled)
alert tcp any any -> $HOME_NET any (content:"|90 90 90 90 90 90 90 90|";
msg:"Possible NOP sled detected"; sid:1000001;)

JA3/JA4 Fingerprinting

JA3 creates a fingerprint of a TLS client by hashing specific fields from the ClientHello message:

  • TLS version
  • Cipher suites offered
  • TLS extensions present
  • Elliptic curves supported
  • Elliptic curve point formats
text
JA3 hash calculation:
Input: TLSVersion,CipherSuites,Extensions,EllipticCurves,ECPointFormats
Example: 769,47-53-5-10-49161-49162-49171-49172-50-56-19-4,0-10-11,23-24-25,0
MD5 hash: ada70206e40642a3e4461f35503241d5

Different applications produce different JA3 hashes because they support different cipher suites and extensions. A specific malware family consistently produces the same JA3 hash across different infections, even when it phones home to different C2 servers. Network defenders can identify known malware by its TLS fingerprint without ever decrypting the traffic.

JA3 has a weakness: it hashes the extensions in the order they appear. Around the start of 2023 Chrome began randomising its TLS extension order (it was on by default from version 110), so a single Chrome build now produces many different JA3 hashes, and other clients have followed. JA4, released by FoxIO in 2023, sorts the extensions before hashing, so the fingerprint survives the shuffle. It also extends the idea with additional fields and finer-grained fingerprinting, improving accuracy and cutting collisions between unrelated applications that happen to share a JA3 hash.

Network Behavioural Analysis

Beyond content inspection, behavioural analysis spots anomalies in traffic patterns:

  • Beaconing: regular, periodic connections to the same destination — the classic C2 cadence
  • DNS anomalies: unusually high query rates, queries to freshly registered domains, or responses with abnormally low TTL values (tunnels and fast-flux networks keep TTLs short so every lookup reaches the attacker's server and answers can change quickly)
  • Data volume asymmetry: large outbound transfers to unfamiliar destinations (possible exfiltration)
  • Protocol anomalies: HTTP traffic on non-standard ports, absurdly long HTTP headers, or protocol behaviour that deviates from the RFC
Layered Network Defence

Useful network security stacks several detection approaches: signature-based (known bad patterns), behavioural (weird patterns), reputation scoring (known bad destinations), and protocol analysis (malformed or abused protocols). No single layer catches everything. Together, they patch over each other's blind spots.