Protocol Reverse Engineering
Advanced Reverse Engineering Techniques · Advanced Reverse Engineering track
6 min read
In this module
Reverse Engineering Protocols Without Documentation
Reverse engineering doesn't stop at executables. Sooner or later you'll meet an application that talks to a server over an undocumented protocol, and now you've got a different problem on your hands: working out the message formats, the field semantics, and the state machine that governs the conversation. That's protocol reverse engineering.
Network Capture with Wireshark
First job: capture the traffic. Wireshark is the standard tool — it pulls packets off a network interface and provides deep dissection for protocols it already knows. For protocols it doesn't know, you're staring at raw bytes.
Workflow looks the same every time. Start the capture, run the target, do something specific in the app — log in, send a message, request data — then stop the capture and inspect what came back. Filter by the target's IP and port to keep the noise down.
For TCP-based protocols, Wireshark's "Follow TCP Stream" stitches the conversation back together byte by byte, in order. For UDP, you're working datagram by datagram.
Binary Protocol Analysis
Most custom protocols are binary, not text. Three landmarks anchor the analysis:
Magic bytes — fixed byte sequences that open every message. PNG's 0x89504E47 is the canonical example. Hunt for constant values at message start.
Length fields — protocols need to know where one message ends and the next begins. Search for a field whose value matches the count of bytes that follow. Try the obvious positions (bytes 0-1, 2-3, 4-7) and the obvious encodings (big-endian, little-endian, varint).
Message type identifiers — somewhere in the header sits a small integer that says what kind of message this is. Request, response, heartbeat, whatever. The value varies across messages of different types but stays consistent across messages that trigger the same in-app behaviour. That stability is how you catch them.
Typical binary protocol message:+--------+--------+--------+--------+--------+---...---+| Magic | Type | Length | SeqNo | Flags | Payload || 2 bytes| 1 byte | 4 bytes| 4 bytes| 1 byte | N bytes |+--------+--------+--------+--------+--------+---...---+The most powerful technique for protocol RE is differential analysis: perform one specific action in the application and compare the captured traffic against a baseline where you did nothing. The differences reveal which bytes correspond to your action. Repeat with slight variations - change one character in a message, change a quantity by one - and watch which bytes change and by how much. This isolates individual fields with high precision.
State Machine Recovery
A protocol is more than its message formats. It's a conversation, with rules about which messages can appear when. A login protocol might insist that the client say Hello first, that the server replies with Challenge, that the client returns AuthResponse, and that the server lands on either AuthSuccess or AuthFailure. Send those messages out of order and the server hangs up.
To recover the state machine, capture lots of complete sessions, classify each message by its type field, and map the resulting sequences onto a graph: nodes are states, edges are message types. Patterns to look for:
- Initial handshakes — identical at the start of every fresh connection
- Authentication phases — early, with success/failure branches
- Heartbeats — periodic, independent of anything the user is doing
- Request-response pairs — type X consistently followed by type Y
- Teardown — always the same on the way out
Run that process on the login protocol above and you'd end up with something like this. Each edge is a message type you saw in the captures; any message that has no edge out of the current state is one the server will reject:
The missing edges are where the fuzzing ideas come from. What does the server do with an AuthResponse sent straight after Hello, before any Challenge?
API Reverse Engineering Through Traffic Analysis
Modern apps tend to speak HTTP with JSON or Protocol Buffer payloads underneath. They're far easier to dissect than custom binary protocols because most of the structure is text you can read.
For HTTPS traffic, you'll need to break TLS. The usual options:
- Proxy tools (mitmproxy, Burp Suite). Point the application at the proxy, which terminates TLS and re-encrypts under its own cert. The catch is the application has to trust the proxy's CA, which means installing it.
- SSLKEYLOGFILE. Plenty of clients — browsers, curl, Python's requests — will write TLS session keys to a file when this env var is set. Hand the file to Wireshark and it decrypts your captures without any proxy at all.
- Frida hooks into the application's TLS library to grab plaintext before encryption (technique covered in Function Hooking & Instrumentation).
Once you've got plaintext HTTP, mapping the API is straightforward. Identify endpoints (URL paths), methods (GET, POST, PUT, DELETE), authentication scheme (Bearer tokens, API keys, cookies), request parameters, and response shape. Save the requests in curl or Postman so you can replay and tweak them at will.
Protocol Buffers and Serialisation Formats
Plenty of modern protocols use binary serialisation formats — Protocol Buffers, FlatBuffers, MessagePack. They're more structured than rolled-by-hand binary protocols, but you still need to reverse them if the schema isn't published.
Protobuf is unusually friendly here. protoc --decode_raw will give you field numbers, wire types, and values without ever needing the .proto file. You won't get field names, but the structure is enough to start with. Pair the decoded output with differential analysis — change one value in the app, see which field number flipped — and the meaning of each field falls out one by one.
$ echo -n '<binary data>' | protoc --decode_raw1: 422: "hello"3 { 1: 100 2: 200}Practical Methodology
A working order of operations. Capture traffic from several sessions with different in-app actions. Identify message boundaries and the header layout. Classify messages by type field. Use differential analysis to assign semantics to fields. Recover the state machine from sequence patterns. Write a parser or, ideally, a Wireshark Lua dissector — they're surprisingly easy to author. Validate everything by writing a client that speaks the protocol back. Each step rests on the one before it, and the cumulative effect is the slow conversion of raw bytes into understood communication.
A word on legality. Protocol RE for interoperability is explicitly permitted in many jurisdictions — the EU's Software Directive is the textbook example. That doesn't mean it's universally legal, and it certainly doesn't override a service's terms of use. Check the rules where you're working, and remember the aim is understanding, not abuse.