Bitwise Operations
Bitwise operations zoom in on the individual bits inside a value. Arithmetic treats a register as a number; bitwise instructions treat it as 64 independent on/off switches. They're the bedrock of bit masking, flag fiddling, encoding, hashing, crypto.
For the theory behind bit manipulation, revisit the Logic Gates module.
AND, OR, XOR, NOT
Four instructions, each running the corresponding Boolean operation across every bit position in parallel:
; AND destination, source (bit-by-bit AND)and rax, rbx ; each bit of rax AND'd with corresponding bit of rbxand rax, 0xFF ; mask: keep only the lowest 8 bits, zero the rest
; OR destination, source (bit-by-bit OR)or rax, rbx ; each bit of rax OR'd with corresponding bit of rbxor rax, 0x80 ; set bit 7 without disturbing other bits
; XOR destination, source (bit-by-bit exclusive OR)xor rax, rbx ; each bit: 1 if bits differ, 0 if bits matchxor rax, rax ; zero rax (any value XOR'd with itself = 0)
; NOT destination (flip every bit)not rax ; one's complement: 0 bits become 1, 1 bits become 0Truth tables for reference:
AND: 0&0=0 0&1=0 1&0=0 1&1=1 (both must be 1)OR: 0|0=0 0|1=1 1|0=1 1|1=1 (at least one must be 1)XOR: 0^0=0 0^1=1 1^0=1 1^1=0 (exactly one must be 1)NOT: ~0=1 ~1=0 (flip the bit)Common Bit Manipulation Patterns
You'll see these patterns again and again in systems code:
; Masking: extract specific bitsand rax, 0x0F ; keep lowest 4 bits (nibble), clear the restand rax, 0xFF00 ; keep bits 8-15, clear everything else
; Setting bits: turn on specific bits without changing othersor rax, 0x01 ; set bit 0 (make value odd)or rax, 0x80000000 ; set bit 31
; Clearing bits: turn off specific bitsand rax, ~0x01 ; clear bit 0 (make value even) ; NASM: and rax, 0xFFFFFFFFFFFFFFFE
; Toggling bits: flip specific bitsxor rax, 0x01 ; toggle bit 0
; Checking if a bit is settest rax, 0x04 ; test bit 2jnz .bit_2_is_set ; jump if bit 2 was 1Try two of these patterns on an 8-bit value. The operand box takes decimal, so 0x80 is 128 and 0xFE is 254. First, pull out the 3-bit field in bits 5-3 of 0xB6: shift it down to the bottom, then mask off everything above it. In assembly that's shr al, 3 then and al, 7.
Bit Manipulation Puzzle
Second, turn 0x35 into 0xB4 by setting bit 7 and clearing bit 0 (or al, 0x80 then and al, 0xFE). There's also a one-step answer with XOR if you can spot which bits need flipping.
Bit Manipulation Puzzle
XOR has a unique property: A XOR B XOR B = A. Applying the same XOR mask twice returns the original value. This makes XOR the basis of simple encryption/encoding schemes and is used extensively in: zeroing registers (xor rax, rax is the fastest way), simple checksums and hashes, toggling between two states, and swap-without-temporary (xchg is usually clearer though). In reverse engineering, seeing repeated XOR operations is often a sign of encoded or obfuscated data.
Shift Operations
Shifts slide every bit left or right by a given count:
; SHL destination, count (shift left -- multiply by 2^count)shl rax, 1 ; shift left by 1 (multiply by 2)shl rax, 3 ; shift left by 3 (multiply by 8)shl rax, cl ; shift left by the value in CL
; SHR destination, count (shift right logical -- unsigned divide by 2^count)shr rax, 1 ; shift right by 1 (unsigned divide by 2)shr rax, 4 ; shift right by 4 (unsigned divide by 16)
; SAR destination, count (shift right arithmetic -- signed divide by 2^count)sar rax, 1 ; preserves the sign bit during shiftThe difference between SHR and SAR:
SHR: shifts in zeros from the left 0xFF00 >> 4 = 0x0FF0 (unsigned: 65280 / 16 = 4080)
SAR: copies the sign bit (MSB) into vacated positions 0xFF00 >> 4 = 0xFFF0 (signed: -256 / 16 = -16, as a 16-bit value)For powers of two, shifts beat multiplication and division on speed. Compilers happily swap * 8 for shl rax, 3 and / 4 for shr rax, 2 without asking.
Rotate Operations
Rotates are shifts that don't drop bits on the floor. What falls off one end reappears on the other:
; ROL destination, count (rotate left)rol rax, 4 ; bits that shift off the left appear on the right
; ROR destination, count (rotate right)ror rax, 4 ; bits that shift off the right appear on the leftROL by 4 on a byte: 0xAB = 1010_1011 becomes 1011_1010 = 0xBA
ROR by 4 on a byte: 0xAB = 1010_1011 becomes 1011_1010 = 0xBA (same for 4-bit rotate on a byte)Nothing escapes a rotate. Cryptographic algorithms (SHA, the AES key schedule) and hash functions lean on them precisely because no information leaks.
Practical Example: Extracting Bit Fields
Plenty of data formats stuff several values into a single integer. To pull one back out, you shift and mask:
; Extract bits [11:8] from a 32-bit value in eax; (This is a 4-bit field starting at bit 8)
mov ebx, eax ; copy the valueshr ebx, 8 ; shift field down to bit position 0and ebx, 0x0F ; mask to keep only 4 bits; ebx now contains the extracted field
; Example: eax = 0x00000A00; After shr 8: ebx = 0x0000000A; After and 0F: ebx = 0x0000000A (value is 10)Practical Example: Setting a Bit Field
Going the other way and writing a value into a specific bit position:
; Set bits [11:8] of eax to the value in ecx (0-15); Step 1: Clear the target fieldand eax, ~(0x0F << 8) ; NASM: and eax, 0xFFFFF0FF
; Step 2: Position the new value and OR it inand ecx, 0x0F ; ensure value fits in 4 bitsshl ecx, 8 ; shift to bit position 8or eax, ecx ; insert the fieldShellcode often cannot contain certain byte values (like 0x00, null bytes) because they would terminate strings in vulnerable programs. A common technique is to XOR the shellcode with a key, producing a payload with no null bytes. A small decoder stub at the beginning XORs the payload back to the original instructions before executing them. Recognizing XOR decode loops is one of the first skills in malware analysis. The pattern typically looks like: load a byte, XOR it with a key, store it back, advance the pointer, loop.
XOR Decode Loop Example
A simple XOR decoder, the kind that turns up daily in malware and shellcode:
; XOR decode stub; Decodes len bytes at encoded_data using single-byte key lea rsi, [rel encoded_data] ; pointer to encoded payload mov rcx, payload_len ; number of bytes mov al, 0xAA ; XOR key
.decode_loop: xor byte [rsi], al ; decode one byte in place inc rsi ; advance pointer dec rcx ; decrement counter jnz .decode_loop ; loop until done
; Execution continues into the now-decoded payloadBit Test Instructions
For poking at individual bits by index:
; BT source, bit_number (bit test: copy bit N to CF)bt rax, 5 ; CF = bit 5 of rax
; BTS destination, bit (bit test and set: copy to CF, then set the bit)bts rax, 5 ; CF = old bit 5, then set bit 5 to 1
; BTR destination, bit (bit test and reset: copy to CF, then clear the bit)btr rax, 5 ; CF = old bit 5, then clear bit 5 to 0
; BTC destination, bit (bit test and complement: copy to CF, then flip)btc rax, 5 ; CF = old bit 5, then toggle bit 5Summary of Flags Effects
| Instruction | CF | ZF | SF | OF |
|---|---|---|---|---|
| AND/OR/XOR/TEST | 0 | yes | yes | 0 |
| NOT | unchanged | unchanged | unchanged | unchanged |
| SHL/SHR/SAR | last bit shifted out | yes | yes | defined for count=1 |
| ROL/ROR | last bit rotated | unchanged | unchanged | defined for count=1 |
What is Next
The full kit for moving and reshaping data is now in your hands. Next module: control flow. Putting those flags to work to make decisions, build loops, and steer the program around.
Challenges
Write a program that takes the value 0xAB in rax, extracts the high nibble (upper 4 bits) into rbx and the low nibble (lower 4 bits) into rcx, then prints both as decimal numbers on separate lines.