Skip to main content
Be Bitwise
x86 AssemblyData Movement & Arithmetic

Bitwise Operations

Bitwise operations zoom in on the individual bits inside a value. Arithmetic treats a register as a number; bitwise instructions treat it as 64 independent on/off switches. They're the bedrock of bit masking, flag fiddling, encoding, hashing, crypto.

For the theory behind bit manipulation, revisit the Logic Gates module.

AND, OR, XOR, NOT

Four instructions, each running the corresponding Boolean operation across every bit position in parallel:

nasm
; AND destination, source (bit-by-bit AND)
and rax, rbx ; each bit of rax AND'd with corresponding bit of rbx
and rax, 0xFF ; mask: keep only the lowest 8 bits, zero the rest
; OR destination, source (bit-by-bit OR)
or rax, rbx ; each bit of rax OR'd with corresponding bit of rbx
or rax, 0x80 ; set bit 7 without disturbing other bits
; XOR destination, source (bit-by-bit exclusive OR)
xor rax, rbx ; each bit: 1 if bits differ, 0 if bits match
xor rax, rax ; zero rax (any value XOR'd with itself = 0)
; NOT destination (flip every bit)
not rax ; one's complement: 0 bits become 1, 1 bits become 0

Truth tables for reference:

c
AND: 0&0=0 0&1=0 1&0=0 1&1=1 (both must be 1)
OR: 0|0=0 0|1=1 1|0=1 1|1=1 (at least one must be 1)
XOR: 0^0=0 0^1=1 1^0=1 1^1=0 (exactly one must be 1)
NOT: ~0=1 ~1=0 (flip the bit)

Common Bit Manipulation Patterns

You'll see these patterns again and again in systems code:

nasm
; Masking: extract specific bits
and rax, 0x0F ; keep lowest 4 bits (nibble), clear the rest
and rax, 0xFF00 ; keep bits 8-15, clear everything else
; Setting bits: turn on specific bits without changing others
or rax, 0x01 ; set bit 0 (make value odd)
or rax, 0x80000000 ; set bit 31
; Clearing bits: turn off specific bits
and rax, ~0x01 ; clear bit 0 (make value even)
; NASM: and rax, 0xFFFFFFFFFFFFFFFE
; Toggling bits: flip specific bits
xor rax, 0x01 ; toggle bit 0
; Checking if a bit is set
test rax, 0x04 ; test bit 2
jnz .bit_2_is_set ; jump if bit 2 was 1

Try two of these patterns on an 8-bit value. The operand box takes decimal, so 0x80 is 128 and 0xFE is 254. First, pull out the 3-bit field in bits 5-3 of 0xB6: shift it down to the bottom, then mask off everything above it. In assembly that's shr al, 3 then and al, 7.

Bit Manipulation Puzzle

2 steps left
Current
1
0
1
1
0
1
1
0
1820xB6
Target
0
0
0
0
0
1
1
0
60x06

Second, turn 0x35 into 0xB4 by setting bit 7 and clearing bit 0 (or al, 0x80 then and al, 0xFE). There's also a one-step answer with XOR if you can spot which bits need flipping.

Bit Manipulation Puzzle

2 steps left
Current
0
0
1
1
0
1
0
1
530x35
Target
1
0
1
1
0
1
0
0
1800xB4
XOR: The Swiss Army Knife

XOR has a unique property: A XOR B XOR B = A. Applying the same XOR mask twice returns the original value. This makes XOR the basis of simple encryption/encoding schemes and is used extensively in: zeroing registers (xor rax, rax is the fastest way), simple checksums and hashes, toggling between two states, and swap-without-temporary (xchg is usually clearer though). In reverse engineering, seeing repeated XOR operations is often a sign of encoded or obfuscated data.

Shift Operations

Shifts slide every bit left or right by a given count:

nasm
; SHL destination, count (shift left -- multiply by 2^count)
shl rax, 1 ; shift left by 1 (multiply by 2)
shl rax, 3 ; shift left by 3 (multiply by 8)
shl rax, cl ; shift left by the value in CL
; SHR destination, count (shift right logical -- unsigned divide by 2^count)
shr rax, 1 ; shift right by 1 (unsigned divide by 2)
shr rax, 4 ; shift right by 4 (unsigned divide by 16)
; SAR destination, count (shift right arithmetic -- signed divide by 2^count)
sar rax, 1 ; preserves the sign bit during shift

The difference between SHR and SAR:

c
SHR: shifts in zeros from the left
0xFF00 >> 4 = 0x0FF0 (unsigned: 65280 / 16 = 4080)
SAR: copies the sign bit (MSB) into vacated positions
0xFF00 >> 4 = 0xFFF0 (signed: -256 / 16 = -16, as a 16-bit value)

For powers of two, shifts beat multiplication and division on speed. Compilers happily swap * 8 for shl rax, 3 and / 4 for shr rax, 2 without asking.

Rotate Operations

Rotates are shifts that don't drop bits on the floor. What falls off one end reappears on the other:

nasm
; ROL destination, count (rotate left)
rol rax, 4 ; bits that shift off the left appear on the right
; ROR destination, count (rotate right)
ror rax, 4 ; bits that shift off the right appear on the left
c
ROL by 4 on a byte:
0xAB = 1010_1011
becomes 1011_1010 = 0xBA
ROR by 4 on a byte:
0xAB = 1010_1011
becomes 1011_1010 = 0xBA (same for 4-bit rotate on a byte)

Nothing escapes a rotate. Cryptographic algorithms (SHA, the AES key schedule) and hash functions lean on them precisely because no information leaks.

Practical Example: Extracting Bit Fields

Plenty of data formats stuff several values into a single integer. To pull one back out, you shift and mask:

nasm
; Extract bits [11:8] from a 32-bit value in eax
; (This is a 4-bit field starting at bit 8)
mov ebx, eax ; copy the value
shr ebx, 8 ; shift field down to bit position 0
and ebx, 0x0F ; mask to keep only 4 bits
; ebx now contains the extracted field
; Example: eax = 0x00000A00
; After shr 8: ebx = 0x0000000A
; After and 0F: ebx = 0x0000000A (value is 10)

Practical Example: Setting a Bit Field

Going the other way and writing a value into a specific bit position:

nasm
; Set bits [11:8] of eax to the value in ecx (0-15)
; Step 1: Clear the target field
and eax, ~(0x0F << 8) ; NASM: and eax, 0xFFFFF0FF
; Step 2: Position the new value and OR it in
and ecx, 0x0F ; ensure value fits in 4 bits
shl ecx, 8 ; shift to bit position 8
or eax, ecx ; insert the field
Security Note: XOR Encoding in Shellcode

Shellcode often cannot contain certain byte values (like 0x00, null bytes) because they would terminate strings in vulnerable programs. A common technique is to XOR the shellcode with a key, producing a payload with no null bytes. A small decoder stub at the beginning XORs the payload back to the original instructions before executing them. Recognizing XOR decode loops is one of the first skills in malware analysis. The pattern typically looks like: load a byte, XOR it with a key, store it back, advance the pointer, loop.

XOR Decode Loop Example

A simple XOR decoder, the kind that turns up daily in malware and shellcode:

nasm
; XOR decode stub
; Decodes len bytes at encoded_data using single-byte key
lea rsi, [rel encoded_data] ; pointer to encoded payload
mov rcx, payload_len ; number of bytes
mov al, 0xAA ; XOR key
.decode_loop:
xor byte [rsi], al ; decode one byte in place
inc rsi ; advance pointer
dec rcx ; decrement counter
jnz .decode_loop ; loop until done
; Execution continues into the now-decoded payload

Bit Test Instructions

For poking at individual bits by index:

nasm
; BT source, bit_number (bit test: copy bit N to CF)
bt rax, 5 ; CF = bit 5 of rax
; BTS destination, bit (bit test and set: copy to CF, then set the bit)
bts rax, 5 ; CF = old bit 5, then set bit 5 to 1
; BTR destination, bit (bit test and reset: copy to CF, then clear the bit)
btr rax, 5 ; CF = old bit 5, then clear bit 5 to 0
; BTC destination, bit (bit test and complement: copy to CF, then flip)
btc rax, 5 ; CF = old bit 5, then toggle bit 5

Summary of Flags Effects

InstructionCFZFSFOF
AND/OR/XOR/TEST0yesyes0
NOTunchangedunchangedunchangedunchanged
SHL/SHR/SARlast bit shifted outyesyesdefined for count=1
ROL/RORlast bit rotatedunchangedunchangeddefined for count=1

What is Next

The full kit for moving and reshaping data is now in your hands. Next module: control flow. Putting those flags to work to make decisions, build loops, and steer the program around.

Challenges

Write a program that takes the value 0xAB in rax, extracts the high nibble (upper 4 bits) into rbx and the low nibble (lower 4 bits) into rcx, then prints both as decimal numbers on separate lines.

Assembly (NASM 2.14.02)