Anti-Rootkit Techniques
Low-Level Detection & Instrumentation · Defense & Forensics track
6 min read
In this module
Finding What Hides
Rootkits are built to be invisible. They modify the operating system to hide their presence, concealing processes, files, network connections, and registry entries from anything an administrator might run. Catching them means looking deeper than the OS's own reporting mechanisms and chasing the discrepancies rootkits inevitably leave behind.
DKOM Detection
Direct Kernel Object Manipulation (DKOM) hides objects by modifying kernel data structures. The most common DKOM technique unlinks a process's EPROCESS structure from the ActiveProcessLinks doubly-linked list.
Detection approach - cross-view detection: enumerate processes through multiple independent mechanisms and compare results.
On Windows:
- Walk the PsActiveProcessHead linked list (what Task Manager uses)
- Scan physical memory for EPROCESS structure signatures (what memory forensics tools do)
- Enumerate threads through PspCidTable (the kernel's handle table)
- Check the scheduler's ready queues for running threads
A process found by physical memory scanning but not in the active process list is a DKOM suspect, with one catch. Processes that have already exited also leave EPROCESS structures behind until the memory is reused, so check the exit time first: a structure with no exit time that's missing from the list is the real red flag.
On Linux:
- Walk the task_struct linked list (what
psuses) - Scan /proc for process directories
- Check the scheduler's run queues
- Scan kernel memory for task_struct signatures
Linux rootkits come in a few families. Kernel-module (LKM) rootkits such as Diamorphine hook syscalls or kernel functions to filter what /proc and getdents return. User-space rootkits load through LD_PRELOAD or /etc/ld.so.preload and override libc functions like readdir, so they only fool dynamically linked tools; a statically linked ls sees straight through them. And passive backdoors like BPFDoor, reported in 2022, attach a BPF filter to a raw socket and sit silent until a magic packet arrives, so there's no listening port to find.
A rootkit has to modify the OS to hide, and every modification leaves traces. Unlink a process from the list and you've still left its memory allocated, its threads scheduled, its handles open. Tools that look for those orphaned resources find them. The deeper a rootkit hides, the more anomalies it has to create on the way down.
SSDT Hooking Detection
On older Windows systems (pre-PatchGuard), rootkits hooked the System Service Descriptor Table (SSDT) to intercept system calls. The SSDT holds a pointer to every system call handler. Swap a pointer, and the rootkit reroutes system calls through its own code.
Detection: compare SSDT entries against known-good values. Every SSDT entry should point to an address within ntoskrnl.exe's address range. An entry pointing outside this range indicates a hook.
Normal SSDT entry: NtOpenProcess → ntoskrnl!NtOpenProcess (0xfffff80012345678)Hooked SSDT entry: NtOpenProcess → rootkit.sys+0x1234 (0xfffff880deadbeef)
Detection: for each SSDT entry, verify the target address falls withina legitimate kernel module's address rangeModern 64-bit Windows uses PatchGuard to flag SSDT modifications, so this technique is much rarer now. Still worth knowing — older systems remain in the wild, and the evolution of rootkit and anti-rootkit techniques is hard to follow without it.
IDT Hooking Detection
The Interrupt Descriptor Table (IDT) maps interrupt numbers to handler addresses. A rootkit that hooks IDT entries can intercept hardware interrupts, software interrupts, and exceptions. Hook the page fault handler (INT 14), for instance, and you see every page fault in the system.
Detection mirrors SSDT detection: verify that every IDT entry points to a legitimate kernel module. IDT entries should land inside ntoskrnl.exe or hal.dll.
Cross-View Detection in Practice
The cross-view principle extends beyond processes to all concealable objects:
Hidden files: compare directory listings from the filesystem API (which a rootkit can filter) against raw disk sector scanning (which bypasses the filesystem entirely). Files on disk but invisible through the API are being hidden.
Hidden network connections: compare netstat output (which reads kernel data structures the rootkit can manipulate) with raw packet capture (which records traffic regardless of those structures). Active connections sending traffic but missing from netstat are hidden.
Hidden registry keys: compare registry API queries against raw hive file parsing. Keys present in the hive file but invisible through the API are being hidden.
# Linux: two views of the process list, in the same format# View 1: what a directory listing of /proc shows (a rootkit# filtering getdents/readdir hides PIDs here)ls /proc | grep -E '^[0-9]+$' | sort > listed.txt
# View 2: probe every possible PID directly. Many rootkits filter# the listing but forget direct lookups. Threads also answer at# /proc/<tid>, so keep only thread-group leaders (Tgid == PID).for pid in $(seq 1 "$(cat /proc/sys/kernel/pid_max)"); do tgid=$(awk '/^Tgid:/ {print $2}' "/proc/$pid/status" 2>/dev/null) [ "$tgid" = "$pid" ] && echo "$pid"done | sort > probed.txt
# PIDs that answer when asked directly but never appear in the listingcomm -13 listed.txt probed.txtProcesses start and exit while the loop runs, so re-check any hit before trusting it. The unhide tool automates this kind of comparison across several more views of the process table.
Integrity Checking
Rather than chasing specific rootkit behaviours, integrity checking sets a known-good baseline and alerts on any deviation:
Code integrity: hash all kernel code pages and re-check the hashes periodically. Any change to kernel code means patching by a rootkit (or, occasionally, a bug). Windows HVCI enforces code integrity at the hypervisor level and blocks kernel code modification outright.
Data structure integrity: check that kernel data structures stay self-consistent. Every process in the scheduler's run queue should also appear in the active process list, for instance. A mismatch is a DKOM tell.
Module verification: maintain a list of expected kernel modules and alert on anything unfamiliar. Compare loaded module code against the original file on disk to catch in-memory patching.
Hardware-Assisted Rootkit Detection
The most advanced detection techniques use hardware features:
Hypervisor-based detection: run the detection tool in a hypervisor that owns the monitored OS. From there, it can watch every memory modification, intercept attempts to touch kernel structures, and verify integrity while remaining invisible to the rootkit.
IOMMU monitoring: use the IOMMU to catch DMA-based attacks where a device (or a rootkit driving a device) writes to kernel memory directly via DMA, sidestepping CPU-level protections.
Intel PT (Processor Trace): hardware-level instruction tracing that records every instruction the CPU executes. The trace comes from the CPU itself rather than from software the rootkit can patch. That isn't a guarantee: PT is configured through model-specific registers, so ring-0 code can switch tracing off or redirect its output buffer, which is why PT-based detection works best when a hypervisor owns that configuration. Replay the trace and you see every code path, including rootkit code that software monitoring missed.
Anti-rootkit tools running at the same privilege level as the rootkit start the fight already losing — the rootkit can mess with the detection tool itself. The trend, accordingly, is detection from a higher privilege level: hypervisor-based monitoring, hardware-assisted tracing, and external forensics performed from a separate system. The detection tool needs to outrank the rootkit, or it can't be trusted.