Endpoint Detection Internals
Low-Level Detection & Instrumentation · Defense & Forensics track
6 min read
In this module
How EDR Really Works
Endpoint Detection and Response (EDR) products are the frontline defence on modern endpoints. But what's actually happening inside these tools? Defenders need to know what their tools can and cannot see, and researchers need to map the detection landscape. This lesson digs into the low-level hooking mechanisms EDR products rely on.
Userland Hooks on Windows
The most common EDR technique on Windows is inline hooking of ntdll.dll functions. ntdll.dll is the lowest-level user-space DLL, holding the syscall stubs that transition from user mode to kernel mode. Hook functions there and you intercept every API call before it reaches the kernel.
The hooking process:
- The EDR driver registers a kernel callback to be notified of every new process creation
- When a process starts, the EDR injects its monitoring DLL into the process
- The monitoring DLL overwrites the first bytes of target ntdll functions with a
JMPto the EDR's handler - When the application calls a hooked function, execution redirects to the EDR
- The EDR inspects the call, logs it, and decides whether to allow or block it
- If allowed, the EDR executes the original instructions and jumps back to continue the function
; Original NtAllocateVirtualMemory in ntdll.dll (Windows 10 x64)4C 8B D1 mov r10, rcxB8 18 00 00 00 mov eax, 0x18 ; syscall numberF6 04 25 08 03 FE 7F 01 test byte ptr [0x7FFE0308], 175 03 jne +3 ; legacy int 2e path0F 05 syscallC3 ret
; After EDR hooking: a 5-byte jmp rel32 overwrites the first 5 bytesE9 xx xx xx xx jmp EDR_Hook_Handler00 00 00 ; leftover tail of mov eax, never executedF6 04 25 08 03 FE 7F 01 test byte ptr [0x7FFE0308], 1... ; rest of the stub untouched
; Trampoline, in the EDR's own memory: replays what the jmp destroyedmov r10, rcxmov eax, 0x18jmp ntdll!NtAllocateVirtualMemory+8 ; back to the test instructionCommon functions hooked by EDR products include: NtAllocateVirtualMemory (memory allocation), NtWriteVirtualMemory (writing to other processes), NtCreateThreadEx (thread creation), NtMapViewOfSection (DLL loading), and NtProtectVirtualMemory (changing memory protections).
Userland hooks have a built-in weakness: they live in the same address space as the monitored process. An attacker with code execution in the process can read the hooked function, spot the EDR's jump instruction, and either restore the original bytes (unhooking) or call the syscall directly, bypassing ntdll entirely. That's why EDR products lean increasingly on kernel-mode components alongside userland hooks.
Kernel Callbacks on Windows
Windows provides official kernel callback mechanisms that EDR drivers register:
Process creation callbacks (PsSetCreateProcessNotifyRoutineEx): the driver gets notified of every process creation and termination, complete with process name, parent PID, and command line.
Thread creation callbacks (PsSetCreateThreadNotifyRoutine): every thread creation fires a callback. Useful for catching remote thread creation, which is a classic injection tell.
Image load callbacks (PsSetLoadImageNotifyRoutine): fires whenever an executable image (EXE, DLL) loads into a process, so the EDR can scan it.
Object access callbacks (ObRegisterCallbacks): can filter and block process and thread handle requests, stopping tools like Mimikatz from grabbing handles to LSASS.
Registry callbacks (CmRegisterCallbackEx): notification of every registry operation. Persistence mechanisms light up here.
Minifilter Drivers
For file system monitoring, EDR products use minifilter drivers, the Windows file system filter framework for intercepting all file I/O operations:
Application → I/O Manager → EDR Minifilter → File System → Disk
Pre-operation callback: EDR inspects the file before the operationPost-operation callback: EDR inspects the result after the operationMinifilters can scan files for malware signatures during reads, detect ransomware patterns (mass file renaming/encryption), and block access to sensitive files. Because they run in the kernel, an ordinary user-mode process can't simply patch them out the way it can unhook ntdll. They aren't untouchable, though: an attacker with administrator rights can try to unload the filter or detach it from a volume, or load a vulnerable signed driver to tamper with it, which is why EDRs guard and monitor their own drivers.
Living in the kernel cuts both ways. On 19 July 2024 a faulty content update for CrowdStrike's Falcon sensor (Channel File 291) made its kernel driver read out of bounds, and Windows machines running it crashed into a boot loop. Microsoft estimated that about 8.5 million devices were hit. The same privilege that lets an EDR see and block everything means a bug in it takes the whole machine down, and since then Microsoft has been working with security vendors to move more EDR logic out of the kernel.
ETW (Event Tracing for Windows)
ETW is a high-performance tracing infrastructure baked into the Windows kernel. Where hooks intercept and can block, ETW just watches passively, surfacing system events as a stream.
Key ETW providers for security:
- Microsoft-Windows-Kernel-Process: Process creation, termination, and image loads
- Microsoft-Windows-Kernel-File: File operations
- Microsoft-Windows-Kernel-Network: TCP/IP connection events
- Microsoft-Windows-DotNETRuntime: .NET assembly loads and JIT compilation
- Microsoft-Windows-PowerShell: PowerShell command execution
- Microsoft-Antimalware-Scan-Interface: AMSI scan events
- Microsoft-Windows-Threat-Intelligence (ETW-TI): events raised by the kernel itself for operations such as allocating, writing or protecting memory in another process and queuing APCs to another process's threads. Only protected (PPL) anti-malware processes can consume it, and since the events originate in kernel code, user-mode ETW patching doesn't silence them
EDR products consume ETW events as a telemetry source alongside hook-based detection. ETW is kernel-integrated and runs at very high throughput with negligible overhead.
Detection Philosophy
Modern EDR products combine multiple data sources:
- Userland hooks: Deep inspection of API call parameters
- Kernel callbacks: Process, thread, image, and registry monitoring
- Minifilter drivers: File system activity monitoring
- ETW consumers: High-level event correlation
- Network monitoring: DNS queries, connection metadata
Each source has blind spots the others cover. Userland hooks can be bypassed by direct syscalls, but kernel callbacks still see the operation. ETW can be patched in user space, but kernel-mode ETW consumers get events before any user-space patching takes effect. The combination is a detection mesh that's a great deal harder to evade than any single mechanism on its own.
Follow a classic CreateRemoteThread injection through the sensors below. Each API call on the left trips one or more of them. Now imagine the attacker switches to direct syscalls: cross out every ntdll hook box and count what still fires.
Every hooking mechanism can be evaded, and every evasion technique can be detected. Direct syscalls bypass ntdll hooks, but kernel ETW still captures the events. Unhooking ntdll works once, then the EDR notices its own hooks have vanished. The arms race drives innovation on both sides, and understanding the mechanics from both angles is the only way to do useful security research.