Skip to main content
Be Bitwise

Kernel-Level Monitoring

Low-Level Detection & Instrumentation · Defense & Forensics track

5 min read

In this module

Watching from Below

The kernel is the most dependable vantage point for watching a system. User-space monitoring can be evaded by malware that operates below it, while kernel-level instrumentation runs at ring 0 alongside the OS and sees every syscall, process and connection go past. It isn't untouchable, though: malware that gets into the kernel itself can blind it, which is the subject of Anti-Rootkit Techniques. Modern Linux systems provide powerful frameworks that security tools use for real-time threat detection — and this is where most serious EDR tooling actually lives.

eBPF: Programmable Kernel Instrumentation

eBPF (extended Berkeley Packet Filter) has transformed kernel monitoring. Originally designed for network packet filtering, it's grown into a general-purpose in-kernel virtual machine that can attach programs to almost any kernel event.

An eBPF program is:

  • Written in a restricted C dialect and compiled to eBPF bytecode
  • Verified by the kernel's eBPF verifier (ensures no infinite loops, no invalid memory access, bounded execution time)
  • JIT-compiled to native machine code for near-zero overhead
  • Attached to kernel hooks (tracepoints, kprobes, syscall entry/exit, network events)
c
// eBPF program that monitors all execve() syscalls
SEC("tracepoint/syscalls/sys_enter_execve")
int trace_execve(struct trace_event_raw_sys_enter *ctx) {
char comm[TASK_COMM_LEN];
char filename[256];
bpf_get_current_comm(&comm, sizeof(comm));
bpf_probe_read_user_str(&filename, sizeof(filename),
(void *)ctx->args[0]);
bpf_printk("execve: %s executed %s\n", comm, filename);
return 0;
}

Security tools built on eBPF include Falco (runtime security), Cilium (network security), Tetragon (process monitoring), and bpftrace (ad-hoc tracing). These tools can monitor file access, network connections, process creation, privilege escalation, and kernel module loading - all from within the kernel with minimal performance impact.

eBPF Safety Model

The eBPF verifier is critical for security. It statically analyses every eBPF program before loading to ensure it cannot crash the kernel, access arbitrary memory, or run indefinitely. Programs that fail verification are rejected. This makes eBPF much safer than loadable kernel modules, which have unrestricted kernel access and can crash the system with a single bug.

Here's the path an eBPF-based sensor takes from source to alert, with the older audit framework alongside for comparison. Note where the verifier sits: before anything is attached, so an unsafe program never runs at all. Audit, by contrast, is a fixed set of hooks built into the kernel that sends records to a daemon over netlink.

Kprobes and Uprobes

Kprobes (kernel probes) insert breakpoints at any instruction in the kernel. When the probed instruction executes, the kprobe handler runs, can inspect registers and arguments, and then the original instruction continues.

Uprobes (user-space probes) do the same for user-space functions. A uprobe on SSL_write in libssl, for example, can capture unencrypted data before it is encrypted - useful for monitoring encrypted communications.

bash
# Using bpftrace to monitor file opens with kprobe
# arg1 (filename) is a user-space pointer, but bpftrace assumes kprobe
# arguments point into the kernel, so mark it with uptr() or str() fails
bpftrace -e 'kprobe:do_sys_openat2 {
printf("%s opened %s\n", comm, str(uptr(arg1)));
}'
# Monitor malloc calls in a specific process with uprobe
bpftrace -e 'uprobe:/lib/x86_64-linux-gnu/libc.so.6:malloc {
printf("%s malloc(%d)\n", comm, arg0);
}'

Syscall Auditing

The Linux Audit Framework (auditd) provides full-coverage system call auditing. Unlike eBPF-based monitoring, which requires custom programs, the audit framework uses declarative rules:

bash
# Audit all execve calls
auditctl -a always,exit -F arch=b64 -S execve -k process_execution
# Audit file access to sensitive files
auditctl -w /etc/shadow -p rwa -k shadow_access
# Audit module loading
auditctl -a always,exit -S init_module -S finit_module -k module_load
# View audit logs
ausearch -k process_execution --start today

The audit framework logs to /var/log/audit/audit.log and provides tools for searching and reporting. It is widely used for compliance (PCI DSS, HIPAA) and security monitoring. The main limitation is performance: verbose audit rules on busy systems can generate enormous log volumes.

Inside the Audit Framework

The audit system consists of several components:

  • Kernel audit subsystem: Generates audit events based on configured rules
  • auditd: Daemon that receives events from the kernel and writes them to disk
  • auditctl: Tool for managing audit rules
  • ausearch/aureport: Tools for searching and reporting on audit logs
  • audisp: Audit dispatcher that forwards events to other systems (like a SIEM)

For security monitoring, the most valuable audit events are syscalls (execve, connect, open, ptrace), file access to sensitive paths, authentication (login, sudo, SSH), kernel module loading, and network events (socket creation, bind, connect).

Combining the audit framework with eBPF gives you coverage at both ends: audit rules for broad event categories, eBPF for deep inspection of specific events you care about.

Performance Considerations

Kernel monitoring always has a performance cost. Each eBPF program run adds anywhere from tens of nanoseconds to a few microseconds, depending on what it does. Audit rules add overhead proportional to the number of matching events. kprobes add overhead for each probe hit. The key is to monitor selectively: instrument the critical events (process creation, network connections, privilege changes) rather than every system call. Tuned that way, the overhead is usually small, often a few per cent of CPU or less, though the real figure depends on the workload and the rule set, so measure it on your own systems.