Skip to main content
Be Bitwise

NUMA & Modern Memory

Cache & Performance · Memory & Systems track

5 min read

In this module

Non-Uniform Memory Access

On a single-socket system, every byte of memory is equally cheap to access. On a multi-socket system, each CPU comes with its own memory controller and its own directly attached RAM. Local memory is fast. Reaching memory attached to another socket means crossing an interconnect, and the latency hit is real. Welcome to NUMA — Non-Uniform Memory Access.

Key Concept

On NUMA hardware, where a thread allocates and which socket it runs on both matter. Memory has a "home" node, and reaching another node's memory costs a multiple of local-access latency. Pinning matters.

NUMA Architecture

A typical two-socket server has two NUMA nodes:

The dotted arrow is the expensive one. Picture a thread that first touched its working set while running on core 3, so under first-touch (below) the pages came from node 0. Then the scheduler moves it to core 20 on socket 1. The thread's code hasn't changed and nor has its data, but every cache miss now takes the dotted path: out through socket 1, across the interconnect, through socket 0's memory controller and back.

Local memory access: ~100ns. Remote memory access (crossing the interconnect): ~150-200ns. That 50-100% penalty applies to every cache miss that lands on remote memory. For memory-intensive workloads, NUMA-blind allocation can quietly slice performance in half.

NUMA-Aware Allocation

On Linux, the default allocation policy is "first-touch": a page lands on whichever NUMA node the thread that first accessed it is running on. Fine when the thread allocating data is the same thread using it.

The trouble starts when:

  • One thread initialises data that other threads (possibly on different nodes) consume
  • Threads migrate between cores on different nodes
  • A large allocation needs more than a single node's memory can hold

The kernel can paper over some of this on its own. With automatic NUMA balancing (kernel.numa_balancing, on by default on many distributions when the machine has more than one node), it periodically unmaps pages, watches which node faults on them, and migrates pages towards the threads that use them. That helps after a migration, but it takes time to react and costs page faults, so explicit placement still wins for workloads you understand.

Controlling NUMA Placement

Linux provides several mechanisms for NUMA-aware allocation:

  • numactl --membind=0 ./program: restrict all allocations to node 0
  • numactl --interleave=all ./program: spread allocations across all nodes (good for shared data)
  • mbind() system call: set NUMA policy for specific memory ranges
  • libnuma: programmatic access to NUMA topology and allocation policies

Thread pinning (taskset or pthread_setaffinity_np) keeps threads on specific cores, preventing migration that would turn local accesses into remote ones.

Memory Interleaving

When data is hit by every core roughly equally — a shared read-only lookup table, say — the right move is interleaving: scattering pages across all NUMA nodes in round-robin. The bandwidth gets spread across every memory controller in the system, so no single node turns into a chokepoint.

bash
$ numactl --interleave=all ./shared_data_server

Interleaving trades locality (half your accesses are remote now) for balanced bandwidth. It works best when the data is genuinely shared across all nodes, the access pattern is random, or bandwidth is the bottleneck rather than latency.

CXL Memory Expansion

Compute Express Link (CXL) is an emerging interconnect standard that lets memory plug into the system through PCIe-like connections. CXL Type 3 devices add memory that the OS sees as another NUMA node — higher latency than local DRAM, but much faster than traditional storage.

CXL opens the door to memory pooling: multiple servers share a pool of CXL-attached memory and pull capacity dynamically as needed. That addresses memory stranding in data centres, where one rack's servers sit with unused memory while another rack's are starving.

To software, CXL memory just looks like a NUMA node with higher latency. NUMA-aware allocation can park performance-critical data on local DRAM while pushing less-critical stuff (caches, logs, cold data) onto CXL.

High Bandwidth Memory (HBM)

HBM (High Bandwidth Memory) is 3D-stacked DRAM that delivers enormous bandwidth by stacking memory dies vertically and wiring them together with through-silicon vias (TSVs). You'll find HBM in GPUs, some server CPUs (Intel Sapphire Rapids with HBM), and high-performance computing systems.

HBM offers 2-8x the bandwidth of DDR4/DDR5 but its capacity is sharply limited, usually 16-64 GB per stack. When it sits alongside standard DDR, the system ends up with two tiers: fast and small, slow and large. The OS or application has to decide what goes where.

Intel's "flat mode" presents HBM and DDR as two NUMA nodes, putting the placement decision in your hands. "Cache mode" uses HBM as a transparent hardware cache for DDR, automatically caching hot data. Flat mode gives the programmer control; cache mode is hands-off but won't always cache optimally for every access pattern.

Persistent Memory

Persistent memory (exemplified by Intel Optane DC Persistent Memory) parks itself between DRAM and storage on both latency and capacity. Byte-addressable access like DRAM, data that survives power cycles like storage.

Persistent memory runs in one of two modes:

  • Memory mode: extends DRAM capacity, with local DRAM acting as a hardware cache for the persistent memory. Software doesn't have to know.
  • App Direct mode: persistent memory gets mapped into the address space and accessed with load/store instructions. The application treats it as ultra-fast storage (via mmap on persistent files) or as non-volatile data structures.

Writing code for persistent memory means thinking hard about crash consistency. If power dies mid-update, the persistent memory has to be in a recoverable state when it comes back. That demands explicit cache line flushes (clwb) and memory barriers (sfence) to control the order in which data hits the persistent medium.

Intel killed off Optane in 2022, but the persistent memory programming model (SNIA's Programming Model, libpmem) is still alive and well as CXL-attached persistent memory devices arrive from other vendors. The core puzzle — building data structures that survive crashes without giving up performance — applies to any byte-addressable persistent medium.