Skip to main content
Be Bitwise

Memory Performance Analysis

Cache & Performance · Memory & Systems track

6 min read

Part ofSystems Programming Companion, week 7

In this module

Measuring What Matters

Optimising cache behaviour without measurement is guesswork dressed up as engineering. Modern CPUs expose hardware performance counters, special registers that count events like cache misses, branch mispredictions, and instructions retired. Tools like perf, Intel VTune, and Cachegrind read those counters and show you exactly where your program is bleeding time, and why.

Key Concept

Hardware performance counters give you the truth: how many cycles, how many cache misses, how many branch mispredicts. Without them you're guessing where the slow path is. With them, the slowness has an address.

perf stat: The Starting Point

The perf stat command runs your program and dumps summary statistics from the hardware performance counters.

text
$ perf stat ./program
Performance counter stats for './program':
2,500.45 msec task-clock
5,234,567,890 instructions # 2.10 insn per cycle
2,492,365,421 cycles
312,456,789 cache-references
45,678,901 cache-misses # 14.62% of all cache refs
123,456,789 branch-instructions
8,901,234 branch-misses # 7.21% of all branches

The numbers worth squinting at:

  • Instructions per cycle (IPC): modern superscalar CPUs can retire 3-5+ instructions per cycle. An IPC below 1 means the CPU is stalling, almost always on memory latency or branch misprediction.
  • Cache miss ratio: percentage of cache references that miss. Above 5-10% for LLC misses points squarely at poor locality.
  • Branch miss ratio: above 5% suggests unpredictable branches and may be a hint to look at branchless algorithms.

perf record and perf report: Finding Hot Spots

perf stat gives totals. perf record is the next step up: it samples the program counter at regular intervals (or on specific events) and maps every sample to a source code line.

bash
$ perf record -e cache-misses ./program
$ perf report
# Overhead Command Shared Object Symbol
# 35.2% program program [.] matrix_multiply
# 22.1% program program [.] hash_lookup
# 15.8% program libc.so [.] memcpy

Drill into each function and you get per-line cache miss counts, which puts a finger on the exact code that's causing the trouble. That needs debug info: build with -g, or perf can only map samples to functions and raw instructions. Also, perf is Linux-only. On macOS, Instruments plays the same role, and WSL often hides the hardware counters, so valgrind --tool=cachegrind (below) is the fallback there.

For surgical analysis, perf annotate drops down to the assembly level.

text
$ perf annotate matrix_multiply
2.10 │ mov (%rax,%rcx,8),%rdx ; 2.10% of cache misses here
35.20 │ add (%rbx,%rcx,8),%rdx ; 35.20% of cache misses here
0.30 │ mov %rdx,(%rdi,%rcx,8)
Sampling vs Counting

Performance counters operate in two modes. Counting mode (perf stat) counts all events, giving exact totals but no location information. Sampling mode (perf record) generates an interrupt after every N events and records the program counter, giving approximate per-location breakdowns. Sampling has overhead (the interrupts themselves take time) but provides the spatial information needed to optimise specific code.

Intel VTune Profiler

Intel VTune is a heavyweight performance analysis tool that digs deeper into the hardware than perf does. The standout features:

  • Microarchitecture Exploration: classifies each cycle as useful work or a specific type of stall (front-end bound, back-end bound, memory bound, core bound). This top-down view tells you whether to attack cache optimisation, branch prediction, or instruction-level parallelism.
  • Memory Access Analysis: maps memory accesses onto the NUMA topology, showing local vs remote accesses and which data structures churn out the most LLC misses.
  • Platform Profiler: watches system-wide memory bandwidth, power, and frequency.

VTune is free for individual developers and surfaces insights that are awkward to coax out of perf.

Cachegrind: Simulated Cache Analysis

Valgrind's Cachegrind tool simulates the cache hierarchy in software, counting hits and misses on every source line. Where perf samples, Cachegrind counts every access deterministically.

text
$ valgrind --tool=cachegrind ./program
$ cg_annotate cachegrind.out.12345
---------------------------------------------------
Ir I1mr ILmr Dr D1mr DLmr Dw
---------------------------------------------------
100,000 0 0 50,000 25,000 5,000 20,000 matrix.c:42
50,000 0 0 25,000 100 10 10,000 matrix.c:43

Cachegrind reports instruction cache stats (Ir, I1mr, ILmr) and data cache stats (Dr, D1mr, DLmr, Dw) on each line. That's enough granularity to point at exactly which array accesses miss the cache and why.

The catch is Cachegrind's eye-watering overhead (a 20-50x slowdown), plus the fact that it simulates a fixed cache configuration rather than your actual hardware. Despite all that, nothing else gives you the deterministic, per-line picture it does.

Hardware Performance Counters

Cache misses aren't the only thing modern CPUs count. They expose hundreds of performance counters. A few particularly useful ones:

  • mem_load_retired.l1_hit: loads satisfied by L1 cache
  • mem_load_retired.l2_hit: loads satisfied by L2 cache
  • mem_load_retired.l3_hit: loads satisfied by L3 cache
  • mem_load_retired.l3_miss: loads that missed all caches
  • offcore_response: loads from local DRAM, remote DRAM, or other sockets
  • dtlb_load_misses.walk_completed: TLB misses that required a page table walk

Together, these counters paint a detailed picture of how the memory hierarchy is actually behaving. Use them to tell apart the different flavours of cache miss: compulsory misses (first access), capacity misses (working set too large), and conflict misses (addresses fighting over the same cache set).

Memory Bandwidth

Total memory bandwidth is a finite resource, and one of the easiest to forget. A system with DDR4-3200 in dual-channel has roughly 50 GB/s of theoretical bandwidth. If your program is doing 10 billion 8-byte loads per second, it wants 80 GB/s. Tough luck.

To measure bandwidth use on Linux, Intel's pcm-memory tool gives a live per-channel readout. On CPUs with Memory Bandwidth Monitoring (MBM, or AMD's equivalent), the kernel's resctrl filesystem reports bytes moved per group of tasks under /sys/fs/resctrl. You may find old guides using perf stat -e intel_cqm/... events. That interface was removed from the kernel in Linux 4.14 and resctrl replaced it. When your program is bandwidth-limited, shrinking the data (smaller types, SoA layout, compression) does more for performance than trimming instructions ever will.