Skip to main content
Be Bitwise

The Memory Wall Problem

Near-Memory Computing & Processing-in-Memory · Frontiers track

9 min read

In this module

The Bottleneck You Cannot See

We've spent this course pulling processors apart in tremendous detail: instruction sets, pipelines, caches, branch predictors. We've walked the memory hierarchy from registers down to DRAM and watched caches scramble to keep hot data close to the CPU. But there's one problem that no amount of CPU cleverness fully solves: the memory wall.

The memory wall is the widening gulf between how fast processors compute and how fast they can grab data from main memory. It's the single most stubborn bottleneck in modern computing, and you can't really understand why whole new paradigms — near-memory computing, processing-in-memory — are being designed without first understanding it.

The Numbers Tell the Story

In 1980, processor cycle times and memory access times were roughly even, both on the order of a few hundred nanoseconds. Then processors took off at around 60% per year, courtesy of Moore's Law and a steady drumbeat of architectural innovations, while DRAM access latency crawled forward at maybe 7% per year.

By the 2020s, the gap has become enormous:

ComponentTypical Latency
CPU arithmetic operation~0.3 ns (single cycle at 3 GHz)
L1 cache access~1 ns (3-4 cycles)
L2 cache access~3-5 ns
L3 cache access~10-20 ns
DRAM access~50-100 ns
SSD access~10,000-100,000 ns

A CPU twiddling its thumbs waiting on DRAM might stall for 150-300 cycles. That's hundreds of useful operations it could have done instead. To put it on a human scale: if a CPU cycle were one second long, hitting DRAM would take you about five minutes, and hitting an SSD would eat up the rest of your afternoon.

The Memory Wall

The memory wall is the term for the widening gap between processor speed and memory access speed. First described by Wulf and McKee in 1995, it predicts that memory latency will increasingly dominate program execution time, regardless of how fast processors become. Three decades later, this prediction has proven remarkably accurate. Cache hierarchies help but cannot eliminate the problem, because many important workloads have access patterns that caches cannot capture efficiently.

Why Caches Are Not Enough

We've covered cache hierarchies in earlier modules. Caches work by exploiting two patterns in how programs touch data:

  • Temporal locality — touched something recently, you'll probably touch it again soon
  • Spatial locality — touched address X, you'll probably touch X+1 next

For many traditional workloads (tight loops over small arrays, blocked matrix multiplication), caches are magnificent. A well-tuned hierarchy lets the processor feel as if a huge slab of memory is right there and fast.

But many of today's most important workloads have terrible cache behaviour:

Graph analytics — social networks, knowledge graphs, and web graphs are pointer-chasing exercises. Each node points to neighbours scattered across memory more or less at random. Cache hit rates during graph traversal often crawl below 10%.

Database operations — scanning a big table, joining two tables on a key, or building an index burns through huge amounts of data with almost no reuse. The working set is laughably bigger than any cache.

Genomics — DNA sequence alignment compares short reads against a reference genome measured in gigabytes. The access pattern is dictated by the biology, not by anything cache-friendly. It looks essentially random.

Machine learning inference — large neural network models (GPT-class language models carry billions of parameters) stream through colossal weight matrices. Each weight is used exactly once per inference, so the cache might as well not be there.

Key-value stores — look up a value by key in a vast hash table and you compute a hash (fast), then jump to a random memory location (slow). With millions of lookups a second, the memory system is what holds you back.

You can watch this happen on a toy cache. The simulator below is direct-mapped: 8 lines of 4 bytes, so only 32 bytes fit at once. Its access sequence hops between scattered addresses the way a graph traversal chases neighbour pointers. Press Run and the hit rate settles at 1 in 12, about 8%. Then hit Reset and type addresses 0 to 15 in order: the same cache scores 75%, because each miss drags in a 4-byte line and the next three reads find it waiting. The addresses are illustrative, but the gap is the real story. If every miss costs around 200 cycles, the scattered walk averages well over 150 cycles per access while the sequential one averages about 50.

Sequence [0/12]:
500ms
07214036200104168823272140136
Hits: 0Misses: 0Rate: 0.0%

Cache Contents(Direct-Mapped, 8 lines, 4B/line)

IndexVTagData
00--
10--
20--
30--
40--
50--
60--
70--

For these workloads, the processor spends most of its time waiting on data, not computing. They're called memory-bound, the opposite of compute-bound workloads where the processor is the limiting factor.

The Energy Cost of Data Movement

The memory wall is an energy story as well as a performance one. Shifting data around eats a shocking amount of power. Research from the late 2010s showed that in many modern systems, 40-60% of total energy is burnt on moving data rather than actually computing on it.

Consider what happens when a processor needs to add two numbers stored in DRAM:

  1. The processor issues a load request
  2. The request travels through the cache hierarchy (L1 miss, L2 miss, L3 miss)
  3. The memory controller translates the address and sends a command to the DRAM
  4. The DRAM activates the appropriate row, reads the data, and sends it back
  5. The data travels back through the memory bus, through the cache hierarchy, to the register file
  6. The processor performs the addition (a single cycle)
  7. The result travels back through the caches (and eventually to DRAM if evicted)

The energy cost of the addition itself is trivial: maybe 1 picojoule. The energy cost of dragging the data from DRAM to the processor and back can run 100 to 1,000 times higher. We're spending most of the budget chauffeuring bits around.

Data Movement Dominates Energy

A single DRAM access consumes roughly 100-1000x more energy than a simple arithmetic operation. In data centers running memory-intensive workloads, the energy cost of moving data between memory and processors can exceed the energy cost of the actual computation by an order of magnitude. This is a primary motivation for processing-in-memory: if you can compute where the data already lives, you eliminate most of the energy cost.

Bandwidth vs. Latency

The memory wall has two dimensions: latency (how long it takes to get a single piece of data) and bandwidth (how much data you can get per second).

Modern memory systems have made vastly more progress on bandwidth than on latency. DDR5 and HBM (High Bandwidth Memory) post huge throughput numbers — HBM3 punts more than 800 GB/s. Latency, on the other hand, has barely moved. A single random DRAM access still costs 50-100 nanoseconds, roughly what it did twenty years ago.

High bandwidth pays off when you can grab large contiguous blocks of data, the streaming case. The moment your workload needs many small, random accesses — graph analytics, database lookups, pointer-chasing — extra bandwidth gets you nothing. You're latency-limited, and no amount of bandwidth fixes a latency-bound problem.

That's why some researchers argue we have not one but two memory walls: the latency wall and the bandwidth wall. Caches and prefetching chip away at the bandwidth wall. Against the latency wall, with random access patterns, they're nearly powerless.

The Von Neumann Bottleneck

The memory wall is actually a specific manifestation of a deeper architectural limitation known as the von Neumann bottleneck. In the von Neumann architecture (which underlies virtually all modern computers), the processor and memory are separate entities connected by a bus. All data must flow through this bus, which has finite capacity.

John Backus highlighted this problem in his 1977 Turing Award lecture, calling it a "tube" through which data must squeeze one word at a time. Nearly fifty years later, despite enormous advances in bus width and bandwidth, the fundamental bottleneck remains: computation happens in one place, data lives in another, and a channel of limited capacity connects them.

Near-memory computing and processing-in-memory represent a fundamental rethinking of this architecture. Instead of always bringing data to the processor, what if we bring the processor to the data?

What Is Changing

Several trends are making the memory wall worse, creating urgency for new approaches:

Data volumes are exploding. Machine learning models are growing exponentially. Genomics datasets are measured in petabytes. Social media graphs have billions of nodes. The working sets of important applications are growing much faster than cache sizes.

Moore's Law is slowing. Transistor scaling is approaching physical limits. We can no longer count on processors getting dramatically faster each year. This means we cannot outrun the memory wall with faster processors.

Energy constraints are tightening. Data centers already consume a significant fraction of global electricity. As AI workloads grow, the energy cost of data movement is becoming unsustainable. Processing data where it lives could reduce energy consumption by 10-100x for some workloads.

New memory technologies are emerging. High Bandwidth Memory (HBM), which stacks DRAM dies with through-silicon vias, creates an opportunity to embed processing logic on the base logic die. Emerging non-volatile memories like ReRAM and PCM have properties that naturally support in-memory computation.

These converging trends are driving a wave of innovation in near-memory and processing-in-memory architectures, which we will explore in the following lessons.

The Fundamental Insight

The key insight behind near-memory computing is simple but profound: for many workloads, it is cheaper to move computation to the data than to move data to the computation. If data lives in DRAM and you need to perform a simple operation on it, placing a small processor inside or next to the DRAM eliminates the costly round trip through the memory bus.

This does not mean that traditional processors are obsolete. For compute-heavy workloads with good data locality, conventional CPUs and GPUs remain excellent. But for the growing class of memory-bound workloads - and the growing fraction of system energy spent on data movement - processing near or in memory offers a fundamentally better approach.

In the next lesson, we will examine specific near-memory computing architectures that are being built and deployed today.