Skip to main content
Be Bitwise

Near-Memory Computing Architectures

Near-Memory Computing & Processing-in-Memory · Frontiers track

9 min read

In this module

Bringing Compute Closer to Data

The previous lesson established that the memory wall — the widening gap between processor speed and memory latency — is a real bottleneck. The core idea of near-memory computing is disarmingly simple: instead of ferrying data to a powerful processor, put the processing element where the data already lives.

But "close to memory" covers a spectrum. This lesson surveys the major near-memory architectures in development and production today, from HBM-PIM to UPMEM to Samsung's AIM.

The Near-Memory Computing Spectrum

Near-memory computing exists on a spectrum defined by how close the processing element is to the memory:

  1. Near-storage computing - processing near SSDs or storage devices (e.g., computational storage drives)
  2. Near-DRAM computing - processing on the logic die of a 3D-stacked memory (e.g., research designs built on the Hybrid Memory Cube's logic layer)
  3. In-DRAM computing - processing within the DRAM die itself but outside the array (e.g., UPMEM DPUs and Samsung's HBM-PIM)
  4. In-memory computing - computation performed directly within the memory array using analogue properties of the memory cells (covered in a later module on analogue computing)

Each approach makes different trade-offs between the power of the processing element, the bandwidth available to it, and the complexity of integration.

HBM-PIM: Processing Inside High Bandwidth Memory

High Bandwidth Memory (HBM) is a 3D-stacked memory technology used in GPUs, AI accelerators, and high-performance computing. An HBM stack consists of multiple DRAM dies stacked vertically on top of a base logic die, connected by thousands of through-silicon vias (TSVs). The base logic die handles memory control and interfaces with the host processor.

The obvious place to add compute would be that base die, and several research designs do exactly that. Samsung's HBM-PIM (announced in 2021) goes a step further: it embeds lightweight SIMD (Single Instruction, Multiple Data) processing units inside the DRAM dies themselves, next to the banks. These units can perform vector operations - addition, multiplication, and multiply-accumulate - directly on data in the HBM without it needing to leave the stack.

Architecture Details

Samsung's HBM-PIM places one processing unit beside every pair of banks in its PIM-enabled DRAM dies. Each processing element is a programmable compute unit that operates on 16-bit floating-point (FP16) data, optimised for the vector operations common in AI workloads.

The key advantage is bandwidth: a normal read moves one bank's data at a time out through the TSVs and the stack's external interface, but in PIM mode many banks feed their own processing units at once. The combined bank-level bandwidth is several times what the external interface can carry, and HBM-PIM uses it to accelerate memory-bound operations.

Why HBM Is Perfect for PIM

HBM's 3D-stacked architecture creates a natural home for PIM processing. HBM already sits right next to the accelerator in the same package, and Samsung's design keeps the standard HBM2 interface, so adding processing units beside the banks is an incremental change rather than a revolutionary one. Running many banks in parallel gives the PIM units bandwidth well beyond what the external interface delivers, which the host processor can never reach. This makes HBM-PIM a practical, near-term approach to processing-in-memory.

Performance Results

Samsung reported that HBM-PIM more than doubled system performance for certain AI workloads while cutting energy use by more than 70%. The speedup comes from eliminating the data transfer between HBM and the GPU's compute units for operations that the PIM units can handle directly.

HBM-PIM is particularly effective for element-wise operations on large vectors: adding two vectors, scaling a vector by a constant, or applying an activation function. These operations are trivially parallel, require no data reuse, and are severely bandwidth-limited on conventional architectures.

UPMEM: A General-Purpose PIM Architecture

UPMEM is a French company that has developed the first commercially available general-purpose PIM hardware. Unlike HBM-PIM, which adds specialised SIMD units, UPMEM embeds full general-purpose processors - called DRAM Processing Units (DPUs) - inside DRAM chips.

Architecture Details

Each UPMEM DRAM chip contains 8 DPUs. Each DPU is a 32-bit RISC-style processor with:

  • A small instruction memory (24 KB IRAM)
  • A working memory (64 KB WRAM, serving as scratchpad/cache)
  • Access to a 64 MB DRAM bank (called MRAM - Main RAM)
  • A 14-stage pipeline capable of running at several hundred MHz

A standard UPMEM DIMM contains 128 DPUs. A server with 20 DIMMs would have 2,560 DPUs running in parallel, each operating on its own local memory bank. The aggregate memory bandwidth across all DPUs far exceeds what the host CPU could achieve through the normal memory bus.

Programming Model

UPMEM DPUs are programmed in C using a custom SDK. The host CPU runs the show, and one job looks like this (the function names are from the UPMEM host library):

One UPMEM job: the host does all the loading and collecting; the DPUs only ever touch their own bank.

Steps 3 and 5 are where the time goes. Every byte in and out still crosses the ordinary memory bus, so a job only pays off when the DPUs do a lot of work per byte they're handed.

The programming model is similar to SPMD (Single Program, Multiple Data) - each DPU runs the same program but on different data. The critical challenge is data placement: data must be distributed so that each DPU's computation accesses only (or primarily) its own local 64 MB memory bank. Inter-DPU communication is not directly supported; data must be routed through the host CPU.

DPUs: Real Processors in DRAM

UPMEM's DPUs are real 32-bit processors capable of running general-purpose C code, including loops, conditionals, function calls, and pointer arithmetic. This makes UPMEM far more flexible than fixed-function PIM accelerators. However, each DPU is deliberately simple (no cache, no floating-point unit, limited memory) to minimise the area and power overhead of embedding a processor inside a DRAM chip.

Strengths and Limitations

Strengths:

  • Drop-in replacement for standard DIMMs - no new motherboards or interconnects required
  • General-purpose programmability in C
  • Massive parallelism (thousands of DPUs per server)
  • Each DPU has high bandwidth to its local memory bank (~800 MB/s per DPU, aggregate TB/s across all DPUs)

Limitations:

  • No floating-point hardware (must use software emulation or fixed-point)
  • Limited per-DPU memory (64 MB)
  • No direct DPU-to-DPU communication
  • Host-DPU data transfer is still through the standard memory bus
  • Programming requires explicit data partitioning

Samsung AIM (Accelerator-in-Memory)

Samsung's AIM (also called AxDIMM or CXL-PIM in various configurations) takes a different approach by placing an AI accelerator on a DIMM or CXL memory module. Unlike HBM-PIM (which is embedded within an HBM stack attached to a GPU), AIM targets the standard server memory channel.

AIM processors are more capable than UPMEM DPUs but less general-purpose - they are optimised for specific AI operations, particularly the embedding table lookups and reductions that dominate recommendation model inference (the kind of AI that powers ad ranking, content recommendations, and search results at companies like Meta and Google).

Why Recommendation Models Need PIM

Recommendation models are one of the largest consumers of data centre compute worldwide. A typical recommendation model has enormous embedding tables - lookup tables that can be hundreds of gigabytes in size. During inference, the model looks up dozens of entries from these tables and combines them. Each lookup is a random memory access to a different part of a massive table.

This workload is almost purely memory-bound: the computation (adding or averaging the looked-up vectors) is trivial, but the memory accesses are random and the table is too large for any cache. Samsung's AIM places compute capability directly on the memory module where the embedding tables reside, eliminating the round trip to the host processor.

SK Hynix AiM (Accelerator-in-Memory)

SK Hynix has developed its own PIM technology, also called AiM, based on GDDR6 memory. Their approach embeds multiply-accumulate (MAC) units within GDDR6 memory dies, targeting AI inference workloads. Each GDDR6 die contains processing elements that can perform matrix-vector multiplications using the bank-level parallelism inherent in GDDR6's architecture.

The SK Hynix design is particularly interesting because GDDR6 is the standard memory used in gaming GPUs. By adding PIM capability to GDDR6, SK Hynix envisions AI acceleration in consumer devices without requiring expensive HBM.

Comparing Approaches

ArchitectureProcessing ElementTarget WorkloadMemory TypeProgrammability
Samsung HBM-PIMSIMD FP16 unitsAI inferenceHBM2Limited (fixed operations)
UPMEM32-bit RISC DPUsGeneral-purposeDDR4 DIMMHigh (C programming)
Samsung AIMAI acceleratorRecommendation modelsDDR/CXLMedium (AI frameworks)
SK Hynix AiMMAC unitsAI inferenceGDDR6Limited (fixed operations)

The Role of CXL

Compute Express Link (CXL) is a new interconnect standard that is accelerating the development of near-memory computing. CXL provides a cache-coherent, high-bandwidth link between processors and memory/accelerator devices. CXL Type 2 devices can contain both memory and processing capability, making them natural homes for PIM accelerators.

CXL is important for near-memory computing because it provides a standardised way to attach intelligent memory devices to a server. Instead of requiring custom hardware integration (as HBM-PIM does with GPUs), CXL allows PIM devices to plug into standard server infrastructure.

Where a CXL Type 2 PIM device sits: beside the DDR channels, wired like a PCIe card.

The link carries three protocols. CXL.io handles discovery and setup, just like PCIe. CXL.cache lets the accelerator cache host memory coherently, and CXL.mem lets the host load and store into the device's memory as if it were ordinary RAM. That last pair is what makes Type 2 interesting for PIM: the host sees the device's memory in its own address space, and the PIM logic can compute on it without a separate copy step.

Key Takeaways

The near-memory computing landscape is diverse and rapidly evolving. The common thread across all approaches is the recognition that moving data is expensive and computing near data is more efficient. Different architectures make different trade-offs:

  • HBM-PIM offers high bandwidth for AI workloads but limited programmability
  • UPMEM offers general-purpose computing with massive parallelism but lacks floating-point
  • AIM/AiM targets specific AI workloads with optimised accelerators

In the next lesson, we go deeper into true processing-in-memory — computation performed inside the memory array itself — and the computational models it unlocks.