Skip to main content
Be Bitwise

Processing-in-Memory Concepts

Near-Memory Computing & Processing-in-Memory · Frontiers track

9 min read

In this module

Computing Where Data Lives

The previous lesson explored architectures that place processing elements near memory — on an HBM logic die, or on a DIMM module. Those still keep a gap between the memory array and the processing logic. In this lesson we go further: processing-in-memory (PIM), where computation happens inside or immediately adjacent to the array itself.

True PIM dissolves the boundary between storage and computation. Data never leaves the array — computation happens in place. That's data movement eliminated at the most fundamental level possible.

DRAM-Based PIM Operations

Standard DRAM already performs certain operations during normal memory access. When a row is activated (opened), the entire row of data (typically 8 KB) is copied from the memory cells into the row buffer (also called the sense amplifier array). This is effectively a parallel copy operation across thousands of bits simultaneously.

Researchers have discovered that by carefully manipulating the timing and sequencing of DRAM commands, they can perform bulk bitwise operations using the row buffer and the electrical properties of the memory cells. The key techniques include:

Triple Row Activation (Ambit)

The Ambit architecture (proposed by researchers at Carnegie Mellon University) exploits the analogue behaviour of DRAM sense amplifiers to perform AND and OR operations on entire rows simultaneously. The technique works by activating three rows at the same time:

  1. Two rows contain the operands
  2. A third row holds a control value, all zeros or all ones
  3. The sense amplifiers resolve the combined charge, effectively computing a majority function

By controlling which rows are pre-charged and how the sense amplifiers settle, Ambit can compute AND, OR, and NOT on entire rows (thousands of bits) in a handful of row operations, without any data leaving the DRAM chip.

There's a catch hiding in step 3: a triple-row activation overwrites all three rows with the answer. Ambit can't do that to your operands, so the real AND runs like this:

Bulk AND in Ambit. Swap the all-zeros control row for all-ones and the same majority gives OR.

Why does majority give AND? With the third input pinned to 0, a column only reads 1 when both A and B are 1. Pin it to 1 and one more 1 from either operand is enough, which is OR. Every copy in the diagram is a RowClone inside the subarray, so the bits never get near the memory bus.

The speedup is dramatic: a bulk AND across an 8 KB row costs a few row activations, versus hundreds of CPU cycles to read, process, and write back the same data.

Bulk Bitwise PIM

DRAM-based PIM architectures like Ambit can perform bitwise AND, OR, and NOT on entire memory rows (thousands of bits) in a single memory operation. Since these are the fundamental building blocks of digital logic, any Boolean function can be computed in-memory by composing these operations. The key insight is that DRAM sense amplifiers already perform analogue computation during normal operation - PIM simply repurposes this capability for general computation.

RowClone: In-Memory Copy

RowClone is a simpler PIM operation that copies an entire row of data from one location to another within the same DRAM subarray. Normally, copying a row requires reading it out to the processor and writing it back - consuming memory bus bandwidth. RowClone does it internally by:

  1. Activating the source row (loading it into the row buffer)
  2. Activating the destination row while the row buffer still holds the source data
  3. The source data overwrites the destination row

This is 11x faster and uses 74x less energy than a conventional memory copy operation via the processor, according to the original research paper.

Compute-Capable Memory Cells

Beyond exploiting DRAM's existing analogue behaviour, researchers have proposed memory cells that are inherently capable of computation:

Memristive Computing

Memristors (memory resistors) are a type of non-volatile memory element whose resistance depends on the history of current that has flowed through it. When organised in a crossbar array, memristors can perform matrix-vector multiplication using Ohm's law and Kirchhoff's current law - the fundamental physics of electrical circuits.

The basic principle:

  1. Store matrix values as the conductance (1/resistance) of memristors in a crossbar
  2. Apply input voltages to the rows (representing the input vector)
  3. Each memristor multiplies the input voltage by its conductance (V × G = I, Ohm's law)
  4. Currents from all memristors in a column sum together (Kirchhoff's law)
  5. The resulting column currents represent the output of the matrix-vector multiplication

This computation happens in a single step, in O(1) time, regardless of the matrix dimensions. Compare this to O(n²) operations for a conventional digital matrix-vector multiply. We will explore this in much greater detail in the module on analogue and in-memory computing.

Content-Addressable Memory (CAM)

Content-addressable memory is a special type of memory that searches its entire contents in parallel. Instead of providing an address and getting back data (like normal memory), you provide data and get back the address where it is stored - or whether it exists at all.

Ternary CAM (TCAM) supports wildcard matching and is used in network routers for high-speed packet classification. TCAM is a form of processing-in-memory because the search operation is performed entirely within the memory array in a single clock cycle, regardless of the size of the stored dataset.

PIM Operations Taxonomy

PIM operations can be categorised by their complexity:

Simple Operations (Row-Parallel)

These operations can be performed on entire rows simultaneously using basic memory operations:

  • Copy: Duplicate a row (RowClone)
  • Bitwise logic: AND, OR, XOR, NOT on entire rows (Ambit)
  • Initialisation: Set entire rows to all-zeros or all-ones
  • Compare: Bitwise comparison of two rows

Arithmetic Operations

More complex operations that require additional logic but can still be performed in or very near the memory:

  • Addition/subtraction: Can be built from bitwise operations with carry chains
  • Multiplication: Requires sequences of shift-and-add operations
  • MAC (multiply-accumulate): Critical for AI workloads

Data-Intensive Operations

Higher-level operations that are natural fits for PIM because they process large amounts of data with simple logic:

  • Search/scan: Find elements matching a criterion across the entire memory
  • Filter: Select rows from a database table that match a predicate
  • Aggregation: Sum, count, max, min across a data column
  • Histogram: Count occurrences of each value in a dataset
The PIM Sweet Spot

PIM is most effective for operations that are simple but data-intensive - where the computation per data element is low but the total amount of data is enormous. Database scans, bitwise analytics, vector operations, and bulk data transformations are ideal. Complex algorithms with irregular data dependencies, branching, and inter-element communication are poor fits for PIM and are better left to conventional processors.

Programming Models for PIM

Programming PIM systems is fundamentally different from programming conventional processors. Several programming models have been proposed:

Host-Orchestrated Model

The most common model today (used by UPMEM, HBM-PIM):

  1. The host CPU partitions data across PIM units
  2. The host loads programs into PIM units
  3. PIM units execute independently on their local data
  4. The host collects and combines results

This is conceptually similar to MapReduce: the host "maps" work to PIM units and "reduces" their results. The programmer must explicitly manage data placement and partitioning.

Transparent PIM

In a transparent model, the hardware automatically identifies operations that can be offloaded to PIM and executes them there. The programmer writes normal code, and the compiler or runtime decides what runs on the CPU versus PIM.

This is the holy grail of PIM programming but is extremely difficult to implement. Challenges include:

  • Determining when PIM is beneficial (not all operations benefit)
  • Managing cache coherence between the CPU and PIM units
  • Handling virtual memory and address translation within PIM

Domain-Specific Interfaces

For specific workloads, domain-specific interfaces hide PIM complexity behind high-level APIs:

  • SQL operators for database PIM (push filter and aggregate operations to memory)
  • Tensor operations for AI PIM (offload element-wise operations and reductions)
  • Graph operators for graph analytics PIM (neighbour traversal, property aggregation)

Coherence and Consistency Challenges

One of the most significant challenges in PIM is maintaining coherence between data that the CPU might have cached and data that PIM units are modifying directly in memory.

In a conventional system, the cache hierarchy ensures that all processors see a consistent view of memory. But PIM units operate below the cache hierarchy - they modify data in DRAM directly. If the CPU has a cached copy of that data, it becomes stale without the CPU knowing.

Several solutions have been proposed:

  • Flush before PIM: The CPU flushes all relevant cache lines before invoking PIM operations, ensuring PIM sees up-to-date data. After PIM completes, the CPU invalidates its caches.
  • Non-cacheable PIM regions: Mark memory regions used by PIM as non-cacheable, so the CPU always accesses DRAM for these regions.
  • Coherent PIM: Integrate PIM units into the cache coherence protocol. This is the most flexible but also the most complex approach.

Virtual Memory and Address Translation

PIM units operate on physical memory addresses, but programs use virtual addresses. This creates a challenge: how does a PIM unit know where to find data when programs specify virtual addresses?

Approaches include:

  • Host-managed translation: The host CPU translates virtual to physical addresses before dispatching PIM operations
  • PIM-side TLB: Equip PIM units with their own translation lookaside buffers (TLBs) for address translation
  • Contiguous physical allocation: Allocate PIM data in physically contiguous regions, simplifying translation

Each approach trades off complexity, performance, and flexibility. Most current PIM systems use the simplest approach (host-managed translation) because adding TLBs to PIM units significantly increases their area and complexity.

The Compute-Memory Trade-off

PIM represents a fundamental trade-off in computer architecture: by placing processing elements inside or near memory, you gain enormous bandwidth and energy efficiency, but the processing elements must be simple (because memory process technology is optimised for density, not logic speed) and programming becomes more constrained.

The PIM sweet spot is massive data parallelism with simple per-element operations and minimal communication between elements. Next lesson, we look at how these concepts translate into real deployments across AI, databases, and genomics.