Skip to main content
Be Bitwise

GPU Computing Basics

SIMD & Vector Processing · CPU Architectures track

6 min read

In this module

Massively Parallel Processors

CPU SIMD chews through 8-16 elements at a clip. A GPU works on thousands, sometimes millions, at once. The trick is brute-force parallelism — thousands of dumb little cores all running the same program against different data. Knowing roughly how a GPU is wired together is what tells you when offloading work from the CPU is worth the bother and when it's a waste.

GPU Architecture

A GPU is built around Streaming Multiprocessors (SMs on NVIDIA, Compute Units on AMD). Each SM packs together:

  • Multiple CUDA cores (32-128 per SM), each capable of one floating-point or integer operation per cycle
  • Warp schedulers that manage groups of threads
  • Shared memory (fast, programmer-managed on-chip scratchpad, carved from the same SRAM as the L1 cache)
  • Register file (much larger than CPU - tens of thousands of registers per SM)
  • Special function units (SFUs) for transcendental maths (sin, cos, sqrt)

An NVIDIA A100, to put numbers on it, packs 108 SMs at 64 FP32 cores apiece — 6,912 FP32 lanes total. Stack that against a typical server CPU's 8-64 general-purpose cores and the gap speaks for itself.

The CUDA Programming Model

CUDA (Compute Unified Device Architecture) is NVIDIA's GPU programming model. A CUDA program is built around kernels: functions that run on the GPU and get invoked once per thread. Same kernel, different threadIdx, different data.

c
// Kernel: runs on the GPU, one invocation per thread
__global__ void vector_add(float *a, float *b, float *c, int n) {
int i = blockIdx.x * blockDim.x + threadIdx.x;
if (i < n) {
c[i] = a[i] + b[i];
}
}
// Host code: launches the kernel
int main() {
// Allocate GPU memory, copy data...
vector_add<<<numBlocks, threadsPerBlock>>>(d_a, d_b, d_c, N);
// Copy results back...
}

Threads are organised into a hierarchy:

  • Thread: one execution instance of the kernel
  • Block: a group of threads (up to 1024) that execute on the same SM and can share data
  • Grid: all blocks for one kernel launch
Warps and SIMT

Inside a block, threads are scheduled in groups of 32 called warps (NVIDIA) or wavefronts on AMD (64 threads on GCN and CDNA, 32 or 64 on RDNA). Broadly, every active thread in a warp executes the same instruction at the same time, which is the SIMT model: Single Instruction, Multiple Threads. When threads in a warp diverge at a branch — some go down the if, others down the else — the warp serialises both paths, masking the inactive threads on each pass. That's why divergent branching is so expensive on GPUs: you pay for the whole warp's worth of work even when half the lanes are idle. (Since Volta, NVIDIA gives each thread its own program counter, so a warp's threads aren't guaranteed to stay in strict lockstep; code that relies on them being in sync needs __syncwarp().)

Memory Hierarchy

GPU memory is organised differently from CPU memory:

  • Registers: fastest, private to each thread. GPUs have enormous register files.
  • Shared memory: ~48-164 KB per SM, shared among all threads in a block. User-managed (not a cache). Used for inter-thread communication within a block.
  • L1 cache: small, per-SM. On some architectures, configurable split between L1 and shared memory.
  • L2 cache: shared among all SMs, typically 4-40 MB.
  • Global memory (HBM/GDDR): large (16-80 GB), high bandwidth (1-3 TB/s), but high latency (~300-500 cycles).

GPU performance comes down to never letting cores go idle. They hide memory latency through occupancy — keeping thousands of threads alive so the warp scheduler always has another ready warp to swap in the moment one stalls on a load.

When GPU Beats CPU

GPUs excel when:

  • The problem is data-parallel: the same operation on millions of elements
  • Arithmetic intensity is high: many operations per byte of data transferred
  • Branching is minimal: all threads follow the same path
  • The data set is large enough to amortize the overhead of CPU-to-GPU data transfer

GPUs are poor at:

  • Sequential algorithms: one-thread-at-a-time work
  • Irregular data access: pointer chasing, graph traversal (cache misses destroy throughput)
  • Small data sets: the overhead of launching a kernel and transferring data exceeds the computation
  • Heavy branching: warp divergence serializes execution

OpenCL and Compute Shaders

OpenCL is the cross-platform answer to CUDA, running on GPUs from every major vendor (NVIDIA, AMD, Intel) and reaching CPUs and FPGAs too. The concepts map across — work-items instead of threads, work-groups instead of blocks, NDRange instead of grid — but the abstractions are familiar once you've squinted at the renaming.

Compute shaders (Vulkan, DirectX, Metal) bolt GPU compute onto graphics APIs and have become the go-to for non-graphics workloads on mobile, where CUDA simply isn't available. Underneath the marketing the model is the same: dispatch a grid of thread groups, each running the same shader against different inputs.

Practical Considerations

Before you start porting things to the GPU, weigh up a few things. First, the data transfer cost: PCIe round-trips aren't free, and small workloads can spend more time moving data than computing on it. Second, your problem needs to be big enough to actually saturate thousands of cores. Third, profile the kernel for warp divergence and memory coalescing — neighbouring threads should be reading neighbouring addresses or you'll bleed memory bandwidth (more on that below). And fourth, ask honestly whether your algorithm even fits SIMT in the first place. Some workloads see 10-100x speedups; others come out flat or worse, dragged down by transfer overhead and parallelism that wasn't really there.

Here's what coalescing means for one warp of 32 threads, each loading a 4-byte float. Modern NVIDIA GPUs fetch global memory in 32-byte sectors, and the memory system merges the warp's requests before it fetches anything:

Same instruction, same amount of useful data, eight times the memory traffic. Laying data out as a structure of arrays, rather than an array of structures, is usually how you get from the second picture to the first.