Introduction to SIMD
SIMD & Vector Processing · CPU Architectures track
5 min read
In this module
Data Parallelism
A huge chunk of real workloads boil down to applying the same operation to every element of a large data set. Adding two arrays element by element, multiplying a matrix, brightening every pixel in an image — all data-parallel. SIMD (Single Instruction, Multiple Data) cashes in on that regularity by chewing through several data elements per instruction.
How SIMD Works
A scalar add instruction processes one pair of values:
ADD: a + b = c (1 operation per instruction)A SIMD add instruction processes multiple pairs simultaneously:
VADDPS: [a0,a1,a2,a3,a4,a5,a6,a7] + [b0,b1,b2,b3,b4,b5,b6,b7] = [c0,c1,c2,c3,c4,c5,c6,c7] (8 operations per instruction)The CPU exposes vector registers: extra-wide registers built to hold multiple data elements at once. A 256-bit register can pack 8 x 32-bit floats, 4 x 64-bit doubles, 32 x 8-bit integers, or other layouts depending on the instruction. Each SIMD instruction then chews on all the lanes in parallel.
Below are two 8-lane float vectors in a 256-bit register. Switch between add, multiply and compare and blend, then flip the width to 128 or 512 bits and watch the instruction names and the loop count change. Try setting N to 100: AVX needs 12 vector iterations plus a 4-element scalar tail. And look at lane 4, where 100000000 + 1 is still 100000000 in single precision.
SIMD lanes
One instruction applies the same operation to every lane of two 32-bit float vectors. Results are rounded to single precision, as the hardware does.
8 lanes × 32 bits = 256 bits, lane 0 on the left
Instructions
AVX
a in ymm1, b in ymm2
| vaddps ymm0, ymm1, ymm2 | ; ymm0 = a + b |
NEON registers are 128 bits, so Arm has no 256-bit NEON form. SVE covers wider vectors.
Loop over an array
- Scalar loop
- 100 iterations
- Vector loop
- 12 iterations of 8 lanes: 12 vector adds
- Scalar tail
- 4 iterations (100 mod 8)
- Total
- 16 loop iterations instead of 100
Counts leave out loads, stores and loop overhead.
x86 SIMD Extensions
x86 has accumulated SIMD extensions like sediment over the decades:
MMX (1997): 64-bit registers (MM0-MM7, aliased to the x87 FP stack), integer only. Obsolete.
SSE (1999): 128-bit registers (XMM0-XMM7, extended to XMM0-XMM15 in x86-64). Floating-point and integer operations. SSE2, SSE3, SSSE3, SSE4.1, SSE4.2 added more operations.
AVX (2011): 256-bit registers (YMM0-YMM15). Three-operand instruction format (destination can differ from both sources). VEX encoding prefix.
AVX-512 (2016+): 512-bit registers (ZMM0-ZMM31), 8 opmask registers for predicated operations, and many new instruction types. Not universally available - varies by CPU model.
Register hierarchy:ZMM0 (512 bits) = [YMM0 (256 bits) | upper 256 bits]YMM0 (256 bits) = [XMM0 (128 bits) | upper 128 bits]Not every CPU lights up every extension. Use the CPUID instruction to ask the chip directly. In C that means __builtin_cpu_supports("avx2") or reading the raw CPUID flags. At compile time, -mavx2 tells the compiler to emit AVX2 code. Production software that has to ship to a diverse install base usually compiles several code paths and picks the best one at startup via runtime feature detection.
ARM SIMD: NEON and SVE
ARM took a different route:
NEON (Advanced SIMD): 128-bit registers (Q0-Q31 on AArch64), supporting integer (8/16/32/64-bit) and floating-point (16/32/64-bit) operations. NEON is ubiquitous on modern ARM chips - all ARMv8-A processors include it.
SVE (Scalable Vector Extension): where x86 nails vector widths down at the ISA level, SVE leaves them open. Vector lengths range anywhere from 128 to 2048 bits, with the hardware implementer picking a width. The same SVE binary runs on any compliant chip regardless of how wide its vectors actually are. SVE2 layers on extra operations aimed at general-purpose code.
That length-agnostic design pays off in deployment: ship one binary, let it auto-scale to whatever the silicon underneath supports. Apple's M-series chips run 128-bit NEON; Fujitsu's A64FX (the chip behind the Fugaku supercomputer) runs 512-bit SVE. Same source, different vector widths.
Auto-Vectorisation
Modern compilers will happily transform scalar loops into SIMD code on your behalf:
// Scalar code:for (int i = 0; i < N; i++) c[i] = a[i] + b[i];
// Compiler auto-vectorizes to something like:int i = 0;for (; i + 8 <= N; i += 8) { __m256 va = _mm256_loadu_ps(&a[i]); // unaligned loads: no alignment is promised __m256 vb = _mm256_loadu_ps(&b[i]); __m256 vc = _mm256_add_ps(va, vb); _mm256_storeu_ps(&c[i], vc);}for (; i < N; i++) // scalar tail for the last N % 8 elements c[i] = a[i] + b[i];Auto-vectorisation thrives on simple loops with no cross-iteration dependencies. It chokes on data-dependent conditionals, pointer aliasing (slap a restrict on offending pointers), non-contiguous memory access, and floating-point reductions like summing every element. Integer sums vectorise happily, but FP addition isn't associative, so splitting a float sum across lanes changes the rounding and the compiler won't do it unless you allow it with -ffast-math (or the narrower -fassociative-math). Knowing where the compiler reaches its limits is half the battle of writing vector-friendly code.
The flags you'll want close to hand: GCC vectorises fully at -O3 or with the explicit -ftree-vectorize. Since GCC 12, -O2 vectorises too, but with a "very cheap" cost model that skips loops needing runtime checks or a scalar tail, so an integer sum over an unknown length typically only vectorises at -O3. -fopt-info-vec-missed tells you exactly which loops GCC gave up on and why. Clang's equivalents are -Rpass=loop-vectorize and -Rpass-missed=loop-vectorize.
When to Use SIMD
SIMD pays off hardest on element-wise array work, image processing (per-pixel kernels), audio (per-sample DSP), bulk string operations like searching or comparing, and scientific number-crunching where the inner loops are matrix operations or simulation steps.
It struggles when the work isn't uniform: pointer-chasing through linked lists, heavily branching code where every lane wants to do something different, and irregular access patterns that defeat contiguous loads.
The theoretical ceiling tracks vector width: 4x for 128-bit SSE on 32-bit floats, 8x for 256-bit AVX, 16x for 512-bit AVX-512. Real-world numbers usually land in the 2-6x range once you factor in load/store overhead, shuffles, and the awkward tail when N isn't a clean multiple of the vector width.