Heterogeneous Computing
Multi-Core & Coherence · CPU Architectures track
6 min read
In this module
Not All Cores Are Created Equal
The era of scaling performance through identical cores is over. What's replacing it is heterogeneous computing: systems that mix different types of processing elements, each tuned for specific workloads. From big.LITTLE on phones to GPUs and DPUs in data centres, the future is specialised.
big.LITTLE and Hybrid Architectures
ARM's big.LITTLE (2011) pioneered heterogeneous CPU design. A chip contains two types of cores: "big" cores (high-performance, power-hungry, wide out-of-order) and "LITTLE" cores (efficient, lower-performance, simpler pipelines). The OS scheduler assigns demanding tasks to big cores and background tasks to LITTLE cores.
Apple's A-series and M-series chips implement this as Performance (P-cores) and Efficiency (E-cores). The M1 has 4 high-performance Firestorm cores and 4 high-efficiency Icestorm cores. The P-cores are several times faster than E-cores for single-threaded work (how many times depends heavily on the workload) but consume much more power.
Intel adopted the same approach with Alder Lake (12th gen): P-cores (Golden Cove, hyperthreaded, wide OoO) and E-cores (Gracemont, no hyperthreading, efficient). Intel's Thread Director is a hardware monitoring unit that observes each thread's behaviour (IPC, memory access patterns, instruction mix) and provides hints to the OS scheduler about which core type is best suited for each thread.
Heterogeneous cores create scheduling complexity. A thread running a tight compute loop should be on a P-core. A thread mostly waiting on I/O should be on an E-core. A thread that transitions between compute and I/O needs to migrate. The scheduler must balance performance, power, and fairness. Thread migration between core types also has a cost: the thread's cache state is lost on the source core and must be rebuilt on the destination. Frequent migration can hurt performance more than running on a suboptimal core.
Data Processing Units (DPUs) and SmartNICs
DPUs (NVIDIA BlueField, AMD Pensando, Intel IPU) are programmable network cards with their own ARM or RISC-V cores, memory, and hardware accelerators. They offload infrastructure tasks from the host CPU:
- Network processing: packet filtering, routing, encryption/decryption, load balancing
- Storage virtualisation: NVMe-oF, iSCSI, compression, deduplication
- Security: firewalls, intrusion detection, DDoS mitigation
- Management: BMC functionality, remote system management
By moving these tasks to the DPU, the host CPU's cores are freed entirely for application workloads. In cloud environments, this is significant: hypervisor networking and storage I/O can consume 10-30% of host CPU capacity. Offloading to a DPU returns those cores to revenue-generating workloads.
DPUs run their own operating system (typically Linux) and can be programmed independently of the host. They represent a shift towards infrastructure processing as a separate compute domain with its own dedicated hardware.
Domain-Specific Accelerators
General-purpose CPUs are, by design, jacks of all trades. Domain-specific accelerators achieve much higher performance and efficiency for specific workloads:
GPUs (Graphics Processing Units): thousands of simple cores for data-parallel computation. 10-100x more efficient than CPUs for matrix operations, neural network inference, and physics simulations. Used extensively for AI/ML training and inference.
TPUs (Tensor Processing Units): Google's custom ASICs for neural network computation. Optimised for matrix multiply-accumulate operations, the core computation in deep learning. Achieve higher efficiency than GPUs for specific ML workloads.
FPGAs (Field-Programmable Gate Arrays): reconfigurable hardware that can implement custom digital circuits. Used for network packet processing (SmartNICs), financial trading (ultra-low latency), and prototyping custom accelerators. Reprogrammable, unlike ASICs.
Custom ASICs: purpose-built chips for specific algorithms. Bitcoin mining ASICs, video encoding chips, and neural network inference accelerators (Apple Neural Engine, Google Edge TPU) achieve the highest efficiency for their target workload but cannot be repurposed.
The System Architecture Shift
Modern servers increasingly resemble heterogeneous systems-on-chip rather than traditional CPU-centric architectures, where every core was equal and everything else hung off a bus somewhere. Here's the block layout of Apple's M1, the chip that made the idea mainstream on laptops:
Apple's M-series exemplifies this: CPU cores (P+E), GPU, Neural Engine, media engines, image signal processor, and Thunderbolt controllers on a single SoC with unified memory accessible to all components.
Sharing Memory Coherently
An accelerator is only as useful as the path its data takes to get there. The classic discrete GPU has its own memory on the far side of a PCIe link, so the programmer copies inputs over, launches the kernel and copies results back. The two memories aren't coherent: if the CPU changes a buffer after the copy, the GPU never finds out.
Extending cache coherence to accelerators removes those copies, and it's hard for the same reasons as on a multi-core chip, only worse. The link is slower and longer than an on-die interconnect, the GPU's caches are huge and organised differently, and snooping thousands of GPU threads' accesses would swamp the CPU. A few approaches have stuck:
- CXL (Compute Express Link) runs over the PCIe physical layer. CXL.cache lets a device cache host memory and stay coherent with the CPU's caches; CXL.mem lets the CPU reach memory attached to a device with ordinary loads and stores.
- NVLink-C2C joins NVIDIA's Grace CPU and Hopper GPU on one module with a coherent 900 GB/s link, so each can follow pointers into the other's memory.
- Unified memory on Apple's M-series skips the link altogether: CPU, GPU and Neural Engine share one physical pool of DRAM behind a shared system level cache.
For programmers, the change is that a pointer means the same thing on every processor. You pass the accelerator a data structure instead of serialising it into a buffer, and the explicit copy step, often the slowest part of offloading small jobs, disappears.
Software Implications
Heterogeneous computing demands software that can identify which workloads map to which accelerator, manage data movement between processing elements, handle different programming models (CUDA for GPUs, OpenCL for FPGAs, and so on), and degrade gracefully when a specific accelerator isn't present.
Frameworks like oneAPI (Intel), ROCm (AMD), and SYCL try to provide unified programming models across devices. In practice, squeezing maximum performance still requires device-specific work.
The trend over the last decade has been clear: more of each chip's area goes to specialised accelerators alongside the general-purpose cores. Multi-core programming is the floor. The ceiling is multi-architecture programming — knowing which processing element fits each part of the workload, and wiring them together.