Skip to main content
Be Bitwise

Multi-Core Architecture

Multi-Core & Coherence · CPU Architectures track

6 min read

In this module

Beyond Single-Core Performance

For decades, CPU performance climbed by cranking up clock speed and making individual cores more sophisticated. Around 2004, power consumption and heat hit a wall — literally — and clock frequencies plateaued. The industry's answer was multi-core: more cores instead of faster cores. Understanding that architecture is table stakes for writing software that actually scales.

Symmetric Multiprocessing (SMP)

In a symmetric multiprocessing (SMP) system, all cores are identical and have equal access to memory and I/O. Any core can run any thread, and the OS scheduler distributes work across cores dynamically. Most desktop and laptop computers are SMP systems.

SMP simplifies software: a multi-threaded program works the same regardless of which cores its threads are assigned to. The OS handles affinity (which thread runs on which core) transparently, though programmers can pin threads to specific cores when needed for performance.

Core Complexes

Modern multi-core designs organise cores into core complexes (CCXs on AMD, clusters on ARM). A core complex is a group of cores that share a local L3 cache and communicate through a fast internal interconnect.

AMD's Zen architecture groups cores into CCXs, which sit on a chiplet called a CCD (Core Complex Die). On Zen 1 and Zen 2 a CCX holds 4 cores, two to a CCD; from Zen 3 onwards one 8-core CCX fills the whole CCD and shares a 32MB L3 cache. The cores within a CCX share their L3 and communicate with low latency. Communication between CCXs crosses the Infinity Fabric interconnect, adding latency.

This hierarchical design means that threads sharing data perform best when they run on cores in the same CCX (accessing the shared L3 cache) rather than across CCXs (requiring inter-CCX communication).

Interconnects

Cores must communicate with each other, with the shared cache, with the memory controller, and with I/O devices. The interconnect provides these communication paths.

Ring bus: cores are connected in a circle. Data packets travel around the ring to reach their destination. Simple and effective for small core counts (up to ~10), but latency grows linearly with the number of cores on the ring.

Mesh interconnect: cores are arranged in a 2D grid with row and column connections. A packet can take the shortest path between any two cores. Scales better than a ring because the maximum distance grows as the square root of core count rather than linearly. Intel uses mesh interconnects in Xeon Scalable processors (Skylake-SP and later).

Chiplets and Die-to-Die Communication

Rather than building one monolithic chip with all cores, modern processors use chiplets - multiple smaller dies connected on a package. AMD's EPYC processors use up to 12 CCD chiplets in the Zen 4 generation and up to 16 in Zen 5 (8 cores each on the standard parts), all connected to a central I/O die. This improves manufacturing yields (smaller dies have fewer defects) and allows mixing different process nodes (compute chiplets on cutting-edge 5nm, I/O die on mature 6nm). The die-to-die interconnect (AMD Infinity Fabric, Intel EMIB) adds some latency compared to a monolithic design, but the manufacturing and cost benefits outweigh this.

Hop counts make the scaling difference concrete. Take the worst case for a core-to-core transfer, assuming a ring that can send in both directions and a mesh laid out as a near-square grid (real chips differ in the details):

CoresRing (worst case)Mesh (worst case)
84 hops4 hops (2 × 4 grid)
3216 hops10 hops (4 × 8 grid)

Chiplets change the question again. Inside one chiplet, cores share an L3 and talk cheaply. Between chiplets, every transfer leaves through the die-to-die link, crosses the I/O die and comes back in, however close the two cores sit physically:

That's why core-to-core latency on a chiplet CPU comes in two clear tiers, and why schedulers try to keep threads that share data on the same chiplet.

Cache Hierarchy in Multi-Core

The cache hierarchy becomes more complex with multiple cores:

  • L1 cache: private per core. Each core has its own L1I and L1D. Fastest access (~4 cycles).
  • L2 cache: typically private per core (Intel's big cores, every AMD Zen generation). Some designs share L2 between pairs of cores. ~12 cycles.
  • L3 cache: shared among cores in a CCX or across the entire chip. Distributed across multiple slices (one per core), connected by the interconnect. ~30-40 cycles for a local slice, more for remote slices.

When Core A reads data that Core B recently wrote, the data may be in Core B's L1 or L2 cache. The cache coherence protocol (covered in the next lesson) ensures Core A gets the correct, up-to-date value. This inter-core data transfer typically takes 40-80 cycles - much faster than going to main memory (~200 cycles) but much slower than a local cache hit (~4 cycles).

Scaling Challenges

Adding more cores does not proportionally increase performance. Amdahl's Law states that the speedup from parallelisation is limited by the sequential portion of the program. If 10% of a program is inherently sequential, the maximum speedup from infinite cores is 10x.

Before running the calculator below, predict the speedup for that 10%-serial program on 8 cores and on 64. The second column adds a small cost per extra core for locks and coherence traffic; guess where it peaks.

Try itPredict the 8-core and 64-core speedups, then change serial and cost to model your own program.
C (GCC 9.2.0)

Eight cores buy less than 5x, and 64 cores less than 9x. With even a tiny per-core cost the curve turns over: past a point, extra cores make the program slower.

Beyond Amdahl's Law, practical scaling is limited by:

  • Synchronisation overhead: locks, barriers, and atomic operations become more expensive as more cores contend for them
  • Cache coherence traffic: more cores sharing data means more invalidation messages on the interconnect
  • Memory bandwidth: all cores share the same memory controllers, and total bandwidth is finite
  • NUMA effects: in multi-socket systems, accessing remote memory is slower than local memory

Software must be designed for scalability from the start: minimise shared mutable state, use lock-free algorithms where contention matters, partition data to avoid false sharing, and be NUMA-aware on large machines. Throwing more cores at a poorly designed parallel program doesn't help — it often makes things worse.