Skip to main content
Be Bitwise

PCIe Architecture

Bus Architectures & Interconnects · Hardware Foundations track

10 min read

In this module

PCIe: A Layered Serial Protocol

Unlike the shared parallel bus of the original PCI, PCI Express is a point-to-point, packet-switched, layered serial interconnect. Every device gets a dedicated link — straight to the root complex, or to a switch — and communication moves in packets rather than shared bus cycles.

PCIe's architecture splits into three layers, each owning its own job. Knowing the layers pays off in hardware programming, driver work, and any security review of a peripheral interface.

The Three-Layer Stack

The Physical Layer

The physical layer at the bottom does the electrical work. Each lane is two differential pairs — one for transmitting, one for receiving. Data flows both directions at once, so PCIe is full-duplex by design.

c
One PCIe Lane:
Tx+ ──────────────────────> Rx+ (device A to device B)
Tx- ──────────────────────> Rx-
Rx+ <────────────────────── Tx+ (device B to device A)
Rx- <────────────────────── Tx-
Total: 4 wires per lane (2 differential pairs)

The physical layer also runs link training — the dance two connected devices do to settle on lane width, link speed, and lane polarity when they first link up or after a reset. The whole thing is automatic. Software never sees it.

The data link layer ensures reliable delivery. It adds two fields to every TLP on its way out:

  • A sequence number so the receiver can detect missing or out-of-order packets
  • A 32-bit LCRC (Link CRC) so the receiver can detect bit errors

The transmitter keeps a copy of each TLP in a replay buffer until the receiver ACKs it; a NAK (or a timeout) triggers a resend. Those ACKs and NAKs travel in Data Link Layer Packets (DLLPs), small packets of the link layer's own that also carry flow-control credit updates and power-management handshakes. DLLPs never wrap TLPs; they run alongside them.

PCIe is therefore a reliable transport at the link level. Every packet either arrives intact or gets retransmitted, with software none the wiser.

The Transaction Layer

The transaction layer is where meaningful work happens. This is the layer software actually talks to. It produces and consumes Transaction Layer Packets (TLPs).

Transaction Layer Packets (TLPs)

TLPs are the fundamental currency of PCIe. Each one carries a header that names the transaction type, address, length, and requester/completer IDs.

TLP Types

PCIe defines four main categories of TLP: memory transactions (read and write to memory-mapped addresses), I/O transactions (legacy port-based I/O), configuration transactions (access to a device's config space), and messages (interrupts, error signalling, power management). Memory transactions dominate in modern systems, as almost all device communication uses memory-mapped I/O (MMIO).

Memory Read and Write

Memory reads and writes dominate the TLP mix. Whenever the CPU touches a device's BAR (Base Address Register) region, the chipset emits a memory TLP.

Memory Write: a posted transaction. The requester sends the TLP with data and walks away. Faster, but the sender never finds out whether the write was actually processed.

Memory Read: a non-posted (split) transaction. The requester sends a read request TLP. The completer responds with a completion TLP carrying the data. The two-phase shape is unavoidable — the data has to come back across the link.

c
Memory Write (posted - no response):
Requester ──── MWr TLP (header + data) ────> Completer
Memory Read (split transaction):
Requester ──── MRd TLP (header only) ────> Completer
Requester <─── CplD TLP (header + data) ──── Completer

The visualiser below steps through those TLPs between a GPU and the root complex, with the GPU as requester this time, plus the MSI write from the next section. Press play, then compare the MRd and CplD steps: the tag and requester ID in the completion are how the GPU matches it to the read it sent. The MWr step has no partner at all.

Bus Transaction Flow

Endpoint
GPU
TLP: MRd
Memory Read Request
Root Complex
CPU

The GPU issues a Memory Read request TLP with a 64-bit address, requester ID, tag, and byte count. The TLP traverses the PCIe switch hierarchy toward the root complex.

Configuration Transactions

Every PCIe device exposes a configuration space of up to 4096 bytes — extended from PCI's original 256. The OS uses configuration read/write TLPs to enumerate devices, hand out resources, and toggle features. The space holds vendor ID, device ID, BARs, capability structures, and control registers.

Message TLPs

Messages handle signalling that has no business in memory or I/O space. The big ones:

  • MSI/MSI-X (Message Signalled Interrupts): interrupts delivered as memory writes rather than dedicated interrupt lines
  • Error messages: correctable, non-fatal, and fatal error reporting
  • Power management: messages for transitioning between power states

Lane Configurations

PCIe links come in 1, 2, 4, 8, 12, 16, or 32 lane configurations — written as x1, x2, x4, x8, x12, x16, x32. The physical layer stripes data across whatever lanes the link has, byte-level round-robin.

ConfigurationTypical Use
x1Network adapters, sound cards, basic SSDs
x4NVMe SSDs, RAID controllers
x8High-end network adapters, some GPUs
x16Discrete GPUs, high-performance accelerators

A device can sit in a bigger slot than it needs and just negotiate fewer lanes. A x4 NVMe card slots happily into a x16 slot, using 4 of the 16 lanes and ignoring the rest.

PCIe Generations

Each PCIe generation roughly doubles the per-lane data rate compared to the previous one:

GenerationTransfer RateEncodingEffective BW per Lane (each direction)
Gen 1 (2003)2.5 GT/s8b/10b~250 MB/s
Gen 2 (2007)5.0 GT/s8b/10b~500 MB/s
Gen 3 (2010)8.0 GT/s128b/130b~985 MB/s
Gen 4 (2017)16.0 GT/s128b/130b~1969 MB/s
Gen 5 (2019)32.0 GT/s128b/130b~3938 MB/s
Gen 6 (2022)64.0 GT/s1b/1b (PAM4, FLIT mode with FEC)~7563 MB/s
Gen 7 (2025)128.0 GT/s1b/1b (PAM4, FLIT mode with FEC)~15125 MB/s
Key Concept
Encoding Overhead

Early PCIe generations used 8b/10b encoding, which maps every 8 data bits to a 10-bit symbol for DC balance and clock recovery. This imposes a 20% overhead. Gen 3 and later switched to 128b/130b encoding, reducing overhead to about 1.5%. Gen 6 introduces PAM4 signalling (4 voltage levels per symbol, encoding 2 bits), effectively doubling the data rate without doubling the frequency.

The Gen 7 figure is for the specification, published in June 2025; shipping devices lag the spec by a few years. Gen 6 and 7 numbers are approximate because FLIT mode replaces line encoding with fixed-size packets that carry their own CRC and FEC overhead.

The same physical layer also carries CXL (Compute Express Link), which runs over PCIe 5.0 and later and adds cache-coherent memory and accelerator protocols alongside ordinary PCIe traffic. It's how servers attach memory expanders and share memory with accelerators.

Watch the units: GT/s (gigatransfers per second) is the raw symbol rate. Effective data bandwidth ends up lower because encoding eats some.

PCIe Topology

PCIe arranges itself as a tree, rooted at the Root Complex — the bridge between the CPU (and its memory controller) and the PCIe fabric.

Root Complex

The root complex straddles the CPU/memory domain and the PCIe domain. During enumeration it issues configuration transactions; in steady state it routes TLPs by address and delivers interrupts. On modern processors it lives on the CPU die or SoC.

Switches

A PCIe switch fans multiple downstream devices off one upstream port. Inside, every port has a virtual PCI-to-PCI bridge. The switch routes TLPs between ports by address range, and where the topology supports it, devices can talk peer-to-peer without bouncing through the root complex.

Endpoints

Endpoints are the leaves: GPUs, NVMe controllers, NICs, USB host controllers. Each endpoint hosts one or more functions, and every function brings its own configuration space, BARs, and capability list.

Flow Control: Credits

PCIe uses credit-based flow control to keep buffers from overflowing. A transmitter cannot send a TLP without enough credits for the receiver's buffer. The receiver advertises its available credits; the transmitter decrements them as it sends. As the receiver drains TLPs and frees buffer space, it sends credit updates back.

The result: no data loss, no stop-and-wait round-trips. The transaction layer keeps multiple TLPs in flight simultaneously and the link runs near full utilisation.

PCIe Enumeration

At boot, firmware (BIOS/UEFI) or the OS walks the PCIe tree by firing configuration read TLPs at every possible address to discover devices. For each bus/device/function, it grabs vendor and device IDs, hands out memory and I/O ranges via the BARs, and wires up interrupt routing. The logical model has been unchanged since PCI, which is why the PCI-to-PCIe transition was almost a non-event for software.

"Hands out memory ranges via the BARs" hides a neat trick. Software doesn't know how big a device's BAR is, so it writes all ones and reads the register back: the device hard-wires the address bits below its size to zero, and the size falls out of the result. Here's that dance against a simulated device with three BARs:

Try itLook at the gap the 256 MB BAR forces, then reorder the BARs largest-first and see how much address space you get back.
C (GCC 9.2.0)
bash
# On Linux, inspect the PCIe device tree
lspci -tv
# Show detailed configuration space for a device
lspci -vvv -s 01:00.0

Here's the part of lspci -vvv you'll read most often, from a desktop GPU. What's the link doing, and how much bandwidth has the card actually got?

text
LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L1, Exit Latency L1 <4us
LnkSta: Speed 8GT/s (downgraded), Width x8 (downgraded)
Show the answer

The card can do Gen 4 x16 (LnkCap) but trained at Gen 3 x8 (LnkSta). Using the table above, that's about 985 MB/s × 8 ≈ 7.9 GB/s each way instead of roughly 31.5 GB/s. Common causes: the slot is wired x8 electrically, or shares lanes with an M.2 slot that's populated; the CPU or board only supports Gen 3; or a riser cable is forcing a lower speed. One harmless case to rule out first: many GPUs drop link speed when idle to save power, so check again under load.

Why It Matters

PCIe is the spine of modern PC and server I/O. Knowing the layered architecture helps you reason about driver behaviour, debug weird hardware issues, and assess security boundaries. PCIe devices can issue memory reads and writes directly into system memory via DMA, which makes the PCIe topology a load-bearing part of the system's security model. The lesson on SoC interconnects and DMA picks that thread up.