[Updated January 23, 2024] The PCI Express® 6.0 (PCIe® 6.0) specification was released by PCI-SIG® in January 2022. This new generation of the ubiquitous PCIe standard brought with it many exciting new features designed to boost performance for compute-intensive workloads including data center, AI/ML and HPC applications. PCIe 6.0 has now evolved to version 6.1 of the standard.
Find out all about PCIe 6.1 in the article below.
Contents
What is PCIe 6.1?
Since PCIe 3, each new generation of the standard has seen a doubling in the data rate. PCIe 6.1 boosts the data rate to 64 gigatransfers per second (GT/s), twice that of PCIe 5.0. For a x16 link, which is typical of graphics and network cards, the bandwidth of the link reaches 128 gigabytes per second (GB/s). As in previous generations, the PCIe 6.1 link is full duplex, so it can deliver that 128 GB/s bandwidth in both directions simultaneously for a total bandwidth capacity of 256 GB/s.
PCIe has proliferated widely beyond servers and PCs, with its economies of scale making it attractive for data-centric applications in IoT, automotive, medical and elsewhere. That being said, the initial deployments of PCIe 6.1 will target applications requiring the highest bandwidth possible and those can be found in the heart of the data center: AI/ML, HPC, networking and cloud graphics.
The following chart shows the evolution of the PCIe specification over time:
| PCie Specification | Data Rate per Lane (GT/s) |
Encoding | x16 Unidirectional Bandwidth (GB/s) | Specification Ratification Year |
|---|---|---|---|---|
| 1.x | 2.5 | 8b/10b | 4 | 2003 |
| 2.x | 5 | 8b/10b | 8 | 2007 |
| 3.x | 8 | 128b/130b | 15.75 | 2010 |
| 4.0 | 16 | 128b/130b | 31.5 | 2017 |
| 5.0 | 32 | 128b/130b | 63 | 2019 |
| 6.x | 64 | PAM4/FLIT | 128 | 2022 |
What’s new with PCIe 6.1?
To achieve the 64 GT/s, PCIe 6.1 introduces new features and innovations:
1. PAM4 Signaling:
On the electrical layer, PCIe 6.1 uses PAM4 signaling (“Pulse Amplitude Modulation with four levels”) that combines 2 bits per clock cycle for 4 amplitude levels (00, 01, 10, 11) vs. PCIe 5.0, and earlier generations, which used NRZ modulation with 1 bit per clock cycle and two amplitude levels (0, 1).
- Comparison of NRZ modulation and PAM4 modulation
2. Forward Error Correction (FEC)
There are always tradeoffs, and the transition to PAM4 signal encoding introduces a significantly higher Bit Error Rate (BER) vs. NRZ. This prompted the adoption of a Forward Error Correction (FEC) mechanism to mitigate the higher error rate. Fortunately, the PCIe 6.1 FEC is sufficiently lightweight to have minimal impact on latency. It works in conjunction with strong CRC (Cyclic Redundancy Check) to keep Link Retry probability under 5×10-6. This new FEC feature targets an added latency under 2ns.
While PAM4 signaling is more susceptible to errors, channel loss is not affected compared to PCIe 5.0 due to the nature of the modulation technique, so the reach of PCIe 6.1 signals on a PCB will be the same as that of a PCIe 5.0.
3. FLIT Mode:
PCIe 6.1 introduces FLIT mode, where packets are organized in Flow Control Units of fixed sizes, as opposed to variable sizes in past PCIe generations. The initial reason for introducing FLIT mode was that error correction requires working with fixed size packets; however, FLIT mode also simplifies data management at the controller level and results in higher bandwidth efficiency, lower latency, and smaller controller footprint. Let’s address bandwidth efficiency for a minute: with fixed-size packets, the framing of packets at the Physical Layer is no longer needed, that’s a 4-byte savings for every packet. FLIT encoding also does away with 128B/130B encoding and DLLP (Data Link Layer Packets) overhead from previous PCIe specifications, resulting in a significantly higher TLP (Transaction Layer Packet) efficiency, especially for smaller packets.
4. Other changes in PCIe 6:
- L0p mode – enabling traffic to run on a reduced number of lanes to save power
- A new PIPE specification – for the PHY to Controller interface
PCIe 6.1 Fun Fact: the x32 and x12 interface widths from earlier generations are dropped. While these widths are available in PCIe 5.0 and earlier specifications, these widths were never implemented in the market.
Why PCIe 6.1 now?
Before 2015, the PCIe specification was well ahead of the market in terms of available bandwidth required for use cases. After 2015, global data traffic has exploded. Data centers transitioned to 100G Ethernet (and up) pushing the bottleneck to the PCIe interconnects in servers and network devices.
The PCIe 6.1 specification fully supports the transition to 800G Ethernet in data centers: 800 gigabit per second (Gb/s) requires 100 GB/s of unidirectional bandwidth which falls within the 128 GB/s envelope of a x16 PCIe 6.1 link; 800G Ethernet, like PCIe, is full duplex. Further, data center general compute and networking are not the sole driving forces behind PCIe 6.1. AI/ML accelerators have an insatiable need for more bandwidth. Processing AI/ML training models is all about speed, and the faster accelerators can move data in and out, the more efficient and cost effective the training can be executed.
Conclusion
PCIe is everywhere in modern computing architectures, and we expect PCIe 6.1 will gain quick adoption in performance-critical applications in AI/ML, HPC, cloud computing and networking.
Rambus offers PCIe 6.1 controller IP, featuring an Integrity and Data Encryption (IDE) engine which provides state-of-the-art security for the PCIe links and the valuable data transferred over them.
Leave a Reply