Choosing the Right Redriver or Retimer Device to Extend PCIe Protocol Signal Range

The data rate requirements of high-performance cloud computing systems continue to advance and create a significant signal integrity challenge for a wide range of components in data center equipment deployments. The increase in data rate results in a decrease in signal transmission distance and can limit system scalability. While a redriver or a retimer device can help to resolve this limitation, each comes with advantages and disadvantages. 

This article describes how these devices can extend the Peripheral Component Interface Express (PCIe®) protocol signal range, and how to select the best one for compute system and NVMe™ storage applications.  

 

The PCIe Signal Integrity Challenge

The PCIe interface standard is one of the most popular interfaces used in today’s high-performance computing systems and data centers. PCIe’s data rates have evolved from first-generation (Gen1) 2.5GT/s to fifth-generation (Gen5) 32GT/s. Sixth-generation 6 (Gen6) will again double the previous version’s data rate. As frequencies have increased to support ever higher data rates, maintaining sufficient signal integrity at a reasonable system cost has become a challenge. Redriver and retimer devices are both solutions that can help to bridge the gap. 

FR4 PCBs are the most popular and cost-effective material in the electronics industry. FR4 PCB material performs well at relatively low frequencies with acceptable attenuation below 10GHz. As the data rate increases, however, FR4 material frequency response decreases.

Other PCB materials such as Megtron 6 have a better frequency response and experience less signal loss but come at a significant cost premium. For example, Megtron 6 is about seven times the cost of FR4. Other materials that can operate in the microwave frequency range have an even higher cost premium. The equation below approximates the signal loss in dB/inch due to trace loss and dielectric loss.

 

Where:

• W = trace width in mil, assuming 5 mil for this calculation
• F = Frequency in GHz
• Df = dissipation factor or loss tangent (dependent on PCB material)
• Dk = dielectric constant (dependent on PCB material)

Figure 1 shows a plot of the PCB attenuation for both FR4 and Megtron 6 PCB materials. Depending on the complexity and size of the PCB, the cost to move to a high-quality material PCB could be cost-prohibitive.  

 

Figure 1. Attenuation versus Frequency as a function of PCB material

 

Some applications may require connectors to deliver signals to other portions of the design such as backplanes and off-board add-in cards. Connectors are an additional contributor to signal loss. A PCIe CEM connector adds about 1.5dB loss at 32Gbps. The PCIe Gen5 standard prescribes an allowable channel loss budget is 36dB end to end.

Using either a redriver or a retimer can help to maintain PCIe signal integrity. Making the right choice requires a basic understanding of the differences between the two.

 

Redriver Explained

A redriver is a high-bandwidth amplifier with receive (RX) side equalizer (EQ) to compensate for frequency-dependent attenuation due to PCB traces or cables. The Continuous Timeline Equalizer's (CTLE) primary function is to open the closed eye of the distorted waveform. The transmit (TX) side can include a pre-emphasis function (transmit equalizer) to pre-shape the transmit waveform.

The signal integrity of serial interfaces such as DisplayPort, USB, Thunderbolt, HDMI, and PCIe can benefit from placing a redriver in its path if the trace or cable length is beyond their standard reach. The analog amplifier does not differentiate between any particular protocol standard because it has no link training process. Since it is protocol agnostic, the link can become non-compliant to any interface standards. It does not require a clock because of its analog circuit nature. 

The major disadvantage of a redriver is that it not only amplifies the data signal but also amplifies any noise that is in the signal path. The amplifier has a noise floor itself and can add its own noise to the overall noise figure of the signal. A typical linear redriver equalizer adds 8ps of intrinsic jitter to the signal and does correct for Inter Symbol Interference (ISI) jitter. A redriver cannot compensate for non-ISI jitter. When compared to a retimer, a redriver, in some cases, has lower power consumption and overall cost. A typical redriver latency is around 100ps. 

Figure 2 outlines the key building blocks of an analog single lane redriver.

 

Figure 2. Single lane redriver block diagram

 

The high-bandwidth amplifier in a redriver can be either a linear or limiting (non-linear). A linear amplifier may provide some pseudo link training functionality for PCIe protocol, depending on the design implementation. A limiting amplifier does not support any type of link training sequence for any protocol. A limiting amplifier supports only two threshold levels to determine the received signal’s condition. Since most link training pulses require the detection of intermediate thresholds, it is very difficult for a redriver to support training sequences. This is the limiting amplifier’s “blind spot.”  

 

Redrivers Have Their Limits

Redrivers can support PCIe Gen 1 to Gen 3 data rates when the application is sufficiently small and of limited complexity for boosting signal transmission distance. However, as design scale and complexity increase, the redriver can no longer compensate for the signal loss while still using cost-effective materials. Cascading two redrivers to overcome the problem is not practical.  Any noise or random jitter will get amplified along with the desired signal. An analog amplifier cannot reset any noise or timing budget. Therefore, cascading two redrivers will actually double the amount of noise to the data.  

PCIe Gen 4 at 16Gbps data rates pose an even greater challenge from signal integrity point of view. The majority of PCIe Gen 4 interface applications are in cloud storage, servers, and high-performance computing platforms, where the 16Gbps links are required to be driven over long traces, connectors, cables, slots, and add-in cards (AICs). The redriver is simply not usable at these data rates in data center infrastructure use cases.

The release of PCIe 5.0 in 2019 bumped the data rate to 32Gbps. High-end networking systems utilizing 400Gbps Ethernet, multi-200Gbps InfiniBand, and accelerator/GPU components and technologies are the primary drivers for the deployment of ever-increasing PCIe link rates. NVMe SSD components deployed in Enterprise server and storage systems are another driver for these higher data rates. Other serial protocols USB4.0, DisplayPort 2.0, and Thunderbolt 3.0 are also doubling their data rate over time. 

 

Retimer to the Rescue

Given the requirement to solve these high-speed signal integrity issues, the PCIe standard, beginning with PCIe Gen 4, has defined PCIe retimer requirements. The standard defines a retimer as a component that is “Physical Layer protocol aware and must interoperate with any pair of Components with any compliant channel on each side of the Retimer.” Retimers have a much higher degree of complexity than a redriver as a result. Section 4.3 of both the PCIe Gen 4 and PCIe Gen 5 specifications covered the retimer requirements in detail.

Figure 3 illustrates the high-level block diagram of a single lane bi-directional retimer. 

 

Figure 3. Retimer block diagram

 

The PCIe standard refers to this as a PCIe x1 configuration. Most PCIe retimers are either x4 (8 total lanes: 4 RX and 4 TX), x8 (16 lanes) or x16 (32 lanes).

The physical layer is the Physical Medium Attachment (PMA: Physical Sub-Block) where the Serializer/De-Serializer (SERDES) that receive and transmit data are located. The PMA is a mixed signal building block. At the receiver side, the distorted signal is equalized and noise filtered using a CTLE.

The heart of a retimer is the Clock and Data Recovery (CDR) block. The CDR recovers the embedded clock along with the data in the parallel domain. The PMA block serializes parallel data for transmission and de-serializes received data into the Physical Coding Sublayer (PCS) block.

The eye monitoring blocks generate a real-time waveform capture of the receive eye pattern for debugging purposes. The PCS handles the Link Training Status State Machine (LTSSM) and the PIPE (PHY Interface for PCIe) functions. The PCS is a purely digital section. 

Table 1 summarizes the key differences between a redriver and retimer.

 

Table 1. Redriver and Retimer Comparison

 

Examples of Retimers in a PCIe Application

The PCIe standard is the primary interface standard used on components deployed in the data center for storage, server, and network infrastructure. CPUs utilize high-speed PCIe interfaces to deliver I/O transactions as a PCIe root complex to attached SSD drives or other endpoint components. Figure 4 illustrates the topology from the CPU to these endpoints as an example. A PCIe switch provides additional fanout to support a larger number of endpoint destinations. Retimers are now required components to support signal expansion over CPU boards, backplanes, cables, and add-in cards.

 

Figure 4. Server example with PCIe retimer

 

In addition, PCIe retimers are often used to support signal conditioning when cables and/or multiple connectors are in the data path. Retimers are often utilized between the CPU and endpoints as listed below and illustrated in Figure 5:

• CPU <— Retimer —> Add-In Card (AIC) 
• CPU <— Retimer —> Riser Card —> AIC
• CPU <— Retimer —> Cable —> Switch —> AIC 
• CPU <— Retimer —> Cable —> AIC 

 

Figure 5. Retimer on Riser Card to AIC Retimer on motherboard to AIC

 

Redrivers and retimers are helpful to maintain signal integrity in many data center system applications. Depending on the complexity and data rate of the equipment design, redrivers can be useful for smaller system that operate at lower data rates.

For data rates above 16Gbps, redrivers do not have sufficient capability to compensate for the significant signal degradation. PCIe 4.0 and 5.0 require the use of retimers for compliance. Other serial protocols such as USB 4.0 and Thunderbolt 3.0 also specify retimer requirements in their perspective specifications. 

Because retimers reset the signal jitter budget and regenerates a flesh clean signal for retransmission, there is no insertion loss and designers can realize the full performance benefits of their compute system and NVMe™ storage applications at a reasonable system cost. 

 

References

1. High-Speed Serial Bus Repeater Primer (PDF)
2. PCI-SIG® Educational Webinar Series 2019 (PDF)
3. AN 766: Intel® Stratix® 10 Devices, High Speed Signal Interface Layout Design Guideline (PDF)
4. PCI Express Base Specification Revision 5.0 Version 1.0, 2019

Industry Articles are a form of content that allows industry partners to share useful news, messages, and technology with All About Circuits readers in a way editorial content is not well suited to. All Industry Articles are subject to strict editorial guidelines with the intention of offering readers useful news, technical expertise, or stories. The viewpoints and opinions expressed in Industry Articles are those of the partner and not necessarily those of All About Circuits or its writers.