Speedster22i Interlaken User Guide UG032 – May 15, 2014 UG032, May 15, 2014 1
Copyright Info Copyright © 2014 Achronix Semiconductor Corporation. All rights reserved. Achronix is a trademark and Speedster is a registered trademark of Achronix Semiconductor Corporation. All other trademarks are the property of their prospective owners. All specifications subject to change without notice. NOTICE of DISCLAIMER: The information given in this document is believed to be accurate and reliable.
Table of Contents Copyright Info.................................................................................. 2 Table of Contents ............................................................................ 3 Introduction ..................................................................................... 6 Design Overview ............................................................................. 8 Hierarchy ...........................................................................................
TX Rate Limiting .............................................................................................. 48 Example: Programming the Rate Limiter ............................................................... 49 Error Handling ................................................................................................. 50 Revision History ..............................................................................................
Table of Figures Figure 1: Shows a block diagram of IIPC with SerDes and user logic interface ............... 8 Figure 2: IIPC Hierarchy .................................................................................................... 9 Figure 3: RX Clock Domains ............................................................................................ 13 Figure 4: TX Clock Domains.............................................................................................
Introduction Interlaken is a scalable chip-to-chip interconnect protocol designed to enable transmission speeds from 10Gbps to 100Gbps and beyond. Using the latest SerDes technology and a flexible protocol layer, Interlaken minimizes the pin and power overhead of chip-to-chip interconnect and provides a scalable solution that can be used throughout an entire system. In addition, Interlaken uses two levels of CRC checking and a self-synchronizing data scrambler to ensure data integrity and link robustness.
• Segment-mode and Packet-mode receive format • BurstMax size can be programmed from 64 bytes to 256 bytes in steps of 64 bytes • Support for minimum BurstShort requirement of 64 bytes up until 256 bytes in steps of 64 bytes but not exceeding BurstMax • In-band flow control • Support for link-level flow control • Flow control mechanism supports stopping packets in mid-stream – head of line blocking • Rate matching with granularity of 1 Gbps • Meta Frame Length programmable between 128 to 8K w
Design Overview Figure 1 shows a block diagram of the IIPC with SerDes and user logic interfaces implemented in Speedster22i. The right hand side is the user interface and the left hand side is the SerDes interface. The IIPC is shown in green and the SerDes blocks in orange.
Hierarchy The interfaced IIPC and SerDes IP is configured using the Achronix Cad Environment (ACE) Interlaken GUI. The upper-levels of hierarchy are shown in Figure 2. Note that the text in Figure 2 represents the name of the module at that level of hierarchy. Interlaken_1 Interlaken_1_serdes ACX_INTERLAKEN_TOP (IIPC core) Figure 2: IIPC Hierarchy The top-level wrapper module, Interlaken_1, contains the instantiated hard IIPC block and an instantiated hard SerDes block for each lane.
Typical Operation The IIPC can be used in a variety of applications, and provides the user all of the flexibility offered by the Interlaken Protocol. The IIPC handles all protocol-related functions needed to communicate to another device’s Interlaken interface. All handshaking, synchronizing and error checking are handled by the IIPC. The LBUS is designed to match commonly used packet bus protocols made common by the SPI4.2 protocol; a detailed description is given in the User Interfaces section.
scheduling function. The user need not be concerned about any of the lower level Interlaken Protocol details.
Clocking The IIPC has three major clock domains: 1. 2. 3. LBUS clock Domain • The clk input port is used to clock the protocol processing of the IIPC. This includes all logic in the TX and RX paths responsible for protocol layer processing including Control Word, Meta frame and the LBUS interface. • The frequency of the clk domain is 470MHz RX SerDes clock Domain • Each SerDes is assumed to provide its recovered clock to the IIPC.
IIPC Core serDes hs_if per_lane_rx rx_serdes_clk[1] serDes hs_if per_lane_rx rx_serdes_clk[2] serDes hs_if per_lane_rx rx_serdes_clk[3] serDes hs_if per_lane_rx rx_serdes_clk[4] serDes hs_if per_lane_rx rx_serdes_clk[5] serDes hs_if per_lane_rx rx_serdes_clk[6] serDes hs_if per_lane_rx rx_serdes_clk[7] serDes hs_if per_lane_rx rx_serdes_clk[8] serDes hs_if per_lane_rx rx_serdes_clk[9] serDes hs_if per_lane_rx rx_serdes_clk[10] serDes hs_if per_lane_rx rx_serdes_cl
IIPC Core hs_if per_lane_tx serDes hs_if per_lane_tx serDes hs_if per_lane_tx serDes hs_if per_lane_tx serDes hs_if per_lane_tx serDes hs_if per_lane_tx serDes hs_if per_lane_tx serDes hs_if per_lane_tx serDes hs_if per_lane_tx serDes hs_if per_lane_tx serDes hs_if per_lane_tx serDes hs_if per_lane_tx tx_striping in the FPGA Core serDes clk Interface to user logic tx_serdes_refclk Figure 4: TX Clock Domains UG032, May 15, 2014 14
Port List Table 1 shows the port list for the IIPC. More detail on the use of each signal is given in the User Interfaces section. Table 1 – Port Description Name Direction Clock Description Register Interface sbus_req Input sbus_clk Asserted for 9-cycles in case of read and for 11-cycles in case of write sbus_datain[1:0] Input sbus_clk Carries read/write indication, address and data to write Output sbus_clk Acknowledgment from register i/f once read or write is completed thru SBUS.
Name Direction Clock Description corresponding bit of this bus. rx_serdes_resetn[11: 0] Input Reset for each RX SerDes lane. The recovered clock for each SerDes lane has associated with it an active-low reset. This signal is low until transmit and receive data-paths of each SerDes are ready. tx_serdes_refclk Input Common clock for the all Tx SerDes interfaces. Input Active low reset for TX Reference clock.
Name Direction Clock Description indicates the channel number of the inflight packet and is only valid in cycles that rx_enaout is sampled as 1. rx_enaout rx_sopout rx_eopout rx_errout rx_mtyout[5:0] Output Output Output Output Output clk Receive LBUS Enable. This signal qualifies the other signal of the RX LBUS Interface. Signals of the RX LBUS Interface are only valid in cycles that rx_enaout is sampled as a 1. clk Receive LBUS Start-Of-Packet.
Name tx_ovfout tx_datain[511:0] tx_chanin[7:0] tx_enain tx_sopin tx_eopin tx_errin tx_mtyin[5:0] UG032, May 15, 2014 Direction Output Input Input Input Input Input Input Input Clock Description clk Transmit LBUS Overflow. This signal indicates whether the user has violated the back pressure mechanism provided by the tx_rdyout signal. If tx_ovfout is sampled as a 1, a violation has occurred.
Name Direction Clock Description tx_eopin are sampled as 1. When tx_eopin and tx_errin are sampled as 1, the value of tx_mtyin[2:0] is ignored as if it was 000. The other bits of tx_mtyin are used as usual. tx_bctlin Input clk Transmit force insertion of Burst Control word. This input is used to force the insertion of a Burst Control Word.
Name Direction Clock Description Words between two Control Words) was shorter than the value specified by ctl_tx_burstshort. This signal is only asserted if the final Control Word did not contain an EOP. This signal is provided to identify poor scheduler design by the user that will result in reduced throughput. This signal is informational only and may be optionally ignored.
Name Direction Clock Description Definition rev1.2. stat_rx_crc32_err[11 :0] stat_rx_mubits[7:0] stat_rx_synced[11:0] stat_rx_synced_err[ 11:0] stat_rx_mf_len_err[ 11:0] stat_rx_mf_repeat_e rr clk Diagnostic Word CRC32 Error/Invalid. This bus provides indication of an invalid CRC32 in the Diagnostic Word for the respective lane. These signals are asserted with a value of 1 for one LBUS clock cycle each time an error is detected. clk RX Multiple-Use Control Bits.
Name Direction Clock Description de-skewed. A value of 1 indicates all lanes are aligned and de-skewed. When this signal is a 1, the RX path is aligned and can receive packet data. stat_rx_aligned_err stat_rx_crc24_err stat_rx_overflow_er r stat_rx_mf_err[11:0] Output Output Output clk Loss of Lane Alignment/De-Skew. This signal indicates an error occurred during lane alignment or lane alignment was lost. A value of 1 indicates an error occurred. clk Control Word CRC24 Error.
Name Direction Clock Description expected Meta Frame Synchronization Word across all (active) lanes. This signal can be used to collect the statistic “RX_Alignment_Error" as described in Table 5-9 of the Interlaken Protocol Definition rev1.2. This signal is not asserted until the Meta Frame Synchronization Word has been received at least once across all lanes. A value of 1 indicates the error occurred.
User Interfaces The IIPC handles the intricate details of transporting data over an Interlaken link. The user interface is a simple packet interface designed to allow the user to easily integrate the IIPC into a system. The LBUS consists of three separate interfaces: 1. Transmitter (TX) interface 2. Receiver (RX) interface 3. Status/Control interface 4.
TX LBUS Interface The synchronous TX Local bus interface accepts packet oriented data of arbitrary length. It accepts data in either packet mode, or segment mode. All signals are synchronous relative to the rising-edge of the clk port. Figure 5 shows a sample waveform for data transaction for a 257 byte packet. 63 63 Figure 5: Sample TX Waveform with a 512-bit Data Bus Data is written into the interface on every clock cycle when tx_enain is asserted.
1, the value of tx_mtyin[2:0] is ignored as if it was 000. The other bits of tx_mtyin are used as usual. Data can be safely written, i.e. tx_enain asserted, whenever tx_rdyout is asserted. After tx_rdyout is negated, additional 16 writes, using tx_enain, can be safely performed provided tx_ovfout is never asserted. Once tx_rdyout is asserted again, additional data can be written. If, at any time, the back-pressure mechanism is violated, the tx_ovfout is asserted to indicate the violation.
• It also strongly recommended that the changing of channels be such that the number of bytes between Control Words, whether forced (via tx_bctlin) or implied, be a multiple of ctl_tx_burstmax. Except for the last burst of a packet, no burst should ever be less than ctl_tx_burstshort. Use of tx_bctlin The tx_bctlin input operates in a similar manner to tx_sopin: both signals cause a Burst Control Word to be injected into the data stream.
RX LBUS Interface The synchronous RX Local bus interface provides packet oriented data much like the TX Local bus interface accepts. All signals are synchronous with the rising-edge of the Local bus clock. Figure 6 shows a sample waveform for data transaction for a 257 byte packet. 63 63 Figure 6: Sample RX Waveform with a 512-bit Data Bus Data is supplied by the IIPC on every clk clock cycle when rx_enaout is asserted. This signal qualifies the other outputs of the RX Local bus interface.
• The user logic must be capable of receiving data when rx_enaout is asserted, and use the Interlaken flow control mechanism to stop the far device (the one sending data to the IIPC) from sending more data if needed. The data provided by the RX Local bus interface is in the same sequence as it is received from the Interlaken bus. Packets may be interleaved and are distinguished using the channel number presented on rx_chanout.
Status/Control Interface The Status/Control interface allows the user to setup the IIPC configuration and to monitor the status of the IIPC directly. All these signals listed below are available to user directly. User can decide how to use these signals. The following sections describe the various Status and Control signals. Note: Proper understanding of the status signals’ description below entails a solid understanding of the Interlaken Protocol. Refer to the Interlaken Protocol Definitionrev1.
• Four consecutive invalid Meta Frame Synchronization Words were detected in the corresponding lane, or • Three consecutive invalid Scrambler State Control Words were detected in the corresponding lane. The bits of the bus remain asserted until word boundary synchronization occurs or until some other error/failure is signaled for the corresponding lane.
This signal is asserted for one clock period each time a CRC24 error is detected. stat_rx_msop_err Packets received with a particular channel address must begin with a valid Start-of-Packet (SOP). If data is detected for a particular channel without a valid SOP, this signal is asserted for a single Local bus clock cycle. Additionally, the required SOP is inserted before the data and an error is signaled in the End-of-Packet (EOP) cycle via the rx_errout signal.
CRC32 Diagnostics Checking Interlaken implements a CRC32 check for each lane of the interface in order for the user to monitor the health of each lane. The IIPC has two signals for this function that are listed below. All signals are synchronous with the rising-edge of clk. stat_rx_crc32_valid[11:0] When a bit of this bus is 1, it indicates two things: 1. The CRC32 in the most recently received Diagnostic Word on the corresponding lane was valid, and 2.
Interlaken Status Messaging for the Transmitter The Transmitter is capable of inserting the Status Messaging as described in the Interlaken Protocol into the Meta Frame Diagnostics words. User’s task: Feed these inputs based on the health of the Receiver. All signals are synchronous with the rising-edge of clk and a detailed description of each signal follows. ctl_tx_diagword_intfstat This input is transmitted on bit[32], the interface health (Status Bit 0), of every Diagnostic Word on all of the lanes.
Transmitter Flow-Control Inputs The IIPC implements the Interlaken in-band flow control mechanism. This mechanism communicates XON/XOFF for different channels using the In-Band Flow Control bits of Control words. Additionally, the Multiple-Use bits of Control Words may be used in a similar manner as described in the Transmitter Multiple-Use Bits and Receiver Multiple-Use Bits sections.
Register Interface The register interface provides access to control and status registers for the IIPC. These registers can be accessed via the FPGA’s SBus. IIPC is a slave on SBus. Users need to design the master interface in the fabric to drive SBus. For information about the SBus, see https://www.achronix.com/wpcontent/uploads/docs/Speedster22i_sBus_User_Guide_UG047.pdf.
TX Rate Limiter Update Interval Register – R/W 0x0110 txrlimintv Bits 7:0 specify the interval, in Local bus clock cycles, that the token bucket will be updated. It is recommended that this value be greater than or equal to 8. This value should not be changed when the rate limiter is enabled. (ctl_tx_rlim_intv) TX Multi-Use Bits Register – R/W 0x011C txmubits Bits 7:0 Specify a value contained in bits 31-24 of subsequent Control Words generated by the TX.
TX Status Register – R/W Bit 3: Set to a value of 1 if tx_ovfout is asserted on the TX LBUS. Write 1 to clear. 0x014A txstat Bit 2: Set to a value of 1 if an overflow occurs. This bit should never be set and indicates a critical failure. Write 1 to clear. Bit 1: Set to a value of 1 if a burst, less than BurstShort, not ending with an EOP, is written into the TX. Write 1 to clear. Bit 0: Set to a value of 1 if the serial data rate is faster than the maximum data rate on the Local bus. Write 1 to clear.
RX In-band Flow Control Registers (0-15) - RO 0x0210 to 0x022E Provides the most recent value for in-band flow control for lanes 0255. rxfcstat0-15 Register at 0x0210 – Bit 0 = Lane 0, Bit 15 = lane 15 Register at 0x0212 – Bit 0 = Lane 16, Bit 15 = lane 31 Continues Up to 0x22E where Bit 0 = lane 240 and Bit 15 = lane 255 RX Status Register – R/W Bit 8 – When 1, RX CRC24 error was detected. Write 1 to clear. Bit 7 – When 1, RX Burst error was detected. Write 1 to clear.
RX CRC32 Error Statistics Registers – RO 0x0880 Bit 15:0 – returns the value contained in bits 15-0 (LSB) of RX CRC32 Error Statistics register for lane 0 0x0880 to 0x08AE statistics counters 0x0882 Bit 15:0 – returns the value contained in bits 31-16 (MSB) of RX CRC32 Error Statistics register for lane 0 0x0884 Bit 15:0 – returns the value contained in bits 15-0 (LSB) of RX CRC32 Error LSB Statistics register for lane 1 0x0886 Bit 15:0 – returns the value contained in bits 31-16 (MSB) of RX CRC32 Error S
0x0980 to 0x09AE statistics counters RX Word Boundary Synchronization Error Statistics Registers – RO 0x0980 Bit 15:0 – returns the value contained in bits 15-0 (LSB) of RX Word Boundary Synchronization Error Statistics register for lane 0 0x0982 Bit 15:0 – returns the value contained in bits 31-16 (MSB) of RX Word Boundary Synchronization Error Statistics register for lane 0 0x0984 Bit 15:0 – returns the value contained in bits 15-0 (LSB) of RX Word Boundary Synchronization Error Statistics register for
0x0A80 to 0x0AAE statistics counters RX Bad Type Error Statistics Registers – RO 0x0A80 Bit 15:0 – returns the value contained in bits 15-0 (LSB) of RX Bad Type Error Statistics register for lane 0 0x0A82 Bit 15:0 – returns the value contained in bits 31-16 (MSB) of RX Bad Type Error Statistics register for lane 0 0x0A84 Bit 15:0 – returns the value contained in bits 15-0 (LSB) of RX Bad Type Error Statistics register for lane 1 0x0A86 Bit 15:0 – returns the value contained in bits 31-16 (MSB) of RX Bad T
0x0B80 to 0x0BAE statistics counters RX Meta Frame Length Error Statistics Registers – RO 0x0B80 Bit 15:0 – returns the value contained in bits 15-0 (LSB) of RX Meta Frame Length Error Statistics register for lane 0 0x0B82 Bit 15:0 – returns the value contained in bits 31-16 (MSB) of RX Meta Frame Length Error Statistics register for lane 0 0x0B84 Bit 15:0 – returns the value contained in bits 15-0 (LSB) of RX Meta Frame Length Error Statistics register for lane 1 0x0B86 Bit 15:0 – returns the value conta
0x0C80 to 0x0CAE statistics counters RX Descrambler Error Statistics Registers – RO 0x0C80 Bit 15:0 – returns the value contained in bits 15-0 (LSB) of RX Descrambler Error Statistics register for lane 0 0x0C82 Bit 15:0 – returns the value contained in bits 31-16 (MSB) of RX Descrambler Error Statistics register for lane 0 0x0C84 Bit 15:0 – returns the value contained in bits 15-0 (LSB) of RX Descrambler Error Statistics register for lane 1 0x0C86 Bit 15:0 – returns the value contained in bits 31-16 (MSB)
Description of Features Lane Decommission The IIPC features two different modes of lane decommissioning: 1. Disabling consecutive lanes, and 2. Disabling a single lane. Disabling consecutive lanes allows the same instantiation of the IP to be used for multiple configurations. Disabling a single lane enables lane resiliency where one lane with too many errors can be disabled in the same physical link. Both modes of decommissioning can be used at the same time.
Disabling a Single Lane Single Transmit Lane To disable one of the lanes in the range between 0 to ctl_tx_last_lane, the parameters ctl_tx_has_bad_lane and ctl_tx_bad_lane are used. To enable single lane decommissioning, ctl_tx_has_bad_lane is set to 1, and the ctl_tx_last_lane is set to the number of the SerDes lane that is decommissioned.
Link Level Flow Control The Interlaken Protocol does not restrict the use of the calendar entries to a particular flow control implementation. As such, the user can decide if one or more of the calendar entries should be designated to control the flow control of the whole interface (i.e. link level flow control). For example, if the calendar length is 32, the user can decide to use calendar slots 0 and 16 as link level flow control bits.
TX Rate Limiting The IIPC rate limiter can be used to reduce the overall Data Word transmission rate. This is achieved by transmitting Idle Control Words in between packet segments to limit the effective data transfer rate. The purpose of transmitter rate limiting is to reduce buffering requirements by the receiving device and reduce the amount of flow control stalling that may otherwise be required.
ctl_tx_rlim_intv[11:0] This input specifies how many tokens are to be added to the bucket after each interval. A token is equal to 1 byte. This value must be greater than 0. The value of this input should not be changed when ctl_tx_rlim_enable is a value of 1. Note: This value should be calculated based on the desired rate and the value in ctl_tx_rlim_intv.
Error Handling The IIPC performs robust checking of all possible error conditions as described in the Interlaken Protocol Definition including the following errors: • a loss of lane alignment, • a CRC24 error, • a BurstShort violation, • an illegal Control Word Type, or • an illegal framing pattern is detected If any of the above conditions are detected, the IIPC takes the following actions: All open channels are marked as being in error and the packet in flight for these channels will indicate the error co
Revision History The following table shows the revision history for this document. UG032, May 15, 2014 Date Version 4/26/2013 4/28/2014 5/15/14 1.0 1.1 1.
