ConnectX® IB HCA Cards With QSFP Connectors User’s Manual P/N: MHRH19-XTC, MHRH29-XSC, MHRH29-XTC, MHQH19-XTC, MHQH29-XSC, MHQH29-XTC Rev 1.
© Copyright 2008. Mellanox Technologies, Inc. All Rights Reserved. Mellanox Technologies, InfiniHost®, and ConnectX® are registered trademarks for Mellanox Technologies, Inc. All other marks and names mentioned herein may be trademarks of their respective companies. ConnectX® IB InfiniBand Adapter Cards With QSFP Connectors User’s Manual Document Number: 2909 Mellanox Technologies, Inc. 350 Oakmead Parkway Sunnyvale, CA 94086 U.S.A. www.mellanox.
ConnectX IB InfiniBand Adapter Cards With PCI Express x8 User’s Manual 3 Table of Contents Table of Contents List of Tables List of Figures Revision History About this Manual 3 4 5 6 7 Chapter 1 8 Chapter 2 Chapter 3 Overview 1.1 Adapter Cards 1.2 Mellanox Part Numbering Legend 1.3 Finding the GUID and Serial Number on the Adapter Cards 8 9 10 HCA Card Installation 11 2.1 Hardware and Software Requirements 2.2 Installation Instructions 2.
List of Tables Table 1: Revision History Table 6 Table 2: Documents List 7 Table 3: HCA Cards List 8 Table 4: Mellanox HCA Cards Part Numbering Key 9 Table 5: Hardware and Software Requirements 11 Table 6: LEDs 15 Table 7: Jumper Configuration 16 Table 8: VPD Layout for MHRH[12]9-X[ST]C 16 Table 9: VPD Layout for MHQH[12]9-X[ST]C 17 Table 10: HCA Cards Certification Status 20 Table 11: Specifications for MHRH19-XTC 23 Table 12: Specifications for MHRH19-XTC 24 Table 13: S
ConnectX IB InfiniBand Adapter Cards With PCI Express x8 User’s Manual 5 List of Figures Figure 1: MH[RQ]H19-XTC Card 8 Figure 2: MH[RQ]H29-XTC Card 9 Figure 3: Card Product Label 10 Figure 4: Port Numbering 14 Figure 5: Physical and Logical Link Indications 15 Figure 6: I2C Connector 15 Figure 7: Flash Jumper 16 Figure 8: Schematic of the ConnectX IB HCA Card with QSFP Connectors 19 Figure 9: I2C-Compatible Connector Plug and Pinout 27 Figure 10: QSFP Connector Pinout 28 Figur
Revision History This document was printed on December 29, 2008 11:57 am. Table 1 - Revision History Table Date Rev December 2008 1.3 Fixed links to website Added single port cards August 2008 1.05 Added French Translation of Safety Warnings Removed Watermark April 2008 0.20 Added MHRH Changed Link from OFED to Mellanox ODFED Removed Confidential March 2008 0.10 Initial Release Rev 1.
ConnectX IB InfiniBand Adapter Cards With PCI Express x8 User’s Manual 7 About this Manual This User’s Manual describes Mellanox Technologies MH[RQ]H[12]9-X[ST]C ConnectX IB PCI Express x8 HCA Adapter cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documentation. Intended Audience This manual is intended for the installer and user of these cards.
Overview 8 1 Overview This document is a User’s Manual for Mellanox Technologies host channel adapter (HCA) cards based on the MTץ25408 ConnectX® IB integrated circuit device. The cards described in this manual have the following main features: • • • • • • • IBTA v1.2 compliant QSFP ports for connecting InfiniBand traffic Compliant with QSFP MSA spec Rev. 1.0 Compatible with copper cable and optical cable with the use of a QSFP connector PCI Express 2.0 (1.
ConnectX IB InfiniBand Adapter Cards With PCI Express x8 User’s Manual 9 Table 3 - HCA Cards List Ordering Part Number IB Port (OPN) Speed MHRH29-XTC 20Gb/s (DDR) Dual port On-board Memory Size Short / Tall Bracket No Mem Tall Figure 2: MH[RQ]H29-XTC Card MHRH29-XSC MHQH29-XTC HCA Card with Brackets Short 40Gb/s (QDR) Dual port No Mem MHQH29-XSC Tall Short Short Bracket Tall Bracket These cards are RoHS-R5 Compliant. 1.
Overview 10 Table 4 - Mellanox HCA Cards Part Numbering Key HCA Card OPN MHTS#I-XBR Field Decoder I Host Interface X = PCI-X, 4 = PCIe x4, 8 = PCIe x8, 9 = PCIe (SerDes @ 5.0 GT/s) G Generation = Initial product generation - Separator X Memory Size X = MemFree, 1=128MB, 2=256MB, 3=512MB B Bracket S = Short, T = Tall, N = None R RoHS = non RoHS, C = RoHS w/ Exemption, R = RoHS Lead-Free 1.
ConnectX IB InfiniBand Adapter Cards With PCI Express x8 User’s Manual 11 2 HCA Card Installation 2.1 Hardware and Software Requirements Before installing the HCA Adapter card, please make sure that the system meets the hardware and software requirements listed in Table 5. Table 5 - Hardware and Software Requirements Requirement Hardware Software Operating Systems/Distributions Description PCI Express x8 or x16 slots • • For Windows see http://www.mellanox.
HCA Card Installation 12 2.3 Safety Warnings 1. Installation Instructions Read all installation instructions before connecting the equipment to the power source. 2. Over-temperature This equipment should not be operated in an area with an ambient temperature exceeding the maximum recommended: 55°C (131°F). To guarantee proper air flow, allow at least 8cm (3 inches) of clearance around the ventilation openings. 3.
ConnectX IB InfiniBand Adapter Cards With PCI Express x8 User’s Manual 13 3 Driver Software and Firmware 3.1 Driver Software For Linux or Windows, download and install the latest OpenFabrics Enterprise Distribution (OFED) software package available via the Mellanox OpenFabrics Web site at http://www.mellanox.com > Downloads > InfiniBand SW/ Drivers. Follow the installation instructions included in the download package. 3.
Adapter Card Interfaces 14 4Adapter Card Interfaces 4.1 I/O Interfaces Each HCA card includes the following interfaces: • • • • QSFP InfiniBand Optical Connectors PCI Express x8 edge connector I/O panel LEDs I2C compatible connector (for debug) 4.1.1 InfiniBand Interface The ConnectX®IB(MT25408) device is compliant with the InfiniBand Architecture Specification, Release 1.2. It has compliant InfiniBand ports, 1 or 2, each having four Tx/Rx pairs of SerDes.
ConnectX IB InfiniBand Adapter Cards With PCI Express x8 User’s Manual 15 4.1.3 LED Assignment The board has I/O LEDs located on the I/O panel- 2 LEDs per port. The green LED, when lit, indicates that the InfiniBand driver is running and a valid physical connection between nodes exists. If the green LED is blinking, it indicates a problem with the physical link. The yellow LED when lit, indicates a valid data activity link, this is the logical link.
Adapter Card Interfaces 16 4.3.1 System Memory Each of the HCA cards utilizes the PCI Express interface to store and access IB fabric connection information and packet data on the system memory. 4.3.2 Flash Each of the HCA cards includes one 2MB SPI Flash device (M25P80 device by ST Microelectronics) accessible via the Flash interface of the MT25408 ConnectX IB device. There is a jumper on each adapter card that indicates to the device whether an on-board Flash device is to be used.
ConnectX IB InfiniBand Adapter Cards With PCI Express x8 User’s Manual 17 Table 8 - VPD Layout for MHRH[12]9-X[ST]C Offset (Decimal) Item Value Format Description 19 21 VPD Keyword PN STR Add in Card Part Number Length 0x15 22 PN PN %STR_SPC 43 VPD Keyword EC STR 45 Length 0x2 Engineering Change Level of the card (rev) 46 Revision A1 %STR PCB revision 48 VPD Keyword SN STR Serial Number 50 Length 0x18 51 SerialNumber %STR_SPC “00..00XXXX..
Adapter Card Interfaces 18 Table 9 - VPD Layout for MHQH[12]9-X[ST]C Offset (Decimal) Item 19 21 Rev 1.3 Value Format Description VPD Keyword PN STR Add in Card Part Number Length 0x15 22 PN PN %STR_SPC 43 VPD Keyword EC STR 45 Length 0x2 Engineering Change Level of the card (rev) 46 Revision A1 %STR PCB revision 48 VPD Keyword SN STR Serial Number 50 Length 0x18 51 SerialNumber %STR_SPC “00..00XXXX..XX” 75 VPD Keyword V0 STR Misc.
ConnectX IB InfiniBand Adapter Cards With PCI Express x8 User’s Manual 19 Appendix A: Specifications A.1 Board Mechanical Drawing and Dimensions All of the HCA cards covered in this User’s Manual have the same mechanical drawing and share the same dimensions as depicted in Figure 8. Note: All dimensions are in millimeters. Figure 8: Schematic of the ConnectX IB HCA Card with QSFP Connectors 167.65 68.90 56.97 15.00 3.65 J1 – I2C Connector 1.90 J4 – Flash Jumper 33.35 43.17 57.15 96.
A.2 EMC Certification Statements Table 10 lists the approved certification status per HCA card in different regions of the world. Table 10 - HCA Cards Certification Status Rev 1.
ConnectX IB InfiniBand Adapter Cards With PCI Express x8 User’s Manual 21 FCC Statements (USA) Class A Statements: § 15.21 Statement Warning! Changes or modifications to this equipment not expressly approved by the party responsible for compliance (Mellanox Technologies) could void the user's authority to operate the equipment. §15.105(a) Statement NOTE: This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant to Part 15 of the FCC Rules.
A.2.3 VCCI Statements (Japan) Class A Statement: (Translation - "This is a Class A product based on the standard of the Voluntary Control Council for Interference by Information Technology Equipment (VCCI). If this equipment is used in a domestic environment, radio interference may occur, in which case the user may be required to take corrective actions.") Rev 1.
ConnectX IB InfiniBand Adapter Cards With PCI Express x8 User’s Manual 23 A.3 MHRH19-XTC Specifications Table 11 - Specifications for MHRH19-XTC Physical Power and Environmental Size: Air Flow: QSFP 20Gb/s Connector: 2.71in. x6.60in. (68.90mm x 167.65mm) 200LFM @55°C IBTA v1.2, Auto-Negotiation1 (20Gb/s, 5Gb/s) or (10Gb/s, 2.5Gb/s) QoS: 8 InfiniBand Virtual Lanes for each port PCI Express 12V, 3.3V 12W max. for passive cables only 14.5W max.
A.4 MHQH19-XTC Specifications Table 12 - Specifications for MHRH19-XTC Physical Power and Environmental Size: Air Flow: QSFP 40Gb/s Connector: 2.71in. x6.60in. (68.90mm x 167.65mm) 200LFM @55°C IBTA v1.2, Auto-Negotiation1 (20Gb/s, 5Gb/s) or (10Gb/s, 2.5Gb/s) QoS: 8 InfiniBand Virtual Lanes for each port PCI Express 12V, 3.3V 12.6W max. for passive cables only 15.1W max.
ConnectX IB InfiniBand Adapter Cards With PCI Express x8 User’s Manual 25 A.5 MHRH29-X[ST]C Specifications Table 13 - Specifications for MHRH29-XSC/-XTC Physical Power and Environmental Size: Air Flow: QSFP 20Gb/s Connector: 2.71in. x6.60in. (68.90mm x 167.65mm) 200LFM @55°C IBTA v1.2, Auto-Negotiation1 (20Gb/s, 5Gb/s) or (10Gb/s, 2.5Gb/s) QoS: 8 InfiniBand Virtual Lanes for each port PCI Express 12V, 3.3V 13W max. for passive cables only 18W max.
A.6 MHQH29-X[ST]C Specifications Table 14 - Specifications for MHRH29-XSC/-XTC Physical Power and Environmental Size: Air Flow: QSFP 40Gb/s Connector: 2.71in. x6.60in. (68.90mm x 167.65mm) 200LFM @55°C IBTA v1.2, Auto-Negotiation1 (20Gb/s, 5Gb/s) or (10Gb/s, 2.5Gb/s) QoS: 8 InfiniBand Virtual Lanes for each port PCI Express 12V, 3.3V 13.5W max. for passive cables only 18.5W max.
ConnectX IB InfiniBand Adapter Cards With PCI Express x8 User’s Manual 27 Appendix B: Interface Connectors Pinout B.1 I2C-Compatible Connector Pinout Figure 7: I2C-Compatible Connector Plug and Pinout Table 15 - I2C-compatible Connector Pinout 4 3 2 1 5 5 1 2 3 4 Connector Pin Number HCA Signal Name 1 SPSDA 2 SPSCL 3 GND 4 NC 5 NC B.
B.3 QSFP Connector Pinout Figure 8: QSFP Connector Pinout 20 21 22 23 24 25 26 27 28 29 30 GND Rx2n Rx1n 18 Rx2p Rx1p 17 GND GND 16 Rx4n Rx3n Rx4p Rx3p GND GND ModPrsL SDA IntL SCL VccTx Vcc Rx Vcc1 ResetL 15 14 13 12 11 10 9 8 31 LPMode 32 GND GND 33 Tx3p Tx4p 34 Tx3n 35 GND Tx4n GND 36 Tx1p Tx2p 3 Tx1n GND Tx2n 2 1 37 38 Rev 1.
ConnectX IB InfiniBand Adapter Cards With PCI Express x8 User’s Manual 29 Table 16 - Connector Pin Name and Number to Signal Name Correspondence Connector Pin Number Connector Pin Name IB Port A Signal Name 1 GND GND 2 TXN_2 Tx2n 3 TXP_2 Tx2p 4 GND GND 5 TXN_4 Tx4n 6 TXP_4 Tx4p 7 GND GND 8 ModSelL_Port0 ModSelL 9 ResetL_Port0 ResetL 10 VccRx 11 SCL SCL 12 SDA SDA 13 GND GND 14 RXP_3 Rx3p 15 RXN_3 Rx3n 16 GND GND 17 RXP_1 Rx1p 18 RXN_1 Rx1n 19 GND GN
16 17 18 19 SCL SDA GND RX3p RX3n GND RX1p RX1n GND 8 ResetL VccRx 15 7 ModSelL 20 GND 21 RX2n 22 RX2p 23 GND 24 RX4n 25 RX4p 26 GND 27 ModPrsL 28 IntL 29 VccTx 30 Vcc1 31 LPMode 32 GND 14 13 12 11 1 0 9 8 7 6 5 4 3 2 1 GND 14 6 GND 15 TX2n 33 TX3p TX2p 13 5 TX4p 16 GND 34 TX3n TX4n 12 4 TX4n TX4p 19 18 17 GND 35 GND ModSelL 11 1 0 9 3 GND ResetL 36 TX1p SCL 2 TX2p SDA 1 TX2n GND 37 TX1n RX3p 38 GND GND 3 8 TX1n 3 7 TX1p 3 6 GND 3 5 TX3n
ConnectX IB InfiniBand Adapter Cards With PCI Express x8 User’s Manual 31 Appendix C: Replacing a Tall Bracket With a Short Bracket on HCA Cards This appendix provides instructions on how to remove the tall bracket of a standard Mellanox Technologies HCA card and replace it with a short one. It includes the following sections: • • Removing a bracket Installing a new bracket C.
Figure 11: Adapter card With Bracket Removed C.1.2 Installing the New Bracket 4. Place the bracket over the connector cages and push the bracket onto the card until the screw holes line up. Figure 12: Placing the Bracket Note: Do not force the bracket onto the card. You may have to gently push the LEDs using a small screwdriver to align the LEDs with the holes in the bracket. 5. Take the screws and washers saved from Step 1. Place the screws and washers into the holes and screw them in snug.
ConnectX IB InfiniBand Adapter Cards With PCI Express x8 User’s Manual 33 Appendix D: Avertissements de sécurité d’installation (French) 1. Instructions d’installation Lisez toutes les instructions d’installation avant de brancher le matériel à la source d’alimentation électrique. 2. Température excessive Ce matériel ne doit pas fonctionner dans une zone avec une température ambiante dépassant le maximum recommandé de 55°C (131°F).
7. Codes électriques locaux et nationaux Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux. Rev 1.
ConnectX IB InfiniBand Adapter Cards With PCI Express x8 User’s Manual 35 Appendix E: Installation - Sicherheitshinweise (German) 1. Installationsanleitungen Lesen Sie alle Installationsanleitungen, bevor Sie das Gerät an die Stromversorgung anschließen. 2. Übertemperatur Dieses Gerät sollte nicht in einem Bereich mit einer Umgebungstemperatur über der maximal empfohlenen Temperatur von °C (°F) betrieben werden. Es ist ein Luftstrom von 200 LFM bei maximaler Umgebungstemperatur erforderlich.
7. Regionale und nationale elektrische Bestimmungen Dieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden. Rev 1.
