ConnectX® IB HCA Adapter Card with QSFP Connectors User Manual P/N: MHRH19-XTC, MHRH19-XSC, MHRH29-XSC, MHRH29-XTC, MHQH19-XTC, MHQH19-XSC, MHQH29-XSC, MHQH29-XTC Rev 1.5 www.mellanox.
Rev 1.5 NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT.
Rev 1.5 Table of Contents Table of Contents 3 List of Figures 5 List of Tables 6 Revision History 7 About this Manual 8 Intended Audience Related Documentation Online Resources Document Conventions Technical Support Firmware and Software Updates Chapter 1 Overview 1.1 1.2 1.3 1.4 Chapter 2 8 8 8 8 8 9 10 Adapter Cards Covered in this Manual Mellanox Part Numbering Legend Finding the and Serial Number on the Adapter Cards Safety Warnings 11 13 14 14 Adapter Card Interfaces 16 2.1 2.2 2.
Rev 1.5 C.
Rev 1.
Rev 1.
Rev 1.5 Revision History This document was printed on 2/14/11. Table 1 - Revision History Table Date Rev Comments/Changes February 2011 1.5 Added Set up section. August 2009 1.4 Added MH[QR]H 19 Short cards Added gaskets to bracket installation moved Bracket installation from Appendix to Chapter 5 December 2008 1.3 Fixed links to website Added single port cards August 2008 1.05 Added French Translation of Safety Warnings Removed Watermark April 2008 0.
Rev 1.5 About this Manual This User Manual describes Mellanox Technologies MH[RQ]H[12]9-X[ST]C ConnectX IB PCI Express x8 HCA Adapter cards. HCAIt provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documentation. Intended Audience This manual is intended for the installer and user of these cards. The manual assumes basic familiarity with InfiniBand® and networks and architecture specifications.
Rev 1.5 • Tel: +1.408.916.0055 Customers who purchased Mellanox M-1 Global Support Services, please see your contract for details regarding Technical Support. Customers who purchased Mellanox products through a Mellanox approved reseller should first seek assistance through their reseller. Firmware and Software Updates The Mellanox support downloader contains software, firmware and knowledge database information for Mellanox products. Access the data base from the Mellanox Support Web page, http://www.
Rev 1.5 1 Overview Overview This document is a User Manual for Mellanox Technologies host channel adapter (HCA) cards based on MT25408 , ConnectX® IBthe ConnectXintegrated circuit device.
Rev 1.5 1.1 Adapter Cards Covered in this Manual Table 3 lists the InfiniBandHCA cards described in this manual.
Rev 1.5 Overview Table 3 - HCA Cards List Ordering Part Number (OPN) MHRH29-XTC IB Port Speed 20Gb/s (DDR) Dual port Onboard Memory Size Short / Tall Brack et No Mem Tall Figure 1: MH[RQ]H29-XTC Card MHRH29-XSC MHQH29-XTC HCA Card with Brackets Short 40Gb/s (QDR) Dual port No Mem MHQH29-XSC Tall Short Short Bracket These cards are RoHS-R5 Compliant.
Rev 1.5 1.2 Mellanox Part Numbering Legend Table 4 describes the Mellanox Technologies adapter cards part numbering legend.
Rev 1.5 1.3 Overview Finding the and Serial Number on the Adapter Cards All Mellanox adapter cards have a label on the printed side of the adapter card that has the card serial number and the card GUID for InfiniBand protocol . Figure 2: Card Product Label S/N:MT0744X00012 REV: X1 P/N:MHQH29-XTC GUID: 0002C90300025 B06 Made in IL 1.4 Safety Warnings For safety warnings in French see “Avertissements de sécurité d’installation (Warnings in French)” on page 47.
Rev 1.5 5. Equipment Installation This equipment should be installed, replaced, or serviced only by trained and qualified personnel. 6. Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regulations. 7. Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical codes.
Rev 1.5 Adapter Card Interfaces 2 Adapter Card Interfaces 2.1 I/O Interfaces Each HCA Adapter card includes the following interfaces: • QSFP ports • PCI Express x8 edge connector • I/O panel LEDs • I2C compatible connector (for debug) For dual port cards, port 1 connects to connector 1 of the device, while port 2 connects to connector 2 of the device. Figure 3: Port Numbering Port 1 Port 2 2.1.
Rev 1.5 2.1.3 LED Assignment The board has I/O LEDs located on the I/O panel- 2 LEDs per port. The green LED, when lit, indicates that the driver is running and a valid physical connection between nodes exists. If the green LED is blinking, it indicates a problem with the physical link. The yellow LED when lit, indicates a valid data activity link, this is the logical link. The yellow LED lights up when the network is discovered over the physical link.
Rev 1.5 Adapter Card Interfaces 2.2 Power 2.3 Memory The adapter cards support multiple memory devices through the PCI Flash, and I2C. 2.3.1 System Memory The HCA Adapter card utilizes the PCI Express interface to store and access connection information and packet data on the system memory. 2.3.2 Flash Each of the HCA Adapter cards include one 2MB SPI Flash device (M25P80 device by ST Microelectronics) accessible via the Flash interface of the ConnectX-IB device.
Rev 1.5 2.4.
Rev 1.
Rev 1.5 2.4.
Rev 1.
Rev 1.5 3 Driver Software and Firmware 3.1 Driver Software 3.1.1 Linux For Linux, download and install the latest OpenFabrics Enterprise Distribution (OFED) software package available via the Mellanox Web site at: http://www.mellanox.com => Support > Download Center. Follow the installation instructions included in the download package. 3.1.2 Windows For Windows, download and install the latest Mellanox WinOF VPI for Windows software package available via the Mellanox Web site at: http://www.mellanox.
Rev 1.5 Driver Software and Firmware 3.3.2 Installation For installation instructions visit: www.mellanox.com => Support => VPI SW/Driver. 3.3.2.1 Kernel Installation: Use Red Hat Package Manager (RPM) to install the Kernel RPM files: - Install the required RPMs, for example, run: # rpm --force -ivh *.rpm - Modify the boot loader configuration file if needed (e.g., edit /etc/grub.conf) - Reboot the machine with the new kernel # reboot 3.3.2.
Rev 1.5 the latest found on the Mellanox site, if not update to the latest version found on the Mellanox website. Firmware can be updated on the stand alone single card using the flint tool of the Mellanox Firmware Tools (MFT) package. This package is available for download, along with its user manual, from the Mellanox Firmware Tools page. See http://www.mellanox.com => Support > Download Center. A firmware binaries table lists a binary file HCA Adapter per adaptercard.
Rev 1.5 HCA InfiniBand Card Installation 4 HCA InfiniBand Card Installation 4.1 Hardware and Software Requirements Before installing the HCA adapter card, please make sure that the system meets the hardware and software requirements listed inTable 9. Refer to Chapter 3,“Driver Software and Firmware” on page 23 for download and installation instructions.
Rev 1.5 The basic steps to embed Mellanox cards in your computer are: 1. Identify the card in your system. 2. Verify the computer is recognizing the new adapter. 3. Install the adapter drivers. 4. Update the adapter firmware if needed. The configuration of the card is dependant upon your choice to use InfiniBand or Ethernet. 4.3.1 Identify the Card in Your System For instructions to identify your adapter card using specific tools see http://www.mellanox.com/content/pages.
Rev 1.5 HCA InfiniBand Card Installation Figure 8: Hardware Devices 3. Select a PCI Device / InfiniBand Controller entry. 4. Right-click. 5. Select “Properties to display the PCI Device Properties” window. 6. Click the Details tab and select Device Instance Id (Windows 2003) or Hardware Ids (Windows 2008/R2) from the Property pull-down menu.
Rev 1.5 Figure 9: PCI Device 7. In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_6368”: VEN is equal to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 0x6368 – this is a valid Mellanox Technologies PCI Device ID. The list of Mellanox Technologies PCI Device IDs can be found in the PCI ID repository at http://pci-ids.ucw.cz/read/PC/15b3. 8.
Rev 1.5 HCA InfiniBand Card Installation 4.3.1.2 Linux Get the device location on the PCI bus by running lspci and locating lines with the string “Mellanox Technologies”: > lspci | grep Mellanox 2:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0) Make sure that either the MLNX_OFED driver or the MLNX_EN driver is loaded and configured. Check the link status First check the network interface name by running the “ifconfig –a” command.
Rev 1.5 Rate: 10 Base lid: 6 LMC: 0 SM lid: 3 Capability mask: 0x0251086a Port GUID: 0x0002c903000c8711 Link layer: IB Check the OFED version To get the version of the running Mellanox OFED/BXOFED, run the following command: Host# ofed_info | head -1 BXOFED-1.5.1-1.3.7-rc19: Troubleshooting MLNX_OFED Installation For troubleshooting driver installation, please check Mellanox OFED driver user manual: http://www.mellanox.com => Support > Adapter IB/VPI SW.
Rev 1.5 HCA InfiniBand Card Installation The “Usage: openibd {start|stop|restart|status}” command to modify this file and thereby control the drivers. Ethernet Driver Usage and Configuration To assign an IP address to the interface run: #> ifconfig eth where 'n' is the OS assigned interface number. • To check driver and device information run: #> ethtool -i eth Example: #> ethtool -i eth2 driver: mlx4_en (MT_04A0140005) version: 1.5.1 (March 2010) firmware-version: 2.7.
Rev 1.5 When installing cables make sure that the latches engage. Always install and remove cables by pushing or pulling the cable and connector in a straight line with the card. Care should be taken not to impede the air exhaust flow through the ventilation holes. Cable lengths should be used which allow for routing horizontally around to the side of the chassis before bending upward or downward in the rack.
Rev 1.5 Appendix A: Specifications A.1 MHRH19-X[ST]C Specifications Table 10 - Specifications for MHRH19-X[ST]C Physical Power and Environmental Size: 2.71in. x6.60in. (68.90mm x 167.65mm) Voltage: Typ Power: Air Flow: QSFP 20Gb/s Connector: 200LFMa InfiniBand (Copper and optical) Max power per port 2.0 W. Cable power budget class 2 Maximum Power: Temperature: Protocol Support InfiniBand: QoS: RDMA Support: Data Rate: PCI Express 12V, 3.
Rev 1.5 A.2 MHQH19-X[ST]C Specifications Table 11 - Specifications for MHQH19-X[ST]C Physical Power and Environmental Size: 2.71in. x6.60in. (68.90mm x 167.65mm) Air Flow: Voltage: Typ Power: 200LFMa QSFP 40Gb/s Connector: InfiniBand (Copper and optical) Max power per port 2.0 W. Cable power budget class 2 Maximum Power: Temperature: Protocol Support IBTA v1.2, Auto-Negotiationb (20Gb/s, 5Gb/s) or (10Gb/s, 2.5Gb/ s) QoS: 4 InfiniBand Virtual Lanes for each port Data Rate: PCI Express 8.
Rev 1.5 Table 12 - Specifications for MHRH29-XSC/-XTC Physical Power and Environmental Protocol Support Regulatory InfiniBand: IBTA v1.2, Auto-Negotiationb (20Gb/s, 5Gb/s) or (10Gb/s, 2.5Gb/ s) QoS: 8 InfiniBand Virtual Lanes for each port RDMA Support: Data Rate: PCI Express EMC: Yes, All Ports DDR 2.0 SERDES @ 5.0 GT/s Safety: Environmental: RoHS: FCC 47 CFR part 15:2006, subpart B, class A ICES-003:2004 Issue 4, class A VCCI V-3/2007.
Rev 1.5 A.4 MHQH29-X[ST]C Specifications Table 13 - Specifications for MHQH29-XSC/-XTC Physical Power and Environmental Size: 2.71in. x6.60in. (68.90mm x 167.65mm) Air Flow: Voltage: Typ Power: 200LFMa QSFP 40Gb/s Connector: InfiniBand (Copper and optical) Max power per port 2.0 W. Cable power budget class 2 Maximum Power: Temperature: Protocol Support InfiniBand: QoS: RDMA Support: Data Rate: PCI Express 12V, 3.3V 9W for passive cables only 11W for active optic modules 12.
Rev 1.5 All dimensions are in millimeters. All the mechanical tolerances are +/-0.1mm Figure 10: Schematic of the Dual Port Adapter Card MH[RQ]H29B-X[ST]R J1 – I2C Connector J3 – Flash Jumper 167.65 U2 15.00 L12 J3 J9 U10 68.40 56.15 R25 L14 U20 U21 3.65 Q4 Q3 U1 J6 64.40 56.97 J4 J5 J8 U3 Q2 Q1 J1 J2 1.90 33.35 43.17 57.15 96.30 A.6 EMC Certification Statements the approved certification status per HCAcard in different regions of the world.
Rev 1.
Rev 1.5 A.6.1 FCC Statements (USA) Class A Statements: § 15.19(a)(4) This device complies with Part 15 of the FCC Rules. Operation is subject to the following two conditions: 1. This device may not cause harmful interference, and 2. This device must accept any interference received, including interference that may cause undesired operation. § 15.
Rev 1.5 A.6.4 VCCI Statements (Japan) Class A Statement: (Translation - "This is a Class A product based on the standard of the Voluntary Control Council for Interference by Information Technology Equipment (VCCI). If this equipment is used in a domestic environment, radio interference may occur, in which case the user may be required to take corrective actions.") A.6.
Rev 1.5 Appendix B: Interface Connectors Pinout B.1 I2C-Compatible Connector Pinout Figure 11: Compatible Connector Plug and Pinout Table 15 - I2C-Compatible Connector Connector Pin Number Signal Name 1 SPSDA 2 SPSCL 3 GND 4 NC 5 NC B.2 4 3 2 1 5 5 1 2 3 4 PCI Express x8 Connector Pinout These cards use a standard PCI Express x8 edge connector and the PCI Express x8 standard pinout according to the PCI Express 2.0 specification.
Rev 1.5 B.3 PCI Express Connector Pinout B.4 QSFP Connector Pinout Figure 12: Connector and Cage Views Top 18.
Rev 1.
Rev 1.5 Appendix C: Replacing a Tall Bracket With a Short Bracket C.1 Replacing a Bracket To replace the bracket you will need the following parts: • the new bracket of the proper height • one new square gasket for each of the ports • the 2 screws saved from the removal of the bracket • the 2 fiber washers saved from the removal of the bracket C.2 Remove the Existing Bracket from the Adapter Card Figure 13: Remove the Bracket Screws Gaskets LEDs 1. Remove the two screws holding the bracket in place.
Rev 1.5 Figure 14: Gasket Installation 4. Place the bracket onto the card until the screw holes line up. Do not force the bracket onto the card. You may have to gently push the LEDs using a small screwdriver to align the LEDs with the holes in the bracket. 5. Screw on the bracket using the screws and washers saved from the procedure above step 1. Figure 15: Placing the Bracket on the Card Gaskets in place on the bracket. 6. Make sure that the LEDs are aligned onto the bracket holes. 7.
Rev 1.5 Appendix D: Avertissements de sécurité d’installation (Warnings in French) 1. Instructions d’installation Lisez toutes les instructions d’installation avant de brancher le matériel à la source d’alimentation électrique. 2. Température excessive Ce matériel ne doit pas fonctionner dans une zone avec une température ambiante dépassant le maximum recommandé de 55°C (131°F). Un flux d’air de 200LFM à cette température ambiante maximale est nécessaire.
Rev 1.5 Appendix E: Sicherheitshinweise (Warnings in German) 1. Installationsanleitungen Lesen Sie alle Installationsanleitungen, bevor Sie das Gerät an die Stromversorgung anschließen. 2. Übertemperatur Dieses Gerät sollte nicht in einem Bereich mit einer Umgebungstemperatur über der maximal empfohlenen Temperatur von 55°C (131°F) betrieben werden. Es ist ein Luftstrom von 200 LFM bei maximaler Umgebungstemperatur erforderlich. Außerdem sollten mindestens 8 cm (3 in.
Rev 1.5 Appendix F: Advertencias de seguridad para la instalación (Warnings in Spanish) 1. Instrucciones de instalación Antes de conectar el equipo a la fuente de alimentación, leer todas las instrucciones de instalación. 2. Sobrecalentamiento No se debe utilizar el equipo en un área con una temperatura ambiente superior a la máxima recomendada: 55°C(131°F).
Rev 1.5 Códigos eléctricos locales y nacionales Este equipo se debe instalar conforme a los códigos eléctricos locales y nacionales.
