ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual P/N: MCX313A-BCBT, MCX314A-BCBT Rev 1.7 www.mellanox.
Rev 1.7 ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS.
ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.7 Table of Contents Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Rev 1.7 A.3 Board Mechanical Drawing and Dimensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 A.4 Regulatory Statements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Appendix B Interface Connectors Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 B.1 I2C-compatible Connector Pinout . . . . . . . . . . . . . . . . . . . . . . . . . .
ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.7 List of Tables Table 1: Revision History Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Table 2: Documents List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Table 3: Single and Dual-port 40 Gigabit Ethernet Adapter Cards. . . . . . . . . . . . . . . . . . . . . . . .
Rev 1.7 List of Figures Figure 1: Card Product Label (Example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Figure 2: MCX314A-BCBT Card. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Figure 3: Dual-port Bracket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Figure 4: Single-port Bracket . . . . . . . . . . . . . . . . . . . . . . . . . .
ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.7 Revision History This document was printed on November 14, 2012. Table 1 - Revision History Table Date Rev Comments/Changes November 2012 1.7 • Updated Web links in the following locations: - Section 4.3.1, “Hardware and Software Requirements,” on page 27 - Table 7, “MCX313A-BCBT Specifications Table,” on page 30 - Table 8, “MCX314A-BCBT Specifications Table,” on page 31 October 2012 1.
Rev 1.7 About this Manual This User Manual describes Mellanox Technologies ConnectX®-3 40 Gigabit Ethernet Single and Dual QSFP+ port PCI Express x8 adapter cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documentation. Intended Audience This manual is intended for the installer and user of these cards. The manual assumes basic familiarity with Ethernet networks and architecture specifications.
ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.7 Customers who purchased Mellanox products through a Mellanox approved reseller should first seek assistance through their reseller. Firmware and Software Updates The Mellanox support downloader contains software, firmware and knowledge database information for Mellanox products. Access the data base from the Mellanox Support web page, http://www.mellanox.
Rev 1.7 1 Overview Overview This document is a User Manual for Mellanox Technologies Ethernet cards based on the ConnectX®-3 EN integrated circuit device. The cards described in this manual have the following main features: • IEEE Std. 802.3 compliant • PCI Express 3.0 (1.1 and 2.0 compatible) through an x8 edge connector up to 8GT/s • Single and Dual-port options available - ESD protection on the network connector • Compliant with QSFP+ MSA spec Rev. 1.
ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual 1.2 Rev 1.7 Adapter Cards Covered in this Manual Table 3 lists the single and dual-port 40 Gigabit Ethernet adapter cards. Table 3 - Single and Dual-port 40 Gigabit Ethernet Adapter Cards Ordering Part Number (OPN) PCI Express SERDES Speed Data Transmission Rate/ # of ports MCX313A-BCBT PCIe 3.0 x8 8GT/s 40GigE singleport QSFP MCX314A-BCAT (Legacy product) MCX314A-BCBT 1.3 PCIe 3.0 x8 8GT/s PCIe 3.
Rev 1.7 Overview 1. Installation Instructions Read all installation instructions before connecting the equipment to the power source. 2. Over-temperature This equipment should not be operated in an area with an ambient temperature exceeding the maximum recommended: 55°C (131°F). To guarantee proper air flow, allow at least 8cm (3 inches) of clearance around the ventilation openings. 3.
ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.7 8. Hazardous Radiation Exposure Caution – Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure.
Rev 1.7 Adapter Card Interfaces 2 Adapter Card Interfaces 2.1 I/O Interfaces Each adapter card includes the following interfaces: • QSFP+ port(s) • PCI Express (PCIe) x8 edge connector • I/O panel LEDs • I2C-compatible connector (for debug) Figure 2: MCX314A-BCBT Card The adapter cards include special circuits to protect from ESD shocks to the card/server when plugging copper cables. For dual-port cards, Port 1 connects to connector 1 of the device, while Port 2 connects to connector 2 of the device.
ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.
Rev 1.7 Adapter Card Interfaces 2.1.1 Ethernet Interface The network ports of the ConnectX®-3 adapter cards are compliant with the IEEE 802.3 Ethernet standards listed in Section 1.1.1, “10 and 40 Gigabit Ethernet,” on page 10. Ethernet traffic is transmitted through the cards' QSFP+ connectors. 2.1.2 PCI Express Interface The ConnectX®-3 adapter cards support PCI Express 3.0 (1.1 and 2.0 compatible) throughanx8 edge connector.
ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual 2.2 Rev 1.7 Power All adapter cards receive 12V and 3.3V power from the PCI Express edge connector. All other required power voltages are generated by on-board switch mode regulators. See “Specifications” on page 30. 2.3 Power Level The adapter cards support power levels according to SFF Committee SFF-8436 Specification for (Small Form-factor Pluggable) Transceiver. 2.
Rev 1.7 Adapter Card Interfaces 2.4.3 EEPROM Each board incorporates an EEPROM that is accessible through the I2C-compatible interface. The EEPROM capacity is 4Kb. The EEPROM is used for storing the Vital Product Data (VPD). The PCI VPD (Vital Product Data) layout for each of the adapter cards complies with the format defined in the PCI 3.0 Specification, Appendix I.
ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual Offset (Decimal) Item 148 150 Rev 1.
Rev 1.
ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual 3 Ethernet Adapter Card Installation 3.1 Hardware Requirements Rev 1.7 A system with a PCI Express x8 slot is required for installing the card. 3.2 Installation Instructions Read all installation instructions before connecting the equipment to the power source. To change a tall bracket to a short bracket see Replacing a Tall Bracket With a Short Bracket on page 38.
Rev 1.7 Ethernet Adapter Card Installation Figure 6: PCI Device 5. In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 1003 – this is a valid Mellanox Technologies PCI Device ID. If the PCI device does not have a Mellanox adapter ID, return to Step 4 to check another device.
ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual 3.4 Rev 1.7 Cables and Modules Please refer to “Mellanox Products Approved Cable Lists” at www.mellanox.com/related-docs/ user_manuals/Mellanox_approved_cables.pdf. 3.4.1 Cable Installation 1. 2. 3. All cables can be inserted or removed with the unit powered on. To insert a cable, press the connector into the port receptacle until the connector is firmly seated.
Rev 1.7 Driver Software and Firmware 4 Driver Software and Firmware 4.1 Driver Software 4.1.1 Linux For Linux, download and install the latest MLNX_EN driver software package available via the Mellanox web site at: http://www.mellanox.com => Products => Software/Drivers => Ethernet SW/Drivers => ConnectX®-3 EN 10/40GigE Linux Driver => Download. Follow the installation instructions included in the download package (also available from the download page).
ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.7 www.mellanox.comTo ensure that communication has been established follow the instructions below.
Rev 1.7 Driver Software and Firmware Step 3: Select the Information tab from the Properties sheet (example). To save this information for debug purposes, click Save To File and provide the output file name. 4.2 FlexBoot FlexBoot enables remote boot over Ethernet, Boot over Ethernet (BoE) or Boot over iSCSI (BoiSCSI). This technology is based on the Preboot Execution Environment (PXE) standard specification, and FlexBoot software is based on the open source iPXE project (see www.ipxe.org).
ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.7 ConnectX®-3 EN with RoCE based network management is the same as that for any Ethernet and DCB-based network management, eliminating the need for IT managers to learn new technologies. 4.3.1 Hardware and Software Requirements Software: • Mellanox OFED 1.5.1 or WinOF 2.1.2 or later. To download these packages go to http:// www.mellanox.com => Products => Software/Drivers => Ethernet SW/Driver.
Rev 1.7 Driver Software and Firmware > flint -d /dev/mst/mt4099_pci_cr0 q Image type: ConnectX FW Version: 2.9.4000 Device ID: 4099 Chip Revision: 0 Description: Node Port1 Port2 Sys image GUIDs: 000002c900000200 000002c900000201 000002c900000202 000002c900000203 MACs: 000002c90200 000002c90201 Board ID: (MT_1020110019) VSD: PSID: MT_1020110019 2. Compare the programmed firmware version with the latest available. a. Go to Mellanox’s web site: http://www.mellanox.com/supportdownloader. See Figure 8. b.
ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.7 > flint -d /dev/mst/mt4099_pci_cr0 -i burn d. Reboot the computer. e. Enter: mst start. f. Verify that the card firmware was updated successfully. > flint -d /dev/mst/mt4099_pci_cr0 Image type: ConnectX FW Version: 2.9.4100 Device ID: 4099 ...
Rev 1.7 Appendix A: Specifications A.1 MCX313A-BCBT Specifications Table 7 - MCX313A-BCBT Specifications Table Size: 2.12in. x5.6 in. (53.59mm x 142.25 mm) Physical Connector: QSFP Ethernet: 40GBASE-CR4 /-SR4/-LR4; also supports the following when used with a QSA: 10GBASE-CR/-SR/LR; 1000BASE-T/-SR/-LR Protocol Support Data Rate: 1/10/40Gb/s – Ethernet PCI Express Gen3: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1 compatible) Voltage: 12V, 3.3V Typ Power: Passive Cables 6.54W Max Power: Passive Cables 8.
ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual A.2 Rev 1.7 MCX314A-BCBT Specifications Table 8 - MCX314A-BCBT Specifications Table Size: 2.71n. x5.6 in. (68.90mm x 142.25 mm) Physical Connector: QSFP Ethernet: 40GBASE-CR4 /-SR4/-LR4; also supports the following when used with a QSA: 10GBASE-CR/-SR/LR; 1000BASE-T/-SR/-LR Protocol Support Data Rate: 1/10/40Gb/s – Ethernet PCI Express Gen3: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1 compatible) Voltage: 12V, 3.
Rev 1.7 A.3 Board Mechanical Drawing and Dimensions All dimensions are in millimeters. All the mechanical tolerances are +/- 0.1mm Figure 9: Mechanical Drawing of the Single-port MCX313A-BCBT Adapter Card J1 - I2C Connector Flash Jumper 142.24 40.5 49.09 22.61 53.59 45.95 32 Mellanox Technologies 12.
ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.7 Figure 10: Mechanical Drawing of the Dual-port MCX314A-BCBT Adapter Card J1 - I2C Connector Flash Jumper 142.25 64.4 56.15 50.00 68.90 45.96 A.4 12.75 Regulatory Statements For regulatory statements for all ConnectX®-3 cards please refer to: http://www.mellanox.com/related-docs/user_manuals/Regulatory_and_Compliance_Guide.
Rev 1.7 Appendix B: Interface Connectors Pinout B.1 I2C-compatible Connector Pinout Figure 11: Compatible Connector Plug and Pinout Table 9 - I2C-compatible Connector Pinout B.2 Connector Pin Number Signal Name 1 SPSDA 2 SPSCL 3 GND 4 NC 5 NC PCI Express x8 Connector Pinout The adapter cards use a standard PCI Express x8 edge connector and the PCI Express x8 standard pinout according to the PCI Express 3.0 specification.
ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.
Rev 1.7 B.
ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.
Rev 1.7 Appendix C: Replacing a Tall Bracket With a Short Bracket This section provides instructions on how to remove the tall bracket of a standard Mellanox Technologies adapter card and replace it with a short one. C.1 Replacing a Bracket To replace the bracket you will need the following parts: • the new bracket of the proper height • one new square gasket for each of the ports • the 2 screws saved from the removal of the bracket • the 2 fiber washers saved from the removal of the bracket C.
ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual C.3 Rev 1.7 Installing the New Bracket 1. 2. 3. Remove the paper to expose the adhesive on the gasket. Place the gasket for each port onto the new bracket. Make sure to correctly align the gasket with the hole in the bracket. If the old gaskets are still on the card, remove them before installing the new bracket. Make sure that only one gasket per port is used.
Rev 1.7 Figure 17: Placing the Bracket on the Card 3. 4. 40 Make sure that the LEDs are aligned onto the bracket holes. Use a torque driver to apply up to 2 lbs-in torque on the screws.
ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.7 Appendix D: Avertissements de sécurité d’installation (Warnings in French) 1. Instructions d’installation Lisez toutes les instructions d’installation avant de brancher le matériel à la source d’alimentation électrique. 2. Température excessive Ce matériel ne doit pas fonctionner dans une zone avec une température ambiante dépassant le maximum recommandé de 55°C (131°F).
Rev 1.7 7. Codes électriques locaux et nationaux Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux. 8. Exposition au rayonnement grave Mise en garde – l'utilisation de commandes ou de réglages ou l'exécution de procédures autres que ce qui est spécifié dans les présentes peut engendrer une exposition au rayonnement grave.
ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.7 Appendix E: Sicherheitshinweise (Warnings in German) 1. Installationsanleitungen Lesen Sie alle Installationsanleitungen, bevor Sie das Gerät an die Stromversorgung anschließen. 2. Übertemperatur Dieses Gerät sollte nicht in einem Bereich mit einer Umgebungstemperatur über der maximal empfohlenen Temperatur von 55°C (131°F) betrieben werden. Es ist ein Luftstrom von 200 LFM bei maximaler Umgebungstemperatur erforderlich.
Rev 1.7 This equipment should be installed in compliance with local and national electrical codes. 8. Strahlenkontak Achtung – Nutzung von Steuerungen oder Einstellungen oder Ausführung von Prozeduren, die hier nicht spezifiziert sind, kann zu gefährlichem Strahlenkontakt führen..
ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.7 Appendix F: Advertencias de seguridad para la instalación (Warnings in Spanish) 1. Instrucciones de instalación Antes de conectar el equipo a la fuente de alimentación, leer todas las instrucciones de instalación. 2. Sobrecalentamiento No se debe utilizar el equipo en un área con una temperatura ambiente superior a la máxima recomendada: 55°C(131°F).
Rev 1.7 7. Códigos eléctricos locales y nacionales Este equipo se debe instalar conforme a los códigos eléctricos locales y nacionales. 8. Exposición a niveles de radiación peligrosos Precaución: el uso de controles o ajustes o la realización de procedimientos distintos de los que aquí se especifican podrían causar exposición a niveles de radiación peligrosos.
