Third party information brought to you courtesy of Dell. InfiniScale® III M2401G 20Gb/s InfiniBand Blade Switch I/O Module for PowerEdge M-Series User Manual PN: M2401G Rev 1.
Copyright 2008. Mellanox Technologies, Inc. All Rights Reserved. Mellanox Technologies ConnectX, InfiniBlast, InfiniBridge, InfiniHost, InfiniRISC, InfiniScale, and InfiniPCI are registered trademarks of Mellanox Technologies, Ltd. Virtual Protocol Interconnect is a trademark of Mellanox Technologies, Ltd. Dell and the Dell logo are trademarks of Dell Inc. All other marks and names mentioned herein may be trademarks of their respective companies. Third party information brought to you courtesy of Dell.
M2401G 20Gb/s InfiniBand Blade Switch I/O Module User Manual 3 Contents Contents 3 About this Manual 4 Revision History 5 Chapter 1 Overview 6 1.1 1.2 7 7 Chapter 2 Chapter 3 Chapter 4 InfiniBand Connectors Switch Status Lights Installation and Basic Operation 9 2.1 2.2 Installation Safety Warnings Mechanical Installation 2.2.1 Removing an Old Switch From the Chassis 2.2.2 Removing a Blank Cover Plate From the Chassis 2.2.3 Installing the Mellanox M2401G Switch Into the Dell Chassis 2.
About this Manual This manual describes the installation and basic use of the InfiniScale® III M2401G 20Gb/s InfiniBand blade switch I/O module for PowerEdge M-Series. Intended Audience This manual is intended for users and system administrators responsible for installing and setting up the switch platforms listed above. The manual assumes familiarity with the InfiniBand® Architecture Specification.
M2401G 20Gb/s InfiniBand Blade Switch I/O Module User Manual 5 Revision History Table 2 - Revision History Table Date Revision Description Mar. 2009 Rev 1.8 Modified product description and document title Mar. 2009 Rev 1.7 Added Dell flat black Logo Dec. 2008 Rev 1.6 Minor fixes to Rev 1.5 Dec. 2008 Rev 1.5 Minor fixes to Rev 1.4 Added Instructions for Reprogramming Over the InfiniBand Network Nov. 2008 Rev 1.
1 Overview The InfiniScale® III M2401G 20Gb/s InfiniBand blade switch I/O module for PowerEdge MSeries provides a high bandwidth, low latency, low power fabric for Enterprise Data Center and High-Performance Computing environments. Based on the third generation InfiniScale® III InfiniBand switch device, the platform delivers an ideal balance between cost and performance.
M2401G 20Gb/s InfiniBand Blade Switch I/O Module User Manual 7 1.1 InfiniBand Connectors InfiniBand connectivity has 8 microGiGa connectors through the front panel. The remaining 16 interfaces are through the AirMax Midplane Connector out the back of the switch. Figure 1 shows the front 8 ports. Each of the InfiniBand ports has two LEDs located next to the connector. The green LED, when lit, indicates that a valid physical connection to the other system (switch or HCA port) exists.
Table 3 - IOM states and LED configurations: LED Switch Status Indication Green Status OFF OFF OFF ON OFF Boot in Progress Switch not ready ON BLINKING BLUE The CMC is identifying the newly installed switch ON ON BLUE Switch is on and operating Normally ON ON or BLINKING AMBER Fault in System Self-diagnosed OFF ON or BLINKING AMBER Fault in System CMC-detected Blinking Blue Rev 1.
M2401G 20Gb/s InfiniBand Blade Switch I/O Module User Manual 9 2 Installation and Basic Operation Installation and initialization of the switch platform are straightforward processes, requiring attention to the normal mechanical, power, and thermal precautions for rack-mounted equipment.
2.1 Installation Safety Warnings 1. Installation Instructions Read all installation instructions before connecting the equipment to the power source. 2. Over-temperature This equipment should not be operated in an area with an ambient temperature exceeding the maximum recommended: °C (°F). Moreover, to guarantee proper air flow, allow at least 8cm (3 inches) of clearance around the ventilation openings. 3. Stacking the Chassis The chassis should not be stacked on any other equipment.
M2401G 20Gb/s InfiniBand Blade Switch I/O Module User Manual 11 8. Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regulations. 9. Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical codes. 2.2 Mechanical Installation These switches are hot pluggable. It is not necessary to power down the Dell Chassis to install a new switch or to replace an old switch with a new one.
3. Unlock the switch from the chassis by pushing the red latch release button. 4. Pull the locking arm down to a position perpendicular to the front of the chassis. 5. Pull the switch out of the chassis using the locking arm. 6. Install either a new switch or a blank IOM within one minute. 2.2.2 Removing a Blank Cover Plate From the Chassis 1. Remove any locking cables or screws that secure the blank cover plate into the chassis. 2. Unlock the blank cover plate by pushing the red latch release button.
M2401G 20Gb/s InfiniBand Blade Switch I/O Module User Manual 13 2.4 InfiniBand Cable Installation All cables can be inserted or removed with the unit power on. To insert a cable, press the connector into the port receptacle until the connector is firmly seated. The green LED indicator below each port will light when the physical connection is established (that is, when the unit is powered on and a cable is plugged into the port with a functioning port plugged into the other end of the connector).
3 Cluster Management and Firmware 3.1 Network Management and Clustering Software Download and install, on all nodes, the Mellanox OpenFabric software package for Linux, Windows, or other operating systems from the Mellanox software website: http://www.mellanox.com > Downloads > InfiniBand SW/Drivers This software package provides connectivity for server and storage systems utilizing High Performance Computing (HPC) or enterprise data center (EDC) applications across an InfiniBand fabric.
M2401G 20Gb/s InfiniBand Blade Switch I/O Module User Manual 15 Reports regarding cluster links will use direct routes. Loading IBDM from: /usr/local/lib/ibdm1.2 -I- Using port 1 as the local port. -I- Discovering ... 3 nodes (2 Switches & 1 CA-s) discovered. -I---------------------------------------------------I- Bad Guids/LIDs Info -I---------------------------------------------------I- skip option set.
II. Using the ibnetdiscover tool: Run: [root@mymach]# ibnetdiscover | grep 00000006660abcd0 | grep -w Switch Switch 24 "S-00000006660abcd0" "MT47396 Infiniscale-III Mellanox Technologies" base port 0 lid 17 lmc 0 Note: The resulting LID is given as a decimal number. 4. Run mlxburn with the LID retrieved in step #3 above to perform the In-Band burning operation. Burn the InfiniScale III switch: [root@mymach]> mlxburn -dev /dev/mst/SW_MT47396_lid-0x0011 -fw ./IS3FW.
M2401G 20Gb/s InfiniBand Blade Switch I/O Module User Manual 17 4 Troubleshooting As soon as a switch is plugged in make sure that the power LED comes on. The power LED for the switch does not come on: 1. Check that the chassis has power. 2. Remove and reinstall the switch. The status LED for the switch is blinking amber: 1. remove the switch from the chassis and re-insert it (verify that the switch is all the way in the chassis and the lever is firmly closed). 2.
Appendix A: Specifications Table 4 - M2401G Specification Data Physical Power and Environmental H x W x D: 1.16 x 10.04 inches 272.50 x 29.45 x 255.20 mm Weight: 1.5 Kg fully configured Mounting: Vertically mounted rack SerDes Speeds 10, 20 Gb/s per port Connectors: microGiGaCN 8 external CX4 connectors 4x4 internal backplane connectors Protocol Support Maximum Power: Dissipated Power: Power through connector: Temperature: Humidity: Altitude: Shock: Vibration: 50W 33.5W 1.
M2401G 20Gb/s InfiniBand Blade Switch I/O Module User Manual 19 Appendix B: Mechanical Drawing Figure 4 Mechanical Drawing of the Switch Mellanox Technologies Rev 1.
Appendix C: EMC Certification Statements C.1: FCC Statements (USA) Class A Statements: § 15.21 Statement Warning! Changes or modifications to this equipment not expressly approved by the party responsible for compliance (Mellanox Technologies) could void the user's authority to operate the equipment. §15.105(a) Statement NOTE: This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant to Part 15 of the FCC Rules.
M2401G 20Gb/s InfiniBand Blade Switch I/O Module User Manual 21 C.1.3 VCCI Statements (Japan) Class A Statement: (Translation - "This is a Class A product based on the standard of the Voluntary Control Council for Interference by Information Technology Equipment (VCCI). If this equipment is used in a domestic environment, radio interference may occur, in which case the user may be required to take corrective actions.") C.1.
. Appendix D: InfiniBand Connector Pinout Figure 5: InfiniBand Connector Pinout Conn 1X S10 S9 S12 S11 S14 S13 S16 IBtxIp(3) IBtxOn(3) IBtxIn(3) IBtxOp(2) IBtxIp(2) IBtxOn(2) IBtxIn(2) IBtxOp(1) IBtxIp(1) IBtxOn(1) IBtxIn(1) IBtxOp(0) IBtxIp(0) IBtxOn(0) IBtxIn(0) H1 G1 G2 G3 G4 G5 G6 G7 G8 G9 H2 GND GND GND GND GND GND GND Sense-3.3V Vcc GND GND S15 IBtxOp(3) Table 5 - Connector Pin to Signal Name Correspondence Rev 1.
M2401G 20Gb/s InfiniBand Blade Switch I/O Module User Manual 23 Table 5 - Connector Pin to Signal Name Correspondence Connector Pin Number Connector Pin Name Signal Name S12 IBtxOp(2) IB_Px_T3_P S13 IBtxOn(1) IB_Px_T2_N S14 IBtxOp(1) IB_Px_T2_P S15 IBtxOn(0) IB_Px_T1_N S16 IBtxOp(0) IB_Px_T1_P G1-G6, G9, H1-H2 Signal Ground G7 Sense-3.3V G8 Vcc Px in the signal name signifies Port # x. This switch has 8 external ports.
Appendix E: Avertissements de sécurité d’installation 1. Instructions d’installation Lisez toutes les instructions d’installation avant de brancher le matériel à la source d’alimentation électrique. 2. Température excessive Ce matériel ne doit pas fonctionner dans une zone avec une température ambiante dépassant le maximum recommandé de 45°C (113°F). En outre, pour garantir un bon écoulement de l’air, laissez au moins 8 cm (3 pouces) d’espace libre autour des ouvertures de ventilation. 3.
M2401G 20Gb/s InfiniBand Blade Switch I/O Module User Manual 25 8. Elimination du matériel L’élimination de ce matériel doit s’effectuer dans le respect de toutes les législations et réglementations nationales en vigueur. 9. Codes électriques locaux et nationaux Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux. Mellanox Technologies Rev 1.
Appendix F: Installation - Sicherheitshinweise 1. Installationsanleitungen Lesen Sie alle Installationsanleitungen, bevor Sie das Gerät an die Stromversorgung anschließen. 2. Übertemperatur Dieses Gerät sollte nicht in einem Bereich mit einer Umgebungstemperatur über der maximal empfohlenen Temperatur von 45°C (113°F) betrieben werden. Außerdem sollten mindestens 8 cm (3 in.) Freiraum um die Belüftungsöffnungen sein, um einen einwandfreien Luftstrom zu gewährleisten. 3.
M2401G 20Gb/s InfiniBand Blade Switch I/O Module User Manual 27 8. Geräteentsorgung Die Entsorgung dieses Geräts sollte unter Beachtung aller nationalen Gesetze Bestimmungen erfolgen. 9. Regionale und nationale elektrische Bestimmungen Dieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden. Mellanox Technologies Rev 1.
