Third party information brought to you courtesy of Dell. InfiniScale® IV 16+16 Port 40Gb/s InfiniBand Switch for Dell PowerEdge M1000e-series Blade Enclosures User Manual PN: M3601Q Rev 1.
Copyright 2009. Mellanox Technologies, Inc. All Rights Reserved. Mellanox Technologies ConnectX, InfiniBlast, InfiniBridge, InfiniHost, InfiniRISC, InfiniScale, and InfiniPCI are registered trademarks of Mellanox Technologies, Ltd. Virtual Protocol Interconnect is a trademark of Mellanox Technologies, Ltd. Dell and the Dell logo are trademarks of Dell Inc. All other marks and names mentioned herein may be trademarks of their respective companies.
M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual 3 Contents Contents 3 About this Manual 4 Revision History 5 Chapter 1 Overview 6 1.1 1.2 7 8 Chapter 2 Chapter 3 Chapter 4 InfiniBand Connectors Switch Status Lights Installation and Basic Operation 9 2.1 2.2 Installation Safety Warnings Mechanical Installation 2.2.1 Removing an Old Switch From the Chassis 2.2.2 Removing an I/O Module Blank From the Chassis 2.2.3 Installing the Mellanox M3601Q Switch Into the Dell Chassis 2.
M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual 4 About this Manual This manual describes the installation and basic use of the Mellanox M3601Q 16+16 Port 40Gb/s InfiniBand Switch for Dell PowerEdge M1000e-series Blade Enclosures. Intended Audience This manual is intended for users and system administrators responsible for installing and setting up the switch platforms listed above. The manual assumes familiarity with the InfiniBand® Architecture Specification.
M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual 5 Revision History Table 2 - Revision History Table Date Revision Description March 2009 1.1 Added Dell Logo November 2008 1.0 Initial Release Mellanox Technologies Rev 1.
M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual 6 1 Overview The Mellanox M3601Q 40Gb/s InfiniBand Switch Blade for Dell PowerEdge M1000e-series Blade Enclosures provides a high bandwidth, low latency fabric for Enterprise Data Centers, High-Performance Computing and Embedded environments. Based on the fourth generation InfiniScale® IV InfiniBand switch device, this I/O module (IOM) delivers up to 40Gb/s full bisectional bandwidth per port.
Figure 1: Switch Front Panel and Locking Mechanism Locking Mechanism Switch Status Lights 1.1 InfiniBand Connectors InfiniBand connectivity has 16 QSFP connectors through the front panel. The remaining 16 interfaces are through the AirMax Midplane Connector out the back of the switch. Figure 1 shows the front 16 ports. Each of the InfiniBand ports has two LEDs located next to the connector.
M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual 8 Figure 2: Physical and Logical Link Indication LEDs LED Name 1.
2 Installation and Basic Operation Installation and initialization of the I/O module are straightforward processes, requiring attention to the normal mechanical, power, and thermal precautions for rack-mounted equipment. The I/O module does not require any programming or configuration to operate as a basic InfiniBand switch and includes all of the necessary functionality to operate with external standard InfiniBand Subnet Management software.
M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual 10 5. Copper InfiniBand Cable Connecting/Disconnecting Copper InfiniBand cables are heavy and not flexible, as such they should be carefully attached to or detached from the connectors. Refer to the cable manufacturer for special warnings and instructions. 6. Rack Mounting and Servicing When this product is mounted or serviced in a rack, special precautions must be taken to ensure that the system remains stable.
2.2 Mechanical Installation These switches are hot pluggable. It is not necessary to power down the Dell Chassis to install a new switch or to replace an old switch with a new one. Figure 4: Rear View of the Dell PowerEdge M1000e Chassis Latch Release Buttons Locking Arms Channel 1 slots A1 B1 C1 Channel 2 slots C2 B2 A2 Note: M3601Q blades are not allowed in the Fabric A slots. The M3601Q IOM occupies both Fabric B and Fabric C slots and connects to the Fabric C midplane.
M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual 12 2.2.2 Removing an I/O Module Blank From the Chassis 1. Unlock the I/O Module Blank by pushing the red latch release button. 2. Pull the locking arm down to a position perpendicular to the front of the chassis. 3. Pull the I/O Module Blank out of the chassis using the locking arm. 4. Install either a new switch or reinstall a blank IOM within one minute. 2.2.
To remove, disengage the locks and slowly pull the connector away from the port receptacle. Both LED indicators will turn off when the cable is unseated. Care should be taken not to impede the air exhaust flow through the chassis. Note: Cable lengths should be used which allow for routing horizontally around to the side of the chassis before bending upward or downward in the rack. Rev 1.
M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual 14 3 Cluster Management and Firmware 3.1 Network Management and Clustering Software Download and install, on all nodes, the Mellanox OpenFabric software package for Linux, Windows, or other operating systems from the Mellanox software website: http://www.mellanox.com >Downloads > InfiniBand SW/Drivers.
-I- Using port 1 as the local port. -I- Discovering ... 3 nodes (2 Switches & 1 CA-s) discovered. -I---------------------------------------------------I- Bad Guids/LIDs Info -I---------------------------------------------------I- skip option set. no report will be issued -I---------------------------------------------------I- Links With Logical State = INIT -I---------------------------------------------------I- skip option set.
M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual 16 II. Using the ibnetdiscover tool: Run: [root@mymach]# ibnetdiscover | grep 00000006660abcd0 | grep -w Switch Switch 24 "S-00000006660abcd0" "MT48436 Infiniscale-III Mellanox Technologies" base port 0 lid 17 lmc 0 Note: The resulting LID is given as a decimal number. 4. Run mlxburn with the LID retrieved in step #3 above to perform the In-Band burning operation.
4 Troubleshooting As soon as a switch is plugged in make sure that the power LED comes on. The power LED for the switch does not come on: 1. Check that the chassis has power. 2. Remove and reinstall the switch. The status LED for the switch is blinking amber: 1. remove the switch from the chassis and re-insert it (verify that the switch is all the way in the chassis and the lever is firmly closed). 2. If the amber LED continues to blink, replace the switch.
M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual 18 Appendix A: Specifications Table 4 - M3601Q Specification Data Physical Power and Environmental H x W x D: 10.73 x 2.322 x 10.04 inches 272.54 x58.98 x 255.02 mm Weight: 3.
Appendix B: EMC Certification Statements B.1: FCC Statements (USA) Class A Statements: § 15.21 Statement Warning! Changes or modifications to this equipment not expressly approved by the party responsible for compliance (Mellanox Technologies) could void the user's authority to operate the equipment. §15.105(a) Statement NOTE: This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant to Part 15 of the FCC Rules.
M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual 20 (Translation - "This is a Class A product based on the standard of the Voluntary Control Council for Interference by Information Technology Equipment (VCCI). If this equipment is used in a domestic environment, radio interference may occur, in which case the user may be required to take corrective actions.") B.1.
Appendix C: QSFP Interface 20 21 22 23 24 25 26 27 28 29 30 31 32 GND Rx2n Rx1n 18 Rx2p Rx1p 17 GND GND 16 Rx4n Rx3n Rx4p Rx3p GND GND ModPrsL SDA IntL SCL VccTx Vcc Rx Vcc1 ResetL LPMode GND ModSelL GND 33 Tx3p Tx4p 34 Tx3n Tx4n GND 35 36 37 38 Rev 1.
15 16 17 18 19 SCL SDA GND RX3p RX3n GND RX1p RX1n GND 8 ModSelL ResetL VccRx 14 7 GND 20 GND 21 RX2n 22 RX2p 23 GND 24 RX4n 25 RX4p 26 GND 27 ModPrsL 28 IntL 29 VccTx 30 Vcc1 31 LPMode 14 13 12 11 1 0 9 8 7 6 5 4 3 2 1 GND 32 GND TX2n 13 6 TX4p 15 TX2p 33 TX3p 16 GND 34 TX3n TX4n 12 5 TX4n TX4p 19 18 17 GND 35 GND ModSelL 4 GND ResetL 11 1 0 9 3 TX2p SCL 36 TX1p SDA 2 TX2n GND 37 TX1n GND 3 8 TX1n 3 7 TX1p 3 6 GND 3 5 TX3n 3 4 TX3p 3 3 GND
Appendix D: Avertissements de sécurité d’installation 1. Instructions d’installation Lisez toutes les instructions d’installation avant de brancher le matériel à la source d’alimentation électrique. 2. Température excessive Ce matériel ne doit pas fonctionner dans une zone avec une température ambiante dépassant le maximum recommandé de 40°C (104°F). En outre, pour garantir un bon écoulement de l’air, laissez au moins 8 cm (3 pouces) d’espace libre autour des ouvertures de ventilation. 3.
M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual 24 8. Elimination du matériel L’élimination de ce matériel doit s’effectuer dans le respect de toutes les législations et réglementations nationales en vigueur. 9. Codes électriques locaux et nationaux Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux. Mellanox Technologies Rev 1.
Appendix E: Installation - Sicherheitshinweise 1. Installationsanleitungen Lesen Sie alle Installationsanleitungen, bevor Sie das Gerät an die Stromversorgung anschließen. 2. Übertemperatur Dieses Gerät sollte nicht in einem Bereich mit einer Umgebungstemperatur über der maximal empfohlenen Temperatur von 40°C (104°F) betrieben werden. Außerdem sollten mindestens 8 cm (3 in.) Freiraum um die Belüftungsöffnungen sein, um einen einwandfreien Luftstrom zu gewährleisten. 3.
M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual 26 8. Geräteentsorgung Die Entsorgung dieses Geräts sollte unter Beachtung aller nationalen Gesetze Bestimmungen erfolgen. 9. Regionale und nationale elektrische Bestimmungen Dieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden. Mellanox Technologies Rev 1.
