Grid Director 4036/2036 Installation Manual DOC-00467 A09 www.mellanox.
NOTE: MELLANOX TECHNOLOGIES, INC. AND ITS AFFILIATES ("MELLANOX") FURNISH THIS DOCUMENT "AS IS," WITHOUT WARRANTY OF ANY KIND. MELLANOX DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT AND THOSE ARISING FROM A COURSE OF PERFORMANCE, A COURSE OF DEALING, OR TRADE USAGE.
Contents Contents Preface .................................................................................................................................................... 7 1 2 3 4 5 Introduction ..................................................................................................................................... 9 1.1 4036/2036 Switch Overview ................................................................................................... 9 1.
Contents 6.1.1 7 Cooling .................................................................................................................... 31 6.2 Required Tools ..................................................................................................................... 31 6.3 Site Planning......................................................................................................................... 32 6.3.1 Rack and Clearance Requirements .......................................
Contents List of Figures Figure 1: 4036/2036 Front View .............................................................................................................. 9 Figure 2: 4036/2036 Rear View ............................................................................................................... 9 Figure 3: 4036/2036 Cardboard Package (Open) - Standard ............................................................... 12 Figure 4: Removing the 4036/2036 Switch from the Cardboard Package ........
Contents Figure 43: Installing a Cabling Guide Bracket ....................................................................................... 51 Figure 44: Installing a Cabling Guide Bracket (sliding rail) ................................................................... 51 Figure 45: Cable Management .............................................................................................................. 52 Figure 46: Cabling ........................................................................
Grid Director 4036/2036 Installation Manual Preface About this Manual This manual provides installation instructions for the Voltaire high-bandwidth, low-latency scalable 36-port InfiniBand 4036 QDR and 2036 DDR chassis. It includes the product specifications, unpacking and installation information, unit power up, and initiation and troubleshooting procedures. Refer to the official and latest product release notes for last-minute updates.
Preface CAUTION: Alerts you to the risk of personal injury, system damage, or loss of data. WARNING: Warns you that failure to take or avoid a specific action might result in personal injury or a malfunction of the hardware or software. Be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents before you work on any equipment. Typography The following table describes typographical conventions in Mellanox documentation.
Grid Director 4036/2036 Installation Manual 1 Introduction 1.1 4036/2036 Switch Overview The Mellanox Grid Director 4036/2036 is a high performance, low latency and fully non-blocking InfiniBand switch for high performance clusters. Delivering 2.88 Tbps of non-blocking bandwidth with less than 100 nanoseconds of latency (port-to-port), I/O bottlenecks are removed making applications operate at maximum efficiency. The 4036/2036 was designed to fit into today’s densely configured racks.
Introduction 1.2 4036/2036 Switch Main Features Thirty-Six auto-negotiating QSFP InfiniBand ports in a 1U chassis: The 4036 switch is equipped with QDR ports (40 Gbps per port) The 2036 switch is equipped with DDR ports (20 Gbps per port) 1Ux19"x15" non blocking standalone switch using a single switch chip QSFP ports supporting any type of cable, Copper or Optical.
Grid Director 4036/2036 Installation Manual 2 Unpacking 2.1 Package Contents The 4036/2036 switch is supplied in a cardboard box. The switch is delivered with either one or two pre-installed hot-swappable power supplies (according to configuration) and one hot-swappable fan unit.
Unpacking Figure 3: 4036/2036 Cardboard Package (Open) - Standard NOTE: Keep the original packing materials; they will be needed if: The product needs to be moved. For example if it is staged at one location before being delivered at the production site. A module needs to be returned for repair, replacement, or upgrade. 2. Remove the box containing the Rail Kit, the Power Cords, and the multipurpose console cable kit (Mng). 3. Remove the Getting Started Guide which also contains the Product CD. 4.
Grid Director 4036/2036 Installation Manual 5. Carefully remove the 4036/2036 switch. Use the dedicated niche to lift the switch, as shown in the following figure. Figure 4: Removing the 4036/2036 Switch from the Cardboard Package 6. Remove the protective antistatic bag from the 4036/2036 switch. 2.
Unpacking Figure 5: Fixed Rail (KIT-00008) 2.4 Multipurpose (Mng) Console Cable Kit The multipurpose console kit (Mng) contains: RJ45 to RJ45 cable A RJ45 to DB-9 cross-adaptor for RS-232 console connection (CLI). (The DB-9 adaptor has a number 26 printed on it. This means that pins 2 and 6 are crossed). A RJ45 to DB-9 adaptor for RS-232 console connection 2.
Grid Director 4036/2036 Installation Manual Figure 6: Packing the 4036/2036 Switch 6. Insert the switch into the cardboard box and place it on the bottom foam. 7. Place the top foam on top of the switch. 8. Slide the rail kit box in the dedicated niche alongside of the switch, as shown in the illustration in Unpacking the Switch (on page 59). IMPORTANT NOTE: When sending the switch for repair without the rail kit, it is still important to secure the switch by filling this niche with a filler.
Getting Started 3 Getting Started 3.1 Front Panel Description This section details the front panel 4036/2036 chassis. Figure 7: 4036/2036 Front Panel 1. Swappable power supply modules 2. LEDs: System power Info LED SM Fan unit Power supply 3. AC power inlet 4. Reset Button 5. Swappable fan unit Table 1: 4036/2036 Front Panel Components Component 4036 /2036 No. of components Function Power Supply (PSU) 1 or 2 (according to configuration) Provides the 4036/2036 DC supply.
Grid Director 4036/2036 Installation Manual Component 4036 /2036 No. of components Function Reset Button 1 When pressed for more than 1 second, resets the entire unit. LEDs (Status Indicators) 5 System Power – Chassis power OK LED - ON/OFF Info LED – User-defined LED for management use. Sets the Info LED on a desired module. When you enable this LED through the GUI or CLI, you can go on site and immediately physically locate the module.
Getting Started Table 2: Rear Panel Components Component No. Function InfiniBand Port Connector 36 4X QSFP InfiniBand connectors for passive copper cables, active copper cables, active optical cables, and connect optical cables. InfiniBand Port Indicators (LEDs) 64 Indicates the physical and logical status of each InfiniBand port.
Grid Director 4036/2036 Installation Manual 4 Product Options 4.1 4036/2036 Switch Available Configurations The Grid Director 4036/2036 is a 1U entry level 36 InfiniBand 4X ports switch, providing bidirectional 40 Gbps/20 Gbps per port grid connectivity, potentially serving as an edge switch in large fabrics (where the 4036/2036 is serving as core switch) or as a standalone managed unit in smaller clusters. 4.
Product Options The following figures show a simple 36-server single-hop non-blocking configuration using a single 4036/2036 switch. Figure 8: InfiniBand Cluster Using 4036 Switch Single-hop Configuration Figure 9: InfiniBand Cluster Using 2036 Switch Single-hop Configuration The following figures show a triple-hop non-blocking configuration, based on multiple 4036/2036 switches functioning as the cluster building blocks.
Grid Director 4036/2036 Installation Manual 21
Product Options The following figures show another example of a three-hop non-blocking configuration, based on multiple 4036/2036 functioning as the cluster building block. Figure 10: InfiniBand Cluster Using 4036/2036 Switch Three-hop Configuration 4.
Grid Director 4036/2036 Installation Manual Number Name Description VLT-30015 Grid Director 4036 1PS 4036 1U 36 port QDR Switch, back-to-front cooling, single PS Table 4: 4036/2036 Module Ordering Information Number Name Description VLT-30029-F PS-36 AC FRU PS-36 AC, FRU VLT-30030-F FAN-1U Unit Back-To-Front FRU FAN-1U unit back-to-front cooling, FRU VLT-30031-F FAN-1U Unit Front-to-Back FRU FAN-1U unit front-to-back cooling, FRU KIT-00008 Rail Kit KIT-00004 iDataPlex Mounting Kit (No
Technical Specifications and Certifications 5 Technical Specifications and Certifications 5.
Grid Director 4036/2036 Installation Manual Features Details Environmental Operating Temperature: 32°F to 113°F (0°C to 45°C) Humidity: 15% to 80%, non-condensing Altitude: 0 to 9843 ft (3000m) Storage Temperature: -13°F to 185°F (-25°C to +85°C) Humidity: 5% to 90% non-condensing Altitude: 0 ft to 15,000 ft (4570 m) Maximum Ambient Temperature: 45°C 5.
Technical Specifications and Certifications 5.3 Certifications Safety cTUVus (USA/CAN); CE according to EN 60950-1 First Ed. MET Labs (NRTL) CE according to EN 69950-1 CSA C22.2 60950-1 S-Mark Argentina CB Certificate and report, according to IEC 60950-1 First Ed. MIC Korea Gost-R EMC Certifications FCC Part 15, Subpart B, Class A and testing to ANSI 63.
Grid Director 4036/2036 Installation Manual 5.4 Gost-R Certification The Mellanox products comply with the Gost-R Russian safety regulations. 5.5 Declarations 5.5.1 Declaration of Conformity Mellanox products comply with the Radio & Telecommunications Terminal Equipment Directive 99/5/EEC, the EMC Directive 89/336/EEC, and of the Low Voltage Directive 73/23/EEC. 5.5.
Technical Specifications and Certifications 5.6 Label(s) The 4036/2036 label (example) below shows all safety and EMC text and logos.
Grid Director 4036/2036 Installation Manual 5.7 Shock and Vibration Prepare the 4036/2036 switch to ship in a rack by carefully following the rail kit installation and instructions, as detailed in Chapter 6 (on page 31). The 4036/2036 is susceptible to shock and vibration damage if the rail kit installation instructions are not followed precisely. 5.8 Chassis Clearance Requirements Table 8: Clearance Requirements 5.
Technical Specifications and Certifications 5.11 Export Information CTP (Composite Theoretical Performance) is a calculation measured in MTOPS (millions of theoretical operations per second). The 460EX/460GT contain a 440H6 core (not a 460 core; it is called 460 to distinguish it as a 90 nm core). The CTP (in MTOPS) for the PPC440 core = (17/18) * F (where F is the CPU clock rate in MHz). The following table provides CTP values. 5.
Grid Director 4036/2036 Installation Manual 6 Hardware Installation This chapter describes how to prepare your site for installation and how to prepare and install the 4036/2036 switch. The 4036/2036 switch is a 19” rack mounted, 1U high chassis provided with a rail kit accommodating racks of different depths with a front-to-back distance between mounting rails of 26" to 36" (660 mm to 914 mm).
Hardware Installation 6.3 Site Planning Planning the proper location and layout of your equipment rack or wiring closet is essential for successful 4036/2036 switch operation. Equipment placed too close together or in an inadequately ventilated area can cause overheating of the system. In addition, poor equipment placement can render system panels inaccessible and difficult to maintain.
Grid Director 4036/2036 Installation Manual 6.3.4 Site Preparation Checklist To help prepare your site for installing the Grid Director 4036/2036 chassis, use the following checklist. Table 12: Site Preparation Checklist Task Prepared by Date Notes Environmental requirements Locate power sources Measure space for the rack, including specified maintenance clearances Acquire specified cables and connectors 6.
Hardware Installation Table 13: Rail Kit (P/N KIT-00008) Detailed Part List Item P/N 2 x short bracket 290000169 2 x Mounting Bar MEC-0055 6 Illustration Front side of the chassis 34 14 x Pan Head Philips Screw, 8-32 x 1/4" with tooth lock washer and nylon patch FAS-0004 4 Long Bracket (right side) MEC-0055 5 Long Bracket (left side) MEC-0055 4 Rear side of the chassis
Grid Director 4036/2036 Installation Manual 6.5.2 Option 1 The rail assembly described in this section enables front-to-rear switch installation within the rack, meaning that the front of the switch faces the front of the rack. The front of the switch is recessed 5 inches from the vertical rail of the rack to allow connecting the power cords from the front side of the rack. Figure 13: 4036/2036 – Option 1 Assembly NOTE: Standard rack clip nuts and rack screws are not provided.
Hardware Installation Mount the long bracket at the front of the chassis using three 8-32x1/4" pan head Philips screws. Important: Left and right long brackets are different. Make sure that you are using the correct bracket. The mounting ear at the end of the bracket should face out. Figure 14: Mounting the long bracket 2. Position the mounting bar and adjust it according to the rack measurements you made while Measuring the Distance between Mounting Rails (on page 33).
Grid Director 4036/2036 Installation Manual 5. Ensure that the clip nuts are already installed in the rack holes of the correct U level (on all four vertical rack rails) as described in Measuring the Distance between Mounting Rails (on page 33). Note: Rack clip nuts are not provided by Mellanox. Figure 17: Installing the clip nuts and placing the power cord 6. Place the power cords with the female connector facing the front of the rack (or the switch) with approximately 10" hanging out of the front.
Hardware Installation 8. Place the power cords over the indentation of the long brackets. Figure 19: Power cord over the long bracket 9. Secure each power cord with a cable tie wrap so that you can still adjust the length of the cord if required. Figure 20: Securing power cords 10.Connect the power cords to the switch power supplies (PSU). Figure 21: Connecting the power cables 11.Push the switch inwards (from the front side of rack) until the long bracket comes together with the rail.
Grid Director 4036/2036 Installation Manual 12.Fasten the long brackets to the rack’s vertical rails using two screws per bracket and tighten them. Figure 22: Secure the long brackets to the rack rail 13.Mount a short bracket from the inner side of the mounting bar and fasten it using two 8-32x1/4" pan head Philips screws. At this stage, do not fully tighten the screws. Important: Use the lower holes in the short bracket as shown in the figure.
Hardware Installation 14.Adjust the short brackets snuggly onto the rack’s vertical rail using the adjusting holes (make sure that the rail is perfectly aligned by using the same U as used in the front of the rack). Figure 24: Adjusting the short bracket 15.Tighten the rack screws. (Rack screws are not supplied) 16.Tighten the short brackets to the mounting bars.
Grid Director 4036/2036 Installation Manual Result – Rack front view Figure 27: Switch mounted in a rack – Rack front view 41
Hardware Installation 6.5.3 Option 2 The rail assembly described in this section enables rear-to-front switch installation within the rack, meaning that the rear of the switch (InfiniBand connector side) faces the front of the rack. This rear-to-front chassis installation is in-line with the vertical front rails of the rack. Figure 28: 4036/2036 – Option 2 Assembly NOTE: Standard rack clip nuts and rack screws are not provided. To mount the 4036/2036 in a rack using Option 2 1.
Grid Director 4036/2036 Installation Manual On each side of the chassis, mount the short brackets on the rear side of the switch (InfiniBand port side) using three 8-32x1/4" pan head Philips screws, as shown on the picture. Figure 29: Mounting the short bracket 2. Mount the long bracket at the front of the chassis using two 8-32x1/4" pan head Philips screws. Use the long bracket middle row orifices and the holes of top row on the rail. Important: Left and right long brackets are different.
Hardware Installation Note: Make sure that the distance is between 0.5" to 1" greater than the distance you measured between the two vertical rack rails to allow adjustment at a later stage. Figure 31: Positioning and aligning the mounting bar and long bracket Total distance = rack measurements + 0.5 to 1" for adjustment. 4. Secure the rail on the side of the switch using two 8-32x1/4" pan head Philips screws (use the holes on the top row).
Grid Director 4036/2036 Installation Manual Note: Rack clip nuts are not provided by Mellanox. Figure 33: Placing the clip nuts on the vertical rack rails 6. Go to the front of the rack. Note: For ease of installation, at least two qualified people should perform the following steps. Lift and position the switch with the connectors facing towards you and insert it into the rack. Figure 34: Insert the switch into the rack 7.
Hardware Installation On each side, align the long bracket holes with the rack holes of the correct U level and secure them using two rack screws (not provided by Mellanox) and two rail screws. Figure 36: Installing the long bracket 9. Tighten the rack screws. (Rack screws are not supplied) Figure 37: Securing the rack screws 10.Tighten the long brackets to the mounting bars.
Grid Director 4036/2036 Installation Manual Important: This step should be done last. Figure 38: Securing the long bracket screws to the mounting bar Result – Rack front view Figure 39: Switch mounted in a rack – Rack front view 6.6 4036/2036 Switch Power Up Once the 4036/2036 chassis is secure within the rack, electrical cables from the power distribution source can be fitted in preparation for connection. Two power cords are supplied by Mellanox, one per power supply unit.
Hardware Installation 6.7 Field Replaceable Units 6.7.1 Fan Unit The 4036/2036 switch has a hot swappable Fan Unit with three fans that are used to maintain proper cooling, as shown. In normal operation, the two fans work at 50% utilization. In case of fan failure or high temperature detection, the fans go into Turbo mode. In case of fan failure, the fan drawer LED and the PS/FAN LED on the rear panel blink.
Grid Director 4036/2036 Installation Manual When removing a power supply from the chassis, the other supply can power the entire chassis. Figure 41: Replacing the Power Supply NOTE: Power supplies can be replaced while the power is ON. However, it is recommended to turn the power OFF before replacing the power supply. To replace a hot-swappable power supply module 1. Release the captive fasteners at each side of the power supply and gently remove the power supply from its bay. 2.
Cabling 7 Cabling This chapter provides details on the 4036/2036 switch cable guide brackets installation and cable management. 7.1 Cabling Guide Bracket Installation NOTES: Selecting the optimal InfiniBand cable length depends on the overall cluster configuration and requires proper planning to produce orderly and maintainable cabling. The rack space for the 4036/2036 switch should be free from obstructions such as power strips.
Grid Director 4036/2036 Installation Manual 3. Tighten the screws.
Cabling The following shows the final results. Figure 45: Cable Management Figure 46: Cabling 7.2 InfiniBand Cabling - QDR Selecting proper InfiniBand cable lengths depends on the overall cluster configuration and requires proper planning to produce orderly and maintainable cabling. We recommend dressing the cables from each row of QSFP connectors into two groups running them left and right, and fastening them down gently after connectivity was verified.
Grid Director 4036/2036 Installation Manual To connect an InfiniBand cable to a 4036/2036 switch 1. Insert the cable into the Cabling Guide bracket, if it applies. 2. Position the connector opposite a connector located on the rear of the 4036/2036 chassis. Depending upon the InfiniBand connector type, squeeze the tabs on either side of the head shell, and push the connector in place. OR Gently push the latch and push the connector in place. 3.
Cabling This kit contains the Mng cable and the following adaptors: RJ45 to RJ45 cable A RJ45 to DB-9 cross-adaptor for RS-232 console connection (CLI). (The DB-9 adaptor has a number 26 printed on it. This means that pins 2 and 6 are crossed). A RJ45 to DB-9 adaptor for RS-232 console connection IMPORTANT: Both serial and Ethernet management ports of a 4036/2036 switch can be connected simultaneously. However, the Ethernet ports of the front and the rear panels cannot be connected simultaneously.
Grid Director 4036/2036 Installation Manual 8 Operation This chapter describes how to operate the 4036/2036 switch, once the installation is complete. The 4036/2036 switch operation consists of making sure that it starts up properly and that any initialization problems are resolved. 8.1 Powering on the 4036/2036 Switch To start up the 4036/2036 unit: The 4036/2036 chassis should be secured within the rack. The InfiniBand cables should be plugged into the rear panel ports.
Operation SM - Indicates the subnet management functionality: on for active mode, blinking for standby mode. PS/FAN - Lights solid green, indicating that the 4036/2036 switch is properly powered and the fan unit is functioning adequately. 6. If any of these conditions is not met, refer to Solving Startup Problems (on page 58) to isolate and, if possible, resolve the problem. 8.
Grid Director 4036/2036 Installation Manual LED Name Color Location Status Functionality IB physical link Green Rear panel on each QSFP port ON Port physical link is up OFF Port physical link is down or port is disconnected BLINK Errors, Link is not stable ON Logical link is configured by SM OFF Logical link is not configured by SM BLINK traffic on the link ON Eth link is up OFF Eth link is down OFF No eth traffic BLINK Eth tragic activity IB logical link Eth link Amber Green
Troubleshooting 9 Troubleshooting 9.1 Overview This chapter provides suggestions for solving startup problems and useful information when contacting Mellanox Customer Service. 9.2 Solving Startup Problems Try to solve startup problems using the 4036/2036 switch LED indicators. Case 1 Problem with power supply – How do I know that my power supply is not functional? Check the power cord and its connections. Check the installation of the power supply module.
Grid Director 4036/2036 Installation Manual Appendix A: A.1 Option 3 – Chassis Installation in an iDataPlex Rack Unpacking the 4036/2036 Switch NOTE: Keep the original packing materials; they will be needed if: The product needs to be moved. For example if it is staged at one location before being delivered at the production site. A module needs to be returned for repair, replacement, or upgrade. The 4036/2036 chassis is supplied in a box similar to the one shown in the following figure.
Troubleshooting 6. Remove the protective antistatic bag from the 4036/2036 switch. A.1.1 Multipurpose Console Cable Kit Refer to Multipurpose (Mng) Console Cable Kit (on page 53). A.1.2 Packing the Switch Refer to Packing the 4036/2036 Switch (on page 14). A.2 Mounting the iDataplex Kit The 4036/2036 switch is a 19" rack mounted, 1U high chassis provided with a rail kit accommodating iDataplex racks with a front-to-back distance between mounting rails of 15.75" (400 mm).
Grid Director 4036/2036 Installation Manual Assembly in an iDataPlex Rack Figure 48: 4036/2036 – Assembly in an iDataplex Rack NOTE: Standard rack clip nuts and rack screws are not provided. To mount the 4036/2036 in an iDataplex rack 1. Put the chassis on a flat surface. 2. On each side of the switch, attach the mount brackets to the rear of the chassis (connector side) using two 8-32x1/4" pan head Philips screws. Figure 49: Attaching the mount bracket 3.
Troubleshooting Important: Make sure that you install the mount holders with the hook facing out, as shown below. 4. Go to the rear of the rack: 1. Place two clip nuts on each side of the desired U level of the rack rails. 2. Secure the mount plates at the rear side of the rack rail (into the clip nuts) using two screws. Note: Rack clip nuts and screws are not provided by Mellanox.
Grid Director 4036/2036 Installation Manual 5. Install the switch by inserting the mount holders hooks into the dedicated rectangular slot in the mount plates. Figure 51: Inserting the mount holder into the mount plate 6. On the front side of the rack, secure the front of the chassis, aligning the mount brackets holes with the rack holes of the correct U level. Note: The Philips screws are not provided by Mellanox. Figure 52: Securing the front of the chassis 7.
Troubleshooting Appendix B: Cabling Information This appendix provides cabling and port pinout information for the 4036/2036 switch. B.1 IB Port Cable Specifications The following table describes cabling specifications for the 4036/2036 4X InfiniBand ports. Table 16: 4X InfiniBand Port Cabling Specifications Type 4XQDR 4XQDR – 4036/HCA 4X DDR 2036/ISR2012 64 Maximum Cable Length AWG Length (ft.) Length (m) Gore Cable P/N 26 16.4 7 QSN7600-5 26 13.1 4 QSN7600-4 28 9.
Grid Director 4036/2036 Installation Manual B.2 1 GbE Ports (Management) Use modular, RJ-45, straight-through UTP cables to connect the 10/100/1000 Fast Ethernet ports to external switches and routers. Use modular, RJ-45 cross-connect cables to connect to end systems. The following figures show straight-through cables and cross-connect cables. Figure 53: Straight-through Cables Figure 54: Cross-connect Cables The 10/100/1000 Fast Ethernet port supports a RJ-45 connector.
Troubleshooting Appendix C: QSFP Cable QSFP (Quad Small Form-factor Pluggable) Key Features: Small size, high density, hot-pluggable connector with four transmit and four receive channels Supports both types of cables: Copper active or passive cable. Copper passive cable is equipped with two simple passive connectors and a standard copper cable. Copper active cable equipped with an active device to optimize signal quality.
Grid Director 4036/2036 Installation Manual The following figure shows the QSFP module with Copper or Optical cables. You can install any combination of the three cable types in any of the 36 QSFP cages.
