Topspin 270/Cisco SFS 7008 Hardware Guide Release 2.3.
Copyright © 2004 - 2005 Topspin Communications, Inc. All rights reserved. The Topspin Switched Computing System, Topspin Host Channel Adapter, Topspin Element Manager, and collateral software programs and documentation are subject to and made available only pursuant to the license agreement signed by you and Topspin, Communications, Inc. (or if no signed license agreement exists, the license agreement included with your original media) and may only be used in accordance with the terms of that agreement.
iii Regulatory Notices........................................ vii FCC Statement .......................................................................................................................................... vii Safety Information .................................................................................................................................... vii Electrical Cautions ...............................................................................................................
iv Connect Management Devices ..................................................................................................................26 Attach a Serial Console Cable to a PC or Terminal ..................................................................... 26 Connect an Ethernet Cable to the Ethernet Management Port ..................................................... 27 Recommended Management Configuration .................................................................................
v Upgrading a Topspin 270/Cisco SFS 7008 with the CLI ............................................................. 53 Upgrading a Topspin 270/Cisco SFS 7008 with Element Manager ............................................. 54 Upgrading an Individual Controller Card ..................................................................................................56 Upgrading a Controller Card with the CLI ...................................................................................
vi Run Fan Tests ............................................................................................................................................96 Run a Self-Test on a Fan............................................................................................................... 96 Run Power Supply Tests............................................................................................................................98 Run a LED Test on the Power Supply .............................
vii Regulatory Notices FCC Statement This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant to Part 15 of the FCC Rules. These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment.
viii CAUTION: Grounding is supplied by the ground-prong on the 3-prong power cable. Do not attach a separate ground cable. Do not use adapter plugs. Do not remove the ground prong from the cable. Ensure the ground connection on the power supply is correct and functioning before applying power to the Topspin system. CAUTION: Always ground yourself before touching any internal Topspin system component to avoid damage from electrostatic discharge (ESD). CAUTION: The Topspin system has two power cables.
ix CAUTION: Never place your hand inside an empty card or module bay. You should never have cause to place a hand anywhere inside the Topspin chassis. Unused card and module bays should always have a Topspin cover over the bay to ensure proper safety, ventilation, and cooling. General Cautions CAUTION: No user is authorized to remove the Topspin system enclosure cover. The internal chassis contains no user-serviceable components. Removing the enclosure cover voids your warranty.
x
1 Features Overview The Topspin 270/Cisco SFS 7008 Server Switch provides data center managers with a high-performance, low-latency interconnect. • “System Architecture” on page 11. • “About the Topspin 270/Cisco SFS 7008 Chassis” on page 13. • “Administrative Features” on page 20. System Architecture Hot Standby On the Topspin 270/Cisco SFS 7008, the inactive controller card remains on hot standby so that it can assume control in the event that the active controller card fails over.
Example 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11.
Fabric For redundancy, InfiniBand Host Channel Adapters (HCAs) can be dual-connected to a redundant pair of Topspin 270/Cisco SFS 7008s. In an InfiniBand fabric that includes more than one Topspin 270/Cisco SFS 7008: if the subnet manager on the Topspin 270/Cisco SFS 7008 that is acting as the master fails, another subnet manager will take over within seconds. All necessary state information is kept in sync.
Figure 1-2: Back Chassis View
Chassis Slot Numbering Slot numbers are printed on the chassis for easy identification.
Slot # 15 Slot # 16 Slot # 17 Slot # 1 Slot # 2 Slot # 3 Slot # 4 Slot # 5 Slot # 6 Slot # 7 Slot # 8 Figure 1-4: Chassis Slot Numbering in Back (Service Side) Dimensions • Height: 6U unit • Width: 17.04” • Depth: 8” - 23.7” • Weight: maximum 106 pounds (full chassis configuration with 4x copper Line Interface Modules) Connections • 96 ports of 10 Gbps 4X Copper InfiniBand. • Two 10/100 Ethernet RJ-45 Management-Ethernet ports for out-of-band management.
• The power supplies are 1U, 48V, 1200W • The supplies provide regulated +48V DC to all other modules in the system. • Each power supply has self contained fans for cooling. Fan Trays Refer to “Installing a Fan Module” on page 41 for installation instructions. The fan trays are redundant, hot-swappable cooling units. • Only one fan tray, in either of the two slots, is required to cool the system. However, a blanking panel is required for any unused bay.
Understanding the Core Slot to Mgmt Interface Module Connection Each Core slot in the chassis (refer to “The Core Slot” on page 18) is connected to 1 port on the Management Interface module. The controller (fabric interface module inserted in the core slot) selects the Management Interface module to which it will connect. The pairing of Management Interface module to core slot is as follows: • The Management Interface module in slot #15 is connected to the Fabric Controller Module in the Core slot #11.
• At least one Fabric Controller module must be installed in the Core slot for the chassis to boot. • When a module is installed in a Core slot, it acts as either the master or standby controller for the chassis, with all modules in the system being slaves to the master controller. • The subnet manager runs on the fabric module installed in the Core slot that has arbitrated to be the master. • The Core slot module also acts as the switch fabric spine, interconnecting the end-nodes.
Administrative Features Real-Time Clock The Topspin 270/Cisco SFS 7008 maintains correct time regardless of power conditions or connectivity. Latency The Topspin 270/Cisco SFS 7008 has port to port latency of less than 600ns.
2 Installing the Topspin 270/Cisco SFS 7008 This chapter describes how to install and manage the Topspin 270/Cisco SFS 7008 system hardware.
• Prepare a management workstation, such as a PC running a terminal program, and a straight-through M/F DB-9 serial cable (included). Mount the Topspin 270/Cisco SFS 7008 Chassis in a Rack This section describes how to install the Topspin 270/Cisco SFS 7008 chassis in an equipment rack. The following procedure is the standard installation, using your own rack shelf. Topspin also has an optional shelf that is appropriate for pre-racking and shipping a unit in the rack.
3. Install a shelf for the chassis, if you have not already done so. Verify that the shelf can support the weight of the chassis, which can be up to 106 pounds with all possible components installed. Figure 2-1: Install Appropriate Rack in Shelf 4. Determine the direction that the switch will be installed and justified in the rack. The direction that the switch will be justified will determine which set of rack ears that need to be removed. 5.
• If you have determined that the switch will be justified toward the back of the rack, then use a #2 phillips screwdriver to unscrew the rack ears from the front of the switch. Figure 2-2: Unscrew Extra Rack Ears from the Chassis 6. Separate the rack ears from the chassis when all screws have been loosened. 7. Use 2 - 3 people to lift the switch into the rack.
8. Push the chassis into the rack until the rack ears are flush with rails. Figure 2-4: Rack Ears Flush with Rails 9. Use screws that fit your rack to attach both rack ears to the rack rails.
Install Optional HA Components If you have purchased a High-Availability (HA) package, install the following components: • Redundant Power Supply. Refer to “Power Supply Modules” on page 35. • Redundant Fan Tray. Refer to “Fan Tray Modules” on page 39. • Redundant Management Interface Module. Refer to “Management Interface Modules” on page 49. Connect Management Devices To connect the management ports, use either a serial cable and/or an Ethernet cable.
• Baud: 9600 bps • Data Bits: 8 • Parity: None • Stop Bits: 1 • Flow control: None Connect an Ethernet Cable to the Ethernet Management Port 3. Connect an Ethernet cable to the Ethernet port of the left Management Interface module (slot number 15). See Figure 2-7. a. If you have two Management Interface modules, you may also want to connect an Ethernet cable to the standby module (slot 16) to maintain Ethernet connectivity in the event of a fail-over.
Power on the Chassis Use only the power cable provided with your InfiniBand system. 4. Remove the power cords from the shipping package. One power cord UL rated 12 Amps/125 VAC or greater is provided by default. If you have a high-availability unit, you will have a second power cord. 5. Inspect the power cord and determine if it provides the proper plug and is appropriately certified for use with your electrical system.
Configure Basic Connectivity 1. Configure an IP address for the Ethernet Management port. It is necessary to configure connectivity to the out-of-box Ethernet management port in order to use Telnet, the Web GUI, or the Java GUI. The system is not pre-configured when it ships from the factory. IP addresses can be assigned statically (which is the default), or dynamically assigned using DHCP.
Assigning an Address Dynamically a. When the system has completed booting, press several times to display the CLI prompt. b. Enter the user name and password. The default user name is super, and the default password is super. c. Enter the enable command. d. Enter the configure command. Login: Login: super Password: super SFS-270> SFS-270> enable SFS-270# SFS-270# configure SFS-270(config)# e. Enter the interface mgmt-ethernet command. SFS-270(config)# interface mgmt-ethernet f.
Connecting InfiniBand Devices Attaching the IB Cable to the IB Port To connect using InfiniBand to other workstations or switches, standard 4X InfiniBand cables are required. InfiniBand cables can be used to connect any two InfiniBand devices, whether switch or host. 1. Plug InfiniBand cables into the InfiniBand switch ports. To plug in an InfiniBand cable, push the connector into the interface until you hear/feel a click. Figure 2-9: Removing a Pinch Connector a.
• Command Line Interface (CLI) - a text-based interface accessible through a direct serial connection, Telnet over IP, or SSH over IP. • Chassis Manager (GUI) - A web-based graphic user interface. • Element Manager (GUI) - An optionally available graphic interface installed on a workstation, and accessible over IP. Refer to the InfiniBand User Guide, the CLI Reference Guide or the Chassis Manager User Guide for more information about managing the InfiniBand systems.
• 2. Mozilla version 1.4 Type the Management IP address of your Server Switch in the address field of your browser and press Enter. (You configured the IP address in Step f.) Figure 2-10: Web Browser Window A log-in window opens. Figure 2-11 displays the log-in window. Figure 2-11: Chassis Manager Log-In Window 3. Enter your Server Switch user name and password in the log-in window and click the OK button. Chassis Manager loads in your browser window.
(Optional) Install the Element Manager GUI For more information regarding the Element Manager tasks, refer to the InfiniBand User Guide. 1. Check that you have sufficient system resources. You will need: • 32 MB free RAM • 50 MB disk space + 50MB additional temporary space during installation • 300 Mhz processor • 800x600 screen resolution, 16bit color depth 2. Insert the Topspin CD-ROM into the CD-ROM drive of a server that has network access to a Topspin management port. 3.
3 Managing Individual Components This chapter describes how to install the Field Replaceable Units (FRUs) in the Topspin 270/Cisco SFS 7008 system.
About a Failed Power Supply If you believe a power supply module has failed, contact technical support. Do not remove a failed power supply unless you have a new power supply to install immediately, or a blanking panel to place over the open bay as overheating may occur. In most cases, vital information can be recovered from the failed module. If you have the appropriate support agreement, you may want to return the unit for diagnosis. Refer to “Vital Product Data Storage” on page 20.
7. Pull the power cord from the chassis. Power Cord (back) Power Supply (front) Figure 3-1: Diagram of Power Supply to the Associated Power Cord 8. Use a #1 phillips screwdriver to unscrew the fasteners that hold the power supply module in place.
9. Use the handle to pull the power supply module from the bay. Figure 3-3: Pulling the Power Supply Module from the Bay 10. Install the blanking panel in place of the power supply module. The device should never be run without either a blanking panel or module in any bay, as overheating may occur. or Install a new Power Supply module. Refer to “Installing a New Power Supply Module” on page 38 if you are installing a new Power Supply module.
7. Insert the new power supply unit into the open slot until it is fully seated. You may need to push the unit with your thumbs to get it completely into the bay. Figure 3-4: Inserting Power Supply Module 8. Secure the captive screws with a #1 phillips screwdriver. 9. Check the LEDs on the power supply to verify the status of the module. Refer to “Power Supply LEDs” on page 69.
4. Use a #1 phillips screw driver to unscrew the captive screws. Figure 3-5: Loosening Captive Screws 5. Carefully begin to pull the fan tray 1” out of the bay. Do not pull the fan tray completely from the bay. CAUTION: The fan tray does not have guards over the blades. You must wait until the fan blades have stopped moving before completely removing the fan tray from the bay, or risk injury. 6. Wait 15 seconds to allow the fan blades to stop moving. Figure 3-6: Partially Removing Fan Tray 7.
or Install a new fan module. Refer to “Installing a Fan Module” on page 41 if you are installing a new fan module. Installing a Fan Module 1. Remove the new fan module from the package. 2. Ground yourself appropriately. 3. Remove the bezel cover from the front of the switch, if you have not already done so. 4. Refer to “Removing a Fan Module” on page 39, if a module is still in place. 5. Remove the blanking panel, if no fan module was in place. 6.
9. Check the LEDs on the fan module to verify the status. Refer to “Fan LEDs” on page 71. Fabric Controllers About the Fabric Controllers Refer to “Fabric Controller Modules” on page 18 for more specific information about this module. About a Failed Fabric Controller If you believe a fabric controller module has failed, contact technical support. Refer to “Fabric Controller LEDs” on page 72 and “Monitor the Cards” on page 81 for monitoring information.
5. Use a #1 phillips screwdriver to unscrew the captive screws from both ends of the Fabric Controller card. Figure 3-9: Loosen Captive Screws Before Removing 6. Pull the levers on both sides of the card toward you. The card will be released. Figure 3-10: Opening Fabric Card Levers 7. Use the levers to pull the card toward you and remove it from the bay. 8. Place a blanking panel over the open bay.
Installing a Fabric Controller To install a Fabric Controller, perform the following steps: NOTE: If the card that you want to install features Anafa 2 chips, verify that your Server Switch runs TopspinOS v2.3.0 or later. 1. Remove the new Fabric Controller module from the packaging. 2. Ground yourself appropriately. 3. Remove the bezel cover, if you have not already done so. 4. Refer to “Removing a Fabric Controller” on page 42, if a Fabric Controller module is still installed. 5.
The version of the firmware is verified during the upgrade process. If a module does not need to be upgraded, the module will be skipped and move on to the next module.
3. Remove the InfiniBand cables from the ports. a. To remove a cable with a pinch connector, pinch both sides of the back of the connector and pull the connector away from the port. Figure 3-12: Removing a Pinch-Type Connector b. To remove a cable with a pull connector, grasp the connector with one hand and push it toward the port, then pull the latch away from the port with your other hand and gently wiggle the connector away from the port. 1. 2. 3.
4. Use a #1 phillips screwdriver to unscrew the captive screws on both sides of the LIM card. Figure 3-14: Loosen the Captive Screws 5. Release the LIM module by pulling both levers toward you, then toward the outside of the chassis. Figure 3-15: Releasing the LIM Module Levers 6. Use the levers to pull the module from the bay.
7. Install a blanking panel in place of the LIM module, or refer to “Installing a LIM Module” on page 48 for instructions on replacing the LIM module. Installing a LIM Module 1. Remove the new LIM module from the package. 2. Refer to “Removing a LIM Card” on page 45, if a LIM module is still in place. 3. Ground yourself appropriately. 4. Remove the blanking panel if a LIM module was not currently in place. 5. Align the LIM module and slide it into the bay as far as possible.
9. Connect the InfiniBand cables to the ports. a. To plug in an InfiniBand cable, push the connector into the interface until you hear/feel a click.Check the InfiniBand port LEDs to determine the status of the port. Refer to “InfiniBand Port LEDs” on page 77. Management Interface Modules About Management Interface Modules Refer to “Management Interface Modules” on page 17 for additional information.
7. If the Master Management Interface module was removed, the standby module becomes active. The “Master” led becomes green. 8. Install a blanking panel in the empty bay, or refer to “Installing a Management Interface Module” on page 50 to replace the Management Interface module. Installing a Management Interface Module 1. Remove the new Management Interface module from the package. 2. Ground yourself appropriately. 3.
3. Use a #1 phillips screwdriver to loosen and remove the screws from the left and right side of the Chassis ID module. Figure 3-20: Loosen the Captive Screws 4. Pull the Chassis ID module from the bay. Figure 3-21: Remove the Chassis ID Module 5. Do not allow the bay to remain empty, as overheating may occur. The Topspin 270/Cisco SFS 7008 should never be run without the replacement module, so you should immediately install the new Chassis ID module.
Installing a Replacement Chassis ID Module 1. Remove the new Chassis ID module from the package. 2. Ground yourself appropriately. 3. Remove the old Chassis ID module, if you have not already done so. Refer to “Removing a Chassis ID Module” on page 50 if a Chassis ID module is still in place. 4. Align the chassis ID module with the slot and push the module completely into the bay. 5. Use the LEDs to verify the status of the module. Refer to “Chassis ID LEDs” on page 76.
4 Administrative Tasks The following sections appear in this chapter: • “Upgrade all Cards in a Chassis” on page 53 • “Upgrading an Individual Controller Card” on page 56 • “Upgrading Individual Node Cards” on page 59 • “Upgrading a New Fabric Controller Module” on page 63 Upgrade all Cards in a Chassis These procedures assumes you are upgrading all fabric controllers and node cards in the chassis.
Upgrading a Topspin 270/Cisco SFS 7008 with Element Manager The following procedure describes the upgrade procedure using the Element Manager GUI. This procedure also assumes you are upgrading all fabric controllers in the chassis. 1. Launch Element Manager for the server switch. 2. Delete any old .img files to make room for the new image. 3. a. Select Maintenance > File Management b. Click on a .img file, if one exists and click the Delete button. c.
Note: all cards in the chassis must have an oper-status of “up” for the upgrade to be effective on all components. a. Select Maintenance > File Management b. Click the new .img file and click the Install button. c. Click Yes when prompted. d. Repeat for all Server Switches you wish to upgrade. Figure 4-2: Watching the File Copy Progress e. 5. Click Ok when the Image Install Completed Successfully message appears. Configure the image for the next boot: a. Select Maintenance > Boot Config b.
6. Select Maintenance > Reboot to reboot the controller cards and activate the image. Click Yes at the prompt. 7. View the new image on the cards: a. Select View > Device Front View. b. Right-click one of the cards and select Properties. c. Click the Inventory tab and view the version name in the Current Image Source field. Figure 4-4: Upgraded Current Image 8. Repeat for additional switches in the fabric.
Upgrading a Controller Card with the CLI For the commands needed to upgrade a single Controller card using the CLI, refer to the CLI Reference Guide. 1. Check the version of your Controller card with the show card-inventory command. In the example below, only the Controller card in slot 12 needs to be upgraded.
2. Show the standby Controller card. Example SFS-270# show card-inventory ================================================================================ Card Resource/Inventory Information ================================================================================ slot-id : 11 used-memory : 18092 (kbytes) free-memory : 109676 (kbytes) used-disk-space : 30891 (kbytes) free-disk-space : 71488 (kbytes) last-image-source : TopspinOS-2.2.0/build531 primary-image-source : TopspinOS-2.2.
5. Activate the image. After downloading the image file to the chassis controller, it must be installed to become active. The install command installs the specified image file into the system. a. In the privileged-execute mode, enter install image:image-file-name.img Example SFS-270# install image:SFS-270-TopspinOS-2.2.0-build531.img *****************operation completed successfully 6. Specify the new boot image to be used. a.
Upgrading a Node Card with Element Manager The following procedure describes the upgrade procedure for individual node cards using the Element Manager GUI. This procedure assumes you are upgrading only one or more node cards, and not the Controller cards. 1. Launch Element Manager for the server switch. 2. Delete any old .img files to make room for the new image. 3. a. Select Maintenance > File Management b. Click on a .img file, if one exists and click the Delete button. c.
Note: all node cards in the must have an oper-status of “up” for the upgrade to be effective. a. Select Maintenance > File Management b. Click the new .img file and click the Install button. c. Click Yes when prompted. Figure 4-6: Watching the File Copy Progress d. 5. Click Ok when the Image Install Completed Successfully message appears. Configure the image for the next boot: a. Select Maintenance > Boot Config b.
6. Activate the image on the node card(s) by enabling and disabling each node card. Do not reboot the chassis as this will not result in the image being applied to the node card(s). a. Right-click a node card and select Properties. b. Click the down radio button and select the Apply button. Figure 4-8: Disable and Enable the Node Card(s) c. 7. Click the up radio button and select the Apply button. View the new image on the node card(s) when the card comes back online: a.
Upgrading Node Card(s) with the Chassis Manager For the procedure needed to upgrade node cards using the Chassis Manager GUI, refer to the Chassis Manager User Guide. Upgrading a New Fabric Controller Module When you add a new fabric controller module to a production Topspin 270/Cisco SFS 7008, you must upgrade the new module so that it runs the same firmware as the existing module.
NOTE: If the appropriate image file already resides on your active fabric controller module, skip this step and proceed to step 5. Example SFS-270# copy ftp://bob:mypassword@10.0.0.5/SFS-270-TopspinOS-2.3.0-build210.img image:SFS-270-TopspinOS-2.3.0-build210.img 5. Enter the install command with the image that you want to install. NOTE: Before you install an image, verify that you have brought up all of the cards on the chassis that you want to run the new image.
Upgrade a New Fabric Controller Module with Element Manager To upgrade the card, perform the following steps: 1. Launch Element Manager and navigate to your Topspin 270/Cisco SFS 7008. 2. Verify that the images that run on the fabric controller modules do not match. If they match, you do not need to upgrade the new module. a. Right-click the running fabric controller module and select Properties. b. Click the Inventory tab and note the image that runs on the card. c.
5 Monitoring the Topspin 270/Cisco SFS 7008 System This chapter describes how to install and manage the Topspin 270/Cisco SFS 7008 system hardware.
Figure 5-1: Front Panel LEDs System-Wide Status LED Locating the System-Wide Status LED The system-wide status LED is located on the front of the chassis, and is visible whether the bezel cover is on or off. This LED indicates the overall health of all modules in the system. If you are facing the front of the chassis, the system-wide status LED is found on the top left corner of the switch. In addition, a mirror of the system-wide LEDs are positioned on the back of the chassis.
Table 5-1: Interpreting the System-Wide LED (see Figure 5-1) Color State Yellow blinking Indication System identify. This state is initiated by the user. This LED can be initiated manually by using the diag LED test. Refer to the Command Line Reference Guide for more information.
In addition, a Power Supply Identify LED is located on the outside of the chassis, next to the corresponding power supply. This LED allows the user to initiate a command that causes the Identify LED to blink, which assists in locating a particular Field Replaceable Unit. Figure 5-3: Power Supply Identify LED Replicas of the Power Supply LEDs are positioned on the back of the chassis. Refer to “Management Interface Module LEDs” on page 73.
Table 5-3: Interpreting the Power Supply Identify LED Color State Indication Yellow blinking Identify. Assists in identifying a particular Field Replaceable Unit on the chassis. (see Figure 5-3) This state is initiated by the user. This LED status is initiated manually by using the diag LED test. Refer to the Command Line Reference Guide for more information.
Fabric Controller LEDs Each Fabric Module has two sets of LED indicators. • The two LEDs (green or yellow) are the module status indicators (Table 5-5). • The single LED (green only) indicates whether of not the module is the active system master (Table 5-6). Location Refer to Figure 5-1 to view the front panel of the chassis.
Interpreting the Back Panel LEDs The back of the Topspin 270/Cisco SFS 7008 chassis has the following types of LED indicators: • “Management Interface Module LEDs” on page 73 • “Chassis ID LEDs” on page 76 • “LIM Status LEDs” on page 77 • “InfiniBand Port LEDs” on page 77 Figure 5-5: Back Panel LEDs Management Interface Module LEDs The Management Interface modules have six LED yellow over green indicator pairs, which show critical system module statuses.
Location The Management Interface modules are located at the top of back the chassis. Refer to Figure 5-5 to view the chassis diagram. Figure 5-6: Management Interface Module LEDs Interpreting the Management Interface LEDs System Status LEDs Table 5-7: Interpreting the Management Interface Module System Status LEDs (see Figure 5-6) Color State Indication Green and Yellow off No module power, or LED failure. Green solid on System running with no errors detected.
Table 5-8: Interpreting the Management Interface Module Status LEDs (see Figure 5-6) Color State Indication Green off No power to the Management Interface module, LED failure, or yellow LED is ON. Yellow solid on Failure of the Management Interface module. Operator attention required. The “!” label indicates a failure. Yellow blinking Identify a Management Interface module. Assists in identifying a particular Field Replaceable Unit on the chassis. This state is initiated by the user.
Table 5-10: Interpreting the Power Supply Status LEDs (see Figure 5-6) Color State Indication Green over Yellow off No fan module installed, or LED failure. Green solid on Power supply is running with no errors detected. Green off No power to the Power supply, LED failure, or yellow LED is On. Yellow solid on Failure of the Power Supply. Operator attention required. The “!” label indicates a failure.
Table 5-13: Interpreting the Chassis ID Status LEDs (see Figure 5-7) Color State Indication Green/Yellow off No module power, or LED failure. Green solid on Chassis ID module is running with no errors detected. Green off No power to the Chassis ID module, LED failure, or yellow LED is On. Yellow solid on Failure of the Chassis ID module. Operator attention required. The “!” label indicates a failure. Yellow blinking Identify a Chassis ID module.
Location The InfiniBand port LEDs are located to the right of each InfiniBand port (if you are facing the LIM card) in the back of the chassis. IB Port LEDs Figure 5-9: InfiniBand Port LEDs Interpreting the InfiniBand Port LEDs Table 5-15: Interpreting the InfiniBand Port Status LED Top (tx/rx) LED Bottom (link) LED Indication off off No link signal detected. off solid on Physical link OK. No logical link established. solid on solid on Physical link OK. Logical link established.
3. The Device Rear View and the Device Front View appear.
Figure 5-11: Chassis Manager Topspin 270/Cisco SFS 7008 Device Front View 4. 5. View the information at the top of the window to quickly check the status color and the number of modules installed. • The numbers associated with the Card field indicate the slot numbers where modules are installed. • The numbers associated with the Power Supplies and Fans fields indicate the ID for the module. • If only one number appears next to any of the modules, then only one item is installed.
• • In the case of the fabric controller cards, the card that is currently on standby (not the master) will be grey. The master fabric controller will be green. A red LED indicates that an error has been detected. Monitor the Cards 1. Launch the Chassis Manager by entering the IP address of the InfiniBand switch into your web browser. 2. Press the Enter key. 3. The Device Rear View and the Device Front View appear. 4. Click open the Chassis file in the left-navigation tree.
normal bootFailed tooHot booting checkingBootImage wrongBootImage rebooting standby recoveryImage 9. Click the radio button next to the card that you want to monitor. 10. Click the Properties button to view details about the card, or to reset the card. Reset a card to return it to the default state. The cards cannot be shut down because they are architected so that they are always in-service once they are inserted into the chassis. Monitor the InfiniBand Ports 1.
7. View the Admin Status and the Oper Status of the ports. The Admin Status refers to the state that the admin has chosen the port to be. The Oper Status refers to the actual state of the port. For example, an admin may have selected a state of “Up” for a port, but the Oper Status would display “down” if a cable had not been connected to the port. 8. Click the radio button next to a port whose details you want to view. 9. Click the Properties button. The Port Properties dialog box appears.
The Power Supplies view appears. Figure 5-15: Chassis Manager Power Supplies Window Monitor the Fans 1. Launch the Chassis Manager by entering the IP address of the InfiniBand switch into your web browser. 2. Log into the chassis. 3. The Device Rear View and the Device Front View appear. 4. View the Fan status at the top of the window to quickly check the status color and the number of modules installed. 5. • If only one number appears, then only one fan is installed.
Monitor the Temperature 1. Launch the Chassis Manager by entering the IP address of the InfiniBand switch into your web browser. 2. Press the Enter key. 3. The Device Rear View and the Device Front View appear. 4. Click open the Chassis file in the left-navigation tree. The individual components appear beneath the Chassis menu. Click Sensors in the left-navigation tree. The Sensors window appears. Figure 5-17: Chassis Manager Sensors Window 5. 6.
Monitoring the System with Element Manager For information regarding installing the optionally licensed Element Manager GUI, refer to the InfiniBand User Guide. 1. Launch the Element Manager. 2. Select Health > Status. The Health Status window opens. Figure 5-18: Element Manager Summary Tab Interpreting the Summary Tab 3. Use the Summary tab to view the status of the Power, Fans, and Temperature Sensors at once.
Interpreting the Power Supplies Tab Use the Health > Status > Power Supplies tab to view the operating status of the power supplies. Figure 5-19: Element Manager Power Supplies Tab Interpreting the Fans Tab Use the Health > Status > Fans tab to view the operating status of the fans. Figure 5-20: Element Manager Fans Tab 4. Note the OperStatus field. A status of “up” means that the fan is operating properly. 5. Note the Speed field. The integer in this field represents a percentage.
Figure 5-21: Element Manager Sensors Tab 6. Note the OperStatus field. A status of “up” indicates that the sensor is functioning properly. 7. Note the Temperature field. Each slot has an individual temperature sensor. The Alarm and Shutdown temperatures vary by slot type. Monitor the System With the CLI For a complete list of CLI commands, refer to the CLI Reference Guide.
Note the Speed field. The integer in this field represents a percentage. The percentage changes based on the ambient temperature of the unit, and will increase as the temperature rises.
6 Hardware Diagnostic Tests This chapter describes how to run diagnostic tests on the Topspin 270/Cisco SFS 7008 system hardware. • “About Diagnostic Tests” on page 91 • “Displaying Hardware Errors” on page 101 About Diagnostic Tests Hardware diagnostic tests can be performed through the CLI or the Element Manager GUI. Refer to the Element Manager User Guide for instruction on running tests in with the Java GUI. A Power-On Self Test (POST) is run automatically on various components at power-on.
• Power Supply page 98 • Rack-Locator Run Card Tests References to “Cards” refers to fabric controllers, management I/O modules, node cards and the chassis ID module. Fans and power supplies are not included in the Card tests; they can be tested through the standard chassis test (page 95), or as individual components.
Example of completed test SFS-270(config-diag-card-11)# exit SFS-270(config)# exit SFS-270# show diagnostic card 11 ================================================================================ Diagnostic Tests For Cards ================================================================================ test : self-test slot-id : 11 iterations : 1 action : start result : success percentage-completed : 100 result-string : Card Test, Final report : PASSED; Please reboot system
3. Use the more file-system:file-name command to display the log file and view the detailed results of the diag test. Example of log results SFS-270# more syslog:hwif_log Thu Mar 4 10:57:49 2004: POST: SEEPROM: PASSED Thu Mar 4 10:57:49 2004: POST: FPGA: PASSED Thu Mar 4 10:57:49 2004: POST: SUMMIT: PASSED Thu Mar 4 10:57:50 2004: POST: RTC: PASSED Thu Mar 4 10:57:54 2004: POST: FAN: PASSED Thu Mar 4 10:57:54 2004: card_startup.
• Standard page 95 • Extended • Extended Memory Run a Chassis Standard Test The chassis standard diag test runs a diagnostic test on all cards in the chassis. 1. Enter the diag chassis test standard and start commands to initiate the test. Example SFS-270> enable SFS-270# config SFS-270(config)# diag chassis test standard SFS-270(config-diag-chassis)# start SFS-270(config-diag-chassis)# 2. Enter the show diag chassis command to show the status of the diag test.
3. Enter the show diag post command to show any errors on individual components.
2. Enter the diag fan command.
4. Use the show diag fru-error command to display any errors that appear in the result-string.
Run a Self-Test on a Power Supply 1. Show the ID of the power supply that you want to diagnose, if you do not already know the number. a. Enter the show power-supply command.
5. Use the show diag fru-error command to display any errors that appear in the result-string.
3. View the results of the test.
You can also view POST error codes with the show diagnostic post command. This command displays the results of each POST that executed at power-on.
105 7 Specifications and Compliance Certifications This chapter details the Topspin 270/Cisco SFS 7008 specifications and compliance certifications. Chassis and Management Interface Table 7-1: Chassis and Management General Specifications Operating Temperature Ranges 0 to 40C Operating Temperature 0 to 42C Non-Operating Temperature -40 to 70C Operating Temperature Gradient 20C max.
106 Table 7-1: Chassis and Management General Specifications Operating Temperature Ranges 0 to 40C Non-Operating Vibration, random 2.09Grms, 3-axis, bottom/top, left/right, front/back Max. Operating Inclination 15 degrees Electrical Specifications Table 7-2: Electrical Specifications Category Specification AC Input Auto-ranging 90-264VAC, 47-63Hz.
107 Product Markings • UL, FCC Statement, VCCI, CE, ICES
108
Index A active controller card 11 admin status 83 C card operational state 81 reset 82 card reset 82 chassis ID about 19 LEDs 76 chassis manager launching 33 LEDs 80 controller card identify active 11 upgrade 59 controllerFabric12x 81 core slot about 18 D diag card 92 diag chassis test standard 95 diag power-supply 99 F fabric controller LEDs general 72 master status 72 module status 72 fabric controller modules about 18 fabric12x 81 fail-over fabric controller 42 management interface module 18 management s
R rack-locator 92 reload 94 reset card 82 reset controller card 59 S safety information vii setup 32 show card 11 show diag chassis 95 show diag fru-error 98, 100 show diag post 96 show power-supply 99 slot numbers 15 speed port 83 starting Chassis Manager 33 switch configuration 32 switch fabric end-nodes about the fabric controller module 18 switch fabric spine about the fabric controller module 18 system controller about the fabric controller module 18 system-wide status LED 68 T temperature monitor with
