Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide Corporate Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
Contents New and Changed Information Preface vii ix Audience ix Organization ix Conventions x Related Documentation x Obtaining Documentation x Cisco.
Contents InfiniBand General Services Managers and Agents SNMP Support 7 SNMP MIBs 7 CHAPTER 2 6 Installation 9 Statement 1019—Main Disconnecting Device 10 Statement 1045—Short-circuit Protection 11 Statement 1074—Comply with Local and National Electrical Codes 13 Statement 1075—Hazardous Voltage or Energy Present on DC Power Terminals 14 Planning the Installation 15 Environmental Requirements 15 Rack Specifications and Recommendations 16 Installing and Routing Cable 16 Power Requirements 17 Installatio
Contents Hot Swapping Components 39 Hot Swapping Spine and Leaf Modules Hot Swapping the Fan Unit 40 Hot Swapping Power Supplies 40 CHAPTER 3 Operations and Administration 39 41 Chassis Viewer 41 The Chassis Viewer Manages 41 Home Page 42 ? (Help) Button 42 Support Button 42 Displaying the Leaf and Spine Module Views 43 Leaf Module View 43 Spine Module View 44 Leaf and Spine Module Component Details Area 44 Leaf and Spine Details Header 45 Leaf and Spine Information Area 45 Displaying the Chassis Vie
Contents Chassis Traps 64 SFS 7012 Port Statistics 67 Port Statistics Field Descriptions 68 Leaf and Spine Module IB Port Statistics 70 Leaf Modules 70 Spine Modules 70 Set Field Thresholds 71 Time Service 73 Configuring the Switch OOB IP Address 76 Configuring the Switch Default Gateway IP Address Spine View Menu 78 Logging 78 Purging the Log 79 Select Boot Image 80 License Keys; Key Administration 80 Adding a New License Key 81 Deleting a License Key 81 APPENDIX A Technical Specifications APPENDIX B
Contents Capture 92 Deprecated 93 APPENDIX C Troubleshooting 95 Hardware Checks 95 Switch 95 Problem 95 Fix 95 Power Supply 95 Problem 95 Fix 95 Problem 96 Fix 96 OOB Ethernet RJ45 Port 96 Problem 96 Fix 96 SFS 7012 Leaf Module IB Ports Problem 96 Fix 96 96 Troubleshooting Scenarios 97 InfiniBand 97 Invalid IP Address entered for SWC via Console Port Bad IB Cable 97 Improperly Seated IB Cable 97 97 Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide v OL-8787-04
Contents Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide vi OL-8787-04
New and Changed Information The Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide applies to the SFS 7012 Release 3.1 or later. Table 1 lists the new and changed features available with each supported SFS 7012 release.
Table 1 Documented Features for the Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide (continued) Changed in Release Feature Description Redundant Management Added redundant management information. 3.3 Where Documented Installing the Spine and Leaf Modules, page 2-26 Updating Management Spine IP Addresses in a Redundant Management Configuration, page 2-31 Rebooting Multiple Managed Spines, page 2-38 Double Data Rate (DDR) capabilities Added DDR information 3.
Preface This preface describes the audience, organization, and conventions of the Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide . It also provides information on how to obtain related documentation. Audience The intended audience for this document are network administrators responsible for configuring and operating network equipment.
Conventions This document uses the following conventions for notes, cautions, and safety warnings. Notes and Cautions contain important information that you should be aware of. Note Caution Means reader take note. Notes contain helpful suggestions or references to material not covered in the publication. Means reader be careful. You are capable of doing something that might result in equipment damage or loss of data.
You can access international Cisco websites at this URL: http://www.cisco.com/public/countries_languages.shtml Product Documentation DVD The Product Documentation DVD is a library of technical product documentation on a portable medium. The DVD enables you to access installation, configuration, and command guides for Cisco hardware and software products. With the DVD, you have access to the HTML documentation and some of the PDF files found on the Cisco website at this URL: http://www.cisco.
To see security advisories, security notices, and security responses as they are updated in real time, you can subscribe to the Product Security Incident Response Team Really Simple Syndication (PSIRT RSS) feed. Information about how to subscribe to the PSIRT RSS feed is found at this URL: http://www.cisco.com/en/US/products/products_psirt_rss_feed.html Reporting Security Problems in Cisco Products Cisco is committed to delivering secure products.
Obtaining Technical Assistance Cisco Technical Support provides 24-hour-a-day award-winning technical assistance. The Cisco Technical Support & Documentation website on Cisco.com features extensive online support resources. In addition, if you have a valid Cisco service contract, Cisco Technical Assistance Center (TAC) engineers provide telephone support. If you do not have a valid Cisco service contract, contact your reseller.
Submitting a Service Request Using the online TAC Service Request Tool is the fastest way to open S3 and S4 service requests. (S3 and S4 service requests are those in which your network is minimally impaired or for which you require product information.) After you describe your situation, the TAC Service Request Tool provides recommended solutions. If your issue is not resolved using the recommended resources, your service request is assigned to a Cisco engineer.
• The Cisco Product Quick Reference Guide is a handy, compact reference tool that includes brief product overviews, key features, sample part numbers, and abbreviated technical specifications for many Cisco products that are sold through channel partners. It is updated twice a year and includes the latest Cisco channel product offerings. To order and find out more about the Cisco Product Quick Reference Guide, go to this URL: http://www.cisco.
Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide xvi OL-8787-04
C H A P T E R 1 Product Overview The Cisco SFS 7012™ is an industry leading modular system used for creating large, single-system Grid/Cluster server fabrics, or as a building block for larger fabrics. The SFS 7012 is designed to maximize performance, streamline operations, and ensure uptime by providing full hardware and software reliability, availability, and serviceability (RAS) features.
Chapter SFS 7012 Product Specifications Mechanical/Thermal/Power Specifications • 7U - 19" rack mount environment • Integrated thermal management • Front-to-back airflow • Hot plug, redundant cooling • Hot plug, redundant power supplies • 90/264 VAC operation • 50/60 Hz Frequency • Redundant, isolated AC inputs • FCC and VCCI compliant • U.L.
Chapter • Available power (all fiber): 879 Watts max • Available air flow: 200 cfm • Thermal status reporting to CME via IBML • Up to one hundred forty-four (144) 4X IB external copper ports • Two (2) RJ-45 connectors for 100BASE Ethernet connection to management LAN. • One (1) RJ-11 serial port per spine module for configuration. Chassis Viewer Software The Chassis Viewer is Cisco”s browser-based management software.
Chapter – Perform maintenance tasks – Manage firmware updates, including uploading new firmware and setting the active version.
Chapter Subnet Manager Every InfiniBand fabric subnet requires a single active Subnet Manager (SM) to initialize and maintain the subnet. The fabric may also have one or more standby Subnet Managers. One of the standby Subnet Managers can be configured to become the active SM in the event of a failure. As Figure 1-2 shows, InfiniBand-enabled fabrics consist of one or more subnets. Each subnet consists of a number of InfiniBand-capable servers connected to target devices.
Chapter An SMA also runs on every InfiniBand-capable server that is equipped with an HCA. As shown in Figure 1-3 the Subnet Manager interacts with these SMAs to discover information about the nodes on the fabric.
Chapter The PM and PMA exchange messages about performance statistics and error information of InfiniBand devices on the fabric. SNMP Support Built-in support for Simple Network Management Protocol (SNMP) allows users to integrate a SFS 7012 into their existing management frameworks. SNMP allows users access to all statistics, trend analysis, alarm handling, filtering, and performance monitoring capabilities supported by these management frameworks.
Chapter Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide 8 OL-8787-04
C H A P T E R 2 Installation This chapter describes how to install the Cisco SFS 7012™ and its components, and it includes the following information: Note • Planning the Installation, page 2-15 • Installation Tasks, page 2-22 • Hot Swapping Components, page 2-39 Before you install, operate, or service the system, read the Regulatory Compliance and Safety Information for the Cisco SFS 7012 and 7024 for important safety information.
Chapter Warning IMPORTANT SAFETY INSTRUCTIONS This warning symbol means danger. You are in a situation that could cause bodily injury. Before you work on any equipment, be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents. Use the statement number provided at the end of each warning to locate its translation in the translated safety warnings that accompanied this device.
Chapter Aviso A combinação ficha-tomada deverá ser sempre acessível, porque funciona como interruptor principal. ¡Advertencia! El conjunto de clavija y toma ha de encontrarse siempre accesible ya que hace las veces de dispositivo de desconexión principal. Varning! Man måste alltid kunna komma åt stickproppen i uttaget, eftersom denna koppling utgör den huvudsakliga frånkopplingsanordningen.
Chapter Aviso Este produto requer proteção contra curto-circuitos (sobreintensidade de corrente), que deve estar instalada nos edifícios. Instale apenas de acordo com as normas de instalação elétrica nacionais e locais. Advertencia Este producto necesita estar conectado a la protección frente a cortacircuitos (sobretensiones) que exista en el edificio. Instálelo únicamente en conformidad con las regulaciones sobre cableado, tanto locales como nacionales, a las que se tenga que atener.
Chapter Statement 1074—Comply with Local and National Electrical Codes Warning Installation of the equipment must comply with local and national electrical codes. Statement 1074 Waarschuwing Bij installatie van de apparatuur moet worden voldaan aan de lokale en nationale elektriciteitsvoorschriften. Varoitus Laitteisto tulee asentaa paikallisten ja kansallisten sähkömääräysten mukaisesti. Attention L'équipement doit être installé conformément aux normes électriques nationales et locales.
Chapter Statement 1075—Hazardous Voltage or Energy Present on DC Power Terminals Warning Hazardous voltage or energy may be present on DC power terminals. Always replace cover when terminals are not in service. Be sure uninsulated conductors are not accessible when cover is in place. Statement 1075 Waarschuwing Op DC-aansluitingspunten kunnen zich gevaarlijke voltages of energieën voordoen.
Chapter ¡Advertencia! Puede haber energía o voltaje peligrosos en los terminales eléctricos de CC. Reemplace siempre la cubierta cuando no estén utilizándose los terminales. Asegúrese de que no haya acceso a conductores descubiertos cuando la cubierta esté colocada. Varning! Farlig spänning eller skadlig energi kan finnas i likströmsterminalerna. Sätt alltid tillbaka höljet när terminalerna inte används. Försäkra att inga oisolerade ledare kan nås när höljet sitter på plats.
Chapter Rack Specifications and Recommendations The SFS 7012 switch is designed to be installed in an existing 19-inch equipment rack or server rack. The SFS 7012 switch is designed for a four-post server cabinet. It is not designed for a two-post telco cabinet. Racks should conform to conventional standards.
Chapter Figure 2-3 Bend Radius Measurement Diagram • Support cable using a cable manager mounted above connectors to avoid unnecessary weight on the cable bundles. • Bundle cables using velcro straps to avoid damaging cables. • Keep all ports and connectors free of dust. • Unshielded Twisted Pair (UTP) cables can build up Electrostatic Discharge (ESD) charges when being pulled into a new installation.
Chapter Installation Tasks Checklist To perform the actual switch installation, the site implementation engineer must perform the following tasks, which are detailed in this section. Caution Be sure to review the Safety Information on page 18 before starting the installation and during the installation process. Step 1 Check the installation site to verify the installation of cabinet power feeds, rails, and grounding. Step 2 Unpack the equipment and inspect for any shipping damage.
Chapter Precautions for Rack-Mountable Products Warning Installing system components in a rack without the front and side stabilizers installed could cause the rack to tip over. Therefore, always install the stabilizers before installing components in the rack. Warning After installing system components in a rack never pull more than one component at one time out of the rack on its slide assemblies. The weight of more than one extended component could cause the rack to tip over.
Chapter Warning This unit may have more than one power cord. To reduce the risk of electrical shock, disconnect both cords before servicing the unit. Warning To avoid potential electrical shock, operate this unit only when the cover is in place. Warning To avoid potential electrical shock, use only a grounded (three wire) electrical outlet. Warning Keep objects that might damage this unit and liquids that might spill clear from this unit.
Chapter Check the Installation Site The SFS 7012 switch is designed to be installed in an existing server cabinet (not a telco cabinet), where it can be mounted in a standard equipment rack. Mounting brackets are integrated with the switch. Be sure of the following: • The cabinet has a full earth ground to provide reliable grounding. • There is enough room to work on the equipment. • The equipment will have enough clearance for front and rear access. • The IB cables can be accessed easily.
Chapter Installation Tasks The following is an overview of the installation tasks detailed in this section: Step 1 If applicable, remove the doors of the rack. Step 2 Mark the rack, allowing 7 U (12.25 inches) of vertical space to install the SFS 7012 switch. Step 3 Install the support rails. Step 4 Rack mount the switch. Step 5 If applicable, replace the rack’s doors.
Chapter Step 1 Install the caged nuts into the 2 back holes (chassis leaf module side) and the top and bottom holes (chassis fan side) in the front of the rack. Note The 2 front holes should match up with the top and bottom holes of the rail front flange. All holes should correspond to the rail mounting positions (i.e., the holes marked with pen or tape). Step 2 Fasten the rail back flange (chassis leaf module side) to the rack by installing two screws into the rail and rack.
Chapter Rack-Mount the Switch Caution Because of its size and weight, it is recommended that either a lifting device or three (3) people install the SFS 7012. To install the switch into the rack, perform the following steps: Step 1 Clear the area of any unnecessary materials. Step 2 Attach the clip of the ESD wristband (strap) to bare metal on the cabinet. Put the wristband around one wrist with the metal button against the skin.
Chapter Figure 7 Secure Chassis to Rack Install 1 screw and caged nut into each hole above and below the handle Step 6 Using two screws, install the lower mounting bracket to the rail assembly and chassis as shown in Figure 8. Figure 8 Install the Lower Mounting Bracket and Heyclip Install the lower rear mounting bracket to the ra assembly and chassis Install the heyclip here Step 7 Install the heyclip to the rail assembly. Step 8 If applicable, reinstall the chassis fascia(s).
Chapter Step 1 On the switch fan side, insert the notches on the top of the fascia into the two slots on the chassis frame. Snap the bottom of the faceplate in place. Removing a Module or Blank The handles are self-locking. To unlock, push up on the handles to disengage from the lock notch. Then gently pull the handles out and slide the module out of the slot. Note If removing, but not replacing a module, remember to replace with a module blank.
Chapter Figure 2-9 b. SFS 7012 Chassis — Spine Module Slot Numbering Leaf Modules— Leaf modules should be populated beginning with slot 1, then slot 2, then slots 3 through 12 respectively. Refer to Figure 2-10: Figure 2-10 SFS 7012 Chassis — Leaf Module Slot Numbering Leaf 11 Leaf 12 Leaf 9 Leaf 10 Leaf 7 Leaf 8 Leaf 5 Leaf 6 Leaf 3 Leaf 4 Leaf 1 Leaf 2 Step 2 To install a module or filler, hold it so that the ejector handles are on the bottom.
Chapter Figure 2-11 SFS 7012 Chassis Front View Figure 2-12 SFS 7012 Chassis Back View Leaf 11 Leaf 12 Leaf 9 Leaf 10 Leaf 7 Leaf 8 Leaf 5 Leaf 6 Leaf 3 Leaf 4 Leaf 1 Leaf 2 AC power inlet for power supplies 1 to 3 RJ-45 for Spine 1 AC power inlet for power supplies 4 to 6 RJ-45 for Spine 2 Step 1 Connect a Category 5 or 6 (Cat 5/6) Ethernet cable to one of the RJ-45 connectors on the SFS 7012. Connect the other end of the Cat 5/6 to the OOB LAN workstation.
Chapter Note When handling IB connectors, make certain to remove the connectors by pulling on the center of the lanyard only as shown in Figure 2-13 below. Pulling abruptly on the lanyard, or pulling on only one side of the lanyard will prevent the latch/unlatch operation from occurring, and could damage or break the lanyard. Figure 2-13 4X IB Cable Connector Connecting Power Step 1 Note Provide strain relief for the power cable(s) by feeding them into the heyclips on the lower rail assemblies.
Chapter Note If the spine module RS-232 port is connected to a terminal emulation program, the user will be able to view the switch boot process. Be certain to use a null-modem/crossover serial cable for the console port. The settings for the terminal emulation device should be: – 8 data bits – no parity bits – 1 stop bit – 56K baud – Use VT100 emulation.
Chapter Step 2 Note Power up the SFS 7012. If using the RS-232 port, skip to Step 6. Steps 3 through 5 are for those users accessing the SFS 7012 via Telnet or SSH. Step 3 If using Telnet or SSH, access the SFS 7012 with the following command: Step 4 The system prompts for a user name. In order to change the IP address and default gateway, the user must be logged in as the administrator. At the prompt enter: Step 5 The system prompts for a password. At the prompt enter: open 192.168.100.
Chapter Step 1 Ensure that the spine module is connected to a COM port on a serial terminal device via the RS-232 port. Step 2 Get to a [boot]: prompt by following Step a or b. a. If the managed spine is running, enter the following command at the console: reboot Then press Enter. b. If the managed spine is not running, power on the switch. Step 3 After selecting image1 or image2, press the Spacebar to interrupt the auto load sequence before the counter expires (within 5 seconds).
Chapter Image Date: Jan 19 2006, 15:03:31 Checking L2 functionality... BCM1125 L2 caches initialized and invalidated CPU0 caches initialized Initialized SMBUS Channels SPD Checksum ok.
Chapter md5 values match! 11088736 Starting at 0x80010000... sysI REGS BOOT Initializing HyperTransport bus HyperTransport initialization completed Found Intel Strata Flash 128 MBit (0x8918). Mounted raw file system on device /image1. (size=6291456 bytes) Mounted raw file system on device /image2. (size=6291456 bytes) Mounted raw file system on device /dump0. (size=1048576 bytes) Mounted flash file system on device /rfa1. (size=2097152 bytes) Network configuration requested.
Chapter The chassis status LED is off when: • There are no functional power supplies present. • There are no management cards in the system • AC power has been removed from the system. Subnet Manager Agent (SMA) LED There is a single SMA LED associated with the system. This LED is Blue if any external InfiniBand links have been established. If no external IB links have been established, the LED if off. RJ45 LEDs The SFS 7012 RJ45 connector has two LEDs, Act and 100.
Chapter • Fast Blink - IB data is flowing through the port. • Off - the physical link is down (port is in the Down state). Spine and Leaf Module Status LED The status LED indicates one of the following conditions: • Steady Green - the module is operating normally. • Blinking Green (twice per second) - transitional to the removal state. • Blinking Green (once every four seconds) - LED test state. • Off - module is in the removable state.
Chapter Accessing On-line Help Online help can be accessed from anywhere within the Chassis Viewer by clicking on the ? buttons as shown in Figure 2-15: Figure 2-15 Chassis Viewer Online Help Button(s) The on-line help screens contain topical information with textual references to more detailed information within the documentation.
Chapter Figure 2-16 Sample Chassis Viewer On-line Help Screen Shutdown Procedures In order to shutdown the SFS 7012: Step 1 Power down the switch by removing the power cords from the AC power inlets. Rebooting Components from Chassis Viewer Step 1 From the Chassis Details header, click Reboot. A screen similar to the following is displayed.
Chapter Step 2 The system prompts for a user name. At the prompt enter: Step 3 The system prompts for a password. At the prompt enter: admin adminpass The system responds with: Welcome to the SFS 7012 CLI. Type 'list' for the list of commands. Step 4 To reboot multiple spines on the SFS 7012 enter the following command: reboot now -m -n where: Note • now initiates the reboot process as soon as the user presses Enter (i.e., no system y/n prompt).
Chapter Step 4 Push the handles in to fully close. Hot Swapping the Fan Unit Step 1 Loosen the captive panel screw. Step 2 Pull the panel screw down to partially disengage the unit. Step 3 Slowly pull the unit. The unit will disengage from the connector. Step 4 Carefully slide the fan out until it is completely removed from its slot. To install a fan unit: Step 1 Place the unit into the slot. Slowly slide the fan unit in until it engages into the connector.
C H A P T E R 3 Operations and Administration This chapter describes how to administer and configure the Cisco SFS 7012™ and its components, and it includes the following information: • Logging, page 3-53 • Firmware Update, page 3-58 • SNMP, page 3-59 • Chassis Traps, page 3-64 • SFS 7012 Port Statistics, page 3-67 Chassis Viewer The Chassis Viewer is Cisco's browser-based management software.
Chapter Home Page Figure 3-1 Chassis Viewer Home Page The Chassis Viewer’s home page provides a high-level overview of the SFS 7012 switch and individual leaf and spine modules. This area is the starting point to more detailed information for the SFS 7012 chassis and components (fans and power supplies), leaf modules, and spine modules. The selected component provides hyperlinks to related menus and information where the user can perform configuration and monitoring tasks.
Chapter Figure 3-4 Support Contact Screen Displaying the Leaf and Spine Module Views Leaf Module View To display the leaf module view for the SFS 7012: Step 1 Mouse over the leaf module to display. The edges of the leaf module are highlighted green as shown in Figure 3-5 below: Figure 3-5 Leaf Module Mouseover Left click the mouse. The leaf module view will be displayed.
Chapter Figure 3-6 Leaf Module View Spine Module View To display the spine module view for the SFS 7012: Step 1 Mouse over the spine module to display. The edges of the spine module are highlighted green as shown in Figure 3-7 below: Figure 3-7 Spine Module Mouseover Left click the mouse. The spine module view will be displayed. Figure 3-8 Spine Module View Leaf and Spine Module Component Details Area The Component Details Area contains three areas.
Chapter • Information area. • Menu Figure 3-9 Leaf and Spine Component Details Area Leaf and Spine Details Header The leaf and spine Details Header allows the user to execute the most common tasks for the SFS 7012 switch: • Display the Home page • View field-replaceable unit (FRU) information.
Chapter Step 1 Mouse over the outer region of the leaf module view. The edges of the chassis are highlighted green as shown in Figure 3-11 below: Figure 3-11 Leaf Module Chassis Mouseover Left click the mouse. The chassis view will be displayed. Step 2 The second way is to mouse over the outer region of the spine module view. The edges of the chassis are highlighted green as shown in Figure 3-12 below: Figure 3-12 Spine Module Chassis Mouseover Left click the mouse. The chassis view will be displayed.
Chapter Figure 3-13 Chassis Viewer Component Details Area Chassis View Component Details Area The Component Details Area contains three areas. • Details Header • Information area.
Chapter Figure 3-15 Reboot Screen Step 2 Select the radio button of the spine(s) to be rebooted, or select the Reboot Entire Chassis radio button to reboot the SFS 7012 chassis and all spines. Step 3 Click Reboot. Rebooting Multiple Managed Spines In a redundant management configuration it may occasionally be necessary for the user to reboot both managed spines. This is accomplished through the CLI of the master spine.
Chapter Chassis View Component Information Area Figure 3-16 Component Information Area The Chassis View Component Information Area allows the user to view important system information. These fields are tied to live data from the selected hardware component. Additionally, the Component Information Area has Apply and Refresh buttons, which perform the following functionality: Apply: Saves any user edits within the white fields to flash memory. Refresh: Refreshes all fields in the information areas.
Chapter Figure 3-17 LEDs and Sensors Tab Note For a detailed explanation of physical LEDs on the hardware components, please refer to the section SFS 7012 Component LEDs, page 2-34. System Tab The System tab displays the overall system information of the SFS 7012. This information includes the following items: Out of Band LAN IP The IP address of the switch. Note that changes to this field only take effect after a switch power cycle (i.e., shutting the switch and powering it back up).
Chapter The textual identification of the contact person and their contact information for this system, assigned by the administrator. System Name The name for the system, assigned by an administrator. One convention is to use the system's fully qualified domain name as the System Name. System Location The location of the system, assigned by an administrator. Apply Button Saves any changes made by the user in the System tab to memory. Refresh Button Refreshes all fields in the System tab.
Chapter Displays the part number of the component. Detail A button for each row that displays additional detail about the component. Additional details include: Model, Version, Manufacturer Name, Product Name, Manufacturer Id, Manufactured Date Apply Button Saves any changes made by the user in the Power tab to memory. Refresh Button Refreshes all fields in the Power tab. Fan Tab The Fan tab displays switch fan information. For descriptions of the fields, see the Power Tab.
Chapter Configuration and Monitoring Features The following section provides detailed, task-oriented descriptions for configuring and monitoring the SFS 7012 switch and its feature functionality. Chassis View Menu Figure 3-19 Chassis View Menu The Chassis View Menu allows the user to execute all configuration and operation tasks available for the SFS 7012 switch. Logging The Logging submenu allows the user to set and reset levels for the message log file.
Chapter Figure 3-20 Logging Submenu Set Level Figure 3-21 Set Level Button The Set Level button allows the user to set log level configuration parameters for all software modules on the SFS 7012. To set log levels: Step 1 From the menu, select Logging. Step 2 From Logging, select Set Level.
Chapter Figure 3-23 Device Tab: Software Module Configurator To change any Log Level settings: Step 1 Click the On-Off checkbox to the right of the setting. Step 2 Click the Submit button to save any changes. The following list describes each of the Log Level configuration parameters. – DUMP – Dump: Indicates that a problem has caused the system to produce a system dump file. In most circumstances, it is recommended that the user retrieve the dump that was produced.
Chapter Debug message levels 1 through 5: Debug messages are for supplier and/or Cisco engineering use and are not necessarily indicative of actions that an end user may need to take. – DEBUG1 – Messages that describe the states of connections and links. – DEBUG2 – Messages that describe major configuration changes or operations. – DEBUG3 – Messages that describe the I/O flow. – DEBUG4 – Messages that contain the packet dumps within an I/O flow. I/O flows contain multiple packets.
Chapter Syslog Tab Figure 3-25 Log System Configurator: Syslog Host Tab The Syslog tab allows the user to save log files to a syslog host. To setup the syslog host: Step 1 In the Host IP dialog box, enter the IP address of the syslog host where the log files are to be saved. Step 2 Click the Apply button to save the IP address. Note Do not change the Port setting. Configure Syslog on a Linux Server Step 1 Edit the /etc/sysconfig/syslog file and ensure that the -r is included in the SYSLOGD_OPTIONS.
Chapter Figure 3-27 Reset Log Levels Window Step 4 To reset the logging levels, click OK. Firmware Update Figure 3-28 Firmware Update Button The Firmware Update button allows the user to select an alternate firmware file for the SFS 7012. These alternate files are reflected in the drop-down lists in the Firmware Update screen. To download firmware: Step 1 From the menu, select Maintenance. Step 2 Click Firmware Update. The Firmware Update screen is displayed.
Chapter SNMP The SNMP submenu allows the user to configure and filter SNMP trap information. Figure 3-30 SNMP Submenu Target Configuration The Target Configuration button displays the SNMP Target Configuration Window, allowing the user to view and edit existing SNMP trap destinations. Figure 3-31 Target MIB Configuration Button To display the Target Configuration window: Step 1 From the menu, select SNMP. Step 2 Select Target Configuration.
Chapter Figure 3-32 SNMP Target Configuration Window The top section of the window, SNMP Target Addresses, allows the user to determine what type of SNMP traps are sent, and where they are sent. The rows provide an area for specifying multiple trap destinations. The bottom section of the window, SNMP Target Parameters, allows the user to configure each trap destination with version, optional security information, and filtering mechanisms.
Chapter Note The combination of the Transport Domain and the Transport Address determines the trap destination. • Port Specifies the TCP port which the SNMP trap will be sent. • Timeout Specifies the time that the trap sender waits on a response before re-sending the trap. • Retry Count Specifies the number of attempts to be made to send the trap after a timeout condition occurs. Note Timeout and Retry Count are SNMP v2.c and above. Not applicable for v1 traps.
Chapter • Security Model The Security Model to be used when generating SNMP messages using this entry. Values for this field are 1, 2, or 3. • Security Name Security name identifies the entity for whom SNMP messages will be generated. Note This is equivalent to the community string in an SNMP get. • Security Level One of three options: – NoAuthNoPriv: No Authentication, no privacy. – AuthNoPriv: Authentication, no privacy.
Chapter Step 1 From the menu, select SNMP Step 2 Click Filter Status. The SNMP Filter Status screen is displayed. Figure 3-34 Filter Status Screen Setting Community Strings The Set Community Stings screen allows the user to set two SNMP community names: • Read Only Community Name • Read/Write Community Name Figure 3-35 Set Community Strings Button To set the Community Strings: Step 1 Click SNMP Step 2 Click Set Community Strings. The Set Community Strings screen is displayed.
Chapter Figure 3-36 Set Community Strings Window The first field, "Read Only Comm. Name," is the community string that when specified in an SNMP client, allows read only access to SNMP fields exported by the SNMP server. The second field, "Read/Write Comm. Name," is the community string that when specified in an SNMP client, allows read and write access to SNMP fields exported by the SNMP server. Step 3 In each field, enter a meaningful name (such as public and private shown above), and click on Apply.
Chapter Step 5 To save settings, click on Apply. Following are definitions for each chassis trap: Chassis Group icsChassisTrapSystemSelfTestFailure This trap indicates that the chassis failed one or more of its self-test(s). icsChassisTrapSystemReboot This trap indicates that the chassis is in the process of rebooting. icsChassisTrapSystemMgmtSrvcStarted This trap indicates that the internal service used to support the management of the chassis is operational.
Chapter This trap indicates that a power supply is not responding to HEARTBEAT poll requests, that are issued by the internal chassis management service. icsChassisTrapPowerSupplyInserted This trap indicates that a power supply was inserted into the chassis. icsChassisTrapPowerSupplyRemoved This trap indicates that a power supply was removed from the chassis. icsChassisTrapPowerSupplyFailed This trap indicates that a power supply has failed and is not operational.
Chapter SFS 7012 Port Statistics The Chassis View Port Statistics area provides IB port information for all of the external and internal ports of the SFS 7012. Figure 3-39 IB Port Statistics To view port statistical information, do the following: Step 1 From the Chassis View Port Statistics submenu, select IB Port Stats.
Chapter Note Spine chips are referenced by the spine number and the switch chip identifier. Each spine module contains two switch chips (Switch chip A and B). Port Statistics Field Descriptions Port Name/Port Number: Corresponds to external leaf module number/port number, spine module number/port number, and all interswtich links. Link State: Indicates whether the InfiniBand link associated with the physical port is up or down. Possible values are no state change, down, init, armed, active, and unknown.
Chapter Receive 32 Bit Words: The number of 32-bit data words received by the port, not including flow control and VCRC data. Transmit Packets: The number of data packets transmitted by the port, not including flow control packets. Receive Packets: The number of data packets received by the port, not including flow control packets. Symbol Errors: The number of times an 8B10B encoding violation, or a disparity violation was detected.
Chapter Leaf and Spine Module IB Port Statistics To access IB port statistics for a specific leaf or spine module, perform the following steps. Leaf Modules Step 1 Select a leaf module. The leaf module view is displayed. 1. From the Leaf Port Stats menu, select IB Port Stats.
Chapter Figure 3-44 Spine Port Statistics Window For information on the each IB port statistic field, refer to the section Port Statistics Field Descriptions, page 3-68 Set Field Thresholds The Set Field Thresholds screen allows the user to set, for a specific parameter(s), an error message threshold for the 12 cable ports on the SFS 7012. Figure 3-45 Set Field Thresholds To change error reporting thresholds, do the following: Step 1 Click on Port Stats. Step 2 Click on Set Field Thresholds.
Chapter Figure 3-46 Set Error Reporting Thresholds Screen Step 3 To change a threshold value for any field: a. Click in the Threshold field. b. Enter a new threshold value. Note c. For those fields with a “Percent of Max” time unit, the user may enter a number from 0 to 100. For those fields with a “1 Second” time unit, the user may enter a number from 1 to 65,535. Click Apply. Note Any port which exceeds the configured thresholds will be displayed as red on the home page.
Chapter Note The thresholds for the following fields are set based upon the number of error message which can occur in one second. The value can be from 1 to 65,535. portSymbolErrThresh The threshold for the number of times a 8B10B encoding violation, or a disparity violation was detected on the port. portLinkErrRecvThresh The threshold for the number of times the link error recovery process happened successfully on the port.
Chapter Figure 3-47 Switch Time Service To set the system time: Step 1 From the menu, click Time Service. Step 2 Click Set System Time. The System Time Information screen is displayed: Figure 3-48 System Time Information Screen To use NTP: Step 1 Click the Use Network Time Protocol? checkbox. Step 2 Enter the IP address for the NTP server. Step 3 To save, click on Apply.
Chapter Step 1 Make sure the Use Network Time Protocol? checkbox is unchecked. Step 2 Set the current date and time using the drop-downs for Month, Day, and Year as well as Hour, Minute, Seconds, and AM/PM. Step 3 To save, click on Apply. Figure 3-50 Time Service - Manual Setup To set time zone and daylight saving time (DST) settings: Step 1 In the Time Zone drop-down, select the correct time zone based upon Greenwich Mean Time (GMT).
Chapter Note Beginning in 2007, Daylight Saving Time in the United States begins at 2 a.m. on the second Sunday of March, and ends at 2 a.m. on the first Sunday in November. – 2006 = Oct. 29 (5th) Additionally, for those US regions that do not observe DST, the start and end dates in the Which, Day, in Month settings, should be set to the exact same date. Configuring the Switch OOB IP Address To configure the Switch IP address: Step 1 From the OOB LAN IP submenu, click Set OOB LAN IP.
Chapter Figure 3-54 Set Switch Default Gateway IP Address Button Step 2 Click in the OOB IP Address field. Figure 3-55 Set Default Gateway IP Window Step 3 Type in the correct switch default gateway IP address. Step 4 Click Apply. Note A reboot is required to activate the new gateway IP address.
Chapter Spine View Menu Figure 3-56 Spine View Menu Note For information on IB port statistics refer to the section SFS 7012 Port Statistics, page 3-67. Logging The Logging submenu allows the user to view, set levels, reset levels, and purge the message log file. Figure 3-57 Logging Submenu Viewing the Log The View Log button allows the user to view the message log.
Chapter Step 1 From the menu, select Logging. Step 2 Click View Log. The log message window is displayed: Figure 3-59 Sample Message Log To save a log message for further analysis, perform the following steps: Step 1 From the Messages window, select Edit, Select All (or CTRL + A). Step 2 Select Edit, Copy (or CTRL + C). Step 3 Open a text editing package, such as Notepad. Step 4 Select Edit, Paste (or CTRL + V). Step 5 Save as a plain text (.txt) file.
Chapter Figure 3-61 Purge Log Confirmation Window Step 3 Click OK. Step 4 The message log file is now purged. Select Boot Image The Select Boot Image button allows the user to choose an alternative boot image for the SFS 7012. To select a boot image: Figure 3-62 Select Boot Image Button Step 1 From the menu, select Maintenance. Step 2 Click Select Boot Image.
Chapter Adding a New License Key Step 1 Click License Keys. The Key Administration button is displayed: Figure 3-64 License Key Submenu Step 2 Click Key Administration. The Key Management window is displayed: Figure 3-65 Key Management Window Step 3 To add a new license key, click the Add Key button. The License Key Management Information window is displayed: Figure 3-66 License Key Management Information Window Step 4 Note Enter the license key information in the New Key text box, and click Apply.
Chapter Figure 3-68 License Key Delete Prompt Step 4 Click OK to delete.
A P P E N D I X A Technical Specifications Table A-1 SFS 7012 Technical Specifications Switch Physical Dimensions Weight Cooling Environmental Power Supply Certification Marks • 7U high (24.5”) • 19" rack mount (17.32" wide) • • • 25 3/4" deep (without cables) 100 lb.
Appendix Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide 84 OL-8787-04
A P P E N D I X B Command Line Interface This appendix includes the following information: • Commands and Functional Groups, page B-86 • Accessing the CLI, page B-87 • Groups and Commands, page B-88 Overview This section details the usage of the Command Line Interface (CLI) feature for the Cisco SFS 7012™. The CLI allows the user to perform remote configuration and management tasks, which in many respects mirrors the functionality of the Chassis Viewer GUI.
Appendix The CLI is also accessible through the RS232 serial port of the SFS 7012. Accessing the CLI through the serial port never requires a login and password and defaults to administrator privileges. Providing access through the serial port means that users will always have access to the SFS 7012, even if Telnet and SSH are not functioning. Commands and Functional Groups The list of available commands can be accessed by typing list.
Appendix Capture: Provides commands for capturing switch-specific information for the purposes of analysis and debugging. To list commands within a functional group, simply type in the functional group name. For example, to list all of the firmware commands, type Firmware.
Appendix – Administrator access: admin Type the appropriate username and press Enter. Step 3 The system prompts for a password. The CLI has the following default passwords: – Operator access: operpass – Administrator access: adminpass Type the appropriate password and press Enter. The system responds with: Welcome to the SFS 7012 CLI. Type 'list' for the list of commands. Groups and Commands The following section lists the CLI’s functional groups along with the commands for each group.
Appendix exit Exit the CLI Deprecated smDisplayLids Use smShowLids smGetPriority Use smShowPriority smDumpGroups Use smShowGroups smDumpServices Use smShowServices smDumpSubscriptions Use smShowSubscriptions smDisplayMasterLMC Use smShowMasterLMC smDisplaySmMasterLid Use smShowMasterLid smDumpLidMap Use smShowLidMap smDisplayMaxLid Use smShowMaxLid smDisplaySMParms Use smShowSMParms ismTune Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide OL-8787-04 89
Appendix Chassis slotInfo Prints out chassis slot occupants hwMonitor Monitor fans, power supplies, and port state showIBNodeDesc Show the IB Node (SMA) Description (or the Default) setIBNodeDesc Set the IB Node (SMA) Description fruInfo Display field replaceable unit information memShow Displays internal memory usage chassisQuery Displays information about the line cards in a chassis showInventory Displays a detailed list of all hardware within the chassis Sample Output: MasterSpine1-> showInventory NAME:
Appendix Network ifShow Displays the interface statistics for the OOB management port routeShow Displays the interface routes for the OOB management port ping Send Ping packets to a specified IP address showChassisIpAddr Displays the chassis IP address setChassisIpAddr Change the chassis IP address showDefaultRoute Displays the default gateway IP address setDefaultRoute Change the default gateway IP address arpShow Displays the link level adress resolution protocol (ARP) table hostShow Displays the host na
Appendix Firmware fwUpdateSlot Update units firmware fwUpdateChassis Update units firmware fwListFiles Lists the contents of the firmware RAM disk fwShowUpdateParams Display firmware default update parameters fwSetUpdateParams Configure firmware default update parameters showCapability Display the capabilities/features fwVersion Display Firmware revisions bootQuery Query boot image information bootSelect Change boot selection Log logShow Display the Log file logClear Clear the Log file logConfigure Config
Appendix IbSwitchInfo ismPortStats Show statistics for all ports ismPortCounters Show Port Counters ismLinearFwdb Show/set Linear Forwarding Database for switch ismMultiFwdb Show Multicast Forwarding Database for switch ismAutoClearConf Configure the port statistics counter auto-clear options ismPortSet12x View/Set/Unset port link width to 12X for specific port[s] ismChassisSet12x View/Set/Unset chassis link width to 12X (including external ports) ismChassisSetSpeed View/Set port link speeds for entire cha
Appendix Capture capture A combination of all capture commands that follow captureFw Capture firmware information captureSm Capture subnet manager information captureIsm Capture IB switch information captureChassis Capture chassis information captureLog Capture log information captureMisc Capture miscellaneous information.
A P P E N D I X C Troubleshooting This appendix describes how to troubleshoot the Cisco SFS 7012™, and it includes the following information: • Hardware Checks, page C-95 • Troubleshooting Scenarios, page C-97 Hardware Checks Switch Problem Fix The Switch has no power Ensure that the power cord(s) is attached to the Switch and the power outlet. Ensure that the power supply(s) is seated properly.
Appendix Fan Problem Fix Red LED is lit Call Tech Support Fan not running Ensure fan is seated properly. If fan continues to be inoperable, call Tech Support OOB Ethernet RJ45 Port Problem Fix The SFS 7012 Ethernet Port(s) have a Ensure that the cable is a straight-through RJ45 Cable with a Noise Problem: Cat 5 cable (not a crossover cable). This problem would occur if there is a RJ45 cable that has poor shielding or If using a straight-through cable and still contact of pins.
Appendix Troubleshooting Scenarios InfiniBand This section documents common problems seen with the SFS 7012 switch. Invalid IP Address entered for SWC via Console Port Symptoms Cannot access the Chassis Viewer browser window. The browser window times out and Chassis Viewer will not come up. Resolution / Workaround Invalid IP Address entered for switch card via the console port.
Appendix Cisco SFS 7012 InfiniBand Server Switch Hardware Users Guide 98 OL-8787-04