User Guide FastFabric 7.
FastFabric 7.0 User Guide Information furnished in this manual is believed to be accurate and reliable. However, QLogic Corporation assumes no responsibility for its use, nor for any infringements of patents or other rights of third parties which may result from its use. QLogic Corporation reserves the right to change product specifications at any time without notice. Applications described in this document for any of these products are for illustrative purposes only.
Table of Contents Preface Intended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Related Materials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Documentation Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . License Agreements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Technical Support. . . . . . . . . . . . . . . . . . . . . . . . . . .
FastFabric 7.0 User Guide Fast Fabric IB Chassis Setup/Admin Menu . . . . . . . . . . . . . . . . . . . . . . . . . Menu Items Description. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Edit the Configuration and Select/Edit Chassis Files . . . . . . . . . Verify Chassis via Ethernet ping . . . . . . . . . . . . . . . . . . . . . . . . . Update Chassis Firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Set up Chassis Basic Configuration . . . . . . . . . . . . . . .
FastFabric 7.0 User Guide Menu Items Description. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Edit Configuration and Select/Edit Hosts Files . . . . . . . . . . . . . . Verify Hosts via Ethernet ping. . . . . . . . . . . . . . . . . . . . . . . . . . . Setup Password-less SSH/SCP . . . . . . . . . . . . . . . . . . . . . . . . . Copy /etc/hosts to all hosts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . Show uname -a for all hosts . . . . . . . . . . . . . . . . . . . .
FastFabric 7.0 User Guide Command Entry. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Common Input Commands. . . . . . . . . . . . . . . . . . . . . . . . . . . . . Screen-Specific Input Commands . . . . . . . . . . . . . . . . . . . . . . . Access to Live and Recent PM Historical Data . . . . . . . . . . . . . . . . . . iba_top TUI Screens. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Summary Screen. . . . . . . . . . . . . . . .
FastFabric 7.0 User Guide Switch Node Screens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Switch Node Selection Screen (500+ Switch Nodes). . . . . . . . . Switch Node Selection Screen (26-500 Switch Nodes) . . . . . . . Switch Node Selection Screen (1-25 Switch Nodes) . . . . . . . . . Switch Node Information Selection Screen . . . . . . . . . . . . . . . . Switch Node Device Information Screen . . . . . . . . . . . . . . . . . . Switch Node Port Selection Screen . . . . . .
FastFabric 7.0 User Guide Error Condition Screens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Switch Group Error Condition Selection Screen . . . . . . . . . . . . Integrity Error Selection Screen . . . . . . . . . . . . . . . . . . . . . . . . . Link Selection Screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Admin Menu Screens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Main Screen . . . . . . . . . . . . . . . . . . .
FastFabric 7.0 User Guide List of Figures Figure Page 1-1 FastFabric Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-3 2-2 QLogic InfiniBand Software Main Menu (Example) . . . . . . . . . . . . . . . . . . . . . . . . . 2-3 2-3 QLogic Fast Fabric InfiniBand Tools Menu (Example) . . . . . . . . . . . . . . . . . . . . . . . 2-5 2-4 Fast Fabric IB Chassis Setup/Admin Menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
FastFabric 7.0 User Guide 4-44 4-45 4-46 4-47 4-48 4-49 4-50 4-51 4-52 4-53 4-54 4-55 4-56 4-57 4-58 4-59 4-60 4-61 4-62 4-63 4-64 4-65 4-66 CA Port Selection Screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . CA Port Information Selection Screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . CA Port General Information Screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . CA Port Statistics Selection Screen . . .
Preface Intended Audience This manual is intended to provide network administrators and other qualified personnel a reference for installation, configuration and administration task information for the FastFabric Toolset. Related Materials QLogic FastFabric Command Line Interface Reference Guide QLogic Fabric Software Installation Guide Documentation Conventions This guide uses the following documentation conventions: NOTE: provides additional information.
To return to the root directory from anywhere in the file structure: Type cd /root and press ENTER. Enter the following command: sh ./install.bin Key names and key strokes are indicated with UPPERCASE: Press CTRL+P. Press the UP ARROW key. Text in italics indicates terms, emphasis, variables, or document titles. For example: For a complete listing of license agreements, refer to the QLogic Software End User License Agreement.
Technical Support Customers should contact their authorized maintenance provider for technical support of their QLogic products. QLogic-direct customers may contact QLogic Technical Support; others will be redirected to their authorized maintenance provider. Visit the QLogic support Web site listed in Contact Information for the latest firmware and software updates.
Knowledge Database The QLogic knowledge database is an extensive collection of QLogic product information that you can search for specific solutions. We are constantly adding to the collection of information in our database to provide answers to your most urgent questions. Access the database from the QLogic Support Center: http://support.qlogic.com.
1 FastFabric Overview Feature Overview The FastFabric Toolset is designed to both simplify and expedite common InfiniBand (IB) cluster management tasks. FastFabric can assist in generic management tasks as well as InfiniBand installation, upgrade, configuration and verification tasks.
1–FastFabric Overview Feature Overview Fabric topology analysis and verification Fabric route analysis Aids in ongoing fabric status and configuration monitoring Fabric Performance, Error and congestion monitoring Automated fabric health checks and configuration baseline compare Automated chassis health checks and configuration baseline compare Automated Subnet Management (SM) health checks and configuration baseline compare Provides tools to accelerate common host admin
1–FastFabric Overview FastFabric Architecture FastFabric Architecture Figure 1-1. FastFabric Architecture FastFabric is typically installed on one or more InfiniBand Management Nodes. The InfiniBand Management Node must be connected to the rest of the cluster through InfiniBand and a management network. The management network may be the primary InfiniBand network (IPoIB) or Ethernet. The management network will be used for FastFabric host setup and administration tasks.
1–FastFabric Overview FastFabric Architecture If remote access to FastFabric is desired, set up remote access to the InfiniBand Management Node using ssh, telnet, X-Windows, VNC or any other mechanism which will allow the remote user to access a Linux Command Line shell. Typically FastFabric is used only by cluster administrators. How FastFabric Works FastFabric consists of a variety of tools to administer hosts, chassis and externally managed switches.
2 FastFabric TUI Menu FastFabric TUI Menu Overview FastFabric is easiest to use from the textual user interface (TUI) menu system. The menu system provides a way to perform all common tasks and presents common options. Additional less common options are available directly, using the Command Line Tools, documented in the QLogic FastFabric Command Line Interface Reference Guide. In the sections that follow, the menu system will be discussed.
2–FastFabric TUI Menu FastFabric TUI Menu Overview If more than one item is selected, the items will be performed in the order shown in the menu. This is the typical order desired during fabric setup. If it's desired to perform items in a different order, select a single item and enter P to perform it by itself. Then repeat for the next item to be performed. An opportunity will be presented after each item is selected to abort as follows: Hit any key to continue (or ESC to abort)...
2–FastFabric TUI Menu QLogic InfiniBand Software Main Menu QLogic InfiniBand Software Main Menu The QLogic InfiniBand Software main menu is the top level menu for the QLogic InfiniBand Software. It can be activated using the iba_config command. This menu is not part of the FastFabric TUI. However, since it is one way of getting to the FastFabric Main Menu it will be summarized here. Figure 2-2 is a example of the QLogic Infiniband Software main menu. QLogic Inc.
2–FastFabric TUI Menu QLogic InfiniBand Software Main Menu Generate Supporting Information for Problem Report Menu item 5) Generate Supporting Information for Problem Report when selected proceeds through the process of generating a report and saving it to a user specified file. Fast Fabric (Host/Chassis/Switch Setup/Admin) Menu item 6) Fast Fabric (Host/Chassis/Switch Setup/Admin) when selected displays the Fast Fabric InfiniBand Tools menu. Refer to FastFabric Main Menu section below.
2–FastFabric TUI Menu FastFabric Main Menu FastFabric Main Menu The FastFabric main menu is the starting point to manage the fabric using the TUI. Selecting 6 from the above menu or executing the fastfabric command at a prompt, displays the Fast Fabric InfiniBand Tools menu (Figure 2-3) QLogic Inc. Fast Fabric InfiniBand Tools Version: VERSION 1) Chassis Setup/Admin 2) Externally Managed Switch Setup/Admin 3) Host Setup 4) Host Verification/Admin 5) Fabric Monitoring X) Exit Figure 2-3.
2–FastFabric TUI Menu Fast Fabric IB Chassis Setup/Admin Menu Fabric Monitoring Menu item 5) Fabric Monitoring when selected displays the Fast Fabric IB Fabric Monitoring Menu. Refer to “Fabric Monitoring” on page 2-29 for detailed information. Fast Fabric IB Chassis Setup/Admin Menu This menu is focused on initial setup and administration of QLogic 12000 internally-managed InfiniBand switches. Pressing the keys corresponding to menu items (0-9) will toggle the Skip/Perform selection for the given item.
2–FastFabric TUI Menu Fast Fabric IB Chassis Setup/Admin Menu Menu Items Description Selecting items 0 through c will change the item from skip to perform. Selecting N will unselect all items and X will exit the menu system. The items are described below. Edit the Configuration and Select/Edit Chassis Files (Switch) This will permit the chassis, ports and fastfabric.conf files to be edited.
2–FastFabric TUI Menu Fast Fabric IB Chassis Setup/Admin Menu Update Chassis Firmware (Switch) This will run the iba_chassis_admin update command to permit the chassis firmware version to be verified and updated as needed. NOTE: Any QLogic or SilverStorm 9000 chassis must be running firmware version 4.0.0.4.3 or later to perform this function. If the chassis is not up to this level, it will need to be manually updated using the chassis GUI. See the SilverStorm 9000 Users Guide for more information.
2–FastFabric TUI Menu Fast Fabric IB Chassis Setup/Admin Menu NTP Server IP Address Time zone and Daylight Savings Time (DST) Maximum MTU size, VL Capability and Link Layer Credit Distribution Link Width Supported IB Node Description (configured to match chassis ethernet name) The IB node description must be a string consisting of the characters A–Z, a–z,0–9, and underscore. No spaces are allowed in the node description string, and it may not begin with a digit.
2–FastFabric TUI Menu Fast Fabric IB Chassis Setup/Admin Menu Configure Chassis Fabric Manager (Switch) The Configure Chassis Fabric Manager selection will assist in configuring the Fabric Manager for any QLogic 12000 chassis with appropriate license keys. This operation will be skipped for other chassis models. Prompts will first guide the user through selection or generation of a qlogic_fm.xml file.
2–FastFabric TUI Menu Fast Fabric IB Chassis Setup/Admin Menu Get Basic Chassis Configuration (Switch) The Get Basic Chassis Configuration supports a new feature to retrieve basic information from chassis such as syslog, NTP configuration, time zone information, MTU Capability, VL Capability, VL Credit Distribution, Link Width and node description. The following is an example of the information retrieved: TEST SUITE getconfig CASE (getconfig.i12k71f.
2–FastFabric TUI Menu Fast Fabric IB Chassis Setup/Admin Menu (All): The answer to Would you like to perform fabric link speed error analysis indicates whether iba_report -o slowlinks should be run. If the user answers y to this question, the Check for links configured to run slower than supported question is asked. If the user answers y, the -o misconfiglinks option will also be used for iba_report.
2–FastFabric TUI Menu Fast Fabric InfiniBand Externally Managed Switch Setup/Admin Menu enableall - Enables Fabric Manager start on master and any slave MMs in selected chassis upon boot/reboot. disable - Disables Fabric Manager start on master and any slave MMs in selected chassis upon boot/reboot. Additional options prompted for: parallel vs serial operation prompting for chassis password (default is to have password in fastfabric.
2–FastFabric TUI Menu Fast Fabric InfiniBand Externally Managed Switch Setup/Admin Menu Fast Fabric IB Switch Setup/Admin Menu Externally Managed Switch List: /etc/sysconfig/iba/ibnodes Setup: 0) Edit Config and Select/Edit Switch Files [ Skip ] 1) Test for Switch Presence [ Skip ] 2) Verify Switch Firmware [ Skip ] 3) Update Switch Firmware [ Skip ] 4) Setup Switch Basic Configuration [ Skip ] 5) Reboot Switch [ Skip ] 6) Report Switch Firmware & Hardware Info [ Skip ] 7) Get Basic Sw
2–FastFabric TUI Menu Fast Fabric InfiniBand Externally Managed Switch Setup/Admin Menu Refer to FastFabric Command Line Interface Reference Guide for more details about the format of the ibnodes and ports file, and about the iba_gen_ibnodes command which can help generate the ibnodes file. Test for Switch Presence (Switch) This will run the iba_switch_admin ping command to test for the presence of the selected switches in the fabric.
2–FastFabric TUI Menu Fast Fabric InfiniBand Externally Managed Switch Setup/Admin Menu NOTE: Since the InfiniBand fabric itself is used to update externally managed switches, updating multiple switches with the reboot option may disrupt parallel update operations. If there are no selected externally managed switches in the path from the InfiniBand Management Node to any other externally managed switch, parallel operations may be used.
2–FastFabric TUI Menu Fast Fabric InfiniBand Externally Managed Switch Setup/Admin Menu NOTE: This only operates on QLogic 12200 switches. Any 9024FC switches selected will be skipped without change. Reboot Switch (Switch) This will run the iba_switch_admin reboot command to reboot all the switches listed in the /etc/sysconfig/iba/ibnodes file that was created in a previous step.
2–FastFabric TUI Menu Fast Fabric InfiniBand Externally Managed Switch Setup/Admin Menu Report Switch VPD Information (Switch) This will run the iba_switch_admin hwvpd command to provide the Virtual Product Data (VPD) for all the selected switches. This information can be useful for inventory and asset control as well as to provide details about the product to customer support.
2–FastFabric TUI Menu Fast Fabric IB Host Setup Fast Fabric IB Host Setup This menu is focused on initial host setup and installation of InfiniBand software on all the hosts. Pressing the keys corresponding to menu items (0-9, a-d) will toggle the Skip/Perform selection for the given item. More than one item may be selected. Once the desired set of items have been selected, enter P. To unselect all items, enter N. Entering X or pressing ESC will exit this menu and return to the Main Menu.
2–FastFabric TUI Menu Fast Fabric IB Host Setup Edit Configuration and Select/Edit Hosts Files (All) This will permit the hosts and fastfabric.conf files to be edited. The hosts file selected and created using this menu should not list the FastFabric host itself. After editing the two files, an opportunity is given to edit them again or continue forward.
2–FastFabric TUI Menu Fast Fabric IB Host Setup Install/Upgrade QLogic IB Software (Host) This will run the iba_host_admin load or iba_host_admin update command to install the QLogicIB software on all the hosts. By default it will look in the current directory for the FF_PRODUCT.FF_PRODUCT_VERSION.tgz file. If it is not found in the current directory, it will prompt for input of a directory name where this file can be found.
2–FastFabric TUI Menu Fast Fabric IB Host Setup Build MPI Test Apps and Copy to Hosts (Host) This will build the MPI sample benchmarks on the InfiniBand Management Node and copy the resulting object files to all the hosts. This is in preparation for execution of MPI performance tests and benchmarks in a later step. NOTE: This option is available for the QLogicIB packaging of OFED, but is not presently available for other packagings of OFED.
2–FastFabric TUI Menu Fast Fabric IB Host Verification/Admin Menu View iba_host_admin result files (All) This permits viewing of the test.log and test.res files that reflect the results from iba_host_admin runs (such as for installing InfiniBand software or rebooting all hosts per menu items above). The user is also given the option to remove these files after viewing them.
2–FastFabric TUI Menu Fast Fabric IB Host Verification/Admin Menu Fast Fabric IB Host Verification/Admin Menu Host List: /etc/sysconfig/iba/allhosts Validation: 0) Edit Config and Select/Edit Hosts Files [ Skip ] 1) Verify Hosts via Ethernet ping [ Skip ] 2) Summary of Fabric Components [ Skip ] 3) Check Status of IB Ports [ Skip ] 4) Verify Hosts see each other [ Skip ] 5) Verify Hosts ping via IPoIB [ Skip ] 6) Refresh ssh Known Hosts [ Skip ] 7) Check MPI Performance [ Skip ] 8)
2–FastFabric TUI Menu Fast Fabric IB Host Verification/Admin Menu Verify Hosts via Ethernet ping (All) This will run the pingall command. All the hosts listed will be pinged through the Management Network. Summary of Fabric Components (All) This will run the fabric_info command to provide a brief summary of the counts of components in the fabric including how many switch chips, hosts, and links are in the fabric.
2–FastFabric TUI Menu Fast Fabric IB Host Verification/Admin Menu (All): The answer to Would you like to perform fabric link speed error analysis indicates when iba_report -o slowlinks should be run. If the user enters y to this question, the Check for links configured to run slower than supported question is asked. If the user enters y, the -o misconfiglinks option will also be used for iba_report.
2–FastFabric TUI Menu Fast Fabric IB Host Verification/Admin Menu Refresh SSH Known Hosts (Linux) This will run the setup_ssh -U command to refresh the SSH known hosts list on this server for the IPoIB and Management Networks. This may be used to update security for this host if hosts are replaced, reinstalled, renamed, or repaired. Check MPI Performance (Host) This will do a quick check of PCI and MPI performance using end to end latency and bandwidth tests.
2–FastFabric TUI Menu Fast Fabric IB Host Verification/Admin Menu settings and any motherboard jumpers related to devices on PCI buses or slot speeds. Check Overall Fabric Health (Host) This will run the all_analysis command to check the overall fabric health. The user will be prompted: Baseline present configuration? [n]: If the user enters y, a new baseline will be created using the present fabric configuration.
2–FastFabric TUI Menu Fabric Monitoring For detail levels 2-4, the additional information is only gathered on the node running the captureall command. The information is gathered for every fabric specified in the /etc/sysconfig/iba/ports file. Run a command on all hosts (Linux) This will run the cmdall command. A Linux shell command (or sequence of commands separated by semicolons) may be specified to be executed against all selected hosts.
2–FastFabric TUI Menu Fabric Monitoring Fabric Performance Monitoring (All) This selection initiates iba_top.
3 iba_top Fabric Performance Monitor Introduction iba_top is a command line tool which displays performance, congestion, and error information about a fabric. Fabric information is divided into two areas performance and error statistics, which are the main starting points for analyzing fabric traffic.
3–iba_top Fabric Performance Monitor iba_top TUI iba_top: Img:Tue Apr 13 14:11:46 2010, Live Summary: Link:21 SW:4 SM:1 NodeFail:0 AvgMBps 0 All Int 1 HCAs TCA-Port:0 PortSkip:3 MaxMBps AvgKPps MinKPps 0 0 0 0 SmaCong:min Secure:min Snd 0 0 0 0 0 Rcv 0 0 0 0 0 Congst:min SmaCong:min Secure:min MaxKPps 0 0 0 Routing:min 0 0 0 0 0 0 Rcv 0 0 0 0 0 0 Congst:min SmaCong:min Secure:min 0 0 0 0 0 0 Snd 0 0 0 0 0 0 Rcv 0 0 0 0 0 Master-SM: LID:0x
3–iba_top Fabric Performance Monitor iba_top TUI Common Input Commands The following input commands are available in every screen: Q: Quit program; u*: Up to previous screen; L: Select Live image; R: Navigate reverse 1 (r*) or 5 (R*) sweeps; F: Navigate forward 1 (f*) or 5 (F*) sweeps; b*: Select (previously) Bookmarked image; B*: Bookmark currently selected image; U*: Unbookmark Bookmarked image; ?: Help Screen-Specific Input Commands The screen-specific input commands will be discussed with each screen
3–iba_top Fabric Performance Monitor iba_top TUI Screens iba_top TUI Screens Additional screens, described in the following paragraphs, are available to display detailed information about: PM configuration, PM sweep (image) configuration, performance statistics, error statistics, port group configuration, and port statistics (port counters). The screens can be navigated in a hierarchal manner to examine the state of a fabric.
3–iba_top Fabric Performance Monitor iba_top TUI Screens Performance and Error Statistics for Each Port Group Fabric performance and error statistics are presented based on four groupings of ports: All (all ports in the fabric), HCAs, TCAs and SWs. These groups provide a natural subdivision of the ports in a fabric for analysis. For more information about Groups and the operation of the PM, refer to the QLogic Fabric Manager User Guide.
3–iba_top Fabric Performance Monitor iba_top TUI Screens Screen-Specific Input Commands The summary screen accepts the following input commands: P: PM Configuration screen; I: Image Information screen 0-3: Select Port Group - All (0), HCAs (1), TCAs (2), SWs (3); Additional Screens After looking at the summary screen a user can decide which area of the fabric (performance or error) and which group of ports most warrants investigation, and can then drill down into that area.
3–iba_top Fabric Performance Monitor iba_top TUI Screens Image Information Screen The Image Information screen (Figure 3-12) displays image information as provided by the PM. Sweep start and duration, numbers of ports in each group, node and port information for the sweep, and SM information is shown. The Image Information screen has no screen-specific input commands. iba_top: Img:Tue Apr 13 15:01:53 2010, Live Image Info: Sweep Start:Tue Apr 13 15:01:53 2010 Sweep Duration:0.
3–iba_top Fabric Performance Monitor iba_top TUI Screens iba_top: Img:Tue Apr 13 15:05:25 2010, Live Group Info Select:All NumIntPorts:43 NumExtPorts:0 Group BW Summary (W) Group Err Summary (E) Group Config (C) Quit up Live/rRev/fFwd/bookmrked Bookmrk Unbookmrk ?help | W E C: Figure 3-13.
3–iba_top Fabric Performance Monitor iba_top TUI Screens The Bandwidth Statistics screen accepts input commands which specify parameters to be used in a group focus query, which will provide a list of ports (in the port group) sorted according to a specified performance criterion. The second line of the Bandwidth Statistics screen displays the group name, and the currently selected focus criterion and number of ports for a group focus query.
3–iba_top Fabric Performance Monitor iba_top TUI Screens iba_top: Img:Tue Apr 13 15:11:47 2010, Live Group Err Stats:All Criteria:Integ Int Max 0+% 25+% 50+% 75+% 100+% Integrity 0 43 0 0 0 0 Congestion 0 43 0 0 0 0 SmaCongest 0 43 0 0 0 0 Security 0 43 0 0 0 0 Routing 0 43 0 0 0 0 Congest %: 0.0 Ext Discard: Number:10 0 Ineffic %: 0.
3–iba_top Fabric Performance Monitor iba_top TUI Screens Congestion: Port Transmit Discards (neighbor port) Port Transmit Congestion (neighbor port) Port Transmit Wait (neighbor port) SmaCongestion: VL15 Dropped Errors Security: Port Receive Constraint Errors Port Transmit Constraint Errors (neighbor port) Routing: Port Receive Switch Relay Errors For each error subgroup five error 'buckets', from 0+% to 100+% in 25% increments, count the number of ports whose 'error compared to error threshold' value cor
3–iba_top Fabric Performance Monitor iba_top TUI Screens Group Configuration Screen The Group Configuration screen (Figure 3-16) displays a list of the ports in a group, including the LID, port number, port GUID and NodeDesc of each. The second line of the screen displays the group name and the number of ports returned in the group config query. If more ports exist than will fit on a screen, the list can be scrolled forward and backward.
3–iba_top Fabric Performance Monitor iba_top TUI Screens Group Focus Screen The Group Focus screen (Figure 3-17) displays a list of the ports the user has selected to focus on within a group, including the LID, port number, focus criterion, port GUID and NodeDesc of each. If the port has a neighbor port, the same information is displayed for the neighbor.
3–iba_top Fabric Performance Monitor iba_top TUI Screens iba_top: Img:Wed Sep 21 13:01:21 2011, Live Group Focus:All GrpNumPorts:43 Ix Integrity LIDx Port NumPorts:3 Node GUID 0x Number:3 NodeDesc 0 0 0001 1 00066A009800EC5B admin1 HCA-1 <-> 0 0002 1 00066A00070014DC i9k066 L02 1 0 0002 2 00066A00070014DC i9k066 L02 <-> 0 0003 1 00066A009800EC51 compute0001 HCA-1 2 0 000C 1 00066A0098007B5E compute0004 HCA-1 <-> 0 0007 24 00066A00D9000108 i9k108 Quit up Live/rRev/fFwd/bookmrk
3–iba_top Fabric Performance Monitor iba_top TUI Screens iba_top: Img:Mon Apr Port Stats:All 5 00:01:28 2010, Live LID:0x28 NodeDesc:i12k71f PortNum:10 Rate: 40g MTU:4096 NodeGUID:0x00066A00E300271F Neighbor:pyro HCA-1 LID:0x7 PortNum:2 Xmit: Data:0 MB (3168 Quads) Pkts:44 Recv: Data:0 MB (1152 Quads) Pkts:16 Integrity: SmaCongest: Symbol:0 VL15 Dropped:0 Link Recovery:0 Link Downed:0 Port Rcv:0 Security: Loc Lnk Integrity:0 Port Rcv Constrain:0 Excess Bfr Overrun*:0 Port Xmt Cons
3–iba_top Fabric Performance Monitor iba_top TUI Screens 4 — STDERR (iba_top) 16 — STDERR PaClient -q/--quiet — disable progress reports -h/--hca hca — hca to send by, default is 1st hca -p/--port port — port to send by, default is 1st active port -i/--interval seconds — obtain performance stats over interval seconds 3-16 IB0054607-01 A
4 Real-Time Fabric Monitor Real-time Fabric Monitor Overview The Real-time Fabric Monitor (RFM) is a TUI based user interactive application that provides real-time fabric monitoring support. It must be run on a host connected to the InfiniBand fabric with FastFabric installed. To use the Performance view of RFM the QLogic Fabric Manager must be installed. The RFM obtains all its data in an InfiniBand Trade Association (IBTA)-compliant manner.
4–Real-Time Fabric Monitor RFM Screen Layout and User Interaction Syntax iba_rfm [-v] [-h] [-p] [-i] [-I] or iba_rfm --help Options -v — verbose output.
4–Real-Time Fabric Monitor RFM Screen Layout and User Interaction > MENU: (A)dmin, (Q)uit FABRIC: Img: [Tue Mar 16 09:16:38 2010, Live], SWs: 8, HCAs: 5, Links: 28, GROUP 1 All TCAs: 0, SWsPorts: 62, TCAsPorts: 0 MinMBps MaxMBps AvgKPps MinKPps MaxKPps 0 0 0 0 0 Snd 0 0 0 0 Integ:Min SmaCong:Min Congst:Min Secure:Min 0 0 0 0 Congst:Min 0 0 Secure:Min Rtng:Min 0 0 Rtng:Min Snd 0 0 0 0 0 0 Rcv 0 0 0 0 0 0 Integ:Min 4 SWs HCAsPorts: 2, 0 Rcv 3 TCAs SMs: 2 Av
4–Real-Time Fabric Monitor Main Screen The screens of RFM are segmented into fixed sections that provide specific information about the fabric environment, and the actions taken by the user. The following subsections describe the different screens and the segments of each screen starting with the main menu shown in Figure 4-19. Main Screen The RFM main screen shown in Figure 4-20 shows a brake down of the segments in the TUI screen. The following subsections describe each segment of the screen.
4–Real-Time Fabric Monitor Main Screen Command-line Segment The Command-line segment of the screen (Figure 4-20) enables the user to submit commands to control RFM. This is the only section of the screen where user typing will appear. All other sections of the screen are output only. Menu Segment The Menu segment of the screen (Figure 4-20) displays any menu selections available for the user to utilize for the screen being displayed. The available selections have a bolded first letter.
4–Real-Time Fabric Monitor Main Screen . NOTE: HlthErrs and Alarms are not presently implemented and will report “Not Avail” Selection Segment The Selection segment of the screen (Figure 4-20) displays the various Views and Contexts selections available. Each View selection has a numeric View Identifier (VID) associated with it and each Context selection has a numeric Context Identifier (CID) associated with it.
4–Real-Time Fabric Monitor View Screen 3 — this option selects the Sma Congestion error category 4 — this option selects the Security error category 5 — this option selects the Routing error category 6 — this option selects the Adaptive Route error category (u)til h — this option selects the High BW utilization category p — this option selects the Packet Utilization High category l — this option selects the Low BW utilization category (p)rvPg — this sub-menu selecti
4–Real-Time Fabric Monitor View Screen > MENU: (H)ome, (F)abricView, (P)erformanceView, (A)dmin, (Q)uit VIEW: [I]:H CONTEXT: Name: Infrastructure View FABRIC: Top: [Wed Sep 21 13:10:13 2011, Live], SWs: 4, HCAs: Links: 21, 8, SWsPorts: HlthErrs: Not Avail, TCAs: 38, CommErrs: 0, Nodes: HCAsPorts: 65631, 12, 8, SMs: 2 TCAsPorts: 0 Alrms: Not Avail INFRASTRUCTURE VIEW INFORMATION CID CATEGORY TOTAL MONTRD OPRTNL HLTH-ERRS COMM-ERRS ALRMS 1 Switches 4 4 4 0 65600 Not Avail
4–Real-Time Fabric Monitor View Screen (F)abricView - this menu selection takes the user to the Fabric View main screen, which enables a user to focus on networking specific elements and characteristics of the fabric. This selection is available when in Infrastructure and Performance View. (I)nfrastructureView - this menu selection takes the user to the Infrastructure View main screen, which enables a user to monitor and query components in the fabric.
4–Real-Time Fabric Monitor View Screen MENU: (H)ome, (F)abricView, (P)erformanceView, (A)dmin, (Q)uit VIEW: [I]:H CONTEXT: Name: Infrastructure View FABRIC: Top: [Wed Sep 21 13:10:13 2011, Live], Figure 4-24. Context Section The Context Section will always contain the name of the context and could include other information at various levels.
4–Real-Time Fabric Monitor View Screen HlthErrs: Not Avail, CommErrs: 65631, Alrms: Not Avail INFRASTRUCTURE VIEW INFORMATION CID CATEGORY TOTAL MONTRD OPRTNL HLTH-ERRS COMM-ERRS ALRMS 1 Switches 4 4 4 0 65600 Not Avail 2 CAs 8 8 8 0 31 Not Avail 3 Chassis 0 0 0 0 0 Not Avail 4 Servers 0 0 0 0 0 Not Avail 5 Cables 0 0 0 0 0 Not Avail 6 Routers 0 0 0 0 0 Not Avail 7 Applications 0 0 0 0 0 Not Avail SUBMENU: (p)rvPg, Fr(Z), Unfr(z) Figure 4-2
4–Real-Time Fabric Monitor View Screen PERF: GrpName: All, GrpNumPrts: 107, LstNumPrts: 10, LstMaxNumPrts: 10, NodesFailed: 0, NodesSkipped: 0, PortsFailed: 0, PortsSkipped: 9 All GROUP ERROR STATS INFORMATION Int Max 0+% 25+% 50+% 75+% 100+% Integrity 0 98 0 0 0 0 Congestion 0 98 0 0 0 0 SmaCongest 0 98 0 0 0 0 Security 0 98 0 0 0 0 Routing 0 98 0 0 0 0 Congest %: 0.0 Discard: 0 Ineffic %: 0.
4–Real-Time Fabric Monitor View Screen PERF: GrpName: All, GrpNumPrts: 107, LstNumPrts: 10, LstMaxNumPrts: 10, NodesFailed: 0, NodesSkipped: 0, PortsFailed: 0, PortsSkipped: 9 All GROUP BW STATS INFORMATION Int TotMBps AvgMBps MinMBps MaxMBps TotKPps AvgKPps MinKPps MaxKPps 0 0 0 0 0 0 0 0 Buckt 0+% 10+% 20+% 30+% 40+% 50+% 60+% 70+% 80+% 90+% 98 0 0 0 0 0 0 0 0 0 SUBMENU: (+), (p)rvPg, (u)til, Ls(t), Im(g)Info Figure 4-27.
4–Real-Time Fabric Monitor View Screen Sub-Menu Section Displays an additional set of minor menu selections that are available. These menu selections exist to assist the user with miscellaneous operations (i.e., maneuvering through the screens). SUBMENU: (p)rvPg, Fr(Z), Unfr(z) Figure 4-28.
4–Real-Time Fabric Monitor Infrastructure View Sub-Screens (cC)rit — forward/reverse switch between utilization and error categories (n)eighbor — displays information about the neighbor port of a link The secondary menu selections supported for the Performance View are the following: (L)v — selects the live image (rR)v — reverse step/skips thru historical images F(wW)d —forward step/skips thru historical images (b)kmrkd —selects the current bookmarked image (B)kmrk —bookmarks the
4–Real-Time Fabric Monitor Infrastructure View Sub-Screens MENU: (H)ome, (I)nfrastructureView, (F)abricView, (P)erformanceView, (T)opContext, (A)dmin, (Q)uit VIEW: [I]:H:I CONTEXT: Name: Switches, Total Switches: 500, Alrms: Not Avail FABRIC: Top: [Mon Feb 1 16:50:13 2010, Hist], Now: Mon Feb 1 16:54:39 2010 SWs: 500, HCAs: 50, TCAs: 0, Nodes: 527, SMs: 1 Links: 48, SWsPorts: 12000, HCAsPorts: 54, TCAsPorts: 0 HlthErrs: Not Avail, CommErrs: 297, Alrms: Not Avail CID SWITCHES ERRORS | CID SWITCH
4–Real-Time Fabric Monitor Infrastructure View Sub-Screens Switch Node Selection Screen (26-500 Switch Nodes) After selecting a block of 500 switch nodes or if there are between 26 to 500 switch nodes found within the fabric Figure 4-31 is an example of the screen that will be displayed.
4–Real-Time Fabric Monitor Infrastructure View Sub-Screens Switch Node Selection Screen (1-25 Switch Nodes) After selecting a block of 25 switch nodes or if there are between 1 to 25 switch nodes found within the fabric, Figure 4-32 is an example of the screen that will be displayed.
4–Real-Time Fabric Monitor Infrastructure View Sub-Screens Switch Node Information Selection Screen Figure 4-33 is an example of the screen for selecting specific information for a switch node within the fabric.
4–Real-Time Fabric Monitor Infrastructure View Sub-Screens Switch Node Device Information Screen Figure 4-34 is an example of the screen for viewing device specific information for a switch node within the fabric.
4–Real-Time Fabric Monitor Infrastructure View Sub-Screens Switch Node Port Selection Screen Figure 4-35 is an example of the screen for selecting a port within a specific switch node within the fabric.
4–Real-Time Fabric Monitor Infrastructure View Sub-Screens Switch Node Port Information Selection Screen Figure 4-36 is an example of the screen for selecting the information to display for a port within a switch node within the fabric.
4–Real-Time Fabric Monitor Infrastructure View Sub-Screens Switch Node General Port Information Screen Figure 4-37 is an example of the screen for viewing general port information, about a specific port within a switch node within the fabric.
4–Real-Time Fabric Monitor Infrastructure View Sub-Screens Switch Node Port Statistics Selection Screen Figure 4-38 is the screen for viewing port statistic information about a specific port within a switch node within the fabric.
4–Real-Time Fabric Monitor Infrastructure View Sub-Screens Switch Node Port Performance Screen Figure 4-39 is the screen for viewing performance information about a specific port within a switch node within the fabric.
4–Real-Time Fabric Monitor Infrastructure View Sub-Screens Channel Adapter (CA) Screens When the user selects the CAs category from the main screen of the Infrastructure View (Refer to “Submenu Segment” on page 4-6), screens are displayed to enable the user to drill-down to a specific channel adapter of interest. The order in which these screens are displayed will depend upon the number of channel adapters found within the fabric.
4–Real-Time Fabric Monitor Infrastructure View Sub-Screens CA Selection Screen (1-25 CAs) After selecting a block of 25 CAs or if there are between 1 to 25 CAs found within the fabric, Figure 4-41 is an example of the screen that will be displayed.
4–Real-Time Fabric Monitor Infrastructure View Sub-Screens CA Information Selection Screen Figure 4-42 is an example of the screen for selecting the information to display for a CA within the fabric.
4–Real-Time Fabric Monitor Infrastructure View Sub-Screens CA Device Information Screen Figure 4-43 is an example of the screen for viewing device specific information for a CA within the fabric.
4–Real-Time Fabric Monitor Infrastructure View Sub-Screens CA Port Selection Screen Figure 4-44 is an example of the screen for selecting a port within a specific CA within the fabric.
4–Real-Time Fabric Monitor Infrastructure View Sub-Screens CA Port Information Selection Screen Figure 4-45 is an example of the screen for selecting the information to be displayed for a port within a CA within the fabric MENU: (H)ome, (I)nfrastructureView, (F)abricView, (P)erformanceView, (T)opContext, (A)dmin, (Q)uit VIEW: [I]:H:I:Ca1:Ca1Info:P1 CONTEXT: Name: FABRIC: Ports Info, Total CAs: 3 Top: [Tue Feb 2 07:31:25 2010, Hist], Now: Tue Feb 2 07:51:50 2010 SWs: 11, HCAs: 3, TCAs: 0, Nodes: 14, S
4–Real-Time Fabric Monitor Infrastructure View Sub-Screens CA Port General Information Screen Figure 4-46 is an example of the screen for viewing general port information for a specific port within a CA within the fabric.
4–Real-Time Fabric Monitor Infrastructure View Sub-Screens CA Port Statistics Selection Screen Figure 4-47 is an example of the screen for viewing port statistic information for a specific port within a CA within the fabric MENU: (H)ome, (I)nfrastructureView, (F)abricView, (P)erformanceView, (T)opContext, (A)dmin, (Q)uit VIEW: [I]:H:I:Ca1:Ca1Info:P1:P1Info CONTEXT: Name: Port Statistics Info, Total CAs: 3 FABRIC: Top: [Tue Feb 2 07:31:25 2010, Hist], Now: Tue Feb 2 07:51:50 2010 SWs: 11, HCAs: 3, TCAs
4–Real-Time Fabric Monitor Infrastructure View Sub-Screens CA Port Performance Screen Figure 4-48 is an example of the screen for viewing performance information for a specific port within a CA within the fabric.
4–Real-Time Fabric Monitor Fabric View Sub-Screens Fabric View Sub-Screens Link Screens When a user selects the Links category from the Fabric View main screen (“Submenu Segment” on page 4-6), the user is able to drill-down to a specific communication link of interest. The order in which these screens are displayed will depend upon the number of links found within the fabric. Link Category Selection Screen Figure 4-49 is an example of the screen for selecting the category of links to view.
4–Real-Time Fabric Monitor Fabric View Sub-Screens Link Selection Screen (24000+ Links) If there are 24000 or more links found within the fabric, Figure 4-50 is an example of the screen that will be displayed first.
4–Real-Time Fabric Monitor Fabric View Sub-Screens Link Selection Screen (1200-24000 Links) After selecting a block of 24000 Links or if there are between 1200 to 24000 Links found within the fabric, Figure 4-51 is an example of the screen that will be displayed.
4–Real-Time Fabric Monitor Fabric View Sub-Screens Link Selection Screen (60-1200 Links) After selecting a block of 1200 Links or if there are between 60 to 1200 Links found within the fabric, Figure 4-52 is an example of the screen that will be displayed.
4–Real-Time Fabric Monitor Fabric View Sub-Screens Link Selection Screen (1-60 Links) After selecting a block of 60 Links or if there are between 1 to 60 Links found within the fabric, Figure 4-53 is an example of the screen that will be displayed.
4–Real-Time Fabric Monitor Fabric View Sub-Screens Link Information Selection Screen Figure 4-54 is an example of the screen for selecting the information to display or a link within the fabric.
4–Real-Time Fabric Monitor Fabric View Sub-Screens Link End-Node Information Selection Screen Figure 4-55 is an example of the screen for selecting the information to display for an end-node of a specific link within the fabric MENU: (H)ome, (I)nfrastructureView, (F)abricView, (P)erformanceView, (T)opContext, (A)dmin, (Q)uit VIEW: [F]:H:F:All:Link1:Node1Info CONTEXT: Name: End-Node 1 Info, Total links: FABRIC: 111 Top: [Tue Feb 2 07:31:25 2010, Hist], Now: Tue Feb 2 07:51:50 2010 SWs: 11, HCAs: 3, TC
4–Real-Time Fabric Monitor Fabric View Sub-Screens Link Element A link is an established connection between two ports (cable or backplane/internal). The following are shown in Figure 4-55: 4-42 LnkNo — numeric identifier associated with the link. LnkTyp — type of link: Internal or External. EndNode1/2 — These two fields lists the Node GUIDs of the two nodes connected through the link. NodeDesc — lists the Node Description of the node.
4–Real-Time Fabric Monitor Fabric View Sub-Screens Link End-Port Information Selection Screen Figure 4-56 is an example of the screen for selecting the information to display for an end-port of a specific link within the fabric MENU: (H)ome, (I)nfrastructureView, (F)abricView, (P)erformanceView, (T)opContext, (A)dmin, (Q)uit VIEW: [F]:H:F:All:Link1:Node1Info:Ca1Info:P1 CONTEXT: Name: End-Port 1 Info, Total links: FABRIC: 111 Top: [Tue Feb 2 07:31:25 2010, Hist], Now: Tue Feb 2 07:51:50 2010 SWs: 11,
4–Real-Time Fabric Monitor Fabric View Sub-Screens Slow Link Selection Screen Figure 4-57 is an example of the screen for selecting which types of slow link performance to analyze.
4–Real-Time Fabric Monitor Fabric View Sub-Screens SM Screens SM Selection Screen Figure 4-58 is an example of the initial screen for selecting a SM within the fabric.
4–Real-Time Fabric Monitor Fabric View Sub-Screens SM Detailed Information Screen Figure 4-59 is an example of the screen showing detailed information about an SM within the fabric.
4–Real-Time Fabric Monitor Performance View Sub-Screens Performance View Sub-Screens Bandwidth Utilization Screens When a user selects the (G)rp<1-4> sub-menu selection from the Main screen (“Submenu Segment” on page 4-6), the user is able to drill-down to the utilization statistics for a specific group of ports of interest. Switch Group Bandwidth Utilization Selection Screen Figure 4-60 is an example of the screen for selecting the utilization category of the Switch group to view.
4–Real-Time Fabric Monitor Performance View Sub-Screens Low Bandwidth Utilization Selection Screen After selecting the (u)til sub-menu selection, for ports associated with links with a low bandwidth utilization, a list of links is displayed to the user. Figure 4-61 is an example of the screen that will be displayed.
4–Real-Time Fabric Monitor Performance View Sub-Screens Error Condition Screens When a user selects the (G)rp<1-4> sub-menu selection from the Main screen (“Submenu Segment” on page 4-6), the user is able to drill-down to the error condition statistics for a specific group of ports of interest. Switch Group Error Condition Selection Screen Figure 4-62 is an example of the screen for selecting the error category of the Switch group to view.
4–Real-Time Fabric Monitor Performance View Sub-Screens Integrity Error Selection Screen After selecting the (e)rr<1> sub-menu selection, for ports associated with links that have integrity related errors, a list of links is displayed to the user. Figure 4-63 is an example of the screen that will be displayed.
4–Real-Time Fabric Monitor Performance View Sub-Screens Link Selection Screen Figure 4-64 is an example of the screen for selecting the information to display for a specific link within the fabric.
4–Real-Time Fabric Monitor Admin Menu Screens Admin Menu Screens Main Screen Figure 4-65 is the main screen of the Admin menu selection, which enables a user to perform administration related operations with RFM MENU: (H)ome, (Q)uit VIEW: [A]:H CONTEXT: Name: Admin CID ADMIN OPERATIONS 1 Performance View Configuration 2 Infrastructure View Configuration 3 Fabric View Configuration 4 General Configuration SUBMENU: (p)rvPg Figure 4-65.
4–Real-Time Fabric Monitor Admin Menu Screens Fabric Discovery Screen Figure 4-66 is the screen for viewing results from the execution of a fabric discovery operation. # iba_rfm Please wait, while the fabric is being discovered... Beginning Fabric Discovery... Getting general information about all PM groups Getting All Node Records... Done Getting All Node Records Done Getting All Link Records Done Getting All SM Info Records Getting All PA Port Counters...
4–Real-Time Fabric Monitor Admin Menu Screens 4-54 IB0054607-01 A
5 Configuration of IPoIB Name Mapping The FastFabric tools support the concept of a management network and an IPoIB network. For some clusters the management network will be a low speed network such as 10/100 Ethernet. For other clusters IPoIB may serve double duty as the host management network. NOTE: When using IPoIB as the management network, the initial installation of InfiniBand software cannot be done using FastFabric.
5–Configuration of IPoIB Name Mapping 5-2 IB0054607-01 A
A FastFabric Configuration Files Table A-2 list the configuration files that are used by FastFabric. The description in the table also list the following sections that have detailed descriptions of each file. For a given release consult the files with -sample at the end of the file name for a sample file with the defaults of the given release. Table A-2. FastFabric Configuration Files Configuration File IB0054607-01 A Description /etc/sysconfig/fastfabric.conf Overall configuration file.
A–FastFabric Configuration Files FastFabric Configuration File Table A-2. FastFabric Configuration Files Configuration File /etc/sysconfig/iba/topology.0:0.xml Description Fabric topology input file used by iba_reports and fabric health tools. Refer to Fabric Topology Input File. FastFabric Configuration File The FastFabric tools support a configuration file /etc/sysconfig/fastfabric.conf. This file can be used to provide default settings for most of the FastFabric command line options.
A–FastFabric Configuration Files FastFabric Configuration File export CONFIG_DIR fi # Override default location for HOSTS_FILE export HOSTS_FILE=${HOSTS_FILE:-$CONFIG_DIR/iba/hosts} # Override default location for CHASSIS_FILE export CHASSIS_FILE=${CHASSIS_FILE:-$CONFIG_DIR/iba/chassis} # Override default location for ESM_CHASSIS_FILE export ESM_CHASSIS_FILE=${ESM_CHASSIS_FILE:-$CONFIG_DIR/iba/esm_chassis} # Override default location for IBNODES_FILE export IBNODES_FILE=${IBNODES_FILE:-$CONFIG_DIR/iba/ibno
A–FastFabric Configuration Files FastFabric Configuration File # $1 = hostname provided (could be ethernet or IPoIB name) echo "$1"|sed -e "s/$FF_IPOIB_SUFFIX\$//" # comment out line above and uncomment line below if using prefixes #echo "$1"|sed -e "s/^$FF_IPOIB_PREFIX//" } fi # IP netmask for IPoIB subnet [-m option] # if "" default will be determined based on class of IP address [A, B, C] export FF_IPOIB_NETMASK=${FF_IPOIB_NETMASK:-} # Maximum parallel processes for ibtest and -p option on other command
A–FastFabric Configuration Files FastFabric Configuration File # local HCA port/fabric selection string (for example 0:0 or 1:2) for # the fabric being selected (see PORTS_FILE for more information) # if this file is not found, or the value of this parameter is "NONE" # no topology input file will be used export FF_TOPOLOGY_FILE=${FF_TOPOLOGY_FILE:-$CONFIG_DIR/iba/topology.%P.
A–FastFabric Configuration Files FastFabric Configuration File export FF_CHASSIS_CMDS=${FF_CHASSIS_CMDS:-showInventory fwVersion showIBNodeDesc ismShowPStatThresh ismChassisSet12x timeZoneConf timeDSTConf snmpCommunityConf snmpTargetAddr showChassisIpAddr showDefaultRoute} # other possible additions (if running newer chassis FW which supports these) # ismIslSet12x, ismIslSetSpeed # single CLI command to issue to check overall health during chassis_analysis # hwCheck is prefered, but is not supported on old
A–FastFabric Configuration Files Port Statistics Thresholds Configuration File Port Statistics Thresholds Configuration File The /etc/sysconfig/iba/iba_mon.conf configuration file defines port statistics thresholds for use by iba_report, fabric_analysis, all_analysis and iba_mon. This file lists a threshold for each port statistic. If the threshold for a given statistic is not defined or is set to 0, the given statistic will not be checked.
A–FastFabric Configuration Files Port Statistics Thresholds Configuration File PortRcvConstraintErrors LocalLinkIntegrityErrors ExcessiveBufferOverrunErrors #VL15Dropped 10 3 3 100 # expected to optimize SM sweep time NOTE: When this file is used by iba_mon, the thresholds represent counts per “Interval”.
A–FastFabric Configuration Files Signal Integrity Thresholds Configuration File Signal Integrity Thresholds Configuration File The /etc/sysconfig/iba/iba_mon.si.conf configuration file defines port counter signal integrity thresholds. This file allows analysis for any non-zero error counters related to Signal Integrity (bad cables, etc) and can be used by adding the -c option to iba_report, iba_extract_error and other related fastfabric tools.
A–FastFabric Configuration Files Signal Integrity Thresholds Configuration File # alternative value is Greater # Normal Data Movement PortXmitData 0# as MB/second PortRcvData 0# as MB/second PortXmitPkts 0# as packets/second PortRcvPkts 0# as packets/second # Error Counters A-10 SymbolErrorCounter 1 LinkErrorRecoveryCounter 1 LinkDownedCounter 1 PortRcvErrors 1 PortRcvRemotePhysicalErrors 0# side effect of errors elsewhere, ignore PortRcvSwitchRelayErrors 0# not related to SI PortXmit
A–FastFabric Configuration Files Host List Files Host List Files The /etc/sysconfig/iba/hosts and /etc/sysconfig/iba/allhosts files are used to specify the hosts which FastFabric will operate against for many operations. If desired alternate filenames may be specified in fastfabric.conf, using environment variables or on the command line. Refer to FastFabric Command Line Interface Reference Guide for more information. Below is a sample host list file: # [ICS VERSION STRING: @(#) .
A–FastFabric Configuration Files Chassis List Files Chassis List Files The /etc/sysconfig/iba/chassis and /etc/sysconfig/iba/esm_chassis files are used to specify the QLogic InfiniBand chassis that FastFabric will operate against for many operations. If desired alternate filenames may be specified in fastfabric.conf, using environment variables or on the command line. Refer to FastFabric Command Line Interface Reference Guide for more information.
A–FastFabric Configuration Files Externally Managed Switch List File However, in some cases it may be desirable to perform operations against a specific subset of cards within the chassis. In this case the chassis IP address, name within a chassis list or a chassis file can be augmented with a list of slot numbers to operate on. This is done in the form: chassis:slot1,slot2,… For example: i9k229:0 i9k229:0,1,5 192.168.0.5:0,1,5 NOTE: There must be no spaces within the chassis name and/or slot list.
A–FastFabric Configuration Files Externally Managed Switch List File The following is a sample ibanodes file: # # # # # # # # # # # # # # # # # # # # # [ICS VERSION STRING: @(#) ./fastfabric/ib_tools/ibnodes x_x_x_x_x [MM/DD/YY hh:mm] This file lists all the QLogic 9000 and 12000 Externally Managed Switches specify one line per switch of the form guid,nodeDesc,distance guid - node guid of the switch nodeDesc - optional node description which should be programmed into the switch by FastFabric.
A–FastFabric Configuration Files Externally Managed Switch List File The GUID will be used to select the switch and on firmware update operations, the node description will be written to the switch such that other FastFabric tools (such as iba_aquery and iba_report) can provide a more easily readable name for the switch. The node description can also be updated as part of switch basic configuration. The hca:port may be used to specify which local port (subnet) to use to access the switch.
A–FastFabric Configuration Files Externally Managed Switch List File The ordering is controlled by an optional distance field in the ibnodes file or the ibnodes provided on the command line. The distance field indicates the relative distance from the FastFabric node for each switch. Any ibnodes file entries which do not specify a distance value are treated as having a value larger than any others in the file.
A–FastFabric Configuration Files Port List File Port List File The /etc/sysconfig/iba/ports file is used to specify the local HCA ports (i.e., subnets) that FastFabric will use for assorted commands (such as iba_reports, fabric_info, iba_switch_admin, fabric_analysis, all_analysis) for fabric access. Alternate filenames may be specified in fastfabric.conf, using environment variables or on the command line. Refer to FastFabric Command Line Interface Reference Guide for more information.
A–FastFabric Configuration Files Fabric Topology Input File Fabric Topology Input File The /etc/sysconfig/iba/topology.0:0.xml file is used to specify the expected fabric topology and augmented fabric information (such as cable labels, types, lengths, SM details, node details, link details, etc). If present this file will be used by assorted FastFabric commands (such as iba_reports, fabric_analysis, all_analysis).
A–FastFabric Configuration Files Fabric Topology Input File 1 CA mindy2 HCA-1 0x00066a0007000e6d 4 SW i9k159 Leaf 5, Chip A 0x0002c9020025a678 mindy2 HCA-1 mindy2 only HCA PAGE 140A–FastFabric Configuration Files Fabric Topology Input File A-20 IB0054607-01 A
Corporate Headquarters QLogic Corporation 26650 Aliso Viejo Parkway Aliso Viejo, CA 92656 949.389.6000 www.qlogic.com International Offices UK | Ireland | Germany | France | India | Japan | China | Hong Kong | Singapore | Taiwan © 2011 QLogic Corporation. Specifications are subject to change without notice. All rights reserved worldwide. QLogic, the QLogic logo, and FastFabric are registered trademarks of QLogic Corporation.
