53-1001253-01 09 June 09 Brocade Adapters Troubleshooting Guide Supporting CNA Models BR-1010, BR-1020 Supporting HBA Models BR-815, BR-825, BR-415, BR-425 ®
Copyright © 2009 Brocade Communications Systems, Inc. All Rights Reserved. Brocade, the B-wing symbol, BigIron, DCX, Fabric OS, FastIron, IronPoint, IronShield, IronView, IronWare, JetCore, NetIron, SecureIron, ServerIron, StorageX, and TurboIron are registered trademarks, and DCFM, Extraordinary Networks, and SAN Health are trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries.
Contents About this Document In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix How this document is organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Supported CNA hardware and software . . . . . . . . . . . . . . . . . . . . . . . CNA support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fabric OS and switch support . . . . . . . . . . . . . . . . . . . . . . . . . . . .
General HBA and CNA problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Adapter not reported under server’s PCI subsystem. . . . . . . . . . 8 No adapters reported though BCU adapter --list command . . . . 8 Port link is not active . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Errors when installing brocade_driver_linux_.tar.gz package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Adapter driver installation verification. . . . . . . . . . . . . . . . . . . . . . . . 34 Confirming driver package installation with HCM . . . . . . . . . . . 34 Confirming driver package installation in Windows systems . . 35 Confirming driver package installation in Linux systems . . . . . 35 Confirming driver package installation in Solaris systems . . . . 36 Confirming driver package installation in VMware systems . . . 36 Additional references for isolating problems. . . . . . . . . . . . . . . . . .
Diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 Beaconing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 Internal and external loopback tests . . . . . . . . . . . . . . . . . . . . . 63 Ethernet port loopback test (CNA only) . . . . . . . . . . . . . . . . . . . 64 PCI loopback test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 Memory test . . . . . . . . . . . . . . . . . . . .
Appendix A Event Message Reference Index Brocade Adapters Troubleshooting Guide 53-1001253-01 vii
viii Brocade Adapters Troubleshooting Guide 53-1001253-01
About this Document In this chapter • Supported CNA hardware and software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x • Supported HBA hardware and software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi • What’s new in this document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii • Document conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii • Notice to the reader . . . . . . . . . . . . . . . . . .
Supported CNA hardware and software This section describes adapter hardware and software support. CNA support The following FCoE CNAs are supported in this release: • Brocade BR-1010. Single-port CNA with a per-port maximum of 10Gbps. • Brocade BR-1020. Dual-port CNA with a per-port maximum of 10 Gbps. NOTE Install only Brocade-branded SFPs in these CNAs. Fabric OS and switch support Brocade CNAs must connect to Fibre Channel SANs and Ethernet data networks through a compatible FCoE switch.
FCoE support • • • • • • Windows Server 2003, version R2 with SP2 Windows Server 2008 Windows Server Core Linux RHEL4, RHEL5, SLES10, and SLES11 Solaris 10 (x86 and SPARC) VMware ESX Server 3.5 NOTE Drivers, BCU, and HCM Agent are supported only on the VMware “console” Operating System. HCM is supported only on the guest operating system on VMware. Specific operating system service pack levels and other patch requirements are detailed in the current CNA release notes.
• Although you can install an 8 Gbps SFP+ into a Brocade 415 or 425 HBA, only 4 Gbps maximum port speed is possible. • Install only Brocade-branded SFPs in these HBAs. Fabric OS and switch support For a current list of servers, switches, and applications compatible with Brocade HBAs, refer to the latest HBA compatibility matrix. Log into Brocade Connect or the Partner Network through www.brocade.com. After login, select Compatibility Matrices from Quick Links.
Document conventions This section describes text formatting conventions and important notice formats used in this document.
Notes, cautions, and warnings The following notices and statements are used in this manual. They are listed below in order of increasing severity of potential hazards. NOTE A note provides a tip, guidance or advice, emphasizes important information, or provides a reference to related information. ATTENTION An Attention statement indicates potential damage to hardware or data.
Notice to the reader This document may contain references to the trademarks of the following corporations. These trademarks are the properties of their respective companies and corporations. These references are made for informational purposes only. Corporation Referenced Trademarks and Products Microsoft Corporation Windows, Windows Server 2003, Windows Server 2008, Vista, XP, PE for Windows, Hyper V for Windows, Windows Automated Installation Kit (WAIK) Sun Microsystems, Inc. Solaris Red Hat Inc.
Other industry resources For additional resource information, visit the Technical Committee T11 Web site. This Web site provides interface standards for high-performance and mass storage applications for Fibre Channel, storage management, and other applications: http://www.t11.org For information about the Fibre Channel industry, visit the Fibre Channel Industry Association Web site: http://www.fibrechannel.
• BCU adapter --list command. This command lists all adapters in the system and information such as model and serial numbers. 3. Port World-Wide Port Name (PWWN). Determine this through the following resources: • Label affixed on adapter card provides the WWPN for each port. • Brocade BIOS Configuration Utility. Select the appropriate adapter port from the initial configuration utility screen, then select Adapter Settings to display the WWNN and PWWN for the port.
• Enter the port --query BCU command: Look for the FCoE MAC. The FCoE Forwarder (FCP) MAC address is the address of the attached FCoE switch. Select an FCoE port in the HCM device tree, then click the Properties tab in the right pane to display the port Properties panel. Look for the FCF MAC field. You can also determine port MAC addresses using the following BCU commands. Command Function port ---query Displays port information, including the MAC addresses.
Chapter Introduction to troubleshooting 1 In this chapter • How to use this manual for troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . 1 • Gathering problem information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 How to use this manual for troubleshooting An adapter, such as an HBA or CNA, is one component of a larger network consisting of switches, storage devices, host systems and the cabling and connections to these components.
1 Gathering problem information 3. Use the BCU commands, HCM features, and host operating system commands described in Chapter 3, “Tools for Collecting Data” to gather data for resolving problems. Although many of these tools are specifically referenced for problems described in Chapter 2, many more are included that can provide helpful data for troubleshooting, such as event logs, operating statistics, and diagnostics. 4.
Gathering problem information 1 • Gather information for support: - Use appropriate tools on storage targets to gather information such as disk, tape, and controller model and firmware levels. - Run the bfa_supportsave BCU command on the host system and save output to a file on your system. This command captures all driver, internal libraries, firmware, and other information needed to diagnose suspected system issues.
1 4 Gathering problem information Brocade Adapters Troubleshooting Guide 53-1001253-01
Chapter 2 Isolating Problems In this chapter • How to use this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 • General HBA and CNA problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 • HBA problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 • Ethernet network interface problems (CNA only) . . . . . . . . . . . . . . . . . . . . . 22 • FCoE and Fibre Channel problems. . .
2 How to use this chapter TABLE 2 6 Isolate adapter problems Problem Category OS Specific “Failed to connect to agent on host... error when using HCM” “General HBA and CNA problems” All “Driver event messages appearing in host system log files” “General HBA and CNA problems” All “Files needed for bfad.
How to use this chapter TABLE 2 2 Isolate adapter problems Problem Category OS Specific “Network stack runs out of heap” “Ethernet network interface problems (CNA only)” VMware “NIC numbering unexpected on VMware systems” “Ethernet network interface problems (CNA only)” VMware “Poor network performance” “Ethernet network interface problems (CNA only)” Linux Windows “VLAN creation and operation problems” “Ethernet network interface problems (CNA only)” Windows “Loss of sync and loss of sig
2 General HBA and CNA problems General HBA and CNA problems This section provides resolution for common problems that could with installed CNAs or HBAs, such as the adapter not being reported under the server’s PCI subsystem or BCU adapter --list command, or the port link to the switch is not active. Adapter not reported under server’s PCI subsystem The adapter is installed but not visible as a device in the host system’s PCI subsystem.
General HBA and CNA problems 2 Port link is not active The link between the adapter and switch port does not appear to be active because of adapter LED operation, lack of data over the link, or BCU port --query or port --list command shows that the link state is down. Refer to “Displaying the port list” on page 73 for more information. Refer to the following descriptions of possible causes and recommended actions or fixes for the problem. 1. Possible Cause: SFP or cable problems.
2 General HBA and CNA problems Errors when installing brocade_driver_linux_.tar.gz package If errors occur when installing the noarch brocade_driver_linux_.tar.gz driver package, refer to the following descriptions of possible causes and recommended actions to help resolve the problem. Possible Cause: Appropriate distribution kernel development packages are not installed on your host system for the currently running kernel.
General HBA and CNA problems 2 c. Reseat SFPs in the adapter. Determine whether the installed SFPs are faulty by observing LED operation by the adapter ports. If all LEDs are flashing amber, the SFP is invalid and may not be a required Brocade model. You can also verify SFP operation by replacing them with SFPs in known operating condition. If the problem is resolved after replacement, original SFP is faulty. d. Check for conflicts with attached devices.
2 General HBA and CNA problems Failed to connect to agent on host... error when using HCM An “Adapter failed to connect to agent on host...” message indicates that the client application cannot connect to the HCM Agent listening on the configured port - normally TCP port 34568. Refer to the following descriptions of possible causes and recommended actions to help resolve the problem. Possible Causes: • • • • The agent is not running. The agent is not accepting connections on the expected port.
2 General HBA and CNA problems 5. If HCM is still unable to connect to the HCM agent after using the preceding steps, collect the following data and send to your Support representative for analysis: • Data collected from the previous step in SupportSaveController.do. • Data from the HCM application SupportSave feature. Select Tools > SupportSave to generate a supportsave file. The data file name and location displays when the SupportSave feature runs.
2 General HBA and CNA problems NOTE You can change the default communication port (34568) for the agent using procedures in the Brocade Adapters Installation and Reference Manual. Refer to the section on modifying HCM agent operation. 6. If the hcmagent is running and listening on port 34568 and there are no firewall issues (as explained in step 5), but you get the same “Failed to connect to agent on host..." error when using HCM, collect the following data.
General HBA and CNA problems 2 Files needed for bfad.sys message appears If a “Files needed for bfad.sys” message appears on Windows systems when removing a driver, refer to the following descriptions of possible causes and recommended actions to help resolve the problem. Possible Cause: This occurs if you perform the following sequence of steps: 1. Install the driver using the driver installer program (brocade_installer.bat) or Brocade Adapters Software Installer (GUI or command-based application). 2.
2 HBA problems I/O data traffic issues I/O data traffic issues are occurring, such as an application is not receiving data, FTP problems on an Ethernet network, ping failures, or data is not reaching a target on a Fibre Channel network. 1. Possible Cause: Ethernet traffic problem (CNAs only). Action: Run the Ethernet loopback serdes test on the suspected Ethernet port using the BCU ethdiag --loopback -t serdes command. This tests internal adapter hardware components.
HBA problems 2 1. Possible Cause: QoS is not enabled on both switch and adapter port. Action: Verify if QoS is enabled for an adapter port using the qos -query BCU command. Verify if it is enabled on the switch using the islShow command. Action: Verify zones on the switch using the Fabric OS cfgActvShow command. 2. Possible Cause: QoS zones not created properly on switch for high, medium, and low priority targets.
2 HBA problems Boot devices not available in host’s Boot Manager menu Fibre Channel attached boot devices do not appear in the EFI Boot Manager Menu or a boot device appears, but it is not functioning. Following are descriptions of possible causes and recommended actions or fixes for the problem. 1. Possible Cause: Adapter is not seated properly. Action: Select the EFI Shell from the EFI Boot Menu and use the devices command to determine if EFI has detected the Brocade adapter.
HBA problems 2 Drv[25] : Brocade Fibre Channel Adapter Bus Driver Drv[26] : SCSI Bus Driver ... Child Controllers Child[70] : SCSI Child[71] : SCSI Child[72] : SCSI : Disk Device Disk Device Disk Device The SCSI Disk Devices under “Child Controllers” are the LUNs that the Fibre Channel Adapter can access. If an expected Fibre Channel attached disk does not appear in the “dh -d” list for a Brocade adapter, check the cabling, the adapter’s link status LEDs, and the Fibre Channel switch configuration.
2 HBA problems 4. Possible Cause: There is a problem with the remote port. Action: Verify that the remote target port (rport) is reporting itself online by comparing rport online and rport offline statistics. Refer to “Remote port statistics” on page 59 for details on displaying these statistics. The rport online counter should be one greater than the rport offline counter. If not, clear the counters and try connecting to the remote port again. Verify the rport online and rport offline statistics again.
HBA problems 2 3. Possible Cause: The link between the adapter port and target is not active yet. Action: Check that the speeds for the adapter port and the connected switch port match. The best approach is to set both speeds to “auto” or “autonegotiate.” Unable to boot from the stored boot device settings in the adapter The host is unable to boot from the boot device settings stored in the adapter. A “No boot LUNs configured” message will display next to the adapter value when booting. 1.
2 Ethernet network interface problems (CNA only) 7. Press F10 to save the changes and exit the BIOS Setup. NOTE Action: Refer to Hewlett Packard (HP) Customer Advisory Document c01199684 on the HP technical support website for detailed information. Adapter : BIOS not installed displays during boot process An “Adapter : BIOS not installed” message displays when booting from an adapter.
Ethernet network interface problems (CNA only) 2 To disable NetworkManager for SLES systems, perform the following steps. a. Open YaST. b. Select the Network Devices Network Card. c. On the first screen set the Network Setup Method option to Traditional Method with ifup. Loss of adapter hardware address in Linux The ifconfig command displays HW Addr as 00:00:00:00:00:00. Possible Cause: The CNA failed to initialize for some reason.
2 Ethernet network interface problems (CNA only) Action: Check for other instances of this diagnostic using ps -ef for Linux and VMware, and Task Manager for Windows systems. Action: Wait a few minutes before retrying the command (Check if the other instance is done using ps -ef command or Task Manager). 5. Symptom: The loopback test returns “Missing frame check and replace SFP/cable.” Possible Cause: The loopback cable was pulled during the test.
Ethernet network interface problems (CNA only) 2 6. Reboot the host system. Ping to remote server is failing Pings generated between servers are failing. 1. Possible Cause: Ethernet interface on either server is in the following states: • Administratively down. Running the Linux or VMware ifconfig command shows that the UP flag is not set. • Administratively up, but link is down. Running the Linux or VMware ifconfig command shows that the RUNNING flag is not set.
2 Ethernet network interface problems (CNA only) VLAN creation and operation problems VLAN creation fails with BCU command or HCM or pass-through VLAN stops working after creating with Device Manager. These problems result when VLANs are created using HCM or BCU commands and also using Device Manager. Follow these guidelines to avoid problems: • If you need to create a single VLAN and VLANs have not been created using BCU commands or HCM, you can use Device Manager.
Ethernet network interface problems (CNA only) 2 Poor network performance Poor network performance apparent for Windows and Linux systems. 1. Symptom: Checksum offloads are disabled. Action: For Windows, verify if checksum offload parameters are enabled using the Advanced tab on the Network Adapters > Properties dialog box in Device Manager. Action: For Linux, run the ethtool -K command. If offload parameters are on, information similar to the following displays in the output.
2 FCoE and Fibre Channel problems FCoE and Fibre Channel problems This section provides resolution of problems related to Fiber Channel and FCoE. Loss of sync and loss of signal errors in port statistics If the port is having loss of synchronization and signal errors, refer to the following descriptions of possible causes and recommended actions to help resolve the problem. Learn more about displaying port statistics in “Port statistics” on page 59. Possible Cause: Possible physical link problem.
FCoE and Fibre Channel problems 2 Disk I/O requests causes low throughput and high latency on Linux If a high number of I/O requests is causing low throughput and high latency on Linux systems, refer to the following descriptions of possible causes and recommended actions to help resolve the problem. Possible Cause: The maximum input/output operations per second are too low on Linux hosts. Action: Refer to “Linux tuning” on page 77 for suggestions to optimize adapter performance in Linux systems.
2 FCoE and Fibre Channel problems Adapter not registering with the name server or cannot access storage If the adapter is not registering with the name server or cannot access storage, refer to the following descriptions of possible causes and recommended actions to help solve the problem. 1. Possible Cause: Adapter cannot log in to the name server. Action: Display logical port statistics (refer to “Logical port statistics” on page 58 for details on displaying these statistics).
FCoE and Fibre Channel problems 2 4. Possible Cause: The FC-MAP on the FCoE switch is not set for a VLAN with FCF capability. Action: Verify if the FC-MAP on the switch is set for a VLAN with FCF capability using appropriate Fabric OS command on the attached switch. Refer to the Fabric OS Command Reference Manual for more information. Action: Set the FC-MAP for a VLAN with FCF capability using the appropriate Fabric OS command on the attached switch.
2 CEE network problems (CNA only) CEE network problems (CNA only) This section provides information to resolve problems of CNA operation on the converged enhanced Ethernet (CEE) network. CEE is not enabled The CEE state does not show “CEE Linkup” when you run the bcu port --query command. 1. Possible Cause: The link between the CNA port and switch is down.
Verifying Fibre Channel and CEE links 2 NOTE Also verify LED operation on a switch port that is connected to an adapter port. Refer to the switch Hardware Reference Manual to analyze LED meaning. Common link problems can be caused by the following: • Damaged cables. (Note that damaged cables can also cause errors and invalid data on links.) • Cables that are not rated or compatible with adapter port speeds. Refer to cable specifications in the Brocade Adapters Installation and Reference Manual.
2 Adapter driver installation verification Adapter driver installation verification Problems with adapter operation may be due to improper hardware or software installation, incompatibility between the adapter and your host system, unsupported SFPs installed on the adapter, improper cable connected to the fabric, or the adapter not operating within specifications.
Adapter driver installation verification 2 Confirming driver package installation in Windows systems Use the Device Manager to determine driver installation. Verify if the driver is installed and Windows is recognizing the adapter using the following steps. 1. Open the Device Manager. • For HBAs and CNAs, when you expand the list of SCSI and RAID controllers, an instance of Brocade 10G FCoE HBA should display for adapter port installed.
2 Adapter driver installation verification • # dmesg This command prints kernel boot messages. Entries for bfa (storage driver) and bna (network driver) should display to indicate driver activity if the hardware and driver are installed successfully. • These commands display the location of the driver modules if loaded to the system: - The following command displays the storage driver module location. The module will have a bfa prefix.
Additional references for isolating problems 2 • cat /proc/vmware/version This displays the latest versions of installed drivers. For storage drivers look for a bfa entry and related build number. For network drivers, look for a bna entry and related build number. • rpm -qa|grep -i bfa This command prints the names of the Brocade adapter storage driver package (bfa) if installed. • rpm -qa|grep -i bna This command prints the names of the Brocade adapter network driver package (bna) if installed.
2 38 Additional references for isolating problems Brsocade Adapters Troubleshooting Guide 53-1001253-01
Chapter 3 Tools for Collecting Data In this chapter • For detailed information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Data to provide support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Data collection using host system commands . . . . . . . . . . . . . . . . . . . . . . . • Data collection using BCU commands and HCM . . . . . . . . . . . . . . . . . . . . . • Data collection using Fabric OS commands . . . . . . . . . .
3 Data to provide support Data to provide support When problems occur requiring support assistance, provide a detailed description of the problem, as well as output collected from the following HCM and BCU tools: • • • • • • Support Save Diagnostics Port logs Port statistics and properties Adapter properties Host operating system error logs Data collection using host system commands The following table describes commands common to each supported operating system that you can use to gather information f
Data collection using host system commands TABLE 5 3 Host system data collection commands Task Linux Windows VMware Solaris Checking for Brocade network (BNA) driver module lsmod | grep bna driverquery /v | findstr bnad vmkload_mod -l | grep bfa NA Displaying driver information • Use lsmod command for general driver information. Use ethtool options to query network driver information and settings.
3 TABLE 5 Data collection using BCU commands and HCM Host system data collection commands Task Linux Windows VMware Solaris Ethernet statistics ethtool -S netstat ethtool -S N/A Ethernet link status ethtool interface_name> netstat esxcfg-nics -l N/A Data collection using BCU commands and HCM You can collect a variety of information on installed Brocade adapters, such as firmware version installed, operational status, port speed, WWN, PCI data, configuration
Data collection using BCU commands and HCM • • • • • • • • • • • 3 Environment information Data.
3 Data collection using BCU commands and HCM Initiating Support Save through the internet browser Initiate bfa_supportsave through an internet browser. 1. Open an Internet browser and type the following URL: https://localhost:34568/JSONRPCServiceApp/SupportSaveController.do where localhost is the IP address of the server from which you want to collect the bfa_supportsave information. 2. Log in using the factory default user name (admin) and password (password).
Data collection using Fabric OS commands 3 • The Converged Network Adapter (CNA) adapter Properties panel displays information about the adapter, such as the MAC address, enabled or disabled status, maximum speed supported, hardware path, serial number, temperature, installed driver name, driver version, firmware version, and BIOS version. This panel also displays PCI information for the adapter, such as vendor ID, device ID, subsystem ID, current number of lanes, and PCI generation.
3 Data collection using Fabric OS commands • fcpProbeShow Use this command to display the Fibre Channel Protocol daemon (FCPd) device probing information for the devices attached to a specified F_Port or FL_Port. This information includes the number of successful logins and SCSI INQUIRY commands sent over this port and a list of the attached devices. • nsShow Use this command to display local NS information about all devices connected to a specific switch.
Adapter event messages 3 Adapter event messages When applicable events occur during adapter operation, the adapter driver generates event messages. These messages are captured in your host system logs. These messages are also captured in an agtEvent.log file by the HCM agent and displayed in the HCM master log.
3 Logs TABLE 7 Event message files Event Catalog File Content bfa_aen_rport.html Remote port (R_Port) events. hba_error_codes.
Logs 3 Syslog support You can configure the HCM agent to forward events to a maximum of three system log destinations using the Syslog option on the HCM Configure menu. These events will display in the operating system logs for systems such as Solaris and Linux. For procedures to configure syslog destinations, refer to the Brocade Adapters Administrator’s Guide. HCM logs You can view data about adapter operation through HCM logs that display in HCM.
3 Logs - IP over FC - Events pertaining to IP over Fibre Channel. - RPORT - Events pertaining to a specific remote port (could be an initiator or target). VLAN - Events pertaining to a virtual LAN. PORT - Events pertaining to a physical port. LPORT - Events pertaining to a specific logical port (one logical port always exists per physical port). ITNIM - Events pertaining to an initiator-target nexus. RSVD - Reserved. AUDIT - Audit events subcategory. • Subcategory of main category.
Logs 3 Adjusting the adapter event logging level Specify the number of event messages logged by the host system log for the storage driver using this BCU command. Port logging level Adjust logging level for port logs using BCU commands and HCM. Adjusting the port logging level through HCM Adjust port logging level through HCM using the following steps: 1. Select Configure > Basic Port Configuration from HCM. The Basic Port Configuration dialog box displays. 2.
3 Statistics Adjusting the Ethernet logging level through BCU Adjust port logging level for each Ethernet driver using this BCU command. ethlog --level [] where: ID of port for which you want to change the logging level. Adjusting the HCM logging level Adjust logging levels for the following logs using HCM: • Agent Communication Log, where all messages are exchanged between the HCM GUI application and the HCM agent.
Statistics 3 This section provides an overview of these statistics and how to access them. For more detail, refer to the Brocade Adapter’s Administrator’s Guide. CEE statistics (CNA only) Use BCU commands and HCM to display converged enhanced Ethernet (CEE) statistics, such as the following: • Logical link layer discovery protocol (LLDP) frames transmitted, received, timed out, discarded, with error, type-length-values (TLVs) discarded, and TLVs unrecognized.
3 Statistics • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • Transmitted broadcast octets Transmitted broadcast frames Transmitted broadcast VLANs Transmitted errors Transmitted VLAN filters Transmitted filter MAC source addresses Received unicast octets Received unicast frames Received unicast VLANs Received multicast octets Received multicast frames Received multicast VLANs Received broadcast octets Received broadcast frames Received broadcast VLANs Received frame drops Received packets Tra
Statistics 3 Displaying statistics through HCM To display the Ethernet Statistics dialog box, use the following steps: 1. Select an Ethernet port from the device tree. 2. Select Monitor > Statistics > Eth Statistics from the main menu. OR Right-click the Ethernet port and select Statistics > Eth Statistics from the list. The Ethernet Statistics dialog at the host level displays. Ethernet IOC statistics (CNA only) Use HCM options and BCU to display statistics relevant to the Ethernet IO controller (IOC).
3 Statistics port_id Specifies the ID of the Ethernet port for which you will display the statistics. --statsclr Clears the Ethernet IOC-level statistics. port_id Specifies the ID of the Ethernet port for which you will reset the statistics. FCoE statistics (CNA only) Use HCM to display statistical information related to a selected Fibre Channel over Ethernet (FCoE) port. Statistics include the number of transmitted and received packets and transmitted and received bytes.
Statistics 3 IOC statistics Use BCU and HCM to display port-level statistics for the I/O controller through the BCU and HCM. The I/O controller refers to the firmware entity controlling the port. The following types of IOC statistics are displayed: • • • • • • • IOC driver IOC firmware Firmware IO Firmware port FPG Firmware port PHYSM Firmware port LKSM Firmware port SNSM Displaying IOC statistics through BCU Use the ioc command to display IOC statistics.
3 Statistics lpwwn Logical port world wide name. This is an optional argument. If the -l lpwwn argument is not specified, the base port is used. rpwwn Remote port world wide name. Displaying FCP initiator mode statistics through HCM Use the following steps to display FCP IM Statistics dialog box: 1. Launch the HCM. 2. Select the base adapter port from the device tree window. 3. Click Monitor > Statistics > Remote port statistics > FCP IM Statistics.
Statistics 3 Displaying logical port statistics through BCU Use the lport --stats command to display logical port statistics. lport –-stats [-l lpwwn] where: port_id ID of the port for which you want to display statistics. lpwwn Logical port world wide name for which you want to display statistics. This is an optional argument. If the -l lpwwn argument is not specified, the base port is used. rpwwn Remote port world wide name for which you want to display statistics.
3 Statistics • Hardware abstraction layer (HAL) activity As an example of using these statistics for troubleshooting, if the host cannot see the target, you can verify that the remote port (rport) is reporting itself online by comparing the rport offline and rport online statistics. The rport online counter should be one greater than the rport offline counter. If not, clear the counters and retry connecting to the remote port. Verify the rport online and rport offline statistics again.
Statistics 3 Virtual port statistics (HBA only) Use HCM and BCU to display logical port statistics for fabric discovery (FDISK) activity, logouts (LOGO) activity, NPIV support, number of fabrics online and offline, and fabric cleanups. Use these statistics to isolate NPIV login problems. Following are examples of what to check if virtual devices are not listed in the name server: • If FDISK sent and FDISK accept statistics do not match, the fabric or switch may not be ready for data transmission.
3 Diagnostics Displaying virtual port statistics through BCU Use the vlan --query command to display VLAN statistics. vlan --query where: port_id Specifies the Ethernet port associated with the VLAN. vlan_id Specifies the VLAN identifier. The range for the VLAN ID is 1 to 4094. Diagnostics Diagnostics, available through BCU commands and HCM, evaluate the integrity of adapter hardware and end-to-end connectivity in the fabric.
Diagnostics 3 fcdiag --linkbeacon {on | off} [] where: port_id ID of the port for which you want to run a link beacon test. on | off Toggle on or off. If turned on, you can specify duration. duration Length of time between blinks. Enabling beaconing through HCM Enable link and port beaconing using the following steps. 1. Launch the HCM. 2. Select the base adapter port from the device tree window. 3. Click Configure > Beacon. 4.
3 Diagnostics Performing loopback tests through HCM Use the Port Tests tab on the Diagnostics dialog box to perform a loopback test. 1. Launch the HCM. 2. Select Configure > Diagnostics 3. Click the Port Tests tab. 4. Select Loopback Test. You can modify the following test parameters • • • • Subtest - The three options are Internal, Serdes, and External. Link Speed (HBA only) - For 8G adapter, 2G, 4G and 8G. For 4G adapter, 1G, 2G and 4G. Frame Count - Integer from 0 to 4,294,967,295.
Diagnostics -c frame count -p pattern 3 Specifies the frame count. Specifies the pattern (must be one hex word). NOTE You must disable the port before running a loopback test. PCI loopback test Use BCU commands or HCM to perform a PCI loopback test for a specific port. In this test, a data pattern is sent from the host to adapter firmware through the PCI bus. The returned data is validated to determine PCI operation. NOTE You need to disable the port before you run loopback tests.
3 Diagnostics Memory test Use the BCU or the HCM to perform a memory test for the adapter. NOTE Performing the memory test disables the adapter. Performing a memory test through BCU Use the diag --memtest command to test the adapter’s memory blocks. diag --memtest where: ad_id ID of the adapter. Performing a memory test through HCM Use the Port Tests dialog box to perform a memory test. 1. Launch the HCM. 2. Select Configure > Diagnostics. 3. Click the Port Tests tab. 4. Select Memory Test.
Diagnostics 3 Issuing a ping command to end points through BCU Use the fcdiag --fcping BCU command to test the connection to a Fibre Channel end point. fcdiag -–fcping [-l lpwwn] where: port_id ID of the adapter port from which you want to ping the remote port. rpwwn Remote port WWN that you want to ping. lpwwn Logical port WWN. 0 indicates the base port.
3 Diagnostics 4. Select FC Trace Route. 5. Select the adapter port and target for which you wish to trace the route. 6. Enter a test cycle if desired. 7. Click Start. Echo test Use the BCU and HCM to initiate an echo test between the adapter port and a Fibre Channel end point. This sends an ECHO command and response sequence between the adapter port and target port to verify connection with the target.
Collecting LLDP data (CNA only) lpwwn 3 Logical port WWN. 0 indicates the base port; otherwise. Test Logs While running a diagnostic test in HCM, a log of the test results displays at the bottom of the Diagnostic dialog box. Display details of the test log by double-clicking a row in the log. Collecting LLDP data (CNA only) Collect information on the Link Layer Discovery Protocol (LLDP) associated with a specific CNA using the HCM LLDP Properties panel.
3 Collecting port data 3. Click the SFP tab in the right pane. Port power on management Use the Port POM properties panel to monitor the SFP attributes. A notification is given for any parameters that are not within the power, temperature, voltage, and current specification. 1. Select a port in the device tree. 2. Click the POM tab in the right pane. Collecting port data This section provides an overview of BCU commands and HCM features that provide information on adapter ports.
Collecting port data • • • • • • • • • • • • • 3 Operating speed Receive and transmit BB credit Frame data field size MPIO enabled or disabled status Queue depth Interrupt control coalesce on or off status Interrupt control latency and delay values Boot over SAN configuration Beaconing state Target rate limiting on or off status FC-SP parameter statistics MPIO on or off state Health status of Fibre Channel security protocol parameters To display CEE port properties, use the following steps: 1.
3 Collecting port data • • • • • • • • • • • Maximum transmission unit (MTU) Target rate limiting (TRL) enabled or disabled status SCSI queue depth Beaconing status Fibre Channel Initialization Protocol (FCIP) online or offline state Priority group ID Bandwidth percentage for priority group MAC address for FCoE forwarder Fabric WWN Fibre Channel map ID FCoE forwarder writing or nonwriting mode To display the HCM FCoE Port Properties panel, use these steps. 1. Select an FCoE port in the device tree. 2.
Authentication settings 3 Displaying the port log Use the HCM debug --portlog BCU command to display a log of Fibre Channel frames and other main control messages that were sent out and received on a specific port. You can use this information to isolate adapter and Fibre Channel protocol problems. debug --portlog where: port_id The ID of the port for which you want to display the port log. NOTE If the port log is disabled, a warning message displays.
3 QoS and target rate limiting settings (HBA only) 1. Select a port from the device tree. 2. Select Configure > FC-SP > Authentication. The Fibre Channel Security Protocol Configuration dialog box displays. This displays the current CHAP secret, hashing algorithm, and group value. Displaying authentication settings through BCU Use the BCU auth --show command to display authentication settings. auth --show where: port_id ID of the port for which you want to display authentication settings.
Persistent binding 3 where: port_id Specifies the ID of the port for which you want to query attributes of a remote port. rpwwn Remote port WWN. You can obtain the RPWWN from the BCU rport --list command. Determining QoS and other settings through HCM Use HCM in the following ways to determine current status and configuration for QoS and target rate limiting settings: • Use the Port Properties panel in HCM to display configured QoS parameters. 1. Select a port in the device tree. 2.
3 Persistent binding 3. Select Configure > Persistent Binding. OR Right-click on an adapter or port in the device tree and select Persistent Binding from the list. The Persistent Binding dialog box at the host level displays.
Chapter Performance optimization 4 In this chapter • Tuning storage drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 • Tuning network drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 Tuning storage drivers This section provides resources for optimizing performance in HBAs and CNAs by tuning the unified storage drivers on Linux, Windows, Solaris, and VMware systems.
4 Tuning storage drivers Solaris tuning To increase I/O transfer performance, set the following parameters on your system: • Set the maximum device read/write directive (maxphy). • Set the disk maximum transfer parameter (ssd_max_xfer_size). Please refer to Sun StorageTek SAM File System Configuration and Administration Guide document for details of the two parameters.” Windows tuning Windows tuning involves configuring the driver and operating system tunable parameters.
Tuning network drivers 4 MSI-X is supported in Windows Vista and Windows Server 2008. To enable MSI-X, set the following registry key value to 0. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\bfad\Parameters\Device\msix_ disable OS tunable parameters Please see the section “Storage Stack Drivers” in Disk Subsystem Performance Analysis for Windows Server 2003 optimizations located on the following website. http://download.microsoft.
4 Tuning network drivers Parameter Default TCP-UDP Checksum Offload Enabled Network Address N/A Receive Buffers 32 Transmit Buffers 16 Priority and VLAN Disable Receive Side Scaling (RSS) Enabled Header Data Split (HDS) Enabled Large Segmentation Offload V1 IPv4 (LSOv1) Enabled Large Segmentation Offload V2 IPv4 (LSOv2) Enabled Large Segmentation Offload V2 IPv6 (LSOv2) Enabled FlowControl, Transmit (Tx) and Receive (Rx) Enabled Interrupt Moderation Enabled VLAN ID Disabled Prio
Tuning network drivers 4 VMware tuning All VMware tunable parameters for the network driver are optimized for best performance using default values. For details on configuring these parameters, refer to the “Adapter Configuration” chapter in the Brocade Adapters Installation and Reference Manual. Specific recommendations for jumbo packet size and NetQueue follow. Parameters are listed in the following table.
4 Tuning network drivers For the Brocade driver, you cannot directly configure the number of NetQueue and filters per NetQueue. By default, these values are based on the number of receive queue sets used, which is calculated from the number of CPUs in the system. You can change the number of NetQueues from the default (number of CPUs) by setting the number of receive queue sets (bnad_rxqsets_used).
Appendix A Event Message Reference This appendix provides details on event messages generated by adapter drivers. These events display in host system logs and the HCM master log. Events are organized as originating from the network driver only, storage driver only, or network and storage driver.
A TABLE 9 Event Message Reference Driver event messages Message Severity Event Type Category Sub Category Cause Action New virtual port created using proprietary interface: WWN = [logical port WWN], Role: [initiator, target, IPFC mode etc.]. Information Storage driver 3 (LPORT) Created. New virtual port created. No action required. New virtual port created using standard interface: WWN = [logical port WWN], Role: [initiator, target, IPFC mode etc.
Event Message Reference TABLE 9 A Driver event messages Message Severity Event Type Category Sub Category Cause Action Virtual port (WWN = [logical port WWN]) login failed. Max NPIV ports already exist in fabric/fport. Warning Storage driver 3 (LPORT) Failed Max NPIV ports already exist in fabric/fport. Check fabric and fport configuration Virtual port (WWN = %s) login failed. Warning Storage driver 3 (LPORT) Failed Unknown error. Check fabric/fport configuration.
A TABLE 9 Event Message Reference Driver event messages Message Severity Event Type Category Sub Category Cause Action IOC [IOC instance number] is disabled Warning Network and storage driver 9 (TOC) Disabled Adapter firmware stopped by user command. No action required. New adapter found: SN = [adapter serial number] base port Hardware address = [Bae port WWN or MAC] Information Network and storage driver 1 (Adapter) Added Adapter added to host. No action required.
Index A adapter diagnostics, 62 adapter BIOS not installed message, 22 adapter event message files, 47 adapter IP address lost, 23 adapter list command, 45 adapter not registering with name server, 30 adapter not reported under PCI subsystem, 8 adapter not showing in fabric, 29 adapter properties panel, 45 adapter query command, 45 adapter statistics, 52 application log, 49, 50 authentication settings, 73 CNA MAC addressing, xvii collecting data using BCU, 45 collecting data using event logs, 48 collectin
diagnostics adapter, 62 beaconing, 62 enabling through BCU, 62 enabling through HCM, 63 echo test enabling through BCU, 68 enabling through HCM, 68 Ethernet loopback tests enabling through BCU, 64 Ethernete loopback tests enabling through HCM, 64 HBA temperature, 66 loopback tests, 63 enabling through BCU, 63 enabling through HCM, 64 memory test, 66 enabling through BCU, 66 enabling through HCM, 66 PCI loopback tests, 65 enabling through BCU, 65 enabling through HCM, 65 ping end points, 66 enable through HC
HBA and CNA problems problem general HBA and CNA, 8 HBA memory test, 66 HBA problem resolving BIOS boot problems, 19 UEFI boot, 17 unable to create more than 126 virtual ports, 17 HBA problems, 16 HCM logs, 49 HCM options to collect data, 42 HCMlogs logging levels, 52 host commands for collecting data, 40 host freezes or crashes, 10 host not booting from remote LUN, 17 host system logs, 48 host syxtem freezes, 10 I I/O data traffic issues, 16 I/O problem on FCoE device, 31 I/Os not failing over on path fai
operating system support, x, xii P PCI loopback tests, 65 enabling through BCU, 65 enabling through HCM, 65 performance optimization Linux network driver tuning, 80 Linux stoirage driver tuning, 77 Solaris storage driver tuning, 78 VMware network driver tuning, 81 VMware tuning, 79 Windows network driver tuning, 79 Windows storage driver tuning, 78 persistent binding settings, 75 ping end points diagnostics, 66 enable through HCM, 67 enabling through BCU, 67 ping to server failing, 25 poor network performa
properties panel for HBA, 45 publication references, 37 Q QoS performance issues, 16 QoS settings, 74 QoS statistics, 60 displaying through BCU, 60 displaying through HCM, 60 R references for isolating problems, 37 remote LUNs not visible, 21 remote port properties, 72 remote port statistics, 59 displaying through BCU, 60 displaying through HCM, 60 S SCSI target ID mappings, 75 SCSI test, 68 serial number location, xvi SFP diagnostics enable through BCU, 69 SFP properites, 69 SFP properties enable throug
temperature diagnostics, 66 trace route, 67 enable through HCM, 67 enabling through BCU, 67 troubleshooting gathering information, 2 host not booting from remote LUN, 17 introduction, 1 no target devices found, 20 using this manual, 1 Windows storage driver tunable parameters, 78 Windows storage driver tuning, 78 WWPN of HBA, xvii U UEFI boot, 17 UEFI boot problems, 17 unable to boot from device, 21 unable to create NPIV ports, 17 V verifying Fibre Channel and CEE links, 32 verifying installation, 34 vir
