HyperFabric Administrator’s Guide HP-UX 11i v2 Edition 14 Manufacturing Part Number : B6257-90043 March 2004 United States © Copyright 2004 Hewlett-Packard Development Company L.P. All rights reserved.
Legal Notices The information in this document is subject to change without notice. Hewlett-Packard makes no warranty of any kind with regard to this manual, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. Hewlett-Packard shall not be held liable for errors contained herein or direct, indirect, special, incidental or consequential damages in connection with the furnishing, performance, or use of this material.
Contents About This Document 1. Overview of the HyperFabric Product About HyperFabric. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . HyperFabric Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . HyperFabric Adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Switches and Switch Modules . . . . . . . . . . . . . . . . . . . . . .
Contents Online Addition and Replacement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Planning and Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Critical Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Card Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Installing the Software . . . . . . .
Contents 5. Managing HyperFabric Starting HyperFabric. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Using the clic_start Command. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Using SAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Verifying Communications within the Fabric. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Contents Radio Frequency Interference (Japan Only) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Declarations of Conformity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Physical Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Environmental . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Figures Figure 2-1. TCP/UDP/IP Point-To-Point Configurations . . . . . . . . . . . . . . . . . . . . . . 22 Figure 2-2. TCP/UDP/IP Basic Switched Configuration . . . . . . . . . . . . . . . . . . . . . . 23 Figure 2-3. TCP/UDP/IP High Availability Switched Configuration . . . . . . . . . . . . 24 Figure 2-4. TCP/UDP/IP Hybrid Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Figure 2-5. HMP Point-To-Point Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Figures viii
Tables Table 1. HP-UX 11i Releases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii Table 2. Publishing History Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii Table 3. Organization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii Table 2-1. HF2 Throughput and Latency with TCP/UDP/IP Applications. . . . . . . . 19 Table 2-2. Supported HyperFabric Adapter Configurations .
Tables x
About This Document This document describes how to install, configure, and troubleshoot the HyperFabric product on the HP-UX 11i v2 (HP-UX 11.23) operating system. Before you install the HyperFabric software, ensure that the operating system software and the appropriate files, scripts, subsets are installed. The document printing date and part number indicate the document’s current edition. The printing date will change when a new edition is printed.
HP-UX Release Name and Release Identifier Each HP-UX 11i release has an associated release name and release identifier. The uname (1) command with the -r option returns the release identifier. Table 1 shows the releases available for HP-UX 11i. Table 1 HP-UX 11i Releases Release Identifier Supported Processor Architecture Release Name B.11.11 HP-UX 11i v1 PA-RISC B.11.20 HP-UX 11i v1.5 Intel Itanium B.11.22 HP-UX 11i v1.6 Intel Itanium B.11.
Table 2 Publishing History Details (Continued) Edition Number Publication Date Eleventh June 2001 Twelfth September 2002 Thirteenth July 2003 Fourteenth March 2004 What Is in This Document HyperFabric Administrator’s Guide is divided into several chapters, each of which contains information about installing, configuring, or troubleshooting HyperFabric. The appendixes contain supplemental information. The following list describes the content in more detail.
New and Changed Information in This Edition This edition includes information about the transparent local failover feature of Hyper Messaging Protocol (HMP). This feature is available with the HyperFabric version B.11.23.01. Typographic Conventions This document uses the following typographic conventions: Book Title Italic (slanted) type indicates document and book names. daemon Courier font type indicates daemons, files, commands, manpages, and option names.
1 Overview of the HyperFabric Product This chapter contains the following sections that give general information about HyperFabric: Chapter 1 1
Overview of the HyperFabric Product 2 • “About HyperFabric” on page 3 • “HyperFabric Products” on page 4 • “HyperFabric Concepts” on page 7 Chapter 1
Overview of the HyperFabric Product About HyperFabric About HyperFabric HyperFabric is an HP high-speed, packet-based interconnect for node-to-node communications. HyperFabric provides higher speed, lower network latency and uses less CPU than other industry standard protocols (for example, Fibre Channel and Gigabit Ethernet). Instead of using a traditional bus-based technology, HyperFabric is built around switched fabric architecture, providing the bandwidth necessary for high speed data transfer.
Overview of the HyperFabric Product HyperFabric Products HyperFabric Products HyperFabric hardware consists of host-based interface adapter cards, interconnect cables, and optional switches. HyperFabric software resides in Application Specific Integrated Circuits (ASICs) and firmware on the adapter cards and includes user-space components and HP-UX drivers. Currently, fiber-based HyperFabric hardware are available. In addition, a hybrid switch that has 8-fiber ports is available to support HF2 clusters.
Overview of the HyperFabric Product HyperFabric Products • HP-UX 11.0: HyperFabric software version B.11.00.09 • HP-UX 11i v1: HyperFabric software version B.11.11.00 • HP-UX 11i v2: HyperFabric software version B.11.23.00 The A6386A HyperFabric2 adapter is supported beginning with the following HyperFabric software versions: • HP-UX 11.0: HyperFabric software version B.11.00.11 • HP-UX 11i v1: HyperFabric software version B.11.11.01 • HP-UX 11i v2: HyperFabric software version B.11.23.
Overview of the HyperFabric Product HyperFabric Products IMPORTANT HF2 adapters and switches are not supported by software versions earlier than those listed in “HyperFabric Adapters” on page 4 and “Switches and Switch Modules” on page 5.
Overview of the HyperFabric Product HyperFabric Concepts HyperFabric Concepts This section briefly describes some of the basic HyperFabric concepts and terms. The fabric is the physical configuration that consists of all the HyperFabric adapters, the HyperFabric switches (if any), and the HyperFabric cables connecting them. The network software controls data transfer over the fabric. The HyperFabric configuration contains two or more HP 9000 systems and optional HyperFabric switches.
Overview of the HyperFabric Product HyperFabric Concepts reason (for example, if it is killed), all HyperFabric-related communications on that node are stopped immediately. This makes the node unreachable by other components in the fabric. When you start HyperFabric, the fabric is verified automatically. This is because each node performs a self diagnosis and verification over each adapter installed in the node.
2 Planning the Fabric This chapter contains the following sections that include general guidelines and protocol-specific considerations for planning HyperFabric clusters that run TCP/UDP/IP or HMP applications.
Planning the Fabric 10 • “Preliminary Considerations” on page 11 • “HyperFabric Features, Parameters and Supported Configurations for TCP/UDP/IP and HMP Applications” on page 13 • “TCP/UDP/IP” on page 14 • “Hyper Messaging Protocol (HMP)” on page 26 Chapter 2
Planning the Fabric Preliminary Considerations Preliminary Considerations Before assembling a fabric physically, do the following to address all of the appropriate issues: Step 1. Read Chapter 1, “Overview of the HyperFabric Product,” on page 1, to get a basic understanding of HyperFabric and its components. Step 2. Read this chapter, Planning the Fabric, to gain an understanding of protocol specific configuration guidelines for TCP/UDP/IP and HMP applications. Step 3.
Planning the Fabric Preliminary Considerations Step 11. Draw the cable connections from each node to switches (if the fabric will contain switches). If you use an HA configuration with switches, it requires more than one switch for complete redundancy and to avoid a single point of failure. For example, each adapter can be connected to its own switch, or two switches can be connected to four adapters.
Planning the Fabric HyperFabric Features, Parameters and Supported Configurations for TCP/UDP/IP and HMP Applications HyperFabric Features, Parameters and Supported Configurations for TCP/UDP/IP and HMP Applications The following sections in this chapter define HyperFabric functionality for TCP/UDP/IP applications and Hyper Messaging Protocol (HMP) applications.
Planning the Fabric TCP/UDP/IP TCP/UDP/IP TCP/UDP/IP is supported on all HF2 hardware. Although some of the HyperFabric adapter cards support both HMP and TCP/UDP/IP applications, in this section, the focus is on TCP/UDP/IP HyperFabric applications. Application Availability All applications, including Oracle 9i and HP-MPI, that use the TCP/UDP/IP stack are supported.
Planning the Fabric TCP/UDP/IP — Logging into a user specified log file with a choice of severity — Email to a user defined email address. For more information on EMS, including instructions for implementing this feature, see “Configuring the HyperFabric EMS Monitor” on page 85 and the EMS Hardware Monitors User’s Guide Part Number B6191-90028 September 2001 Edition. • ServiceGuard: Supported Within a cluster, ServiceGuard groups application services (individual HP-UX processes) into packages.
Planning the Fabric TCP/UDP/IP ServiceGuard uses a “heartbeat” to monitor the cluster. The HyperFabric links cannot be used for the heartbeat. Instead, an alternate LAN connection such as 100BaseT, Ethernet, Token Ring, or FDDI must be made between the nodes for use as a heartbeat link. End-To-End HA: HyperFabric provides end-to-end HA on the entire cluster fabric at the link level.
Planning the Fabric TCP/UDP/IP Switch Management is not supported. Switch management will not operate properly if you enable it on a HyperFabric cluster. • Diagnostics: Supported Diagnostics can be run to obtain information on many of the HyperFabric components using the clic_diag, clic_probe and clic_stat commands, as well as the Support Tools Manager (STM). For more information on HyperFabric diagnostics, see “Running Diagnostics” on page 133.
Planning the Fabric TCP/UDP/IP You can interconnect (mesh) up to 4 switches (16-port fiber, or Mixed 8 fiber ports) in a single HyperFabric cluster. • Trunking Between Switches (multiple connections) You can use trunking between switches to increase bandwidth and cluster throughput. Trunking is also a way to eliminate a possible single point of failure. The number of trunked cables between nodes is limited only by port availability.
Planning the Fabric TCP/UDP/IP • Table 2-1 Throughput and Latency HF2 Throughput and Latency with TCP/UDP/IP Applications Server Class rp7400 Chapter 2 Maximum Throughput 2 + 2 Gbps full duplex per link Latency < 42 microsec 19
Planning the Fabric TCP/UDP/IP Table 2-2 HF Adapter Supported HyperFabric Adapter Configurations Supported HP Systems A6386A PCI (4X) rx2600 servers 11i v2 No 1 A6386A PCI (4X) rx56XX servers 11i v2 No 4 A6386A PCI (4X) zx6000 workstations 11i v2 No 1 A6386A PCI (4X) SD64A servers 11i v2 Yes 8 (maximum 4 per PCI card cage) A6386A PCI (4X) rx7620 servers 11i v2 No 8 (maximum 4 per PCI card cage) A6386A PCI (4X) rx8620 servers 11i v2 Yes 8 (maximum 4 per PCI card cage) A
Planning the Fabric TCP/UDP/IP TCP/UDP/IP Supported Configurations Multiple TCP/UDP/IP HyperFabric configurations are supported to match the cost, scaling, and performance requirements of each installation. In the previous section, “Configuration Parameters” on page 17, the maximum limits for TCP/UDP/IP enabled HyperFabric hardware configurations were outlined. In this section the TCP/UDP/IP enabled HyperFabric configurations that HP supports are explained.
Planning the Fabric TCP/UDP/IP Figure 2-1 22 TCP/UDP/IP Point-To-Point Configurations Chapter 2
Planning the Fabric TCP/UDP/IP Switched Configuration This configuration offers the same benefits as the point-to-point configurations illustrated in Figure 2-1, but it has the added advantage of greater connectivity (see Figure 2-2).
Planning the Fabric TCP/UDP/IP High Availability Switched Configuration This configuration has no single point of failure. The HyperFabric driver provides end-to-end HA. If any HyperFabric resource in the cluster fails, traffic is transparently rerouted through other available resources. This configuration provides high performance and high availability (see Figure 2-3).
Planning the Fabric TCP/UDP/IP Hybrid Configuration You can interconnect servers and workstations in a single heterogeneous HyperFabric cluster. In this configuration, the servers are highly available. In addition, the workstations and the servers can run the same application or different applications (see Figure 2-4).
Planning the Fabric Hyper Messaging Protocol (HMP) Hyper Messaging Protocol (HMP) Hyper Messaging Protocol (HMP) is an HP patented, high performance cluster interconnect protocol. HMP provides reliable, high speed, low latency, low CPU overhead, datagram service to applications running on the HP-UX operating system. HMP was jointly developed with Oracle Corp. The resulting feature set was tuned to enhance the scalability of the Oracle Cache Fusion clustering technology.
Planning the Fabric Hyper Messaging Protocol (HMP) HP MPI is a native implementation of version 1.2 of the Message-Passing Interface Standard. It has become the industry standard for distributed technical applications and is supported on most technical computing platforms.
Planning the Fabric Hyper Messaging Protocol (HMP) nodes in a cluster. ServiceGuard cannot use the HyperFabric interconnect as a heartbeat link. Instead, a separate LAN must be used for the heartbeat. For more information on configuring ServiceGuard, see “Configuring HyperFabric with ServiceGuard” on page 87, as well as Managing MC/ServiceGuard Part Number B3936-90065 March 2002 Edition.
Planning the Fabric Hyper Messaging Protocol (HMP) same process takes place when a resource is removed from a cluster. However, DRU is not supported if you add or remove an adapter from a node that is running an HMP application. This is consistent with the fact that OLAR is not supported when an HMP application is running on HyperFabric.
Planning the Fabric Hyper Messaging Protocol (HMP) HyperFabric clusters running HMP applications are limited to supporting a maximum of 64 adapter cards. However, in local failover configurations, a maximum of only 52 adapters are supported. In point-to-point configurations running HMP applications, the complexity and performance limitations of having a large number of nodes in a cluster make it necessary to include switches in the fabric.
Planning the Fabric Hyper Messaging Protocol (HMP) Table 2-3 • HMP is supported on A400, A500, rp2400, rp2450, rp54xx (N-class), rp74xx (L-class), rp8400, and Superdome servers running 64-bit HP-UX. • HMP is supported on HyperFabric starting HyperFabric versions B.11.00.11, B.11.11.01, and B.11.23.00. • HMP is not supported on the A180 or A180C server. • HMP is not supported on 32-bit versions of HP-UX.
Planning the Fabric Hyper Messaging Protocol (HMP) NOTE 32 The local failover configuration on HMP is supported only on the A6386A HF2 adapters.
Planning the Fabric Hyper Messaging Protocol (HMP) HMP Supported Configurations Multiple HMP HyperFabric configurations are supported to match the performance, cost and scaling requirements of each installation. In the section, “Configuration Parameters” on page 29, the maximum limits for HMP enabled HyperFabric hardware configurations were outlined. This section discusses the HMP enabled HyperFabric configurations that HP supports.
Planning the Fabric Hyper Messaging Protocol (HMP) Figure 2-5 34 HMP Point-To-Point Configurations Chapter 2
Planning the Fabric Hyper Messaging Protocol (HMP) Enterprise (Database) Configuration The HMP enterprise configuration illustrated in Figure 2-6 is very popular for running Oracle RAC 9i. Superdomes or other large servers make up the Database Tier. Database Tier nodes communicate with each other using HMP. Application Tier nodes communicate with each other and to the Database Tier using TCP/UDP/IP.
Planning the Fabric Hyper Messaging Protocol (HMP) Enterprise (Database) - Local Failover Supported Configuration The HMP enterprise configuration is a scalable solution. For high availability and performance, you can easily scale the HMP enterprise configuration with multiple connections between the HyperFabric resources. Any single point of failure in the database tier of the fabric is eliminated in Figure 2-7.
Planning the Fabric Hyper Messaging Protocol (HMP) Technical Computing (Work Stations) Configuration This configuration is typically used to run technical computing applications with HP-MPI. A large number of small nodes are interconnected to achieve high throughput (see Figure 2-8). High availability is not usually a requirement in technical computing environments. HMP provides the high performance, low latency path necessary for these technical computing applications.
Planning the Fabric Hyper Messaging Protocol (HMP) Figure 2-8 38 Technical Computing Configuration Chapter 2
Planning the Fabric Hyper Messaging Protocol (HMP) Figure 2-9 Chapter 2 Large Technical Computing Configuration 39
Planning the Fabric Hyper Messaging Protocol (HMP) 40 Chapter 2
3 Installing HyperFabric This chapter contains the following sections that describe the HyperFabric installation: • Chapter 3 “Checking HyperFabric Installation Prerequisites” on page 43.
Installing HyperFabric 42 • “Installing HyperFabric Adapters” on page 44. • “Installing the Software” on page 51. • “Installing HyperFabric Switches” on page 57.
Installing HyperFabric Checking HyperFabric Installation Prerequisites Checking HyperFabric Installation Prerequisites Before installing HyperFabric, ensure that the following hardware and software prerequisites are met: ✓ Check the HyperFabric Release Notes for known problems, required patches, or other information needed for installation. ✓ Confirm that the /usr/bin, /usr/sbin, and /sbin directories are in your PATH by logging in as root and using the echo $PATH command.
Installing HyperFabric Installing HyperFabric Adapters Installing HyperFabric Adapters This section contains information about installing HyperFabric adapters in HP 9000 systems. Online Addition and Replacement (OLAR) information is provided in the section, “Online Addition and Replacement” on page 44. CAUTION HyperFabric adapters contain electronic components that can easily be damaged by small amount of electricity.
Installing HyperFabric Installing HyperFabric Adapters available on HP 9000 systems that are designed to support OLAR. The system hardware uses the per-slot power control combined with OS support to enable this feature. NOTE OLAR is supported only on TCP/UDP/IP over HF2 adapters. Not all add-in cards have this capability, but over time many cards will be gaining this capability. The HyperFabric Release Notes contains information about which HP 9000 systems and HyperFabric adapters OLAR is supported for.
Installing HyperFabric Installing HyperFabric Adapters Table 3-1 below explains some important OLAR-related terms. Table 3-1 Important OLAR Terms Term IMPORTANT Meaning OLAR All aspects of the OLAR feature including Online Addition (OLA) and Online Replacement (OLR). Power Domain A grouping of 1 or more interface card slots that are powered on or off as a unit. (Note: Multi-slot power domains are not currently supported.
Installing HyperFabric Installing HyperFabric Adapters Planning and Preparation As mentioned previously, for the most part, SAM prevents the user from performing OLAR procedures that would adversely affect other areas of the HP 9000 system. See Configuring HP-UX For Peripherals for detailed information. Critical Resources The effects of shutting down a card’s functions must be considered. Replacing a card that is still operating can have extensive consequences.
Installing HyperFabric Installing HyperFabric Adapters Online Addition (OLA) Multiple cards can be added at the same time. When adding a card online, the first issue to resolve is whether the new card is compatible with the system. Each OLAR-capable PCI slot provides a set amount of power. The replacement card cannot require more power than there is available. The card must also operate at the slot’s bus frequency.
Installing HyperFabric Installing HyperFabric Adapters Online Replacement (OLR) When replacing an interface card online, the replacement card must be identical to the card being replaced (or at least be able to operate using the same driver as the replaced card). This is referred to as like-for-like replacement and should be adhered to, because using a similar but not identical card can cause unpredictable results.
Installing HyperFabric Installing HyperFabric Adapters You are asked if you want to continue. If you reply Yes, client applications are suspended. Replace the adapter according to the procedure described in the “Managing PCI Cards with OLAR” chapter of the Configuring HP-UX Peripherals manual. When an adapter has been replaced, client application activity resumes unless the TCP timers or the application timers have popped.
Installing HyperFabric Installing the Software Installing the Software This section describes the HyperFabric file structure and the steps necessary to load the software. The software must be installed on each instance of the HP-UX operating system in the fabric. File Structure The HyperFabric file structure is shown in Figure 3-1 below. The structure is shown for informational purposes only. The user cannot modify any of the files or move them to a different directory.
Installing HyperFabric Installing the Software The commands and files used to administer HyperFabric typically have a prefix of clic_. CLIC stands for CLuster InterConnect, and it is used to differentiate those HyperFabric commands/files from other commands/files. For example, the HyperFabric command clic_init is different from the HP-UX init command.
Installing HyperFabric Installing the Software • /var/adm/OLDclic_log The log file from the previous time the clic_start command was executed. • /usr/conf/lib/libclic_dlpi_drv.a The kernel library that contains the HyperFabric software. • /usr/conf/lib/libha_drv.a The kernel library that contains the High Availability (HA) software. • /usr/conf/master.d/clic This file is described along with the other master files in the master man page (type man master at the HP-UX prompt).
Installing HyperFabric Installing the Software • /opt/clic/firmware/clic_fw_4x8c The 4X PCI HyperFabric 8-bit CRC firmware. This file must not be modified. • /opt/clic/firmware/clic_fw_4x32c The 4X HyperFabric PCI 32-bit CRC firmware. This file must not be modified. • /opt/clic/firmware/clic_fw_hf28c The HyperFabric2 8-bit firmware. This file must not be modified. • /opt/clic/firmware/clic_fw_hf232c The HyperFabric2 32-bit firmware. This file must not be modified.
Installing HyperFabric Installing the Software Loading the Software Listed below are the steps you must follow to load the HyperFabric software, using the HP-UX swinstall program. Step 1. Log on to the system as root. Step 2. Insert the software media into the appropriate drive. If the software is being loaded from a CD-ROM, go to step 3; otherwise, go to step 4. Step 3.
Installing HyperFabric Installing the Software swinstall loads the fileset, runs the control script for the filesets, and builds the kernel. When the processing is finished, the “Status” field displays a “Ready” message. Select “Done” and then the “Note” window opens. Step 11. Select the OK button in the “Note” window to reboot. The user interface disappears and the system reboots. Step 12. When the system comes back up, log on to the system as root and view the /var/adm/sw/swagent.
Installing HyperFabric Installing HyperFabric Switches Installing HyperFabric Switches This section contains the information you need to install HyperFabric switches. As stated earlier, the term HyperFabric2 (HF2) switch refers to the functional switch (the A6384A switch chassis with one of the switch modules installed).
Installing HyperFabric Installing HyperFabric Switches — If your cables have dust caps over the connectors, keep them in place until you are ready to connect them. This prevents dirt and oils from soiling any important surfaces. — Be careful not to stretch, puncture, or crush the cable. To install an HF2 switch, see “Installing the HF2 Switch” on page 59.
Installing HyperFabric Installing HyperFabric Switches Installing the HF2 Switch This section contains information on installing an HF2 switch. The front of the HF2 switch has a flange—or “wing”—on each side, with two holes for attaching the switch to the rack. The following figures do not show the flanges. Figure 3-2 shows the front of the HF2 switch with an A6388A HF2 8-port fiber switch module installed in the switch’s expansion slot.
Installing HyperFabric Installing HyperFabric Switches Figure 3-3 shows the front of the HF2 switch with an A6389A HF2 4-port copper switch module installed in the switch’s expansion slot.
Installing HyperFabric Installing HyperFabric Switches With the Rail Kit As mentioned earlier, HP strongly recommends installing the HF2 switch using the rail kit. When you install the HF2 switch, you will be putting the front of the switch—the end with the flanges (“wings”)—at the back of the rack. The steps for installing the HF2 switch using the rail kit are as follows: Step 1. Prepare the rack for rail and switch installation. Step 2. Install and secure the rails in the rack, using two screws per rail.
Installing HyperFabric Installing HyperFabric Switches sliding the switch. If so, try lifting the switch over the rail screws. If you cannot do this, remove the rail screws, slide the switch into position, and put the rail screws back in. Step 4. Align the two holes in each flange (“wing”) on the switch’s front with the holes in the rack frame. Fasten each flange of the switch to the rack by putting a screw in each of the four holes in the flanges. Be sure to use screws with over-sized heads.
Installing HyperFabric Installing HyperFabric Switches the flat part). Otherwise, the brackets will not secure the back of the switch properly. The following figure shows the rack with these two brackets installed. rack back fr (w ont ith of fla sw ng itch es ) bracket 2 rack front bracket 1 Step 6. When you are sure the brackets are snug against the beveled part of the switch, tighten all four screws in brackets 1 and 2. Step 7.
Installing HyperFabric Installing HyperFabric Switches Step 9. Plug the switch’s power cord into the rack’s PDU, if it has one. Alternatively, you can plug a power cord that is compatible with your country’s requirements into a power strip or outlet that you want to use for the switch. (In this case, you are responsible for obtaining a compatible power cord.) Step 10. Power on the HF2 switch by plugging the power cord into the AC inlet on the back of the switch. (There is no power switch.) Step 11.
Installing HyperFabric Installing HyperFabric Switches Without the Rail Kit As mentioned earlier, HP strongly recommends installing the HF2 switch using the rail kit (described in the previous section, “With the Rail Kit” on page 61). When you install the HF2 switch, you will be putting the front of the switch—the end with the flanges (“wings”) — at the back of the rack. The steps for installing the HF2 switch without using the rail kit are as follows: Step 1. Prepare the rack for switch installation.
Installing HyperFabric Installing HyperFabric Switches Step 10. Once the power is on, check these LEDs on the integrated Ethernet management LAN adapter card (in the top slot of the switch): ✓ The “Operating/Fault” LED shows solid green. ✓ The “Power A” and “Power B” LEDs show solid green. ✓ The “Ethernet Port Main” and “Ethernet Port Aux” LEDs are showing solid green (connected) or flashing green (Ethernet traffic is flowing to the switch).
4 Configuring HyperFabric This chapter contains the following sections that describe configuring HyperFabric: • Chapter 4 “Configuration Overview” on page 69.
Configuring HyperFabric 68 • “Information You Need” on page 71. • “Performing the Configuration” on page 78. • “Deconfiguring a HyperFabric Adapter with SAM” on page 83. • “Configuring the HyperFabric EMS Monitor” on page 85. • “Configuring HyperFabric with ServiceGuard” on page 87. • “Configuring HMP for Transparent Local Failover Support” on page 96.
Configuring HyperFabric Configuration Overview Configuration Overview You do not need to configure the HyperFabric switch because the HyperFabric management process performs automatic routing and configuring for the switch. So, configuring HyperFabric consists only of creating the HyperFabric /etc/rc.config.d/clic_global_conf global configuration file on each node in the fabric.
Configuring HyperFabric Configuration Overview IMPORTANT In this file, clic_init and SAM add some HyperFabric-related lines that end with the characters #clic. These lines are used by the HyperFabric software — and are not comments — so do not remove them from the file. • The system /etc/rc.config.d/clic_global_conf file. • The /etc/rarpd.conf (Reverse Address Resolution Protocol [RARP]) support file.
Configuring HyperFabric Information You Need Information You Need When you run the clic_init command or use SAM for configuration, you have to provide certain configuration information. So, before you run clic_init or use SAM, you should have the following information: ❏ For each node in the fabric, determine if that node will need to interoperate with other nodes that are using; any HP-UX 11.0 HyperFabric versions earlier than B.11.00.11 or any HP-UX 11i v1 HyperFabric versions earlier than B.11.11.01.
Configuring HyperFabric Information You Need IMPORTANT ✓ The IP address of the switch. ✓ The MAC address of the switch’s Ethernet port. If you do not already know the switch’s MAC address, it is printed on a label on the back of the HF switch and on the front of the HF2 switch. See Figure 3-2 on page 66 for the location of the label on the HF switch, and Figure 3-2 on page 59 and Figure 3-3 on page 60 for the location of the label on the HF2 switch.
Configuring HyperFabric Information You Need IP_address IP_address IP_address Chapter 4 bently6.corp3.com bently4.corp7.com bently2.corp4.
Configuring HyperFabric Information You Need Configuration Information Example This example uses some “dummy” (that is, not valid) addresses to the components in Figure 4-1. The dummy addresses are used only to show the flow of the information provided as input to the clic_init command and SAM. Do not try to use these addresses in your configuration.
Configuring HyperFabric Information You Need Using the configuration information in Figure 4-1, the information you would specify when you run clic_init or SAM on each of the nodes is listed below. This example is not an exact depiction of the prompts produced by clic_init nor the fields in SAM, but merely an example of the flow of information input. In addition, you should not try to use the dummy addresses in your actual configuration. On node A: 1.
Configuring HyperFabric Information You Need 12. What is the Ethernet hardware address of the second switch? (0060b0d00256) 13. What is the Multicast address for the switches to use? (226.10.1.1) 14. What is the IP address for the LAN card on the same subnet as the switches? (193.0.0.10) (In Figure 4-1, this is the IP address for lan0 on node A.) On node B: 1. How many HyperFabric adapters are installed on the node? 2.
Configuring HyperFabric Information You Need 12. What is the Ethernet hardware address of the second switch? (0060b0d00256) 13. What is the Multicast address for the switches to use? (226.10.1.1) 14. What is the IP address for the LAN card on the same subnet as the switches? (193.0.0.11) (In Figure 4-1, this is the IP address for lan0 on node B.
Configuring HyperFabric Performing the Configuration Performing the Configuration As explained in “Configuration Overview” on page 69, you must create the global configuration file (/etc/rc.config.d/clic_global_conf) on each node in the fabric. This consists mostly of specifying HyperFabric adapter-related information. (If you are also going to enable switch management — which HP does not recommend doing — you need to specify additional configuration information.
Configuring HyperFabric Performing the Configuration Using the clic_init Command Run the clic_init command to create the global configuration file. To view the man page for clic_init see “Viewing man Pages” on page 127 of this manual. To use clic_init to configure the Transparent Local Failover feature on HMP, see the section, “Configuring HMP for Transparent Local Failover Support - Using the clic_init command” on page 104.
Configuring HyperFabric Performing the Configuration If you do not specify any of the above parameters, the online help for clic_init is displayed. After you have entered the information for all the adapters in the node and all of the switches (if any) in the fabric, a summary of the configuration information is displayed.
Configuring HyperFabric Performing the Configuration Using SAM This section describes how to use SAM to configure HyperFabric. For information on how to use SAM to configure and deconfigure local failover feature on HMP, see “Configuring HMP for Transparent Local Failover Support - Using SAM” on page 102 and “Deconfiguring HMP for Local Failover support - Using SAM” on page 103.
Configuring HyperFabric Performing the Configuration • Subnet Mask—Optional. The adapter’s subnet mask. If you do not specify this, a default mask is chosen based on the adapter’s IP address. • Interoperability Enabled—Required. Whether you want the adapter to be able to interoperate with adapters that are using; any HP-UX 11.0 HyperFabric versions earlier than B.11.00.11 or any HP-UX 11i v1 HyperFabric versions earlier than B.11.11.01.
Configuring HyperFabric Deconfiguring a HyperFabric Adapter with SAM Deconfiguring a HyperFabric Adapter with SAM To use SAM to deconfigure a HyperFabric adapter on an HP 9000 system running HP-UX 11i v2, complete the following steps: Step 1. Start SAM. Step 2. Select the “Networking and Communications” area. Step 3. Select “HyperFabric.” All HyperFabric adapters installed in the system are listed.
Configuring HyperFabric Deconfiguring a HyperFabric Adapter with SAM is informed and the card pair entry is removed from the /etc/rc.config.d/netconf and /etc/rc.config.d/clic_global_conf files.
Configuring HyperFabric Configuring the HyperFabric EMS Monitor Configuring the HyperFabric EMS Monitor In the HyperFabric version B.11.23.01, the HyperFabric Event Monitoring Service (EMS) monitor allows system administrators to separately monitor each HyperFabric adapter on every node in the fabric, in addition to monitoring the entire HyperFabric subsystem. The monitor can inform the user if the resource being monitored is UP or DOWN.
Configuring HyperFabric Configuring the HyperFabric EMS Monitor 12. Validate by pressing OK NOTE Although EMS is able to monitor each HyperFabric adapter on every node in the fabric, as well as the entire HyperFabric subsystem, EMS is not able to monitor HyperFabric switches. For more detailed information on EMS, including instructions for implementing this feature, see the EMS Hardware Monitors Users Guide Part Number B6191-90028 September 2001 Edition.
Configuring HyperFabric Configuring HyperFabric with ServiceGuard Configuring HyperFabric with ServiceGuard HyperFabric supports the ServiceGuard HA product. NOTE If you plan to configure HyperFabric with ServiceGuard, please read this section. Otherwise, skip this section and go on to the next section, “Configuring HMP for Transparent Local Failover Support” on page 96. ServiceGuard lets you create HA clusters of HP 9000 server systems.
Configuring HyperFabric Configuring HyperFabric with ServiceGuard Figure 4-2 below shows a HyperFabric switch configuration with ServiceGuard. This example shows a four-node configuration with two HyperFabric switches, and redundant heartbeat Ethernet LANs. NOTE 88 Because the HyperFabric network does not currently support ServiceGuard heartbeat connections, you must use an alternative type of connection for the heartbeat, such as FDDI, Token Ring, 100BaseT, or Ethernet (as shown in Figure 4-2).
Configuring HyperFabric Configuring HyperFabric with ServiceGuard Figure 4-2 An ServiceGuard Configuration (with Two HyperFabric Switches) Ethernet Heartbeat LAN 1 Ethernet Heartbeat LAN 0 node B node A HF adapter 1 HF adapter 0 HF adapter 0 HF adapter 1 S S node C Chapter 4 HF switch 1 Ethernet Port node D HF adapter 0 HF adapter 1 HF adapter 1 HF adapter 0 S HF switch 0 S Ethernet Port 89
Configuring HyperFabric Configuring HyperFabric with ServiceGuard How HyperFabric Handles Adapter Failures HyperFabric adapters are handled differently than other types of networking adapters (such as Ethernet, FDDI, and Fibre Channel) in the ServiceGuard environment. In the non-HyperFabric cases, two network links are in a node, and one will be active and one will be idle or in standby.
Configuring HyperFabric Configuring HyperFabric with ServiceGuard Example 1: This example, illustrated by Figure 4-3 below, presents an HA configuration using ServiceGuard with HyperFabric. Both of the HyperFabric adapters are active on node A. The HyperFabric Resource Monitor reports the active status of the HyperFabric resource to the Event Monitoring Service (EMS), which lets ServiceGuard know that the HyperFabric resource is available to Packages A and B.
Configuring HyperFabric Configuring HyperFabric with ServiceGuard Example 2: This example, illustrated by Figure 4-4 below, shows the same node after the failure of one of the HyperFabric adapters. The remaining adapter in node A is now handling all HyperFabric network traffic for the node. Because the HyperFabric resource is still available, ServiceGuard has not been notified; HyperFabric handles the local HyperFabric adapter failover.
Configuring HyperFabric Configuring HyperFabric with ServiceGuard Example 3: This final example, illustrated by Figure 4-5 below, shows a situation in which all of the HyperFabric adapters on node A fail. The HyperFabric Resource Monitor reports to the Event Monitoring Service (EMS). The EMS then notifies the ServiceGuard cmcld daemon that the HyperFabric resource on node A is unavailable.
Configuring HyperFabric Configuring HyperFabric with ServiceGuard Configuring HyperFabric with the ServiceGuard Resource Monitor You can configure the HyperFabric Resource Monitor with ServiceGuard in either of these ways: • Editing an ASCII file. • Using the SAM GUI. For more details, please see the manual Using EMS HA Monitors. NOTE You should configure HyperFabric with ServiceGuard before running the clic_start command or using SAM to start HyperFabric.
Configuring HyperFabric Configuring HyperFabric with ServiceGuard Specify Specify Specify Specify Specify Specify Package Package Package Package Package Package Name and Nodes SUBNET Address Services Failover Options Control Script Location Control Script Information Specify Package Resources Dependencies Add Resource Name (Navigate the Resource Subclass by double-clicking on /net until /net/interfaces/clic/ status shows up in the selection box Resource Name,then select it and click OK.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support Configuring HMP for Transparent Local Failover Support HMP supports Local Failover in the HyperFabric version B.11.23.01. If a HyperFabric resource (adapter, cable, switch or switch port) fails in a cluster, HMP transparently fails over traffic (Local Failover) using another available resource from the card pair. A card pair can be defined as a logical entity comprising of a pair of HF2 adapters on a HP 9000 node.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support • HMP does not support backward compatibility in the local failover and non-local failover mode. However, TCP/UDP/IP supports backward compatibility and interoperability. • When HMP is configured in the local failover mode, all the resources in the cluster are utilized. If a resource fails in the cluster and is restored, HMP does not utilize that resource until another resource fails.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support How Transparent Local Failover Works Consider a hypothetical HyperFabric configuration in a 4-node cluster, with each node having two adapters (see Figure 4-6). In this configuration, there is no single point of failure, and all adapters that are installed on any given node are configured as part of a card pair.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support Case 1: Adapter, Link or Switch Port Failure (see Figure 4-7) If an adapter or a link or a switch port fails, HMP transparently fails over traffic through the other available link.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support Case 2: Switch Failure (see Figure 4-8) Consider the following illustration where node A is connected to node D with traffic being routed through the HF adapter 1 on both the nodes (A and D), and the HF switch 1 fails. HMP transparently fails over traffic through the other available switch (HF switch 0).
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support Thus, if a switch fails, HMP transparently fails over traffic only if at least one member of the card pair is physically reachable through the other switch. Case 3: Cable Failure Between Two Switches (see Figure 4-9) If a cable between two switches fails, HMP traffic fails over to the other available cable between those two switches.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support Configuring HMP for Transparent Local Failover Support - Using SAM To use SAM to configure HMP for Local Failover Support, complete the following steps: Step 1. Start SAM. Step 2. Select the “Networking and Communications” area. Step 3. Select “HyperFabric.” All HyperFabric adapters installed in the system are listed; installed adapters that are not yet configured show Not Configured in the “Status” field.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support of adapters clic0 and clic1 and if the Card-Pair 1, comprises of adapters clic2 and clic3, then the following entries are added to the clic_global_conf file. CARD_PAIR[0] = clic0-clic1 CARD_PAIR[1] = clic2-clic3 If you press Cancel, you remain in the main “HyperFabric Configuration” screen. If you press Help, help text for this task appears. Step 6. Exit SAM. NOTE To view the card pair information from the /etc/rc.config.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support Configuring HMP for Transparent Local Failover Support - Using the clic_init command You can configure the Transparent Local Failover feature of HMP using clic_init also. Let us consider the following example where we have discussed the configuration in detail. This example uses some “dummy” (that is, not valid) addresses to the components in Figure 4-10.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support Using the configuration information in Figure 4-10, the information you would specify when you run clic_init on each of the nodes is listed below. This example is not an exact depiction of the prompts produced by clic_init, but merely an example of the flow of information input. In addition, you should not try to use the dummy addresses in your actual configuration. On node A: 1.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support On node B: 1. How many HyperFabric adapters are installed on the node? 2. Do you want this node to interoperate with nodes running any HyperFabric versions earlier than B.11.00.11 or B.11.11.01? (n) You must answer ‘no’ if you want to run applications using HMP (Local Failover or Non-Local Failover) for communication over HyperFabric. In that case, all nodes in the cluster must be running version B.11.00.11 (or) B.11.11.
5 Managing HyperFabric This chapter contains the following sections that give information about managing HyperFabric: • Chapter 5 “Starting HyperFabric” on page 109.
Managing HyperFabric 108 • “Verifying Communications within the Fabric” on page 112. • “Displaying Status and Statistics” on page 117. • “Viewing man Pages” on page 127. • “Stopping HyperFabric” on page 128.
Managing HyperFabric Starting HyperFabric Starting HyperFabric HyperFabric is started in one of these three ways: • As part of the normal local node boot process (HP 9000 system). • By running the HyperFabric clic_start command (described below). • By starting HyperFabric through SAM (described in “Using SAM” on page 110).
Managing HyperFabric Starting HyperFabric Using the clic_start Command Run the clic_start command on each node to start the HyperFabric management process on that node. If you include /opt/clic/bin in your PATH statement, you can run the command as it is shown below. Otherwise, you must include /opt/clic/bin as part of the command name (that is, /opt/clic/bin/clic_start). You must be logged in as root to run this command.
Managing HyperFabric Starting HyperFabric When HyperFabric starts, a confirmation message displays. Also, the status “HyperFabric: Running” is displayed above the adapter configuration area of the screen. Step 5. Exit SAM.
Managing HyperFabric Verifying Communications within the Fabric Verifying Communications within the Fabric You can verify the communications within the fabric by running the clic_probe command, which is described below. You can also use clic_probe to verify the status of specific adapters.
Managing HyperFabric Verifying Communications within the Fabric The command parameters are as follows: • node_name specifies the node you want to verify. This value is conditionally required—you must specify it when you are verifying traffic to a remote node, unless you use the -r parameter (described below). • -c specifies that you want to use the adapter identified by adapter_ID for the verification. • -r specifies that VRID switch_hopcount is the routing information for the adapter.
Managing HyperFabric Verifying Communications within the Fabric NOTE Also see the clic_diag command to: Probe a specific remote node. Dump and format trace data. Set the tracing level for the HyperFabric software and firmware. The clic_diag command is detailed in the section, “Running Diagnostics” on page 133. Examples of clic_probe Some examples of using clic_probe are shown below.
Managing HyperFabric Verifying Communications within the Fabric CLIC_PROBE: 256 byte packets Source adapter id: bently6.corp3.com:clic0 Target adapter id: bently4.corp7.com:clic3 256 bytes: seq_num = 1. Packet Acknowledged. 256 bytes: seq_num = 2. Packet Acknowledged. 256 bytes: seq_num = 3. Packet Acknowledged. 256 bytes: seq_num = 4. Packet Acknowledged. 256 bytes: seq_num = 5. Packet Acknowledged. 256 bytes: seq_num = 6. Packet Acknowledged. 256 bytes: seq_num = 7. Packet Acknowledged.
Managing HyperFabric Verifying Communications within the Fabric The generated output could look like this: CLIC_PROBE: 256 byte packets sent Source adapter id: bently6.corp3.com:clic0 Target adapter id: bently7.corp4.com:clic1 256 bytes: seq_num = 1. Packet Acknowledged. 256 bytes: seq_num = 2. Packet Acknowledged. 256 bytes: seq_num = 3. Packet Acknowledged. 256 bytes: seq_num = 4. Packet Acknowledged. 256 bytes: seq_num = 5. Packet Acknowledged. --------- bently7.corp4.
Managing HyperFabric Displaying Status and Statistics Displaying Status and Statistics You can get the status of and statistics associated with many of the HyperFabric components by using the clic_stat command, which is described below. The clic_stat Command The following list contains some of the information that the clic_stat command provides: • The current fabric map, in textual format. • The status of one or more HyperFabric adapters.
Managing HyperFabric Displaying Status and Statistics Note that the second line in the above syntax is indented for readability purposes only. When the command is typed there should not be any indentation. The command parameters are as follows: • -p enables/disables performance statistics gathering according to the value of perf_level, which is one of the following: TCP Enables DLPI driver statistics when under the TCP/IP stack. HMP Enables Hyper Messaging Protocol (HMP) statistics gathering.
Managing HyperFabric Displaying Status and Statistics • -c displays the statistics of the adapter identified by adapter_ID. • -n displays statistics for the node identified by nodename. • -s specifies that you want to get the status of the switch identified by switch_ID (assigned by clic_init or SAM). Note that this parameter is meaningful only if you enabled switch management (through the clic_init command). To determine the switch_ID, run the clic_stat command without specifying any parameters.
Managing HyperFabric Displaying Status and Statistics If the local node is bently7, and you want to display the management daemon (clic_mgmtd) configuration and statistics data, issue this command: clic_stat -d CFG The generated output could look like this: ============================================================================ Date: Sat Aug 5 16:08:12 2000 Node: bently7.corp2.
Managing HyperFabric Displaying Status and Statistics The generated output could look like this: ============================================================================ Date: Sat Aug 5 16:08:30 2000 Node: bently7.corp2.com ---------------------------------------------------------------------------Adapter ID: clic1 Instance Number: 1 Adapter Type: 4X HF2 PCI Firmware File: /opt/clic/firmware/clic_fw_hf232c Major Num: 238 Mgmt process driver handle: 5 Version: 1.
Managing HyperFabric Displaying Status and Statistics Firmware reset notification: 0 Data corruption notification: 0 Unsupported QOS message received: 0 Invalid HMP VC ID: 0 Invalid HMP endpoint ID: 0 Invalid HMP endpoint protection key: 0 HMP message order violation: 0 Packet drops: 0 Transmit side congestion events: 1 Receive side congestion events: 0 Other Misc Statistics Mapping message send failures: 0 ============================================================================ b.
Managing HyperFabric Displaying Status and Statistics Frame buffer overflow: 0 Receive on disabled endpoint: 0 Invalid endpoint ID: 0 Invalid endpoint protection key: 0 Interleaved gathered receive: 0 Interleaved multi-frame bulk messages: 0 NQ overflow: 0 Send NQ overflow: 0 Invalid slot key for NQ credit update: 0 DLPI QOS receive buffer shortage: 0 Link congestion events: 0 Max send packet retry exceeded: 0 Link or switch failure events: 0 Link or switch resume events: 1 Bad route detected: 0 Bad optiona
Managing HyperFabric Displaying Status and Statistics ============================================================================ Date: Sat Aug 5 16:08:12 2000 Node: bently6.corp4.
Managing HyperFabric Displaying Status and Statistics The generated output if the nodes are connected through a switch could look like this: ============================================================================ Date: Sat Aug 5 16:08:12 2000 Node: bently6.corp4.
Managing HyperFabric Displaying Status and Statistics ============================================================================ • Example 5 If the local node is bently7, and you want to disable all statistics gathering on bently7, issue this command: clic_stat -p RST The generated output could look like this: ============================================================================ Date: Sat Aug 5 16:08:35 2000 Node: bently7.corp6.
Managing HyperFabric Viewing man Pages Viewing man Pages If you want to be able to view the HyperFabric man pages, you must first add /opt/clic/share/man to your MANPATH environment variable.
Managing HyperFabric Stopping HyperFabric Stopping HyperFabric You can stop HyperFabric only on a local node. Stopping HyperFabric on a node makes all of the HyperFabric adapters in that node unreachable by all other nodes in the fabric. It stops the HyperFabric management process, which stops all interconnect operations on the node. To stop HyperFabric, you use (1) the clic_shutdown command (described below) or (2) SAM (described in “Using SAM” on page 129).
Managing HyperFabric Stopping HyperFabric Using SAM To use SAM to stop HyperFabric on a local HP 9000 system running HP-UX 11i v2, complete the following steps: Step 1. Start SAM. Step 2. Select the “Networking and Communications” area. Step 3. Select “HyperFabric.” Step 4. Pull down the “Actions” menu and select Stop HyperFabric. Note that if HyperFabric is not running on the system, Stop HyperFabric is grayed out and you cannot select it. When HyperFabric stops, a confirmation message displays.
Managing HyperFabric Stopping HyperFabric 130 Chapter 5
6 Troubleshooting HyperFabric This chapter contains these sections that describe troubleshooting HyperFabric: • Chapter 6 “Running Diagnostics” on page 133.
Troubleshooting HyperFabric 132 • “Using Support Tools Manager” on page 140. • “Useful Files” on page 141. • “LED Colors and Their Meanings” on page 143. • “Determining Whether an Adapter or a Cable is Faulty” on page 153. • “Determining Whether a Switch is Faulty” on page 154. • “Replacing a HyperFabric Adapter” on page 156. • “Replacing a HyperFabric Switch” on page 157.
Troubleshooting HyperFabric Running Diagnostics Running Diagnostics Before running HyperFabric diagnostics: 1. Confirm HyperFabric adapters are installed on each node. Use the following command to display a list of HyperFabric adapters installed each node: $ ioscan -funC clic 2. Check to see if HyperFabric software is installed. Use the following command to display the HyperFabric software version installed on each node: $ swlist | grep -i hyperfabric 3. Check the patch level.
Troubleshooting HyperFabric Running Diagnostics $ clic_start All of these commands reside in the /opt/clic/bin directory. 6. Check cabling to make sure all of the HyperFabric adapters are connected to the fabric. 7. Run the following command: $ /opt/clic/bin/clic_stat -dALL If a TCP/UDP/IP application is running: Check the firmware file field to make sure the same version of firmware is downloaded on all of the HyperFabric adapters in the cluster.
Troubleshooting HyperFabric Running Diagnostics Make sure 4X HyperFabric adapters are installed on the nodes. The adapter type field indicates the type of HyperFabric adapter that is installed. HMP will only run on 4X HF2 PCI (A6386A) adapters. Check the firmware file field to make sure the firmware file name for each HyperFabric adapter ends in 32c. All of the HyperFabric adapters in the fabric must have firmware files that end in 32c for HMP to run.
Troubleshooting HyperFabric Running Diagnostics Diagnostics can be run on many of the HyperFabric components by using the clic_diag command. If the HyperFabric subsystem is still not usable, contact your HP support representative with the diagnostics data generated using the clic_diag command described below. The clic_diag Command Use the clic_diag command to run the following diagnostics: • Probe a specific remote node. • Dump and format trace data.
Troubleshooting HyperFabric Running Diagnostics • -r specifies that you want to probe a specific remote node identified by remote_node_name. The probe is done on all operational routes to the remote node.
Troubleshooting HyperFabric Running Diagnostics • -B specifies that you want the size (in bytes) of the trace buffer to be trace_buffer_size. The buffer is dynamically allocated when tracing is enabled. Also, the buffer is circular, which means that when the end of the buffer is reached, the data wraps around to the beginning of the buffer (and overwrites any previous data). If you do not specify this parameter, a default buffer is created with a size of 64k bytes.
Troubleshooting HyperFabric Running Diagnostics Source adapter id: bently6.corp4.com:clic1 Target adapter id: bently8.corp2.com:clic3 256 bytes: seq_num = 0. Packet Acknowledged. 256 bytes: seq_num = 1. Packet Acknowledged. 256 bytes: seq_num = 2. Packet Acknowledged. 256 bytes: seq_num = 3. Packet Acknowledged. 256 bytes: seq_num = 4. Packet Acknowledged. --------- CLIC_PROBE Statistics -------5 packets transmitted, 5 packets received, 0% packet loss.
Troubleshooting HyperFabric Using Support Tools Manager Using Support Tools Manager Use Support Tools Manager (STM) with HyperFabric to gather information about HyperFabric components and to diagnose hardware problems. Two tools are available in STM for HyperFabric: • The Information Tool provides information about the HyperFabric adapter, without resetting the adapter. • The Diagnostics Tool can be used to run tests on the HyperFabric adapter; the tool reports any failures.
Troubleshooting HyperFabric Useful Files Useful Files When you are troubleshooting HyperFabric-related problems, you might find it useful to look at the contents of the following files: • /etc/rc.config.d/clic_global_conf This is the global configuration file. Check it to confirm that the configuration information is correct. • /var/adm/clic_log This is a global log file that contains a history of significant HyperFabric events.
Troubleshooting HyperFabric Useful Files /var/adm/clic_fw.dumpx This is the default file for a memory dump of an HyperFabric adapter, created when the clic_diag -d command is run without specifying an output file. • /etc/rc.config.d/netconf This file contains IP-related configuration information for all of the networking adapters installed in the HP 9000. IMPORTANT: clic_init and SAM modify this file, adding some HyperFabric-related lines that end with the characters #clic.
Troubleshooting HyperFabric LED Colors and Their Meanings LED Colors and Their Meanings Listed below are the possible colors (and the corresponding meaning) of the LEDs on the HyperFabric adapters and switches. Adapter LEDs Table 6-1 below shows the names of the LEDs on each HyperFabric adapter. Note that the LEDs on the A4920A adapter are labeled, but the labels might be hard to see when a cable is connected to the adapter.
Troubleshooting HyperFabric LED Colors and Their Meanings — The adapter is bad. — The cable is bad. — The switch port is bad (if the adapter is connected to a switch). Note that if a switch port is bad, and (for some reason) you cannot use a different port on the switch, you must replace the switch module in the HF2 switch (whichever is applicable). However, you first should try turning the switch’s power off and then back on.
Troubleshooting HyperFabric LED Colors and Their Meanings Table 6-2 HyperFabric Adapter LED Colors and Meanings LED “Connected/Traffic” (A4919A and A6092A adapters) Color None Meaning Adapter is not operational. “Link” (A4920A and A6386A adapters) “Link Connected/Traffic” (A4921A adapter) Chapter 6 Solid green Connection from adapter to switch port or remote adapter is operational. Flashing green Data is flowing between the adapter and the switch port or remote adapter.
Troubleshooting HyperFabric LED Colors and Their Meanings Table 6-2 HyperFabric Adapter LED Colors and Meanings (Continued) LED “Error” 146 Color Meaning Notes None Adapter is not in an error condition. Adapter should be operating normally. Solid yellow Adapter is in an error condition. You must replace the adapter.
Troubleshooting HyperFabric LED Colors and Their Meanings Chapter 6 147
Troubleshooting HyperFabric LED Colors and Their Meanings HF2 Switch LEDs The HF2 switch LED colors and meanings are explained below. ❏ For each “Status” LED on the switch: ✓ If the card/switch module is not operating, the LED is off. — For the integrated Ethernet management LAN adapter card (in the top slot) and the integrated 8-port fiber card (in the middle slot): the card can be safely removed by qualified HP personnel only.
Troubleshooting HyperFabric LED Colors and Their Meanings ❏ For each port on the cards/switch module in the switch: ✓ If the port is not operational, the LED is off. Some of the possible reasons for this happening are the following: — A cable is not attached correctly to the port or no cable is attached at all. — The port is connected to a non-operational adapter in an HP 9000. (See “Adapter LEDs” on page 143 or Table 6-2 on page 145 for some tips about a non-operational adapter.) — The cable is bad.
Troubleshooting HyperFabric LED Colors and Their Meanings Table 6-3 HF2 Switch LED Colors and Meanings LED “Status” “Power A” and “Power B” 150 Color Meaning None The card/switch module is not operating. Solid yellow A fault is occurring on the card/switch module. Solid green The card/switch module passed the self-test and is operating. None Power to switch is off. Solid green Power to switch is on.
Troubleshooting HyperFabric LED Colors and Their Meanings Table 6-3 HF2 Switch LED Colors and Meanings (Continued) LED “Ethernet Port Main” and “Ethernet Port Aux” Color None Meaning Ethernet port is disconnected. Notes This can happen if the Ethernet port is not correctly connected to the Ethernet network, or the integrated Ethernet management LAN adapter card is experiencing a fault.
Troubleshooting HyperFabric LED Colors and Their Meanings Table 6-3 HF2 Switch LED Colors and Meanings (Continued) LED “Port x” 152 Color None Meaning Port is not operational. Solid green Connection from port to adapter is operational. Flashing green Data is flowing between the port and the adapter. Notes • A cable is not attached correctly to the port or no cable is attached at all. • The port is connected to a non-operational adapter in an HP 9000.
Troubleshooting HyperFabric Determining Whether an Adapter or a Cable is Faulty Determining Whether an Adapter or a Cable is Faulty If you suspect that either an HyperFabric adapter or a cable attached to it is faulty, follow these steps to determine which component needs replacing: Step 1. Disconnect the cable from the HyperFabric adapter. Step 2.
Troubleshooting HyperFabric Determining Whether a Switch is Faulty Determining Whether a Switch is Faulty If you suspect that a HyperFabric switch is faulty, follow the steps below to determine if the switch needs replacing. HF2 Switch The steps for determining if an HF2 switch is faulty are as follows: Step 1. Check the cards/switch module in the switch—the integrated Ethernet management LAN adapter card, integrated 8-port fiber card, and switch module in the expansion slot: a.
Troubleshooting HyperFabric Determining Whether a Switch is Faulty Step 4. Check the ports on the cards/switch module: a. Disconnect the cable from the card/switch module port you suspect is faulty. b. Attach a loopback plug to the relevant port: • If the port is on the integrated 8-port fiber card or the A6388A HF2 8-port fiber switch module in the expansion slot, use a fiber loopback plug. (A fiber loopback plug [HP part number A6384-67004] is shipped with each HF2 switch). c.
Troubleshooting HyperFabric Replacing a HyperFabric Adapter Replacing a HyperFabric Adapter If OLAR is supported for the HP 9000 system and the HyperFabric adapter, and you want to do OLR, see “Online Replacement (OLR)” on page 49. The HP HyperFabric Release Notes contains information about which HP 9000 systems and HyperFabric adapters OLAR is supported for. If you cannot or do not want to do OLR to replace a HyperFabric adapter, follow these steps: Step 1.
Troubleshooting HyperFabric Replacing a HyperFabric Switch Replacing a HyperFabric Switch If you need to replace a faulty HyperFabric switch, follow these steps: Step 1. If you do not have an HA environment, stop HyperFabric on all nodes that are connected to the faulty switch. (See “Stopping HyperFabric” on page 128.) If you have an HA environment with two switches (where the backup switch will have taken over when the first switch failed), disconnect all cables attached to the faulty switch. Step 2.
Troubleshooting HyperFabric Replacing a HyperFabric Switch 158 Chapter 6
A Safety and Regulatory Information This appendix contains the following sections that contain safety information and regulatory statements for the HyperFabric hardware Appendix A 159
Safety and Regulatory Information components: • “Safety Symbols” on page 161. • “Regulatory Statements” on page 162. These components were tested for conformance to various national and international regulations and standards. The scope of this regulatory testing includes electrical and mechanical safety, electromagnetic emissions, immunity, acoustics, and hazardous materials. When required, approvals are obtained from third party test agencies. Approval marks appear on the product label.
Safety and Regulatory Information Safety Symbols Safety Symbols The safety-related symbols used in this manual are shown below. WARNING A WARNING denotes a hazard that can cause personal injury. CAUTION A CAUTION denotes a hazard that can damage equipment.
Safety and Regulatory Information Regulatory Statements Regulatory Statements This section contains the regulatory statements for the HyperFabric products.
Safety and Regulatory Information Regulatory Statements if not installed and used in accordance with the instruction manual, may cause harmful interference to radio communications. Operation of this equipment in a residential area is likely to cause harmful interference, in which case the user will be required to correct the interference at his own expense. Any changes or modifications not expressly approved by Hewlett-Packard could void the user’s authority to operate this equipment.
Safety and Regulatory Information Regulatory Statements Radio Frequency Interference (Japan Only) VCCI, Class A (Model A4891-62001 only) 164 Appendix A
Safety and Regulatory Information Regulatory Statements Declarations of Conformity Appendix A 165
Safety and Regulatory Information Regulatory Statements 166 Appendix A
Safety and Regulatory Information Regulatory Statements Appendix A 167
Safety and Regulatory Information Regulatory Statements 168 Appendix A
B Technical Specifications This appendix contains the following sections that contain the technical specifications for the HyperFabric hardware components: • Appendix B “Physical Attributes” on page 171.
Technical Specifications • 170 “Environmental” on page 173.
Technical Specifications Physical Attributes Physical Attributes The physical attributes of the HyperFabric adapters and switches are as follows: • HF2 adapter (A6386A PCI [4X]) — 4.2 inch (10.7 cm) height — 1 inch (2.5 cm) width — 7.1 inch (18 cm) length — 4.
Technical Specifications Physical Attributes • HF2 switch chassis (A6384A) — 3.47 inch (8.8 cm) height — 19 inch (48.2 cm) width (at flanges) 17.18 inch (43.6 cm) width (enclosure) — 17.5 inch (44.5 cm) length (depth from back of flanges to back plate) 18.5 inch (47 cm) length (depth including protrusion of the switch module extractors) — 200 ounces (5680 g) weight • HF2 switch modules — A6388A — 0.8 inch (2 cm) height — 15.75 inch (40 cm) width — 11.5 inch (29.
Technical Specifications Environmental Environmental These environmental specifications are the same for the HyperFabric adapters and switches: • Temperature — -40 to +70 degrees C, non-operating/storage — +5 to +40 degrees C, operating — +20 to +30 degrees C, recommended operating range • Relative Humidity — 15% to 90%, non-operating/storage — 15% to 80% @ 22 degrees C, operating — 15% to 80% @ 22 degrees C, recommended operating range • Altitude — 15,000 feet (4.
Technical Specifications Environmental 174 Appendix B
Index A adapters deconfiguring, 83, 103 faulty, testing for, 153 installing, 44 LED colors and meanings, 143, 145 OLA of, 48 OLR of, 49 regulatory specifications, 162 replacing, 156 C cables, testing for faulty, 153 CLIC, definition of, 8, 52 clic_diag command, 114, 136 clic_init command, 79 after OLA, 48 clic_probe command, 112 to remote node, 137 clic_shutdown command, 128 clic_start command, 110 after OLA, 48 clic_stat command, 117 CLuster InterConnect.
Index IP addresses, relocatable. See relocatable IP addresses.
