HyperFabric Administrator’s Guide HP-UX 11i v1 and HP-UX 11i v2 Edition 14 Manufacturing Part Number : B6257-90057 October 2006 United States © Copyright 2006 Hewlett-Packard Development Company L.P. All rights reserved.
Legal Notices The information in this document is subject to change without notice. Hewlett-Packard makes no warranty of any kind with regard to this manual, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. Hewlett-Packard shall not be held liable for errors contained herein or direct, indirect, special, incidental or consequential damages in connection with the furnishing, performance, or use of this material.
Contents About This Document 1. Overview of the HyperFabric Product About HyperFabric. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . HyperFabric Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . HyperFabric Adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Switches and Switch Modules . . . . . . . . . . . . . . . . . . . . . .
Contents Installing HyperFabric Adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Online Addition and Replacement—HP-UX 11i v1 Only . . . . . . . . . . . . . . . . . . . . . . Planning and Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Critical Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Card Compatibility . . . . . . . . . . . . . . . . . . . .
Contents command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 5. Managing HyperFabric Starting HyperFabric. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Using the clic_start Command. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Using SAM—HP-UX 11.0 and 11i v1 0nly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Contents FCC Statement (USA only) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . DOC Statement (Canada only) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Europe RFI Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Australia and New Zealand EMI Statement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Radio Frequency Interference (Japan Only) . . . . . . . . . .
Figures Figure 2-1. TCP/UDP/IP Point-To-Point Configurations . . . . . . . . . . . . . . . . . . . . . . .26 Figure 2-2. TCP/UDP/IP Basic Switched Configuration . . . . . . . . . . . . . . . . . . . . . . .27 Figure 2-3. TCP/UDP/IP High Availability Switched Configuration . . . . . . . . . . . . .28 Figure 2-4. TCP/UDP/IP Hybrid Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .29 Figure 2-5. TCP/UDP/IP Mixed HF1 and HF2 Configuration. . . . . . . . . . . . . . . . . . .30 Figure 2-6.
Figures viii
Tables Table 1. HP-UX 11i Releases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii Table 2. Publishing History Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii Table 3. Organization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii Table 2-1. HF1 Throughput and Latency with TCP/UDP/IP Applications. . . . . . . . .21 Table 2-2.
Tables x
About This Document This document describes how to install, configure, and troubleshoot the HyperFabric product on the HP-UX 11i v1 operating system. Before you install the HyperFabric software, ensure that the HP-UX 11i v1 operating system software and the appropriate files, scripts, subsets are installed. The document printing date and part number indicate the document’s current edition. The printing date will change when a new edition is printed.
HP-UX Release Name and Release Identifier Each HP-UX 11i release has an associated release name and release identifier. The uname (1) command with the -r option returns the release identifier. Table 1 shows the releases available for HP-UX 11i. Table 1 HP-UX 11i Releases Release Identifier Supported Processor Architecture Release Name B.11.11 HP-UX 11i v1 PA-RISC B.11.20 HP-UX 11i v1.5 Intel Itanium B.11.22 HP-UX 11i v1.6 Intel Itanium B.11.
Table 2 Publishing History Details (Continued) Edition Number Publication Date Eleventh June 2001 Twelfth September 2002 Thirteenth July 2003 What’s in This Document HyperFabric Administrator’s Guide is divided into several chapters, each of which contains information about installing, configuring, or troubleshooting HyperFabric. The appendixes contain supplemental information. The following list describes the content in more detail.
New and Changed Information in This Edition This edition includes information about the transparent local failover feature of Hyper Messaging Protocol (HMP). This feature is available with the HyperFabric version B.11.11.03. Typographic Conventions This document uses the following typographic conventions: Book Title Italic (slanted) type indicates document and book names. daemon Courier font type indicates daemons, files, commands, manpages, and option names.
1 Overview of the HyperFabric Product This chapter contains the following sections that give general information about HyperFabric: Chapter 1 1
Overview of the HyperFabric Product 2 • “About HyperFabric” on page 3 • “HyperFabric Products” on page 4 • “HyperFabric Concepts” on page 9 Chapter 1
Overview of the HyperFabric Product About HyperFabric About HyperFabric HyperFabric is an HP high-speed, packet-based interconnect for node-to-node communications. HyperFabric provides higher speed, lower network latency and uses less CPU than other industry standard protocols (for example, Fibre Channel and Gigabit Ethernet). Instead of using a traditional bus-based technology, HyperFabric is built around switched fabric architecture, providing the bandwidth necessary for high speed data transfer.
Overview of the HyperFabric Product HyperFabric Products HyperFabric Products HyperFabric hardware consists of host-based interface adapter cards, interconnect cables, and optional switches. HyperFabric software resides in Application Specific Integrated Circuits (ASICs) and firmware on the adapter cards and includes user-space components and HP-UX drivers. Currently, both copper and fiber-based HyperFabric hardware are available.
Overview of the HyperFabric Product HyperFabric Products HyperFabric Adapters The HyperFabric adapters are as follows: • A4919A HF1 PCI (1X) adapter with a copper interface. (Discontinued...04-02) • A4920A HF1 HSC adapter with a copper interface. (Discontinued...09-02) • A4921A HF1 EISA/HSC adapter with a copper interface. (Discontinued...09-02) • A6092A HF1 PCI (4X) adapter with a copper interface. (Discontinued...11-02) • A6386A HF2 PCI (4X) adapter with a fiber interface.
Overview of the HyperFabric Product HyperFabric Products • A6384A HF2 fiber switch chassis with one integrated Ethernet management LAN adapter card, one integrated 8-port fiber card, and one expansion slot. For the chassis to be a functional switch, install one of the following switch modules in the expansion slot: — The A6388A HF2 8-port fiber switch module. This gives 16-fiber ports to the switch (8 from the integrated fiber card and 8 from the A6388A switch module).
Overview of the HyperFabric Product HyperFabric Products IMPORTANT HF1 and HF2 adapters and switches are not supported by software versions earlier than those listed in “HyperFabric Adapters” on page 5 and “Switches and Switch Modules” on page 5.
Overview of the HyperFabric Product HyperFabric Products fabric, then the node cannot communicate with any other nodes that are running the specified versions of the HyperFabric software. For more information on using HMP applications in a HyperFabric cluster, see Chapter 2, “Planning the Fabric,” on page 11.
Overview of the HyperFabric Product HyperFabric Concepts HyperFabric Concepts This section briefly describes some of the basic HyperFabric concepts and terms. The fabric is the physical configuration that consists of all the HyperFabric adapters, the HyperFabric switches (if any), and the HyperFabric cables connecting them. The network software controls data transfer over the fabric. The HyperFabric configuration contains two or more HP 9000 systems and optional HyperFabric switches.
Overview of the HyperFabric Product HyperFabric Concepts HyperFabric to run. If the HyperFabric management process on a node stops running for some reason (for example, if it is killed), all HyperFabric-related communications on that node are stopped immediately. This makes the node unreachable by other components in the fabric. When you start HyperFabric, the fabric is verified automatically. This is because each node performs a self diagnosis and verification over each adapter installed in the node.
2 Planning the Fabric This chapter contains the following sections that include general guidelines and protocol-specific considerations for planning HyperFabric clusters that run TCP/UDP/IP or HMP applications.
Planning the Fabric 12 • “Preliminary Considerations” on page 13 • “HyperFabric Features, Parameters and Supported Configurations for TCP/UDP/IP and HMP Applications” on page 15 • “TCP/UDP/IP” on page 16 • “Hyper Messaging Protocol (HMP)” on page 31 Chapter 2
Planning the Fabric Preliminary Considerations Preliminary Considerations Before assembling a fabric physically, do the following to address all of the appropriate issues: Step 1. Read Chapter 1, “Overview of the HyperFabric Product,” on page 1, to get a basic understanding of HyperFabric and its components. Step 2. Read this chapter, Planning the Fabric, to gain an understanding of protocol specific configuration guidelines for TCP/UDP/IP and HMP applications. Step 3.
Planning the Fabric Preliminary Considerations Step 11. Draw the cable connections from each node to switches (if the fabric will contain switches). If you use an HA configuration with switches, it requires more than one switch for complete redundancy and to avoid a single point of failure. For example, each adapter can be connected to its own switch, or two switches can be connected to four adapters.
Planning the Fabric HyperFabric Features, Parameters and Supported Configurations for TCP/UDP/IP and HMP Applications HyperFabric Features, Parameters and Supported Configurations for TCP/UDP/IP and HMP Applications The following sections in this chapter define HyperFabric functionality for TCP/UDP/IP applications and Hyper Messaging Protocol (HMP) applications.
Planning the Fabric TCP/UDP/IP TCP/UDP/IP TCP/UDP/IP is supported on all HF1 and HF2 hardware. Although some of the HyperFabric adapter cards support both HMP and TCP/UDP/IP applications, in this section, the focus is on TCP/UDP/IP HyperFabric applications. Application Availability All applications, including Oracle 9i and HP-MPI, that use the TCP/UDP/IP stack are supported.
Planning the Fabric TCP/UDP/IP — Logging into a user specified log file with a choice of severity — Email to a user defined email address. For more information on EMS, including instructions for implementing this feature, see “Configuring the HyperFabric EMS Monitor” on page 101 and the EMS Hardware Monitors User’s Guide Part Number B6191-90028 September 2001 Edition. • ServiceGuard: Supported Within a cluster, ServiceGuard groups application services (individual HP-UX processes) into packages.
Planning the Fabric TCP/UDP/IP ServiceGuard uses a “heartbeat” to monitor the cluster. The HyperFabric links cannot be used for the heartbeat. Instead, an alternate LAN connection such as 100BaseT, Ethernet, Token Ring, or FDDI must be made between the nodes for use as a heartbeat link. End-To-End HA: HyperFabric provides end-to-end HA on the entire cluster fabric at the link level.
Planning the Fabric TCP/UDP/IP Switch Management is not supported. Switch management will not operate properly if you enable it on a HyperFabric cluster. • Diagnostics: Supported Diagnostics can be run to obtain information on many of the HyperFabric components using the clic_diag, clic_probe and clic_stat commands, as well as the Support Tools Manager (STM). For more information on HyperFabric diagnostics, see “Running Diagnostics” on page 149.
Planning the Fabric TCP/UDP/IP You can interconnect (mesh) up to 4 switches (16-port copper, 16-port fiber, or Mixed 8 fiber ports / 4 copper ports) in a single HyperFabric cluster. • Trunking Between Switches (multiple connections) You can use trunking between switches to increase bandwidth and cluster throughput. Trunking is also a way to eliminate a possible single point of failure. The number of trunked cables between nodes is limited only by port availability.
Planning the Fabric TCP/UDP/IP • Table 2-1 Throughput and Latency HF1 Throughput and Latency with TCP/UDP/IP Applications Server Class rp7400 Table 2-2 1.28 + 1.
Planning the Fabric TCP/UDP/IP Table 2-3 HF1 and HF2 Hardware that Supports TCP/UDP/IP Applications Adapter Part Number A4919A (HF1) HP 9000 Server / Work Station HP-UX Release OLAR N4000 (*) 11.0, 11i v1 (64-bit only) YES (11i v1) 8 V-class (@) 11.0, 11i v1 (64-bit only) NO 7 (Max. 1 per Epic [total 2 per IO cage]) HSC K-class 10.20 (@@), 11.0 NO 2 EISA /HSC D,R-class 10.20 (@@), 11.0 NO 2 PCI 4X A400,A500, B1000, B2000, B2600, C3x, J5x, J6x, J7x 11.
Planning the Fabric TCP/UDP/IP Table 2-3 HF1 and HF2 Hardware that Supports TCP/UDP/IP Applications Adapter Part Number A6386A (HF2) Bus Type PCI 4X HP 9000 Server / Work Station HP-UX Release OLAR Maximum Adapters per Instance of HP-UX OS Superdome (**) 11i v1 YES 8 (Max. 4 per PCI card cage) V-class (@) 11.0, 11i v1 NO 7 (Max. 1 per Epic [total 2 per IO cage]) A400,A500, B1000, B2000, B2600, C3x, J5x, J6x, J7x 11.0, 11i v1 NO 2 rp54xx (L-class) 11.
Planning the Fabric TCP/UDP/IP Menu” to check the system’s firmware version. If the firmware version is earlier than 3.0, the system has a PA-8500 processor (and so the A4919A adapter is supported on it). If the firmware version is 3.0 or later, the A4919A adapter is not supported.
Planning the Fabric TCP/UDP/IP TCP/UDP/IP Supported Configurations Multiple TCP/UDP/IP HyperFabric configurations are supported to match the cost, scaling, and performance requirements of each installation. In the previous section, “Configuration Parameters” on page 19, the maximum limits for TCP/UDP/IP enabled HyperFabric hardware configurations were outlined. In this section the TCP/UDP/IP enabled HyperFabric configurations that HP supports are explained.
Planning the Fabric TCP/UDP/IP Figure 2-1 26 TCP/UDP/IP Point-To-Point Configurations Chapter 2
Planning the Fabric TCP/UDP/IP Switched Configuration This configuration offers the same benefits as the point-to-point configurations illustrated in Figure 2-1, but it has the added advantage of greater connectivity (see Figure 2-2).
Planning the Fabric TCP/UDP/IP High Availability Switched Configuration This configuration has no single point of failure. The HyperFabric driver provides end-to-end HA. If any HyperFabric resource in the cluster fails, traffic is transparently rerouted through other available resources. This configuration provides high performance and high availability (see Figure 2-3).
Planning the Fabric TCP/UDP/IP Hybrid Configuration You can interconnect servers and workstations in a single heterogeneous HyperFabric cluster. In this configuration, the servers are highly available. In addition, the workstations and the servers can run the same application or different applications (see Figure 2-4).
Planning the Fabric TCP/UDP/IP Mixed HF1/HF2 (Copper & Fiber) Configuration You can interconnect all the currently available HyperFabric products in a single HyperFabric cluster. The HF1 and HF2 products are interoperable, enabling user-controlled migration from copper-based to fiber-based technologies (see Figure 2-5).
Planning the Fabric Hyper Messaging Protocol (HMP) Hyper Messaging Protocol (HMP) Hyper Messaging Protocol (HMP) is an HP patented, high performance cluster interconnect protocol. HMP provides reliable, high speed, low latency, low CPU overhead, datagram service to applications running on the HP-UX operating system. HMP was jointly developed with Oracle Corp. The resulting feature set was tuned to enhance the scalability of the Oracle Cache Fusion clustering technology.
Planning the Fabric Hyper Messaging Protocol (HMP) HP MPI is a native implementation of version 1.2 of the Message-Passing Interface Standard. It has become the industry standard for distributed technical applications and is supported on most technical computing platforms. HMP is certified with HP-MPI 1.7 on both HP-UX 11.0 and HP-UX 11i v1. HMP is certified with HP-MPI 1.7.2 which uses HMP as the default interconnect.
Planning the Fabric Hyper Messaging Protocol (HMP) Within a cluster, ServiceGuard groups application services (individual HP-UX processes) into packages. In the event of a single service failure (node or network), EMS provides notification and ServiceGuard transfers control of the package to another node in the cluster, allowing services to remain available with minimal interruption.
Planning the Fabric Hyper Messaging Protocol (HMP) For more information on Transparent Local Failover while running HMP applications, see “Configuring HMP for Transparent Local Failover Support” on page 112. • Dynamic Resource Utilization (DRU): Partially Supported If you add a new HyperFabric resource (node, cable or switch) to a cluster running an HMP application, the HyperFabric subsystem will dynamically identify the added resource and start using it.
Planning the Fabric Hyper Messaging Protocol (HMP) • HMP is supported on the HF2 adapter, A6386A, only. • Although HMP is supported on A6092A HF1 (copper) adapters, the performance advantages that HMP offers are not completely realized unless HMP is used with A6386A HF2 (fiber) adapters and related fiber hardware. See Table 2-6 on page 37 for details. The local failover configuration of HMP is supported only on the A6386A HF2 adapters.
Planning the Fabric Hyper Messaging Protocol (HMP) • Maximum Cable Lengths HF1 (copper): The maximum distance between two nodes or between a node and a switch is 60 ft. (2 standard cable lengths are sold and supported: 35 ft. and 60 ft.) HMP supports up to 4 HF1 switches connected in series with a maximum cable length of 60 ft. between the switches and 60 ft. between switches and nodes. HF2 (fiber): The maximum distance is 200m (4 standard cable lengths are sold and supported: 2m, 16m, 50m and 200m).
Planning the Fabric Hyper Messaging Protocol (HMP) Table 2-5 HF2 Throughput and Latency with HMP Applications Server Class Maximum Throughput rp 7400 Table 2-6 2 + 2 Gbps full duplex per link < 22 microsec HF1 and HF2 Hardware that Supports HMP Applications Adapter Part Number A6092A (HF1) Bus Type PCI 4X Discontinued 11, 2002 A6386A (HF2) Chapter 2 Latency PCI 4X HP 9000 Server/ Work Station HP-UX Release OLAR Maximum Adapters per Instance of HP-UX rp2400 (A-class), rp2450 (A-class), B1
Planning the Fabric Hyper Messaging Protocol (HMP) Table 2-6 HF1 and HF2 Hardware that Supports HMP Applications Adapter Part Number Bus Type HP 9000 Server/ Work Station HP-UX Release OLAR Maximum Adapters per Instance of HP-UX rp74xx (N-class) 11.0, 11i v1 NO 8 rp8400 11i v1 NO 8 Superdome (*) 11i v1 NO 8 (Max.
Planning the Fabric Hyper Messaging Protocol (HMP) HMP Supported Configurations Multiple HMP HyperFabric configurations are supported to match the performance, cost and scaling requirements of each installation. In the section, “Configuration Parameters” on page 34, the maximum limits for HMP enabled HyperFabric hardware configurations were outlined. This section discusses the HMP enabled HyperFabric configurations that HP supports.
Planning the Fabric Hyper Messaging Protocol (HMP) Figure 2-6 40 HMP Point-To-Point Configurations Chapter 2
Planning the Fabric Hyper Messaging Protocol (HMP) Enterprise (Database) Configuration The HMP enterprise configuration illustrated in Figure 2-7 is very popular for running Oracle RAC 9i. Superdomes or other large servers make up the Database Tier. Database Tier nodes communicate with each other using HMP. Application Tier nodes communicate with each other and to the Database Tier using TCP/UDP/IP.
Planning the Fabric Hyper Messaging Protocol (HMP) Enterprise (Database) - Local Failover Supported Configuration The HMP enterprise configuration is a scalable solution. For high availability and performance, you can easily scale the HMP enterprise configuration with multiple connections between the HyperFabric resources. Any single point of failure in the database tier of the fabric is eliminated in Figure 2-8.
Planning the Fabric Hyper Messaging Protocol (HMP) Technical Computing (Work Stations) Configuration This configuration is typically used to run technical computing applications with HP-MPI. A large number of small nodes are interconnected to achieve high throughput (see Figure 2-9). High availability is not usually a requirement in technical computing environments. HMP provides the high performance, low latency path necessary for these technical computing applications.
Planning the Fabric Hyper Messaging Protocol (HMP) Figure 2-9 44 Technical Computing Configuration Chapter 2
Planning the Fabric Hyper Messaging Protocol (HMP) Figure 2-10 Chapter 2 Large Technical Computing Configuration 45
Planning the Fabric Hyper Messaging Protocol (HMP) 46 Chapter 2
3 Installing HyperFabric This chapter contains the following sections that describe the HyperFabric installation: • Chapter 3 “Checking HyperFabric Installation Prerequisites” on page 49.
Installing HyperFabric 48 • “Installing HyperFabric Adapters” on page 51. • “Installing the Software” on page 58. • “Installing HyperFabric Switches” on page 64.
Installing HyperFabric Checking HyperFabric Installation Prerequisites Checking HyperFabric Installation Prerequisites Before installing HyperFabric, ensure that the following hardware and software prerequisites are met: ✓ Check the HyperFabric Release Notes for known problems, required patches, or other information needed for installation. ✓ Confirm that the /usr/bin, /usr/sbin, and /sbin directories are in your PATH by logging in as root and using the echo $PATH command.
Installing HyperFabric Checking HyperFabric Installation Prerequisites ✓ Create a map of the fabric (optional). ✓ Confirm that HP-UX super-user privileges are available; they are necessary to complete the HyperFabric installation. The first HyperFabric installation step is installing HyperFabric adapter cards in the nodes.
Installing HyperFabric Installing HyperFabric Adapters Installing HyperFabric Adapters This section contains information about installing HyperFabric adapters in HP 9000 systems. Online Addition and Replacement (OLAR) information is provided in the section, “Online Addition and Replacement—HP-UX 11i v1 Only” on page 52. CAUTION IMPORTANT HyperFabric adapters contain electronic components that can easily be damaged by small amount of electricity.
Installing HyperFabric Installing HyperFabric Adapters Online Addition and Replacement—HP-UX 11i v1 Only Online Addition and Replacement (OLAR) allows PCI I/O cards, adapters or controllers to be replaced or added to HP 9000 systems, without the need for completely shutting down and rebooting the system, or adversely affecting other system components. This feature is only available on HP 9000 systems that are designed to support OLAR.
Installing HyperFabric Installing HyperFabric Adapters Table 3-1 below explains some important OLAR-related terms. Table 3-1 Important OLAR Terms Term IMPORTANT Meaning OLAR All aspects of the OLAR feature including Online Addition (OLA) and Online Replacement (OLR). Power Domain A grouping of 1 or more interface card slots that are powered on or off as a unit. (Note: Multi-slot power domains are not currently supported.
Installing HyperFabric Installing HyperFabric Adapters Planning and Preparation As mentioned previously, for the most part, SAM prevents the user from performing OLAR procedures that would adversely affect other areas of the HP 9000 system. See Configuring HP-UX For Peripherals for detailed information. Critical Resources The effects of shutting down a card’s functions must be considered. Replacing a card that is still operating can have extensive consequences.
Installing HyperFabric Installing HyperFabric Adapters Online Addition (OLA) Multiple cards can be added at the same time. When adding a card online, the first issue to resolve is whether the new card is compatible with the system. Each OLAR-capable PCI slot provides a set amount of power. The replacement card cannot require more power than there is available. The card must also operate at the slot’s bus frequency.
Installing HyperFabric Installing HyperFabric Adapters Online Replacement (OLR) When replacing an interface card online, the replacement card must be identical to the card being replaced (or at least be able to operate using the same driver as the replaced card). This is referred to as like-for-like replacement and should be adhered to, because using a similar but not identical card can cause unpredictable results.
Installing HyperFabric Installing HyperFabric Adapters WARNING: You have 1 operational HyperFabric card. If you go ahead with this operation you will lose network access via HyperFabric until the on-line replaced HyperFabric card becomes operational. You are asked if you want to continue. If you reply Yes, client applications are suspended. Replace the adapter according to the procedure described in the “Managing PCI Cards with OLAR” chapter of the Configuring HP-UX Peripherals manual.
Installing HyperFabric Installing the Software Installing the Software This section describes the HyperFabric file structure and the steps necessary to load the software. The software must be installed on each instance of the HP-UX operating system in the fabric. File Structure The HyperFabric file structure is shown in Figure 3-1 below. The structure is shown for informational purposes only. The user cannot modify any of the files or move them to a different directory.
Installing HyperFabric Installing the Software The commands and files used to administer HyperFabric typically have a prefix of clic_. CLIC stands for CLuster InterConnect, and it is used to differentiate those HyperFabric commands/files from other commands/files. For example, the HyperFabric command clic_init is different from the HP-UX init command.
Installing HyperFabric Installing the Software • /var/adm/OLDclic_log The log file from the previous time the clic_start command was executed. • /usr/conf/lib/libclic_dlpi_drv.a The kernel library that contains the HyperFabric software. • /usr/conf/lib/libha_drv.a The kernel library that contains the High Availability (HA) software. • /usr/conf/master.d/clic This file is described along with the other master files in the master man page (type man master at the HP-UX prompt).
Installing HyperFabric Installing the Software • /opt/clic/firmware/clic_fw_4x8c The 4X PCI HyperFabric 8-bit CRC firmware. This file must not be modified. • /opt/clic/firmware/clic_fw_4x32c The 4X HyperFabric PCI 32-bit CRC firmware. This file must not be modified. • /opt/clic/firmware/clic_fw_hf28c The HyperFabric2 8-bit firmware. This file must not be modified. • /opt/clic/firmware/clic_fw_hf232c The HyperFabric2 32-bit firmware. This file must not be modified.
Installing HyperFabric Installing the Software Loading the Software Listed below are the steps you must follow to load the HyperFabric software, using the HP-UX swinstall program. Step 1. Log on to the system as root. Step 2. Insert the software media into the appropriate drive. If the software is being loaded from a CD-ROM, go to step 3; otherwise, go to step 4. Step 3.
Installing HyperFabric Installing the Software swinstall loads the fileset, runs the control script for the filesets, and builds the kernel. When the processing is finished, the “Status” field displays a “Ready” message. Select “Done” and then the “Note” window opens. Step 11. Select the OK button in the “Note” window to reboot. The user interface disappears and the system reboots. Step 12. When the system comes back up, log on to the system as root and view the /var/adm/sw/swagent.
Installing HyperFabric Installing HyperFabric Switches Installing HyperFabric Switches This section contains the information you need to install HyperFabric switches. As stated earlier, the term HyperFabric2 (HF2) switch refers to the functional switch (the A6384A switch chassis with one of the switch modules installed).
Installing HyperFabric Installing HyperFabric Switches To prevent overheating, you must leave one rack unit (1 EIA) of empty space above the HyperFabric switch. WARNING ❏ After the HyperFabric switch is mounted in the rack, attach the various cables to the switch. To avoid damage to any of the cables, follow these guidelines: — If your cables have dust caps over the connectors, keep them in place until you are ready to connect them. This prevents dirt and oils from soiling any important surfaces.
Installing HyperFabric Installing HyperFabric Switches Steps for Installing the HF1 Switch This section contains information for installing an HF1 switch. As mentioned earlier, the HF1 switch must be mounted using the rail kit shipped with the switch. Figure 3-2 below shows the locations of the ports, LEDs, and power cord inlet on the back of the HF1 switch.
Installing HyperFabric Installing HyperFabric Switches Step 2. Install and secure the rails in the rack, using two screws per rail. The figure below shows the rack with the rails installed. screws rack front screws rack back Step 3. From the front of the rack, install a bracket on the outside of each rail, using two screws per bracket. Be sure to use the upper screw holes on each bracket. Put the screws in the second and third square holes—counting away from yourself—in each rail.
Installing HyperFabric Installing HyperFabric Switches bracket 2 rack front bracket 1 rack back 68 Chapter 3
Installing HyperFabric Installing HyperFabric Switches Step 4. From the back of the rack, slide the switch—with the front of the switch facing the front of the rack—into the rack, on the rails. Move it until it is touching brackets 1 and 2. Note that you might not have enough clearance between the switch and the rail screws, so that you cannot easily slide the switch. If so, try lifting the switch over the rail screws.
Installing HyperFabric Installing HyperFabric Switches rack front fro nt o fs wi t ch bracket 4 rack back bracket 3 70 Chapter 3
Installing HyperFabric Installing HyperFabric Switches Step 6. Tighten all four screws in brackets 3 and 4. Step 7. From the front of the rack, push the switch so that it barely touches brackets 3 and 4. The figure below shows how to move the switch to this position.
Installing HyperFabric Installing HyperFabric Switches Step 8. Once the switch is snug against brackets 3 and 4, push brackets 1 and 2 in towards the switch, so that they are snug against the switch. The figure below shows brackets 1 and 2 in this position. bracket 2 bracket 4 fro nt of sw it c h rack front bracket 1 rack back bracket 3 Step 9. Tighten the four screws in brackets 1 and 2. Step 10.
Installing HyperFabric Installing HyperFabric Switches Step 12. Plug the switch’s power cord into the rack’s power distribution unit (PDU), if it has one. Alternatively, you can plug a power cord that is compatible with your country’s requirements into a power strip or outlet that you want to use for the switch. (In this case, you are responsible for obtaining a compatible power cord.) Step 13. Power on the HF1 switch by plugging the power cord into the AC inlet on the back of the switch.
Installing HyperFabric Installing HyperFabric Switches Installing the HF2 Switch This section contains information on installing an HF2 switch. The front of the HF2 switch has a flange—or “wing”—on each side, with two holes for attaching the switch to the rack. The following figures do not show the flanges. Figure 3-3 shows the front of the HF2 switch with an A6388A HF2 8-port fiber switch module installed in the switch’s expansion slot.
Installing HyperFabric Installing HyperFabric Switches Figure 3-4 shows the front of the HF2 switch with an A6389A HF2 4-port copper switch module installed in the switch’s expansion slot.
Installing HyperFabric Installing HyperFabric Switches With the Rail Kit As mentioned earlier, HP strongly recommends installing the HF2 switch using the rail kit. When you install the HF2 switch, you will be putting the front of the switch—the end with the flanges (“wings”)—at the back of the rack. The steps for installing the HF2 switch using the rail kit are as follows: Step 1. Prepare the rack for rail and switch installation. Step 2. Install and secure the rails in the rack, using two screws per rail.
Installing HyperFabric Installing HyperFabric Switches sliding the switch. If so, try lifting the switch over the rail screws. If you cannot do this, remove the rail screws, slide the switch into position, and put the rail screws back in. Step 4. Align the two holes in each flange (“wing”) on the switch’s front with the holes in the rack frame. Fasten each flange of the switch to the rack by putting a screw in each of the four holes in the flanges. Be sure to use screws with over-sized heads.
Installing HyperFabric Installing HyperFabric Switches the flat part). Otherwise, the brackets will not secure the back of the switch properly. The following figure shows the rack with these two brackets installed. rack back fr (w ont ith of fla sw ng itc es h ) bracket 2 rack front bracket 1 Step 6. When you are sure the brackets are snug against the beveled part of the switch, tighten all four screws in brackets 1 and 2. Step 7.
Installing HyperFabric Installing HyperFabric Switches Step 9. Plug the switch’s power cord into the rack’s PDU, if it has one. Alternatively, you can plug a power cord that is compatible with your country’s requirements into a power strip or outlet that you want to use for the switch. (In this case, you are responsible for obtaining a compatible power cord.) Step 10. Power on the HF2 switch by plugging the power cord into the AC inlet on the back of the switch. (There is no power switch.) Step 11.
Installing HyperFabric Installing HyperFabric Switches Without the Rail Kit As mentioned earlier, HP strongly recommends installing the HF2 switch using the rail kit (described in the previous section, “With the Rail Kit” on page 76). When you install the HF2 switch, you will be putting the front of the switch—the end with the flanges (“wings”) — at the back of the rack. The steps for installing the HF2 switch without using the rail kit are as follows: Step 1. Prepare the rack for switch installation.
Installing HyperFabric Installing HyperFabric Switches Step 10. Once the power is on, check these LEDs on the integrated Ethernet management LAN adapter card (in the top slot of the switch): ✓ The “Operating/Fault” LED shows solid green. ✓ The “Power A” and “Power B” LEDs show solid green. ✓ The “Ethernet Port Main” and “Ethernet Port Aux” LEDs are showing solid green (connected) or flashing green (Ethernet traffic is flowing to the switch).
Installing HyperFabric Installing HyperFabric Switches 82 Chapter 3
4 Configuring HyperFabric This chapter contains the following sections that describe configuring HyperFabric: • Chapter 4 “Configuration Overview” on page 85.
Configuring HyperFabric 84 • “Information You Need” on page 87. • “Performing the Configuration” on page 94. • “Deconfiguring a HyperFabric Adapter with SAM—HP-UX 11.0 and 11i v1 Only” on page 99. • “Configuring the HyperFabric EMS Monitor” on page 101. • “Configuring HyperFabric with ServiceGuard” on page 103. • “Configuring HMP for Transparent Local Failover Support” on page 112.
Configuring HyperFabric Configuration Overview Configuration Overview You do not need to configure the HyperFabric switch because the HyperFabric management process performs automatic routing and configuring for the switch. So, configuring HyperFabric consists only of creating the HyperFabric /etc/rc.config.d/clic_global_conf global configuration file on each node in the fabric.
Configuring HyperFabric Configuration Overview In this file, clic_init and SAM add some HyperFabric-related lines that end with the characters #clic. These lines are used by the HyperFabric software — and are not comments — so do not remove them from the file. IMPORTANT • The system /etc/rc.config.d/clic_global_conf file. • The /etc/rarpd.conf (Reverse Address Resolution Protocol [RARP]) support file.
Configuring HyperFabric Information You Need Information You Need When you run the clic_init command or use SAM for configuration, you have to provide certain configuration information. So, before you run clic_init or use SAM, you should have the following information: ❏ For each node in the fabric, determine if that node will need to interoperate with other nodes that are using; any HP-UX 11.0 HyperFabric versions earlier than B.11.00.11 or any HP-UX 11i v1 HyperFabric versions earlier than B.11.11.01.
Configuring HyperFabric Information You Need ✓ The IP address of the switch. ✓ The MAC address of the switch’s Ethernet port. If you do not already know the switch’s MAC address, it is printed on a label on the back of the HF switch and on the front of the HF2 switch. See Figure 3-2 on page 66 for the location of the label on the HF switch, and Figure 3-3 on page 74 and Figure 3-4 on page 75 for the location of the label on the HF2 switch.
Configuring HyperFabric Information You Need IP_address IP_address IP_address Chapter 4 bently6.corp3.com bently4.corp7.com bently2.corp4.
Configuring HyperFabric Information You Need Configuration Information Example This example uses some “dummy” (that is, not valid) addresses to the components in Figure 4-1. The dummy addresses are used only to show the flow of the information provided as input to the clic_init command and SAM. Do not try to use these addresses in your configuration.
Configuring HyperFabric Information You Need Using the configuration information in Figure 4-1, the information you would specify when you run clic_init or SAM on each of the nodes is listed below. This example is not an exact depiction of the prompts produced by clic_init nor the fields in SAM, but merely an example of the flow of information input. In addition, you should not try to use the dummy addresses in your actual configuration. On node A: 1.
Configuring HyperFabric Information You Need 12. What is the Ethernet hardware address of the second switch? (0060b0d00256) 13. What is the Multicast address for the switches to use? (226.10.1.1) 14. What is the IP address for the LAN card on the same subnet as the switches? (193.0.0.10) (In Figure 4-1, this is the IP address for lan0 on node A.) On node B: 1. How many HyperFabric adapters are installed on the node? 2.
Configuring HyperFabric Information You Need 12. What is the Ethernet hardware address of the second switch? (0060b0d00256) 13. What is the Multicast address for the switches to use? (226.10.1.1) 14. What is the IP address for the LAN card on the same subnet as the switches? (193.0.0.11) (In Figure 4-1, this is the IP address for lan0 on node B.
Configuring HyperFabric Performing the Configuration Performing the Configuration As explained in “Configuration Overview” on page 85, you must create the global configuration file (/etc/rc.config.d/clic_global_conf) on each node in the fabric. This consists mostly of specifying HyperFabric adapter-related information. (If you are also going to enable switch management — which HP does not recommend doing — you need to specify additional configuration information.
Configuring HyperFabric Performing the Configuration Using the clic_init Command Run the clic_init command to create the global configuration file. To view the man page for clic_init see “Viewing man Pages” on page 143 of this manual. To use clic_init to configure the Transparent Local Failover feature on HMP, see the section, “Configuring HMP for Transparent Local Failover Support - Using the clic_init command” on page 120.
Configuring HyperFabric Performing the Configuration If you do not specify any of the above parameters, the online help for clic_init is displayed. After you have entered the information for all the adapters in the node and all of the switches (if any) in the fabric, a summary of the configuration information is displayed.
Configuring HyperFabric Performing the Configuration Using SAM—HP-UX 11.0 and HP-UX 11i v1 This section describes how to use SAM to configure HyperFabric. For information on how to use SAM to configure and deconfigure local failover feature on HMP, see “Configuring HMP for Transparent Local Failover Support - Using SAM” on page 118 and “Deconfiguring HMP for Local Failover support - Using SAM” on page 119.
Configuring HyperFabric Performing the Configuration • Subnet Mask—Optional. The adapter’s subnet mask. If you do not specify this, a default mask is chosen based on the adapter’s IP address. • Interoperability Enabled—Required. Whether you want the adapter to be able to interoperate with adapters that are using; any HP-UX 11.0 HyperFabric versions earlier than B.11.00.11 or any HP-UX 11i v1 HyperFabric versions earlier than B.11.11.01.
Configuring HyperFabric Deconfiguring a HyperFabric Adapter with SAM—HP-UX 11.0 and 11i v1 Only Deconfiguring a HyperFabric Adapter with SAM—HP-UX 11.0 and 11i v1 Only To use SAM to deconfigure a HyperFabric adapter on an HP 9000 system running HP-UX 11.0 or 11i v1, complete the following steps: Step 1. Start SAM. Step 2. Select the “Networking and Communications” area. Step 3. Select “HyperFabric.” All HyperFabric adapters installed in the system are listed.
Configuring HyperFabric Deconfiguring a HyperFabric Adapter with SAM—HP-UX 11.0 and 11i v1 Only is informed and the card pair entry is removed from the /etc/rc.config.d/netconf and /etc/rc.config.d/clic_global_conf files.
Configuring HyperFabric Configuring the HyperFabric EMS Monitor Configuring the HyperFabric EMS Monitor Starting with the December 2000 releases B.11.00.11 and B.11.11.01, the HyperFabric Event Monitoring Service (EMS) monitor allows system administrators to separately monitor each HyperFabric adapter on every node in the fabric, in addition to monitoring the entire HyperFabric subsystem. The monitor can inform the user if the resource being monitored is UP or DOWN.
Configuring HyperFabric Configuring the HyperFabric EMS Monitor 11. Define a way of notification: SNMP trap, log in syslog with a choice of severity, or email to a user defined email address 12. Validate by pressing OK NOTE Although EMS is able to monitor each HyperFabric adapter on every node in the fabric, as well as the entire HyperFabric subsystem, EMS is not able to monitor HyperFabric switches.
Configuring HyperFabric Configuring HyperFabric with ServiceGuard Configuring HyperFabric with ServiceGuard HyperFabric supports the ServiceGuard HA product. NOTE If you plan to configure HyperFabric with ServiceGuard, please read this section. Otherwise, skip this section and go on to the next section, “Configuring HMP for Transparent Local Failover Support” on page 112. ServiceGuard lets you create HA clusters of HP 9000 server systems.
Configuring HyperFabric Configuring HyperFabric with ServiceGuard Figure 4-2 below shows a HyperFabric switch configuration with ServiceGuard. This example shows a four-node configuration with two HyperFabric switches, and redundant heartbeat Ethernet LANs. NOTE 104 Because the HyperFabric network does not currently support ServiceGuard heartbeat connections, you must use an alternative type of connection for the heartbeat, such as FDDI, Token Ring, 100BaseT, or Ethernet (as shown in Figure 4-2).
Configuring HyperFabric Configuring HyperFabric with ServiceGuard Figure 4-2 An ServiceGuard Configuration (with Two HyperFabric Switches) Ethernet Heartbeat LAN 1 Ethernet Heartbeat LAN 0 node B node A HF adapter 1 HF adapter 0 HF adapter 0 HF adapter 1 S S node C Chapter 4 HF switch 1 Ethernet Port node D HF adapter 0 HF adapter 1 HF adapter 1 HF adapter 0 S HF switch 0 S Ethernet Port 105
Configuring HyperFabric Configuring HyperFabric with ServiceGuard How HyperFabric Handles Adapter Failures HyperFabric adapters are handled differently than other types of networking adapters (such as Ethernet, FDDI, and Fibre Channel) in the ServiceGuard environment. In the non-HyperFabric cases, two network links are in a node, and one will be active and one will be idle or in standby.
Configuring HyperFabric Configuring HyperFabric with ServiceGuard Example 1: This example, illustrated by Figure 4-3 below, presents an HA configuration using ServiceGuard with HyperFabric. Both of the HyperFabric adapters are active on node A. The HyperFabric Resource Monitor reports the active status of the HyperFabric resource to the Event Monitoring Service (EMS), which lets ServiceGuard know that the HyperFabric resource is available to Packages A and B.
Configuring HyperFabric Configuring HyperFabric with ServiceGuard Example 2: This example, illustrated by Figure 4-4 below, shows the same node after the failure of one of the HyperFabric adapters. The remaining adapter in node A is now handling all HyperFabric network traffic for the node. Because the HyperFabric resource is still available, ServiceGuard has not been notified; HyperFabric handles the local HyperFabric adapter failover.
Configuring HyperFabric Configuring HyperFabric with ServiceGuard Example 3: This final example, illustrated by Figure 4-5 below, shows a situation in which all of the HyperFabric adapters on node A fail. The HyperFabric Resource Monitor reports to the Event Monitoring Service (EMS). The EMS then notifies the ServiceGuard cmcld daemon that the HyperFabric resource on node A is unavailable.
Configuring HyperFabric Configuring HyperFabric with ServiceGuard Configuring HyperFabric with the ServiceGuard Resource Monitor You can configure the HyperFabric Resource Monitor with ServiceGuard in either of these ways: • Editing an ASCII file. • Using the SAM GUI. For more details, please see the manual Using EMS HA Monitors. NOTE You should configure HyperFabric with ServiceGuard before running the clic_start command or using SAM to start HyperFabric.
Configuring HyperFabric Configuring HyperFabric with ServiceGuard Specify Specify Specify Specify Specify Package Package Package Package Package SUBNET Address Services Failover Options Control Script Location Control Script Information Specify Package Resources Dependencies Add Resource Name (Navigate the Resource Subclass by double-clicking on /net until /net/interfaces/clic/ status shows up in the selection box Resource Name,then select it and click OK.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support Configuring HMP for Transparent Local Failover Support HMP supports Local Failover in the HyperFabric version B.11.11.03. If a HyperFabric resource (adapter, cable, switch or switch port) fails in a cluster, HMP transparently fails over traffic (Local Failover) using another available resource from the card pair. A card pair can be defined as a logical entity comprising of a pair of HF2 adapters on a HP 9000 node.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support • HMP does not support backward compatibility in the local failover and non-local failover mode. However, TCP/UDP/IP supports backward compatibility and interoperability. • When HMP is configured in the local failover mode, all the resources in the cluster are utilized. If a resource fails in the cluster and is restored, HMP does not utilize that resource until another resource fails.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support How Transparent Local Failover Works Consider a hypothetical HyperFabric configuration in a 4-node cluster, with each node having two adapters (see Figure 4-6). In this configuration, there is no single point of failure, and all adapters that are installed on any given node are configured as part of a card pair.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support Case 1: Adapter, Link or Switch Port Failure (see Figure 4-7) If an adapter or a link or a switch port fails, HMP transparently fails over traffic through the other available link.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support Case 2: Switch Failure (see Figure 4-8) Consider the following illustration where node A is connected to node D with traffic being routed through the HF adapter 1 on both the nodes (A and D), and the HF switch 1 fails. HMP transparently fails over traffic through the other available switch (HF switch 0).
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support Thus, if a switch fails, HMP transparently fails over traffic only if at least one member of the card pair is physically reachable through the other switch. Case 3: Cable Failure Between Two Switches (see Figure 4-9) If a cable between two switches fails, HMP traffic fails over to the other available cable between those two switches.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support Configuring HMP for Transparent Local Failover Support - Using SAM To use SAM to configure HMP for Local Failover Support, complete the following steps: Step 1. Start SAM. Step 2. Select the “Networking and Communications” area. Step 3. Select “HyperFabric.” All HyperFabric adapters installed in the system are listed; installed adapters that are not yet configured show Not Configured in the “Status” field.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support of adapters clic0 and clic1 and if the Card-Pair 1, comprises of adapters clic2 and clic3, then the following entries are added to the clic_global_conf file. CARD_PAIR[0] = clic0-clic1 CARD_PAIR[1] = clic2-clic3 If you press Cancel, you remain in the main “HyperFabric Configuration” screen. If you press Help, help text for this task appears. Step 6. Exit SAM. NOTE To view the card pair information from the /etc/rc.config.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support Configuring HMP for Transparent Local Failover Support - Using the clic_init command You can configure the Transparent Local Failover feature of HMP using clic_init also. Let us consider the following example where we have discussed the configuration in detail. This example uses some “dummy” (that is, not valid) addresses to the components in Figure 4-10.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support Using the configuration information in Figure 4-10, the information you would specify when you run clic_init on each of the nodes is listed below. This example is not an exact depiction of the prompts produced by clic_init, but merely an example of the flow of information input. In addition, you should not try to use the dummy addresses in your actual configuration. On node A: 1.
Configuring HyperFabric Configuring HMP for Transparent Local Failover Support On node B: 1. How many HyperFabric adapters are installed on the node? 2. Do you want this node to interoperate with nodes running any HyperFabric versions earlier than B.11.00.11 or B.11.11.01? (n) You must answer ‘no’ if you want to run applications using HMP (Local Failover or Non-Local Failover) for communication over HyperFabric. In that case, all nodes in the cluster must be running version B.11.00.11 (or) B.11.11.
5 Managing HyperFabric This chapter contains the following sections that give information about managing HyperFabric: • Chapter 5 “Starting HyperFabric” on page 125.
Managing HyperFabric 124 • “Verifying Communications within the Fabric” on page 128. • “Displaying Status and Statistics” on page 133. • “Viewing man Pages” on page 143. • “Stopping HyperFabric” on page 144.
Managing HyperFabric Starting HyperFabric Starting HyperFabric HyperFabric is started in one of these three ways: • As part of the normal local node boot process (HP 9000 system). • By running the HyperFabric clic_start command (described below). • By starting HyperFabric through SAM (described in “Using SAM—HP-UX 11.0 and 11i v1 0nly” on page 126).
Managing HyperFabric Starting HyperFabric /opt/clic/bin/clic_start Using the clic_start Command Run the clic_start command on each node to start the HyperFabric management process on that node. If you include /opt/clic/bin in your PATH statement, you can run the command as it is shown below. Otherwise, you must include /opt/clic/bin as part of the command name (that is, /opt/clic/bin/clic_start). You must be logged in as root to run this command.
Managing HyperFabric Starting HyperFabric Step 3. Select “HyperFabric”. Step 4. Pull down the “Actions” menu and select Start HyperFabric. Note that with a HP-UX 11.0 operating system, if HyperFabric is already running, Start HyperFabric is grayed out and cannot be selected. With HP-UX 11i v1, if HyperFabric is already running, Start HyperFabric is not grayed out. With HP-UX 11i v1, the newly added adapter can be configured and started (using OLA), without doing a clic_shutdown or a clic_start on the node.
Managing HyperFabric Verifying Communications within the Fabric Verifying Communications within the Fabric You can verify the communications within the fabric by running the clic_probe command, which is described below. You can also use clic_probe to verify the status of specific adapters. NOTE Beginning with HyperFabric versions B.11.00.11 and B.11.11.01, clic_probe replaces the clic_ping command. For systems running HP-UX 10.
Managing HyperFabric Verifying Communications within the Fabric clic_probe node_name [-c adapter_ID] [-c adapter_ID -r VRID switch_hopcount] [-l -c adapter_ID] [-s -c adapter_ID] [-p packet_count] [-?] Note that some of the lines in the above syntax are indented for readability purposes only. When you actually type the command, you do not indent anything. The command parameters are as follows: • node_name specifies the node you want to verify.
Managing HyperFabric Verifying Communications within the Fabric NOTE Also see the clic_diag command to: Probe a specific remote node. Dump and format trace data. Set the tracing level for the HyperFabric software and firmware. The clic_diag command is detailed in the section, “Running Diagnostics” on page 149. Examples of clic_probe Some examples of using clic_probe are shown below.
Managing HyperFabric Verifying Communications within the Fabric CLIC_PROBE: 256 byte packets Source adapter id: bently6.corp3.com:clic0 Target adapter id: bently4.corp7.com:clic3 256 bytes: seq_num = 1. Packet Acknowledged. 256 bytes: seq_num = 2. Packet Acknowledged. 256 bytes: seq_num = 3. Packet Acknowledged. 256 bytes: seq_num = 4. Packet Acknowledged. 256 bytes: seq_num = 5. Packet Acknowledged. 256 bytes: seq_num = 6. Packet Acknowledged. 256 bytes: seq_num = 7. Packet Acknowledged.
Managing HyperFabric Verifying Communications within the Fabric The generated output could look like this: CLIC_PROBE: 256 byte packets sent Source adapter id: bently6.corp3.com:clic0 Target adapter id: bently7.corp4.com:clic1 256 bytes: seq_num = 1. Packet Acknowledged. 256 bytes: seq_num = 2. Packet Acknowledged. 256 bytes: seq_num = 3. Packet Acknowledged. 256 bytes: seq_num = 4. Packet Acknowledged. 256 bytes: seq_num = 5. Packet Acknowledged. --------- bently7.corp4.
Managing HyperFabric Displaying Status and Statistics Displaying Status and Statistics You can get the status of and statistics associated with many of the HyperFabric components by using the clic_stat command, which is described below. The clic_stat Command The following list contains some of the information that the clic_stat command provides: • The current fabric map, in textual format. • The status of one or more HyperFabric adapters.
Managing HyperFabric Displaying Status and Statistics Note that the second line in the above syntax is indented for readability purposes only. When the command is typed there should not be any indentation. The command parameters are as follows: • -p enables/disables performance statistics gathering according to the value of perf_level, which is one of the following: TCP Enables DLPI driver statistics when under the TCP/IP stack. HMP Enables Hyper Messaging Protocol (HMP) statistics gathering.
Managing HyperFabric Displaying Status and Statistics • -c displays the statistics of the adapter identified by adapter_ID. • -n displays statistics for the node identified by nodename. • -s specifies that you want to get the status of the switch identified by switch_ID (assigned by clic_init or SAM). Note that this parameter is meaningful only if you enabled switch management (through the clic_init command). To determine the switch_ID, run the clic_stat command without specifying any parameters.
Managing HyperFabric Displaying Status and Statistics If the local node is bently7, and you want to display the management daemon (clic_mgmtd) configuration and statistics data, issue this command: clic_stat -d CFG The generated output could look like this: ============================================================================ Date: Sat Aug 5 16:08:12 2000 Node: bently7.corp2.
Managing HyperFabric Displaying Status and Statistics The generated output could look like this: ============================================================================ Date: Sat Aug 5 16:08:30 2000 Node: bently7.corp2.com ---------------------------------------------------------------------------Adapter ID: clic1 Instance Number: 1 Adapter Type: 1X PCI or HSC Firmware File: /opt/clic/firmware/clic_fw_1x32c Major Num: 238 Mgmt process driver handle: 5 Version: 1.
Managing HyperFabric Displaying Status and Statistics Firmware reset notification: 0 Data corruption notification: 0 Unsupported QOS message received: 0 Invalid HMP VC ID: 0 Invalid HMP endpoint ID: 0 Invalid HMP endpoint protection key: 0 HMP message order violation: 0 Packet drops: 0 Transmit side congestion events: 1 Receive side congestion events: 0 Other Misc Statistics Mapping message send failures: 0 ============================================================================ b.
Managing HyperFabric Displaying Status and Statistics Frame buffer overflow: 0 Receive on disabled endpoint: 0 Invalid endpoint ID: 0 Invalid endpoint protection key: 0 Interleaved gathered receive: 0 Interleaved multi-frame bulk messages: 0 NQ overflow: 0 Send NQ overflow: 0 Invalid slot key for NQ credit update: 0 DLPI QOS receive buffer shortage: 0 Link congestion events: 0 Max send packet retry exceeded: 0 Link or switch failure events: 0 Link or switch resume events: 1 Bad route detected: 0 Bad option
Managing HyperFabric Displaying Status and Statistics ============================================================================ Date: Sat Aug 5 16:08:12 2000 Node: bently6.corp4.
Managing HyperFabric Displaying Status and Statistics The generated output if the nodes are connected through a switch could look like this: ============================================================================ Date: Sat Aug 5 16:08:12 2000 Node: bently6.corp4.
Managing HyperFabric Displaying Status and Statistics ============================================================================ • Example 5 If the local node is bently7, and you want to disable all statistics gathering on bently7, issue this command: clic_stat -p RST The generated output could look like this: ============================================================================ Date: Sat Aug 5 16:08:35 2000 Node: bently7.corp6.
Managing HyperFabric Viewing man Pages Viewing man Pages If you want to be able to view the HyperFabric man pages, you must first add /opt/clic/share/man to your MANPATH environment variable.
Managing HyperFabric Stopping HyperFabric Stopping HyperFabric You can stop HyperFabric only on a local node. Stopping HyperFabric on a node makes all of the HyperFabric adapters in that node unreachable by all other nodes in the fabric. It stops the HyperFabric management process, which stops all interconnect operations on the node. To stop HyperFabric, you use (1) the clic_shutdown command (described below) or (2) SAM (described in “Using SAM—HP-UX 11.0 and 11i v1 0nly” on page 145).
Managing HyperFabric Stopping HyperFabric Using SAM—HP-UX 11.0 and 11i v1 0nly To use SAM to stop HyperFabric on a local HP 9000 system running HP-UX 11.0 or 11i v1, complete the following steps: Step 1. Start SAM. Step 2. Select the “Networking and Communications” area. Step 3. Select “HyperFabric.” Step 4. Pull down the “Actions” menu and select Stop HyperFabric. Note that if HyperFabric is not running on the system, Stop HyperFabric is grayed out and you cannot select it.
Managing HyperFabric Stopping HyperFabric 146 Chapter 5
6 Troubleshooting HyperFabric This chapter contains these sections that describe troubleshooting HyperFabric: • Chapter 6 “Running Diagnostics” on page 149.
Troubleshooting HyperFabric 148 • “Using Support Tools Manager” on page 156. • “Useful Files” on page 157. • “LED Colors and Their Meanings” on page 159. • “Determining Whether an Adapter or a Cable is Faulty” on page 172. • “Determining Whether a Switch is Faulty” on page 173. • “Replacing a HyperFabric Adapter” on page 176. • “Replacing a HyperFabric Switch” on page 177.
Troubleshooting HyperFabric Running Diagnostics Running Diagnostics Before running HyperFabric diagnostics: 1. Confirm HyperFabric adapters are installed on each node. Use the following command to display a list of HyperFabric adapters installed each node: $ ioscan -funC clic 2. Check to see if HyperFabric software is installed. Use the following command to display the HyperFabric software version installed on each node: $ swlist | grep -i hyperfabric 3. Check the patch level.
Troubleshooting HyperFabric Running Diagnostics $ clic_start All of these commands reside in the /opt/clic/bin directory. 6. Check cabling to make sure all of the HyperFabric adapters are connected to the fabric. 7. Run the following command: $ /opt/clic/bin/clic_stat -dALL If a TCP/UDP/IP application is running: Check the firmware file field to make sure the same version of firmware is downloaded on all of the HyperFabric adapters in the cluster.
Troubleshooting HyperFabric Running Diagnostics Make sure 4X HyperFabric adapters are installed on the nodes. The adapter type field indicates the type of HyperFabric adapter that is installed. HMP will only run on 4X HF1 PCI (A6092A) or 4X HF2 PCI (A6386A) adapters. Check the firmware file field to make sure the firmware file name for each HyperFabric adapter ends in 32c. All of the HyperFabric adapters in the fabric must have firmware files that end in 32c for HMP to run.
Troubleshooting HyperFabric Running Diagnostics Diagnostics can be run on many of the HyperFabric components by using the clic_diag command. If the HyperFabric subsystem is still not usable, contact your HP support representative with the diagnostics data generated using the clic_diag command described below. The clic_diag Command Use the clic_diag command to run the following diagnostics: • Probe a specific remote node. • Dump and format trace data.
Troubleshooting HyperFabric Running Diagnostics • -r specifies that you want to probe a specific remote node identified by remote_node_name. The probe is done on all operational routes to the remote node.
Troubleshooting HyperFabric Running Diagnostics • -B specifies that you want the size (in bytes) of the trace buffer to be trace_buffer_size. The buffer is dynamically allocated when tracing is enabled. Also, the buffer is circular, which means that when the end of the buffer is reached, the data wraps around to the beginning of the buffer (and overwrites any previous data). If you do not specify this parameter, a default buffer is created with a size of 64k bytes.
Troubleshooting HyperFabric Running Diagnostics Source adapter id: bently6.corp4.com:clic1 Target adapter id: bently8.corp2.com:clic3 256 bytes: seq_num = 0. Packet Acknowledged. 256 bytes: seq_num = 1. Packet Acknowledged. 256 bytes: seq_num = 2. Packet Acknowledged. 256 bytes: seq_num = 3. Packet Acknowledged. 256 bytes: seq_num = 4. Packet Acknowledged. --------- CLIC_PROBE Statistics -------5 packets transmitted, 5 packets received, 0% packet loss.
Troubleshooting HyperFabric Using Support Tools Manager Using Support Tools Manager Use Support Tools Manager (STM) with HyperFabric to gather information about HyperFabric components and to diagnose hardware problems. Two tools are available in STM for HyperFabric: • The Information Tool provides information about the HyperFabric adapter, without resetting the adapter. • The Diagnostics Tool can be used to run tests on the HyperFabric adapter; the tool reports any failures.
Troubleshooting HyperFabric Useful Files Useful Files When you are troubleshooting HyperFabric-related problems, you might find it useful to look at the contents of the following files: • /etc/rc.config.d/clic_global_conf This is the global configuration file. Check it to confirm that the configuration information is correct. • /var/adm/clic_log This is a global log file that contains a history of significant HyperFabric events.
Troubleshooting HyperFabric Useful Files /var/adm/clic_fw.dumpx This is the default file for a memory dump of an HyperFabric adapter, created when the clic_diag -d command is run without specifying an output file. • /etc/rc.config.d/netconf This file contains IP-related configuration information for all of the networking adapters installed in the HP 9000. IMPORTANT: clic_init and SAM modify this file, adding some HyperFabric-related lines that end with the characters #clic.
Troubleshooting HyperFabric LED Colors and Their Meanings LED Colors and Their Meanings Listed below are the possible colors (and the corresponding meaning) of the LEDs on the HyperFabric adapters and switches. Adapter LEDs Table 6-1 below shows the names of the LEDs on each HyperFabric adapter. Note that the LEDs on the A4920A adapter are labeled, but the labels might be hard to see when a cable is connected to the adapter.
Troubleshooting HyperFabric LED Colors and Their Meanings — An adapter is installed in a slot in the HP 9000, but the cable is attached incorrectly or no cable is attached at all. — The adapter is bad. — The cable is bad. — The switch port is bad (if the adapter is connected to a switch).
Troubleshooting HyperFabric LED Colors and Their Meanings Table 6-2 HyperFabric Adapter LED Colors and Meanings LED “Connected/Traffic” (A4919A and A6092A adapters) Color None Meaning Adapter is not operational. “Link” (A4920A and A6386A adapters) “Link Connected/Traffic” (A4921A adapter) Chapter 6 Solid green Connection from adapter to switch port or remote adapter is operational. Flashing green Data is flowing between the adapter and the switch port or remote adapter.
Troubleshooting HyperFabric LED Colors and Their Meanings Table 6-2 HyperFabric Adapter LED Colors and Meanings (Continued) LED “Error” 162 Color Meaning Notes None Adapter is not in an error condition. Adapter should be operating normally. Solid yellow Adapter is in an error condition. You must replace the adapter.
Troubleshooting HyperFabric LED Colors and Their Meanings HF1 Switch LEDs The HF1 switch LED colors and meanings are explained below. ❏ For each port on the switch: ✓ If the port is not operational, the LED is off. Some of the possible reasons for this happening are the following: — A cable is not attached correctly to the port or no cable is attached at all. — The switch port is connected to a non-operational adapter in an HP 9000.
Troubleshooting HyperFabric LED Colors and Their Meanings ❏ For the “Ethernet” port and LED on the switch: ✓ If the Ethernet port is not connected to the Ethernet network, the LED is off. ✓ If the connection from the Ethernet port to the Ethernet network is operational, the LED on the port shows as solid green. ✓ If data is flowing between the Ethernet port and the Ethernet network, the LED shows as flashing green. Table 6-3 below summarizes the HF1 switch LED information in a table format.
Troubleshooting HyperFabric LED Colors and Their Meanings Table 6-3 HF1 Switch LED Colors and Meanings LED Port “Power” Chapter 6 Color None Meaning Port is not operational. Solid green Connection from port to adapter is operational. Flashing green Data is flowing between the port and the adapter. None Power to switch is off. Solid green Power to switch is on. Notes • A cable is not attached correctly to the port or no cable is attached at all.
Troubleshooting HyperFabric LED Colors and Their Meanings Table 6-3 HF1 Switch LED Colors and Meanings (Continued) LED “Ethernet” 166 Color Meaning Notes None Ethernet port is disconnected. This can happen if the Ethernet port is not correctly connected to the Ethernet network. Management is disabled when the Ethernet port is not operating, however, this does not prevent the fabric from operating. Solid green Connection from Ethernet port to the Ethernet network is operational.
Troubleshooting HyperFabric LED Colors and Their Meanings HF2 Switch LEDs The HF2 switch LED colors and meanings are explained below. ❏ For each “Status” LED on the switch: ✓ If the card/switch module is not operating, the LED is off. — For the integrated Ethernet management LAN adapter card (in the top slot) and the integrated 8-port fiber card (in the middle slot): the card can be safely removed by qualified HP personnel only.
Troubleshooting HyperFabric LED Colors and Their Meanings ❏ For each port on the cards/switch module in the switch: ✓ If the port is not operational, the LED is off. Some of the possible reasons for this happening are the following: — A cable is not attached correctly to the port or no cable is attached at all. — The port is connected to a non-operational adapter in an HP 9000. (See “Adapter LEDs” on page 159 or Table 6-2 on page 161 for some tips about a non-operational adapter.) — The cable is bad.
Troubleshooting HyperFabric LED Colors and Their Meanings Table 6-4 HF2 Switch LED Colors and Meanings LED “Status” “Power A” and “Power B” Chapter 6 Color Meaning None The card/switch module is not operating. Solid yellow A fault is occurring on the card/switch module. Solid green The card/switch module passed the self-test and is operating. None Power to switch is off. Solid green Power to switch is on.
Troubleshooting HyperFabric LED Colors and Their Meanings Table 6-4 HF2 Switch LED Colors and Meanings (Continued) LED “Ethernet Port Main” and “Ethernet Port Aux” Color None Meaning Ethernet port is disconnected. Notes This can happen if the Ethernet port is not correctly connected to the Ethernet network, or the integrated Ethernet management LAN adapter card is experiencing a fault.
Troubleshooting HyperFabric LED Colors and Their Meanings Table 6-4 HF2 Switch LED Colors and Meanings (Continued) LED “Port x” Chapter 6 Color None Meaning Port is not operational. Solid green Connection from port to adapter is operational. Flashing green Data is flowing between the port and the adapter. Notes • A cable is not attached correctly to the port or no cable is attached at all. • The port is connected to a non-operational adapter in an HP 9000.
Troubleshooting HyperFabric Determining Whether an Adapter or a Cable is Faulty Determining Whether an Adapter or a Cable is Faulty If you suspect that either an HyperFabric adapter or a cable attached to it is faulty, follow these steps to determine which component needs replacing: Step 1. Disconnect the cable from the HyperFabric adapter. Step 2.
Troubleshooting HyperFabric Determining Whether a Switch is Faulty Determining Whether a Switch is Faulty If you suspect that a HyperFabric switch is faulty, follow the steps below to determine if the switch needs replacing. HF1 Switch The steps for determining if an HF1 switch is faulty are as follows: Step 1. Disconnect the cable from the switch port you suspect is faulty. Step 2. Attach a copper loopback plug to the relevant port.
Troubleshooting HyperFabric Determining Whether a Switch is Faulty Step 2. Check the switch’s power: a. If the power to the switch is on, the “Power A” and “Power B” LEDs both show as solid green. b. If only one of the “Power” LEDs shows as solid green, the switch is still operational. However, the integrated Ethernet management LAN adapter card should be replaced soon. (Remember, it can be replaced by qualified HP personnel only.) c.
Troubleshooting HyperFabric Determining Whether a Switch is Faulty d. If the port is faulty, do not use it. Instead, use a known good port on that card/switch module if one is available. If no good ports are available, replace that card/switch module. Remember, though, that the integrated 8-port fiber card can be removed by qualified HP personnel only. The A6388A and A6389A switch modules can be removed by you or a qualified HP personnel. Step 5.
Troubleshooting HyperFabric Replacing a HyperFabric Adapter Replacing a HyperFabric Adapter If OLAR is supported for the HP 9000 system and the HyperFabric adapter, and you want to do OLR, see “Online Replacement (OLR)” on page 56. The HP HyperFabric Release Notes contains information about which HP 9000 systems and HyperFabric adapters OLAR is supported for. If you cannot or do not want to do OLR to replace a HyperFabric adapter, follow these steps: Step 1.
Troubleshooting HyperFabric Replacing a HyperFabric Switch Replacing a HyperFabric Switch If you need to replace a faulty HyperFabric switch, follow these steps: Step 1. If you do not have an HA environment, stop HyperFabric on all nodes that are connected to the faulty switch. (See “Stopping HyperFabric” on page 144.) If you have an HA environment with two switches (where the backup switch will have taken over when the first switch failed), disconnect all cables attached to the faulty switch. Step 2.
Troubleshooting HyperFabric Replacing a HyperFabric Switch 178 Chapter 6
A Safety and Regulatory Information This appendix contains the following sections that contain safety information and regulatory statements for the HyperFabric hardware Appendix A 179
Safety and Regulatory Information components: • “Safety Symbols” on page 181. • “Regulatory Statements” on page 182. These components were tested for conformance to various national and international regulations and standards. The scope of this regulatory testing includes electrical and mechanical safety, electromagnetic emissions, immunity, acoustics, and hazardous materials. When required, approvals are obtained from third party test agencies. Approval marks appear on the product label.
Safety and Regulatory Information Safety Symbols Safety Symbols The safety-related symbols used in this manual are shown below. WARNING A WARNING denotes a hazard that can cause personal injury. CAUTION A CAUTION denotes a hazard that can damage equipment.
Safety and Regulatory Information Regulatory Statements Regulatory Statements This section contains the regulatory statements for the HyperFabric products.
Safety and Regulatory Information Regulatory Statements if not installed and used in accordance with the instruction manual, may cause harmful interference to radio communications. Operation of this equipment in a residential area is likely to cause harmful interference, in which case the user will be required to correct the interference at his own expense. Any changes or modifications not expressly approved by Hewlett-Packard could void the user’s authority to operate this equipment.
Safety and Regulatory Information Regulatory Statements Radio Frequency Interference (Japan Only) VCCI, Class A (Model A4891-62001 only) 184 Appendix A
Safety and Regulatory Information Regulatory Statements Declarations of Conformity Appendix A 185
Safety and Regulatory Information Regulatory Statements 186 Appendix A
Safety and Regulatory Information Regulatory Statements Appendix A 187
Safety and Regulatory Information Regulatory Statements 188 Appendix A
B Technical Specifications This appendix contains the following sections that contain the technical specifications for the HyperFabric hardware components: • Appendix B “Physical Attributes” on page 191.
Technical Specifications • 190 “Environmental” on page 193.
Technical Specifications Physical Attributes Physical Attributes The physical attributes of the HyperFabric adapters and switches are as follows: • HF1 adapters — A4919A PCI (1X) and A6092A PCI (4X) — 5.0 inch (12.7 cm) height — 0.85 inch (2.16 cm) width — 6.95 inch (17.65 cm) length — 5 ounces (141.7 g) weight — A4920A HSC — 3.45 inch (8.76 cm) height — 1.15 inch (2.92 cm) width — 5.9 inch (14.99 cm) length — 4 ounces (113.4 g) weight — A4921A EISA/HSC — 4.875 inch (12.38 cm) height — 0.85 inch (2.
Technical Specifications Physical Attributes • HF1 switch (A4891A) — 3.4 inch (8.6 cm) height — 17.0 inch (43.2 cm) width — 12.0 inch (30.5 cm) length — 12 pounds (5.5 kg) weight • HF2 switch chassis (A6384A) — 3.47 inch (8.8 cm) height — 19 inch (48.2 cm) width (at flanges) 17.18 inch (43.6 cm) width (enclosure) — 17.5 inch (44.5 cm) length (depth from back of flanges to back plate) 18.
Technical Specifications Environmental Environmental These environmental specifications are the same for the HyperFabric adapters and switches: • Temperature — -40 to +70 degrees C, non-operating/storage — +5 to +40 degrees C, operating — +20 to +30 degrees C, recommended operating range • Relative Humidity — 15% to 90%, non-operating/storage — 15% to 80% @ 22 degrees C, operating — 15% to 80% @ 22 degrees C, recommended operating range • Altitude — 15,000 feet (4.
Technical Specifications Environmental 194 Appendix B
Index A adapters deconfiguring, 99, 119 faulty, testing for, 172 installing, 51 LED colors and meanings, 159, 161 OLA of, 55 OLR of, 56 regulatory specifications, 182 replacing, 176 C cables, testing for faulty, 172 CLIC, definition of, 10, 59 clic_diag command, 130, 152 clic_init command, 95 after OLA, 55 clic_ping command, 128 clic_probe command, 128 to remote node, 153 clic_shutdown command, 144 clic_start command, 126 after OLA, 55 clic_stat command, 133 CLuster InterConnect.
Index steps for HF2, no rail kit, 80 steps for HF2, rail kit, 76 IP addresses, relocatable. See relocatable IP addresses.
