Managing HP Serviceguard for Linux, Sixth Edition Manufacturing Part Number : B9903-90050 August 2006
Legal Notices The information in this document is subject to change without notice. Hewlett-Packard makes no warranty of any kind with regard to this manual, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. Hewlett-Packard shall not be held liable for errors contained herein or direct, indirect, special, incidental or consequential damages in connection with the furnishing, performance, or use of this material. Warranty.
Trademark Notices HP Serviceguard® is a registered trademark of Hewlett-Packard Company, and is protected by copyright. NIS™ is a trademark of Sun Microsystems, Inc. UNIX® is a registered trademark of The Open Group. Linux® is a registered trademark of Linus Torvalds. Red Hat® is a registered trademark of Red Hat Software, Inc.
Contents 1. Serviceguard for Linux at a Glance What is Serviceguard for Linux? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Viewing Serviceguard Clusters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configuration Roadmap. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Contents What Makes a Package Run? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Before the Control Script Starts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . During Run Script Execution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Normal and Abnormal Exits from the Run Script . . . . . . . . . . . . . . . . . . . . . . . . . . . Service Startup with cmrunserv . . . . . . . . . . . . . .
Contents Disk I/O Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 Hardware Configuration Worksheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 Power Supply Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 Power Supply Configuration Worksheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 Quorum Server Planning. . . . .
Contents Implementing Channel Bonding (SLES9 and SLES10) . . . . . . . . . . . . . . . . . . . . . . . 144 Restarting Networking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 Creating the Logical Volume Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 Displaying Disk Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 Creating Partitions . . . . . . . . . . . . . . . . .
Contents Writing the Package Control Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Customizing the Package Control Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Optimizing for Large Numbers of Storage Units . . . . . . . . . . . . . . . . . . . . . . . . . . . Configuring Disk Monitoring Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Package Control Script Template File . . . . . . . . . . . . . . . . . . . . .
Contents Adding Nodes to the Configuration While the Cluster is Running . . . . . . . . . . . . . Deleting Nodes from the Configuration While the Cluster is Running . . . . . . . . . Managing Packages and Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Starting a Package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Halting a Package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Contents Serviceguard Command Hangs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Cluster Re-formations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . System Administration Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Package Movement Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Node and Network Failures . . . . . . . . . . . . . .
Contents Providing Online Application Reconfiguration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320 Documenting Maintenance Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320 C. Integrating HA Applications with Serviceguard Checklist for Integrating HA Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Defining Baseline Application Behavior on a Single System . . . . . . . . . . . . . . . . .
Printing History Table 1 Printing Date Part Number Edition November 2001 B9903-90005 First November 2002 B9903-90012 First December 2002 B9903-90012 Second November 2003 B9903-90033 Third February 2005 B9903-90043 Fourth June 2005 B9903-90046 Fifth August 2006 B9903-90050 Sixth The last printing date and part number indicate the current edition, which applies to the A.11.16 version of HP Serviceguard for Linux. The printing date changes when a new edition is printed.
Preface This guide describes how to configure and manage Serviceguard for Linux on HP ProLiant and HP Integrity servers under the Linux operating system. It is intended for experienced Linux system administrators. (For Linux system administration tasks that are not specific to Serviceguard, use the system administration documentation and manpages for your distribution of Linux.
Related Publications • Appendix C, “Integrating HA Applications with Serviceguard,” presents suggestions for integrating your existing applications with Serviceguard for Linux. • Appendix D, “Blank Planning Worksheets,” contains a set of empty worksheets for preparing a Serviceguard configuration. The following documents contain additional useful information: • HP Serviceguard for Linux Version A.11.16 Release Notes, Third Edition (B9903-90051) • Serviceguard Manager Version A.05.
Serviceguard for Linux at a Glance 1 Serviceguard for Linux at a Glance This chapter introduces Serviceguard for Linux and shows where to find different kinds of information in this book. The following topics are presented: • What is Serviceguard for Linux • Viewing Serviceguard Clusters • Configuration Roadmap If you are ready to start setting up Serviceguard clusters, skip ahead to Chapter 4, “Planning and Documenting an HA Cluster.
Serviceguard for Linux at a Glance What is Serviceguard for Linux? What is Serviceguard for Linux? Serviceguard for Linux allows you to create high availability clusters of HP ProLiant and HP Integrity servers. A high availability computer system allows application services to continue in spite of a hardware or software failure. Highly available systems protect users from software failures as well as from failure of a system processing unit (SPU), disk, or local area network (LAN) component.
Serviceguard for Linux at a Glance What is Serviceguard for Linux? In the figure, node 1 (one of two SPU's) is running package A, and node 2 is running package B. Each package has a separate group of disks associated with it, containing data needed by the package's applications, and a copy of the data. Note that both nodes are physically connected to disk arrays. However, only one node at a time may access the data for a given group of disks.
Serviceguard for Linux at a Glance What is Serviceguard for Linux? Failover Under normal conditions, a fully operating Serviceguard cluster simply monitors the health of the cluster's components while the packages are running on individual nodes. Any host system running in the Serviceguard cluster is called an active node. When you create the package, you specify a primary node and one or more adoptive nodes.
Serviceguard for Linux at a Glance What is Serviceguard for Linux? Serviceguard is designed to work in conjunction with other high availability products, such as disk arrays, which use various RAID levels for data protection; and HP-supported uninterruptible power supplies (UPS), such as HP PowerTrust, which eliminates failures related to power outage. These products are highly recommended along with Serviceguard to provide the greatest degree of availability.
Serviceguard for Linux at a Glance Viewing Serviceguard Clusters Viewing Serviceguard Clusters Serviceguard Manager is a graphical user interface for creating, updating or viewing the configuration of Serviceguard clusters and observing the status of objects such as clusters, nodes and packages. Use Serviceguard Manager to take and save snapshots of cluster status to document your configurations and to provide a basis for comparison at a later date.
Serviceguard for Linux at a Glance Configuration Roadmap Configuration Roadmap This manual presents the tasks you need to perform in order to create a functioning HA cluster using Serviceguard. These tasks are shown in Figure 1-4. Figure 1-4 Tasks in Configuring a Serviceguard Cluster The tasks in Figure 1-4 are covered in step-by-step detail in chapters 4 through 7. It is strongly recommended that you gather all the data that is needed for configuration before you start.
Serviceguard for Linux at a Glance Configuration Roadmap 24 Chapter 1
Understanding Hardware Configurations for Serviceguard for Linux 2 Understanding Hardware Configurations for Serviceguard for Linux This chapter gives a broad overview of how the server hardware components operate with Serviceguard for Linux. The following topics are presented: • Redundancy of Cluster Components • Redundant Network Components • Redundant Disk Storage • Redundant Power Supplies Refer to the next chapter for information about Serviceguard software components.
Understanding Hardware Configurations for Serviceguard for Linux Redundancy of Cluster Components Redundancy of Cluster Components In order to provide a high level of availability, a typical cluster uses redundant system components, for example two or more SPUs and two or more independent disks. Redundancy eliminates single points of failure. In general, the more redundancy, the greater your access to applications, data, and supportive services in the event of a failure.
Understanding Hardware Configurations for Serviceguard for Linux Redundant Network Components Redundant Network Components To eliminate single points of failure for networking, each subnet accessed by a cluster node is required to have redundant network interfaces. Redundant cables are also needed to protect against cable failures. Each interface card is connected to a different cable and hub or switch. Network interfaces are allowed to share IP addresses through a process known as channel bonding.
Understanding Hardware Configurations for Serviceguard for Linux Redundant Disk Storage Redundant Disk Storage Each node in a cluster has its own root disk, but each node may also be physically connected to several other disks in such a way that more than one node can obtain access to the data and programs associated with a package it is configured for. This access is provided by the Logical Volume Manager (LVM).
Understanding Hardware Configurations for Serviceguard for Linux Redundant Disk Storage disk failure occurs on one node, the monitor will cause the package to fail, with the potential to fail over to a different node on which the same disks are available. Sample Disk Configurations Figure 2-2 shows a two node cluster. Each node has one root disk which is mirrored and one package for which it is the primary node.
Understanding Hardware Configurations for Serviceguard for Linux Redundant Power Supplies Redundant Power Supplies You can extend the availability of your hardware by providing battery backup to your nodes and disks. HP-supported uninterruptible power supplies (UPS), such as HP PowerTrust, can provide this protection from momentary power loss. Disks should be attached to power circuits in such a way that disk array copies are attached to different power sources.
Understanding Serviceguard Software Components 3 Understanding Serviceguard Software Components This chapter gives a broad overview of how the Serviceguard software components work.
Understanding Serviceguard Software Components Serviceguard Architecture Serviceguard Architecture The following figure shows the main software components used by Serviceguard for Linux. This chapter discusses these components in some detail.
Understanding Serviceguard Software Components Serviceguard Architecture Each of these daemons logs to the Linux system logging files. The quorum server daemon logs to the user specified log file, such as, /usr/local/qs/log/qs.log file on Red Hat or /var/log/qs/sq.log on SuSE and cmomd logs to /usr/local/cmom/log/cmomd.log on Red Hat or /var/log/cmom/log/cmomd.log on SuSE.
Understanding Serviceguard Software Components Serviceguard Architecture The cmcld daemon also detects the health of the networks on the system. Finally, this daemon handles the management of Serviceguard packages, determining where to run them and when to start them. The path for this daemon is: $SGLBIN/cmcld. NOTE The file /etc/cmcluster.conf contains the mappings that resolve the symbolic references to $SGCONF, $SGROOT, $SGLBIN etc., used in this manual.
Understanding Serviceguard Software Components Serviceguard Architecture Cluster Object Manager Daemon: cmomd This daemon is responsible for providing information about the cluster to clients—external products or tools such as Serviceguard Manager that depend on knowledge of the state of cluster objects. Clients send queries to cmomd and receive responses from it. This daemon may not be running on your system; it is used only by clients of the object manager.
Understanding Serviceguard Software Components Serviceguard Architecture monitor daemon reads when it starts up. This daemon is automatically started at each reboot or manually with the command cmresmond --start. The path for this daemon is: $SGLBIN/cmresmond. The process for configuring disk monitoring is described in “Creating a Disk Monitor Configuration” on page 206.
Understanding Serviceguard Software Components Serviceguard Architecture Quorum Server Daemon: qs The quorum server daemon provides tie-breaking services when needed during cluster re-formation. The quorum server runs on a system external to the cluster, and it is started by the user, not by Serviceguard. It is normally started out of /etc/inittab, which means that it automatically re-spawns if it fails or is killed. Additionally, the quorum server can be run in a package in another cluster.
Understanding Serviceguard Software Components How the Cluster Manager Works How the Cluster Manager Works The cluster manager is used to initialize a cluster, to monitor the health of the cluster, to recognize node failure if it should occur, and to regulate the re-formation of the cluster when a node joins or leaves the cluster. The cluster manager operates as a daemon process that runs on each node.
Understanding Serviceguard Software Components How the Cluster Manager Works as before. In such cases, packages do not halt or switch, though the application may experience a slight performance impact during the re-formation. If heartbeat and data are sent over the same LAN subnet, data congestion may cause Serviceguard to miss heartbeats during the period of the heartbeat timeout and initiate a cluster re-formation that would not be needed if the congestion had not occurred.
Understanding Serviceguard Software Components How the Cluster Manager Works Automatic Cluster Startup An automatic cluster startup occurs any time a node reboots and joins the cluster. This can follow the reboot of an individual node, or it may be when all nodes in a cluster have failed, as when there has been an extended power failure and all SPUs went down. Automatic cluster startup will take place if the flag AUTOSTART_CMCLD is set to 1 in the $SGCONF/cmcluster.rc file.
Understanding Serviceguard Software Components How the Cluster Manager Works Cluster Quorum to Prevent Split-Brain Syndrome In general, the algorithm for cluster re-formation requires a cluster quorum of a strict majority (that is, more than 50%) of the nodes previously running. If both halves (exactly 50%) of a previously running cluster were allowed to re-form, there would be a split-brain situation in which two instances of the same cluster were running.
Understanding Serviceguard Software Components How the Cluster Manager Works The operation of the lock LUN is shown in Figure 3-2. Figure 3-2 Lock LUN Operation Serviceguard periodically checks the health of the lock LUN and writes messages to the syslog file when a lock disk fails the health check. This file should be monitored for early detection of lock disk problems. You can also configure the Disk Monitor to check the lock LUN.
Understanding Serviceguard Software Components How the Cluster Manager Works The operation of the quorum server is shown in Figure 3-3. When there is a loss of communication between node 1 and node 2, the quorum server chooses one node (in this example, node 2) to continue running in the cluster. The other node halts. Figure 3-3 Quorum Server Operation Types of Quorum Server Configuration The quorum server can be configured as a Serviceguard package or as a stand alone installation.
Understanding Serviceguard Software Components How the Cluster Manager Works If the first cluster created, in a group of clusters, needs a quorum device, that cluster must use a stand alone quorum server or lock LUN. Figure 3-4 illustrates quorum server use across four clusters. Figure 3-4 Quorum Server to Cluster Distribution HP recommends that two clusters running quorum server packages do not provide quorum services for each other.
Understanding Serviceguard Software Components How the Package Manager Works How the Package Manager Works Each node in the cluster runs an instance of the package manager; the package manager residing on the cluster coordinator is known as the package coordinator. The package coordinator does the following: • Decides when and where to run, halt or move packages. The package manager on all nodes does the following: • Executes the user-defined control script to run and halt packages and package services.
Understanding Serviceguard Software Components How the Package Manager Works Configuring Packages Each package is separately configured. You create a package by editing a package ASCII configuration file (detailed instructions are given in Chapter 6). Then you use the cmapplyconf command to check and apply the package to the cluster configuration database. You also create the package control script, which manages the execution of the package’s services. Then the package is ready to run.
Understanding Serviceguard Software Components How the Package Manager Works A package switch involves moving packages and their associated IP addresses to a new system. The new system must already have the same subnetwork configured and working properly, otherwise the packages will not be started. With package failovers, TCP connections are lost. TCP applications must reconnect to regain connectivity; this is not handled automatically.
Understanding Serviceguard Software Components How the Package Manager Works Figure 3-7 shows the condition where Node 1 has failed and Package 1 has been transferred to Node 2. Package 1’s IP address was transferred to Node 2 along with the package. Package 1 continues to be available and is now running on Node 2. Also note that Node 2 can now access both Package 1’s disk and Package 2’s disk.
Understanding Serviceguard Software Components How the Package Manager Works Failover Policy The Package Manager selects a node for a package to run on based on the priority list included in the package configuration file together with the FAILOVER_POLICY parameter, also coded in the file. Failover policy governs how the package manager selects which node to run a package on when a specific node has not been identified and the package needs to be started.
Understanding Serviceguard Software Components How the Package Manager Works Automatic Rotating Standby Using the MIN_PACKAGE_NODE failover policy, it is possible to configure a cluster that lets you use one node as an automatic rotating standby node for the cluster. Consider the following package configuration for a four node cluster. Note that all packages can run on all nodes and have the same NODE_NAME lists.
Understanding Serviceguard Software Components How the Package Manager Works If a failure occurs, any package would fail over to the node containing fewest running packages, as in Figure 3-9, which shows a failure on node 2: Figure 3-9 Rotating Standby Configuration after Failover NOTE Using the MIN_PACKAGE_NODE policy, when node 2 is repaired and brought back into the cluster, it will then be running the fewest packages, and thus will become the new standby node.
Understanding Serviceguard Software Components How the Package Manager Works If these packages had been set up using the CONFIGURED_NODE failover policy, they would start initially as in Figure 3-8, but the failure of node 2 would cause the package to start on node 3, as in Figure 3-10: Figure 3-10 CONFIGURED_NODE Policy Packages after Failover If you use CONFIGURED_NODE as the value for the failover policy, the package will start up on the highest priority node in the node list, assuming that the node is
Understanding Serviceguard Software Components How the Package Manager Works As an example, consider the following four-node configuration, in which FAILOVER_POLICY is set to CONFIGURED_NODE and FAILBACK_POLICY is AUTOMATIC: Figure 3-11 Automatic Failback Configuration before Failover Table 3-2 Node Lists in Sample Cluster Package Name Chapter 3 NODE_NAME List FAILOVER POLICY FAILBACK POLICY pkgA node1, node4 CONFIGURED_NODE AUTOMATIC pkgB node2, node4 CONFIGURED_NODE AUTOMATIC pkgC node3,
Understanding Serviceguard Software Components How the Package Manager Works Node1 panics, and when the cluster reforms, pkgA starts running on node 4: Figure 3-12 Automatic Failback Configuration After Failover After rebooting, node 1 rejoins the cluster. At that point, pkgA will be automatically stopped on node 4 and restarted on node 1.
Understanding Serviceguard Software Components How the Package Manager Works On Combining Failover and Failback Policies Combining a FAILOVER_POLICY of MIN_PACKAGE_NODE with a FAILBACK_POLICY of AUTOMATIC can result in a package’s running on a node where you did not expect it to run, since the node running the fewest packages will probably not be the same host every time a failover occurs.
Understanding Serviceguard Software Components How the Package Manager Works Table 3-3 Package Failover Behavior (Continued) Switching Behavior Parameters in ASCII File If desired, package must be manually returned to its primary node if it is running on a non-primary node. • FAILBACK_POLICY set to MANUAL. (Default) • FAILOVER_POLICY set to CONFIGURED_NODE.
Understanding Serviceguard Software Components How Package Control Scripts Work How Package Control Scripts Work Packages are the means by which Serviceguard starts and halts configured applications. Packages are also units of failover behavior in Serviceguard. A package is a collection of services, disk volumes and IP addresses that are managed by Serviceguard to ensure they are available. Any package can have a maximum of 30 services, and there can be as many as 150 packages per cluster.
Understanding Serviceguard Software Components How Package Control Scripts Work NOTE If you configure the package while the cluster is running, the package does not start up immediately after the cmapplyconf command completes. To start the package without halting and restarting the cluster, you must issue the cmrunpkg or cmmodpkg command. How does the package start up, and what is its behavior while it is running? Some of the many phases of package life are shown in Figure 3-14.
Understanding Serviceguard Software Components How Package Control Scripts Work Before the Control Script Starts First, a node is selected. This node must be in the package’s node list, it must conform to the package’s failover policy, and any resources required by the package must be available on the chosen node. One resource is the subnet that is monitored for the package. If the subnet is not available, the package cannot start on this node.
Understanding Serviceguard Software Components How Package Control Scripts Work 6. Exits with an exit code of zero (0). Figure 3-15 Package Time Line for Run Script Execution At any step along the way, an error will result in the script exiting abnormally (with an exit code of 1). For example, if a package service is unable to be started, the control script will exit with an error.
Understanding Serviceguard Software Components How Package Control Scripts Work Normal and Abnormal Exits from the Run Script Exit codes on leaving the run script determine what happens to the package next. A normal exit means the package startup was successful, but all other exits mean that the start operation did not complete successfully. • 0—normal exit. The package started normally, so all services are up on this node. • 1—abnormal exit, also known as NO_RESTART exit.
Understanding Serviceguard Software Components How Package Control Scripts Work NOTE If you set restarts and also set SERVICE_FAILFAST_ENABLED to YES, the failfast will take place after restart attempts have failed. It does not make sense to set SERVICE_RESTART to “-R” for a service and also set SERVICE_FAILFAST_ENABLED to YES. Disk Monitor Services Services that launch resource monitors such as the disk monitor behave just like any other service.
Understanding Serviceguard Software Components How Package Control Scripts Work When a Service or Subnet Fails What happens when something goes wrong? If a service fails and there are no more restarts, or if the control script fails, then the package will halt on its current node and, depending on the setting of the package switching flags, may be restarted on another node. Package halting normally means that the package halt script executes (see the next section).
Understanding Serviceguard Software Components How Package Control Scripts Work During Halt Script Execution Once the package manager has detected a service failure, or when the cmhaltpkg command has been issued for a particular package, then it launches the halt script (that is, the control script executed with the ‘halt’ parameter. This script carries out the following steps (also shown in Figure 3-16): 1. Halts all package services. 2. Executes any customer-defined halt commands. 3.
Understanding Serviceguard Software Components How Package Control Scripts Work Normal and Abnormal Exits from the Halt Script The package’s ability to move to other nodes is affected by the exit conditions on leaving the halt script. The following are the possible exit codes: • 0—normal exit. The package halted normally, so all services are down on this node. • 1—abnormal exit, also known as NO_RESTART exit. The package did not halt normally. Services are killed, and the package is disabled globally.
Understanding Serviceguard Software Components How Package Control Scripts Work Table 3-4 Error Conditions and Package Movement (Continued) Package Error Condition Results Linux Status on Primary after Error Halt script runs after Error or Exit Package Allowed to Run on Primary Node after Error Package Allowed to Run on Alternate Node Error or Exit Code Node Failfast Enabled Service Failfast Enabled Run Script Exit 1 Either Setting Either Setting Running No Not changed No Run Script Exit 2
Understanding Serviceguard Software Components How Package Control Scripts Work Table 3-4 Error Conditions and Package Movement (Continued) Package Error Condition Results Linux Status on Primary after Error Halt script runs after Error or Exit Package Allowed to Run on Primary Node after Error Package Allowed to Run on Alternate Node Error or Exit Code Node Failfast Enabled Service Failfast Enabled Halt Script Timeout YES Either Setting TOC N/A N/A (TOC) Yes, unless the timeout happened aft
Understanding Serviceguard Software Components How the Network Manager Works How the Network Manager Works The purpose of the network manager is to detect and recover from network card and cable failures so that network services remain highly available to clients. In practice, this means assigning IP addresses for each package to the primary LAN interface card on the node where the package is running and monitoring the health of all interfaces.
Understanding Serviceguard Software Components How the Network Manager Works Adding and Deleting Relocatable IP Addresses When a package is started, a relocatable IP address can be added to a specified IP subnet. When the package is stopped, the relocatable IP address is deleted from the specified subnet. Adding and removing of relocatable IP addresses is handled through the cmmodnet command in the package control script, which is described in detail in the chapter “Configuring Packages and their Services.
Understanding Serviceguard Software Components How the Network Manager Works Bonding of LAN Interfaces On the local node, several LAN interfaces can be grouped together in a process known in Linux as channel bonding. In the bonded group, one interface is used to transmit and receive data, while the others are available as backups. If one interface fails, another interface in the bonded group takes over.
Understanding Serviceguard Software Components How the Network Manager Works associated with a single IP address and MAC address. In this example, the aggregated ports are collectively known as bond0, and this is the name by which the bond is known during cluster configuration. Figure 3-18 shows a bonded configuration using redundant hubs with a crossover cable.
Understanding Serviceguard Software Components How the Network Manager Works After the failure of a card, messages are still carried on the bonded LAN and are received on the other node, but now eth1 has become active in bond0 on node1. This situation is shown in Figure 3-19.
Understanding Serviceguard Software Components How the Network Manager Works Bonding for Load Balancing It is also possible to configure bonds in load balancing mode, which allows all slaves to transmit data in parallel, in an active/active arrangement. In this case, high availability is provided by the fact that the bond still continues to function (with less throughput) if one of the component LANs should fail.
Understanding Serviceguard Software Components How the Network Manager Works Remote Switching A remote switch (that is, a package switch) involves moving packages and their associated IP addresses to a new system. The new system must already have the same subnetwork configured and working properly, otherwise the packages will not be started. With remote switching, TCP connections are lost. TCP applications must reconnect to regain connectivity; this is not handled automatically.
Understanding Serviceguard Software Components Volume Managers for Data Storage Volume Managers for Data Storage A volume manager is a tool that lets you create units of disk storage that are more flexible than individual disk partitions. These units can be used on single systems or in high availability clusters. HP Serviceguard for Linux uses the Linux Logical Volume Manager (LVM) which creates redundant storage groups. This section provides an overview of volume management with LVM.
Understanding Serviceguard Software Components Volume Managers for Data Storage Examples of Storage on Smart Arrays Figure 3-21 shows an illustration of storage configured on a Smart MSA500 storage. Physical disks are configured by an array utility program into logical units or LUNs which are then seen by the operating system. Figure 3-21 Physical Disks Combined into LUNs NOTE LUN definition is normally done using utility programs provided by the disk array manufacturer.
Understanding Serviceguard Software Components Volume Managers for Data Storage Figure 3-22 shows LUNs configured with a Smart Array cluster storage with single pathways to the data.
Understanding Serviceguard Software Components Volume Managers for Data Storage Finally, the Smart Array LUNs are configured into volume groups as shown in Figure 3-23. Figure 3-23 Smart Array LUNs Configured in Volume Groups Figure 3-24 shows an illustration of storage configured on a disk array. Physical disks are configured by an array utility program into logical units or LUNs which are then seen by the operating system.
Understanding Serviceguard Software Components Volume Managers for Data Storage NOTE LUN definition is normally done using utility programs provided by the disk array manufacturer. Since arrays vary considerably, make sure you read the documentation that accompanies your storage unit. Multipathing and LVM How multipathing is implemented depends on the storage sub-system attached to the cluster and the HBA in the servers. Check the documentation that accompanied your storage sub-system and HBA.
Understanding Serviceguard Software Components Volume Managers for Data Storage NOTE Do not use the MD lines in the Serviceguard package control scripts for MD multipath. Instead, for multipath only, MD activation can be done at system boot. Monitoring Disks The devices that are used for shared storage can be monitored by the resource monitoring facility cmresmond. This daemon runs on all nodes in the cluster and keeps track of the status of disks that are configured for monitoring.
Understanding Serviceguard Software Components Responses to Failures Responses to Failures HP Serviceguard responds to different kinds of failures in specific ways. For most hardware failures, the response is not user-configurable, but for package and service failures, you can choose the system’s response, within limits.
Understanding Serviceguard Software Components Responses to Failures Responses to Hardware Failures If a serious system problem occurs, such as a system panic or physical disruption of the SPU's circuits, Serviceguard recognizes a node failure and transfers the packages currently running on that node to an adoptive node elsewhere in the cluster. The new location for each package is determined by that package's configuration file, which lists primary and alternate nodes for the package.
Understanding Serviceguard Software Components Responses to Failures Responses to Package and Service Failures In the default case, the failure of the package or of a service within a package causes the package to shut down by running the control script with the 'stop' parameter, and then restarting the package on an alternate node. If you wish, you can modify this default behavior by specifying that the node should crash (TOC) before the transfer takes place.
Understanding Serviceguard Software Components Responses to Failures 84 Chapter 3
Planning and Documenting an HA Cluster 4 Planning and Documenting an HA Cluster Building a Serviceguard cluster begins with a planning phase in which you gather and record information about all the hardware and software components of the configuration. Planning starts with a simple list of hardware and network components. As the installation and configuration continue, the list is extended and refined.
Planning and Documenting an HA Cluster NOTE Planning and installation overlap considerably, so you may not be able to complete the worksheets before you proceed to the actual configuration. In cases where the worksheet is incomplete, fill in the missing elements to document the system as you proceed with the configuration. Subsequent chapters describe configuration and maintenance tasks in detail.
Planning and Documenting an HA Cluster General Planning General Planning A clear understanding of your high availability objectives will quickly help you to define your hardware requirements and design your system. Use the following questions as a guide for general planning: 1. What applications must continue to be available in the event of a failure? 2. What system resources (processing power, networking, SPU, memory, disk space) are needed to support these applications? 3.
Planning and Documenting an HA Cluster General Planning Serviceguard Memory Requirements The amount of lockable memory required for Serviceguard for Linux depends on the number of packages and resources configured in the cluster.
Planning and Documenting an HA Cluster Hardware Planning Hardware Planning Hardware planning requires examining the physical hardware itself. One useful procedure is to sketch the hardware configuration in a diagram that shows adapter cards and buses, cabling, disks and peripherals. A sample diagram for a two-node cluster is shown in Figure 4-1. Figure 4-1 Sample Cluster Configuration Create a similar sketch for your own cluster, and record the information on the Hardware Worksheet (page 326).
Planning and Documenting an HA Cluster Hardware Planning SPU Information SPU information includes the basic characteristics of the server systems you are using in the cluster. Different servers can be mixed in the same cluster however, they must all be running the same version of the operating system (i.e. IA64, x86-64, etc.) This includes both uniprocessor and multiprocessor computers. On the worksheet, include the following items: Server Series Number Enter the series number, e.g., LXR2000.
Planning and Documenting an HA Cluster Hardware Planning IP Address Enter this node’s host IP address intended to be used on this interface. The IP address is a string of digits separated with periods in the form ‘nnn.nnn.nnn.nnn’. If the interface is a standby and does not have an IP address, enter ‘Standby.’ Kind of LAN Traffic Identify the purpose of the subnet. Valid types include the following: • Heartbeat • Client Traffic Label the list to show the subnets that belong to a bridged net.
Planning and Documenting an HA Cluster Hardware Planning Shared Storage SCSI may be used for up to four node clusters, or FibreChannel can be used for clusters of up to 16 nodes.
Planning and Documenting an HA Cluster Hardware Planning Disk I/O Information This part of the worksheet lets you indicate where disk device adapters are installed. Enter the following items on the worksheet for each disk connected to each disk device adapter on the node: Bus Type Indicate the type of bus. Supported busses are F/W SCSI and FibreChannel. LUN Number Indicate the number of the LUN as defined in the storage unit.
Planning and Documenting an HA Cluster Hardware Planning • mount • vgdisplay -v • lvdisplay -v See the man pages on these commands for information about specific usage. The commands should be issued from all nodes after installing the hardware and rebooting the system. The information will be useful when doing LVM and cluster configuration.
Planning and Documenting an HA Cluster Hardware Planning Hardware Configuration Worksheet The Hardware configuration worksheet on page 326 will help you organize and record your specific cluster hardware configuration. Make as many copies as you need. Complete the worksheet and keep it for future reference.
Planning and Documenting an HA Cluster Power Supply Planning Power Supply Planning There are two sources of power for your cluster which you will have to consider in your design: line power and uninterruptible power supplies (UPS). Loss of a power circuit should not bring down the cluster. No more than half of the nodes should be on a single power source.
Planning and Documenting an HA Cluster Quorum Server Planning Quorum Server Planning The quorum server (QS) provides tie-breaking services for Linux clusters. The QS is described in chapter 3 under “Cluster Quorum to Prevent Split-Brain Syndrome.” For the quorum server, you will need a separate system that is not a part of the cluster, but which communicates with the cluster nodes on the same subnet. A quorum server: • can be used with up to 50 clusters, not exceeding 100 nodes total.
Planning and Documenting an HA Cluster Volume Manager Planning Volume Manager Planning When designing your disk layout using LVM, you should consider the following: • The volume groups that contain high availability applications, services, or data must be on a bus or buses available to the primary node and all adoptive nodes. • High availability applications, services, and data should be placed in volume groups that are separate from non-high availability applications, services, and data.
Planning and Documenting an HA Cluster Cluster Configuration Planning Cluster Configuration Planning A cluster should be designed to provide the quickest possible recovery from failures. The actual time required to recover from a failure depends on several factors: • The length of the cluster heartbeat interval and node timeout.
Planning and Documenting an HA Cluster Cluster Configuration Planning Quorum Server Information The quorum server (QS) provides tie-breaking services for Linux clusters. The QS is described in chapter 3 under “Cluster Quorum to Prevent Split-Brain Syndrome.
Planning and Documenting an HA Cluster Cluster Configuration Planning Lock LUN Information The lock LUN is a disk-based form of tie-breaking services for Linux clusters. The lock LUN is described in chapter 3 under “Cluster Quorum to Prevent Split-Brain Syndrome.
Planning and Documenting an HA Cluster Cluster Configuration Planning QS_POLLING_INTERVAL The time (in microseconds) between attempts to contact the quorum server to make sure it is running. Default is 300,000,000 microseconds (5 minutes).This parameter is not used if CLUSTER_LOCK_LUN is used. QS_TIMEOUT_EXTENSION The quorum server timeout is the time during which the quorum server is not communicating with the cluster. After this time, the cluster will mark the quorum server DOWN.
Planning and Documenting an HA Cluster Cluster Configuration Planning The use of a private heartbeat network is not advisable if you plan to use Remote Procedure Call (RPC) protocols and services. RPC assumes that each network adapter device or I/O card is connected to a route-able network. An isolated or private heartbeat LAN is not route-able, and could cause an RPC request-reply, directed to that LAN, to risk time-out without being serviced.
Planning and Documenting an HA Cluster Cluster Configuration Planning NODE_TIMEOUT The time after which a node may decide that the other node has become unavailable and initiate cluster reformation. Enter a number of microseconds. The default value is 2,000,000 microseconds. Minimum is 2 * (Heartbeat Interval). The maximum recommended value for this parameter is 30,000,000. The default setting yields the fastest cluster reformations.
Planning and Documenting an HA Cluster Cluster Configuration Planning NETWORK_POLLING_INTERVAL The frequency at which the networks configured for Serviceguard are checked. In the ASCII cluster configuration file, this parameter is NETWORK_POLLING_INTERVAL. Default is 2,000,000 microseconds in the ASCII file. Thus every 2 seconds, the network manager polls each network interface to make sure it can still send and receive information. Changing this value can affect how quickly a network failure is detected.
Planning and Documenting an HA Cluster Cluster Configuration Planning Cluster Configuration Worksheet The following worksheet on page 332 will help you to organize and record your cluster configuration. Make as many copies as you need. Complete and save this worksheet for future reference.
Planning and Documenting an HA Cluster Package Configuration Planning Package Configuration Planning Planning for packages involves assembling information about each group of highly available services. Some of this information is used in creating the package configuration file, and some is used for editing the package control script. NOTE Volume groups that are to be activated by packages must also be defined as cluster aware in the cluster configuration file.
Planning and Documenting an HA Cluster Package Configuration Planning • What volume groups are needed? • How much disk space is required, and how should this be allocated in logical volumes? • What file systems need to be mounted for each package? • Which nodes need to import which logical volume configurations. • If a package moves to an adoptive node, what effect will its presence have on performance? Create a list by package of volume groups, logical volumes, and file systems.
Planning and Documenting an HA Cluster Package Configuration Planning Planning for Expansion You can add packages to a running cluster. This process is described in the chapter “Cluster and Package Administration.” When adding packages, be sure not to exceed the value of MAX_CONFIGURED_PACKAGES as defined in the cluster configuration file. When planning on increasing the number of packages, remember that the use of packages requires 6MB plus about 80KB per package of lockable memory on each cluster node.
Planning and Documenting an HA Cluster Package Configuration Planning The policy used by the package manager to select the node on which the package should be automatically started. Default is CONFIGURED_NODE, which means the next available node in the list of node names for the package. The order of node name entries dictates the order of preference when selecting the node.
Planning and Documenting an HA Cluster Package Configuration Planning and will be able to fail over automatically to another node. If AUTO_RUN is set to NO, the package will not start up automatically when the cluster starts running, and will not be able to fail over automatically to another node. Default is YES, which allows a package to start up normally on an available node. Enter YES or NO in the package ASCII file.
Planning and Documenting an HA Cluster Package Configuration Planning This run script contains package run instructions and the halt script contains package halt instructions. It is recommended that you use the same script as both the run and halt script. If you wish to separate the package run instructions and package halt instructions into separate scripts, the package configuration file allows you to do this by naming two separate scripts.
Planning and Documenting an HA Cluster Package Configuration Planning • The control script will exit with status 1. If the halt script timeout occurs, you may need to perform manual cleanup. See “Package Control Script Hangs or Failures” in Chapter 8. SERVICE_NAME Enter a unique name for each service. Define one SERVICE_NAME entry for each service.You can configure a maximum of 30 services per package.
Planning and Documenting an HA Cluster Package Configuration Planning ACCESS CONTROL POLICIES Specify three things for each policy: USER_NAME, USER_HOST, and USER_ROLE. For Serviceguard Manager, USER_HOST must be the name of the Session node. Policies set in the configuration file of a cluster and its packages must not be conflicting or redundant. For more information, see “Username Validation” on page 126.
Planning and Documenting an HA Cluster Package Configuration Planning On starting the package, the script may activate one or more volume groups, and at halt time, the script deactivates each volume group. All volume groups must be accessible on each target node. Include as many volume groups (VGs) as needed. If you are using raw devices, the LV, FS, and FS_MOUNT_OPT entries are not needed. Only cluster aware volume groups should be specified in package control scripts.
Planning and Documenting an HA Cluster Package Configuration Planning FS_UMOUNT_COUNT The number of unmount attempts allowed for each file system during package shutdown. Default is 1. FS_MOUNT_RETRY_COUNT The number of mount retries for each file system. The default is 0. During startup, if a mount point is busy and FS_MOUNT_RETRY_COUNT is 0, package startup will fail and the script will exit with 1.
Planning and Documenting an HA Cluster Package Configuration Planning In the package control script file, these variables are entered in pairs. Example IP[0]=192.10.25.12 and SUBNET[0]=192.10.25.0. (In this case the subnet mask is 255.255.255.0.) HA_APP_SERVER Specify that High Availability dependent servers are being used with this package. One of the key servers for HA is Network File System (NFS).
Planning and Documenting an HA Cluster Package Configuration Planning In the package control script file, enter values into an array known as SERVICE_CMD. Enter one service command string for each service. The SERVICE_NAME, SERVICE_CMD, and SERVICE_RESTART parameters are set in the package control script in groups of three. Service Restart Parameter Enter a number of restarts. One valid form of the parameter is -r n where n is a number of retries. A value of -r 0 indicates no retries.
Planning and Documenting an HA Cluster Package Configuration Planning Timeout: _NO_TIMEOUT_ Package Halt Script: __/usr/local/cmcluster/pkg1/control.
Planning and Documenting an HA Cluster Package Configuration Planning Figure 4-3 Package Control Script Worksheet PACKAGE CONTROL SCRIPT WORKSHEET Page ___ of ___ ================================================================================ Package Control Script Data: ================================================================================ PATH______________________________________________________________ VGCHANGE_________________________________ VG[0]___/dev/gv01____LV[0]___/dev/vg01/1v011___
Building an HA Cluster Configuration 5 Building an HA Cluster Configuration This chapter and the next take you through the configuration tasks required to set up a Serviceguard cluster. These procedures are carried out on one node, called the configuration node, and the resulting binary file is distributed by Serviceguard to all the nodes in the cluster. In the examples in this chapter, the configuration node is named ftsys9, and the sample target node is called ftsys10.
Building an HA Cluster Configuration Preparing Your Systems Preparing Your Systems Before configuring your cluster, ensure that all cluster nodes possess the appropriate security files, kernel configuration and NTP (network time protocol) configuration. Understanding the Location of Serviceguard Files Serviceguard uses a special file, cmcluster.conf, to define the locations for configuration and log files within the Linux file system. The different distributions may use different locations.
Building an HA Cluster Configuration Preparing Your Systems file. For example, if SGCONF is /usr/local/cmcluster/conf, then the complete pathname for file $SGCONF/cmclconfig would be /usr/local/cmcluster/conf/cmclconfig. Enabling Serviceguard Command Access To allow the creation of a Serviceguard configuration, you should complete the following steps on all cluster nodes before running any Serviceguard commands: 1. Make sure the root user’s path includes the Serviceguard executables.
Building an HA Cluster Configuration Preparing Your Systems Editing Security Files Serviceguard daemons grant access to commands by matching incoming hostname and username against defined access control policies. To understand how to properly configure these policies, administrators need to understand how Serviceguard handles hostnames, IP addresses, usernames and the relevant configuration files. For redundancy, Serviceguard utilizes all available IPv4 networks for communication.
Building an HA Cluster Configuration Preparing Your Systems 10.8.1.132 sly.uksr.hp.com sly 15.145.162.150 bit.uksr.hp.com bit NOTE Serviceguard will only recognize the hostname in a fully qualified domain name (FQDN). For example, two nodes gryf.uksr.hp.com and gryf.cup.hp.com could not be in the same cluster as they would both be treated as the same host gryf. Serviceguard also supports domain name aliases.
Building an HA Cluster Configuration Preparing Your Systems Username Validation Serviceguard relies on the ident service of the client node to verify the username of the incoming network connection. If the Serviceguard daemon is unable to connect to the client's ident daemon, permission will be denied. Root on a node is defined as any user who has the UID of 0.
Building an HA Cluster Configuration Preparing Your Systems to server_args = -i -f /user/local/cmom/log/cmomd.log -r /user/local/cmom/run 3. Restart xinetd: /etc/init.d/xinetd restart Access Roles Serviceguard has two levels of access: • Root Access: Users who have been authorized for root access have total control over the configuration of the cluster and packages.
Building an HA Cluster Configuration Preparing Your Systems Setting access control policies uses different mechanisms depending on the state of the node. Nodes not configured into a cluster use different security configurations than nodes in a cluster. The following two sections discuss how to configure these access control policies. Setting Controls for an Unconfigured Node Serviceguard access control policies define what a remote node can do to the local node.
Building an HA Cluster Configuration Preparing Your Systems Using the cmclnodelist File The cmclnodelist file is not created by default in new installations. If administrators want to create this bootstrap file they should add a comment such as the following: ########################################################### # Do Not Edit This File # This is only a temporary file to bootstrap an unconfigured # node with Serviceguard version A.11.
Building an HA Cluster Configuration Preparing Your Systems Though hostsequiv allows defining any user on any node as equivalent to root, Serviceguard will not grant root access to any user who is not root on the remote node. Such a configuration would grant non-root access to that user. Setting Access Controls for a Configured Clusters Once nodes are configured in a cluster, different cluster wide security mechanisms are used. Changes to cmclnodelist file and hostsequiv are ignored.
Building an HA Cluster Configuration Preparing Your Systems NOTE You do not have to halt the cluster or package to configure or modify access control policies. For example: USER_NAME john USER_HOST bit USER_ROLE PACKAGE_ADMIN If these policies are defined in the cluster configuration file, it grants the PACKAGE_ADMIN role for any package to user john on node bit. User john also has the MONITOR role for the entire cluster.
Building an HA Cluster Configuration Preparing Your Systems # Policy 3 USER_NAME ANY_USER USER_HOST ANY_SERVICEGUARD_NODE USER_ROLE MONITOR In the above example, the configuration will fail because user john is assigned two roles. Policy 2 is also redundant because PACKAGE_ADMIN already includes the role MONITOR. Policy 3 does not conflict with either policy even though ANY_USER on ANY_SERVICEGUARD_NODE includes user john. Plan the clusters roles and validate them as soon as possible.
Building an HA Cluster Configuration Setting up the Quorum Server Setting up the Quorum Server If you are using a quorum server for tie-breaking, the quorum server software has to be running during cluster configuration on a system other than the nodes on which your cluster will be running. If you are using a lock LUN for tie-breaking, skip ahead to the section entitled “Setting up the Lock LUN.
Building an HA Cluster Configuration Setting up the Quorum Server Installing the Quorum Server Use the Linux rpm command to install the quorum server, product number B8467BA, on the system or systems where it will be running. You do not need to install the product on nodes that are simply using quorum services. More details on installation are found in the Quorum Server Release Notes. The quorum server executable file, qs, is installed in the /usr/local/qs/bin directory for Red Hat or /opt/qs/bin for SuSE.
Building an HA Cluster Configuration Setting up the Quorum Server Running the Quorum Server The quorum server must be running during the following cluster operations: • when the cmquerycl command is issued. • when the cmapplyconf command is issued. • when there is a cluster re-formation. By default, quorum server run-time messages go to stdout and stderr. It is suggested that you capture these messages by redirecting stdout and stderr to the file /var/log/qs/qs.log.
Building an HA Cluster Configuration Setting up the Quorum Server Creating a Package for the Quorum Server You can run the Quorum Server as a package in another cluster. In fact, a QS package running on one cluster can provide quorum services for any number of other clusters. You can add the Quorum Server to an existing cluster by creating a package with QS as the monitored service. Use the following procedure: 1. Install the Quorum Server software on all nodes, as described above. 2.
Building an HA Cluster Configuration Setting up the Quorum Server Parameter Value SERVICE_FAIL_FAST_ENABLED NO SERVICE_HALT_TIMEOUT 10 SUBNET Specify your subnet here. 5. Create a control script in the same directory: # cmmakepkg -s qs-pkg.ctl 6. Edit the file using the parameters in the following table. Parameter Value IP[0] IP address to be used when accessing the Quorum Server SUBNET[0] Specify your subnet here SERVICE_NAME[0] “qs” SERVICE_CMD[0] HP-UX: “/usr/lbin/qs >> /var/adm/qs/qs.
Building an HA Cluster Configuration Setting up the Lock LUN Setting up the Lock LUN The lock LUN requires a partition of one cylinder of at least 100K defined (via the fdisk command) as type Linux (83). You will need the pathnames for the lock LUN as it is seen on each cluster node. On one node, use the fdisk command to define a partition of 1 cylinder, type 83, on this LUN.
Building an HA Cluster Configuration Setting up the Lock LUN Command (m for help): p Disk /dev/sdc: 64 heads, 32 sectors, 4067 cylinders Units = cylinders of 2048 * 512 bytes Device Boot /dev/sdc1 Start 1 End 1 Blocks 1008 Id 83 System Linux Command (m for help): w The partition table has been altered! NOTE • Do not try to use LVM to configure the lock LUN. • The partition type must be 83. • Do not create any filesystem on the partition used for the lock LUN.
Building an HA Cluster Configuration Implementing Channel Bonding (Red Hat) Implementing Channel Bonding (Red Hat) Use the procedures included in this section to implement channel bonding on Red Hat installations. If you are using a SuSE distribution, skip ahead to the next section. Channel bonding of LAN interfaces is implemented by the use of the bonding driver, which is installed in the kernel at boot time.
Building an HA Cluster Configuration Implementing Channel Bonding (Red Hat) Sample Configuration Configure the following files to support LAN redundancy. For a single failover only one bond is needed. 1. Create a bond0 file, ifcfg-bond0. Create the configuration in the /etc/sysconfig/network-scripts directory. For example, in the file, ifcfg-bond0, bond0 is defined as the master (for your installation, substitute the appropriate values for your network instead of 192.168.1).
Building an HA Cluster Configuration Implementing Channel Bonding (Red Hat) Use MASTER=bond1 for bond1 if you have configured a second bonding interface, then add the following after the first bond (bond0): options bond1 -o bonding1 miimon=100 mode=1 NOTE During configuration, you need to make sure that the active slaves for the same bond on each node are connected the same hub or switch. You can check on this by examining the file /proc/net/bondx/info on each node.
Building an HA Cluster Configuration Implementing Channel Bonding (Red Hat) Viewing the Configuration You can test the configuration and transmit policy with ifconfig. For the example, execution on the above created configuration, the display should appear like this: # /sbin/ifconfig bond0 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4 inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.
Building an HA Cluster Configuration Implementing Channel Bonding (SLES9 and SLES10) Implementing Channel Bonding (SLES9 and SLES10) If you are using a Red Hat distribution, use the procedures described in the previous section. The following applies only to the SLES9 and SLES10 distributions. First run yast/yast2 and configure ethernet devices as DHCP so they create the ifcfg-eth-id- files.
Building an HA Cluster Configuration Implementing Channel Bonding (SLES9 and SLES10) MTU='' NETMASK='255.255.255.0' NETWORK='172.16.0.0' REMOTE_IPADDR='' STARTMODE='onboot' BONDING_MASTER='yes' BONDING_MODULE_OPTS='miimon=100 mode=1' BONDING_SLAVE0='eth0' BONDING_SLAVE1='eth1' The above example configures bond0 with mii monitor equal to 100 and mode active-backup. Adjust the IP, BROADCAST, NETMASK, NETWORK accordingly for your configuration.
Building an HA Cluster Configuration Implementing Channel Bonding (SLES9 and SLES10) NOTE It is better not to restart the network from outside the cluster subnet, as there is a chance the network could go down before the command can complete. If there was an error in any of the bonding configuration files, the network might not function properly. If this occurs, check each configuration file for errors, then try to restart the network again.
Building an HA Cluster Configuration Creating the Logical Volume Infrastructure Creating the Logical Volume Infrastructure Serviceguard makes use of shared disk storage. This is set up to provide high availability by using redundant data storage and redundant paths to the shared devices. Storage for a Serviceguard package is logically composed of LVM Volume Groups that are activated on a node as part of starting a package on that node. Storage is generally configured on logical units (LUNs).
Building an HA Cluster Configuration Creating the Logical Volume Infrastructure CAUTION The minor numbers used by the LVM volume groups must be the same on all cluster nodes. This means that if there are any non-shared volume groups in the cluster, create the same number of them on all nodes, and create them before you define the shared storage. NOTE Except as noted in the sections that follow, you perform the LVM configuration of shared storage on only one node.
Building an HA Cluster Configuration Creating the Logical Volume Infrastructure Displaying Disk Information To display a list of configured disks, use the following command: # fdisk -l You will see output such as the following: Disk /dev/sda: 64 heads, 32 sectors, 8678 cylinders Units = cylinders of 2048 * 512 bytes Device Boot /dev/sda1 * /dev/sda2 /dev/sda5 /dev/sda6 /dev/sda7 Start 1 1002 1002 4003 5004 End 1001 8678 4002 5003 8678 Blocks 1025008 7861248 3073008 1025008 3763184 Id 83 5 83 82 83 Sy
Building an HA Cluster Configuration Creating the Logical Volume Infrastructure 1. Run fdisk, specifying your device file name in place of : # fdisk Respond to the prompts as shown in the following table, to define a partition: Prompt Response Action Performed 1. Command (m for help): n Create a new partition 2. Command action e extended p primary partition (1-4) p Creation a primary partition 3. Partition number (1-4): 1 Create partition 1 4.
Building an HA Cluster Configuration Creating the Logical Volume Infrastructure Disk /dev/sdc: 64 heads, 32 sectors, 4067 cylinders Units = cylinders of 2048 * 512 bytes Device Boot /dev/sdc1 Start 1 End Blocks Id System 4067 4164592 83 Linux Command (m for help): w The partition table has been altered! 2. Respond to the prompts as shown in the following table to set a partition type: Prompt Response Action Performed 1. Command (m for help): t Set the partition type 2.
Building an HA Cluster Configuration Creating the Logical Volume Infrastructure Command (m for help): w The partition table has been altered! 3. Repeat this process for each device file that you will use for shared storage. # fdisk /dev/sdd # fdisk /dev/sdf # fdisk /dev/sdg 4. If you will be creating volume groups for internal storage, make sure to create those partitions as well, and create those volume groups before you define the shared storage.
Building an HA Cluster Configuration Creating the Logical Volume Infrastructure NOTE At this point, the setup for VG activation protection is complete. As of Serviceguard for Linux A.11.16.07, the package control script adds a tag matching the value of node (as specified in step 3 above) when it activates the volume group, and deletes the tag when it deactivates it, preventing the volume group from being activated by more than one node at the same time.
Building an HA Cluster Configuration Creating the Logical Volume Infrastructure 1. Update the LVM configuration and create the /etc/lvmtab file. You can omit this step if you have previously created volume groups on this node. # vgscan NOTE The files /etc/lvmtab and /etc/lvmtab.d may not exist on some distributions. In that case, ignore references to these files. 2. Create LVM physical volumes on each LUN.
Building an HA Cluster Configuration Creating the Logical Volume Infrastructure Figure 5-1 shows these two volume groups as they are constructed for the MSA500 Storage.
Building an HA Cluster Configuration Creating the Logical Volume Infrastructure Building Volume Groups and Logical Volumes Step 1. Use Logical Volume Manager (LVM) to create volume groups that can be activated by Serviceguard packages. For an example showing volume-group creation on LUNs, see “Building Volume Groups: Example for Smart Array Cluster Storage (MSA 500 Series)” on page 153.
Building an HA Cluster Configuration Creating the Logical Volume Infrastructure NOTE Be careful if you use YAST or YAST2 to configure volume groups, as that may cause all volume groups on that system to be activated. After running YAST or YAST2, check to make sure that volume groups for Serviceguard packages not currently running have not been activated, and use LVM commands to deactivate any that have. For example, use the command vgchange -a n /dev/sgvg00 to deactivate the volume group sgvg00.
Building an HA Cluster Configuration Creating the Logical Volume Infrastructure # reboot The partition table on the rebooted node is then rebuilt using the information placed on the disks when they were partitioned on the other node. NOTE You must reboot at this time. 3. Run vgscan to make the LVM configuration visible on the new node and to create the LVM database on /etc/lvmtab and /etc/lvmtab.d.
Building an HA Cluster Configuration Creating the Logical Volume Infrastructure Step 2.
Building an HA Cluster Configuration Creating the Logical Volume Infrastructure Storing Volume Group Configuration Data When you create volume groups, LVM creates a backup copy of the volume group configuration on the configuration node.
Building an HA Cluster Configuration Creating the Logical Volume Infrastructure Setting up Disk Monitoring HP Serviceguard for Linux includes a Disk Monitor which you can use to detect problems in disk connectivity. This lets you fail a package from one node to another in the event of a disk link failure. To identify the set of disks you wish to monitor, you run a script after all packages are configured. (The script is run again any time you add, change or remove packages.
Building an HA Cluster Configuration Configuring the Cluster Configuring the Cluster Use the cmquerycl command to specify a set of nodes to be included in the cluster and to generate a template for the cluster configuration file. The following is an example of the command as issued from node ftsys9: # cmquerycl -v -C $SGCONF/clust1.config \ -n ftsys9 -n ftsys10 -q lp-qs The example creates an ASCII template file in the default cluster configuration directory $SGCONF.
Building an HA Cluster Configuration Configuring the Cluster Using Serviceguard Manager to Configure the Cluster Before you start: 1. Configure the volume groups. 2. If you are using a quorum server to manage the cluster lock, make sure it is running. To configure the cluster using Serviceguard Manager, proceed as follows. Online Help is available at each step to help you make decisions. 1. Create a session on Serviceguard Manager. Select the option for discovering unused nodes.
Building an HA Cluster Configuration Configuring the Cluster Specifying a Lock LUN If you will be using a lock LUN, be sure to specify the -L lock_lun_device option with the cmquerycl command. If the name of the device is the same on all nodes, enter the option before the node names, as in the following example: # cmquerycl -v -L /dev/cciss/c0d0p1 -n lp01 -n lp02 \ -C $SGCONF/lpcluster.
Building an HA Cluster Configuration Configuring the Cluster # this file.
Building an HA Cluster Configuration Configuring the Cluster # # # # # # # # LUN Lock Disk Parameters. Use the CLUSTER_LOCK_LUN parameter to define the device on a per node basis. The device may only be used for the purpose and by only a single cluster. # # # # # # # # # # # # # # # # # # # # # # Quorum Server Parameters. Use the QS_HOST, QS_POLLING_INTERVAL and QS_TIMEOUT_EXTENSION parameters to define a quorum server.
Building an HA Cluster Configuration Configuring the Cluster # Configuration/Reconfiguration Timing Parameters (microseconds). AUTO_START_TIMEOUT NETWORK_POLLING_INTERVAL 600000000 2000000 # Package Configuration Parameters. # Enter the maximum number of packages which will be configured in the cluster. # You can not add packages beyond this limit. # This parameter is required. MAX_CONFIGURED_PACKAGES 3 # Access Control Policy Parameters.
Building an HA Cluster Configuration Configuring the Cluster # If set in a package configuration file, PACKAGE_ADMIN applies to that # package only. # # Conflicting or redundant policies will cause an error while applying # the configuration, and stop the process. The maximum number of access # policies that can be configured in the cluster is 200.
Building an HA Cluster Configuration Configuring the Cluster Modifying Cluster Timing Parameters The cmquerycl command supplies default cluster timing parameters for HEARTBEAT_INTERVAL and NODE_TIMEOUT. Changing these parameters will directly impact the cluster’s reformation and failover times. It is useful to modify these parameters if the cluster is reforming occasionally due to heavy system load or heavy network traffic.
Building an HA Cluster Configuration Configuring the Cluster Verifying the Cluster Configuration If you have edited an ASCII cluster configuration file, use the following command to verify the content of the file: # cmcheckconf -v -C $SGCONF/clust1.config This command checks the following: 170 • Network addresses and connections. • Quorum server connection. • All lock LUN device names on all nodes refer to the same physical disk area. • One and only one lock LUN device is specified per node.
Building an HA Cluster Configuration Configuring the Cluster • The network interface device files specified are valid LAN device files. • Other configuration parameters for the cluster and packages are valid. If the cluster is online or the cmcheckconf command also verifies that all the conditions for the specific change in configuration have been met.
Building an HA Cluster Configuration Managing the Running Cluster Managing the Running Cluster This section describes some approaches to routine management of the cluster. Additional tools and suggestions are found in Chapter 7, “Cluster and Package Maintenance.” Checking Cluster Operation with Serviceguard Manager Serviceguard Manager lets you see all the nodes and packages within a cluster and displays their current status. Refer to the section on “Using Serviceguard Manager” in Chapter 7.
Building an HA Cluster Configuration Managing the Running Cluster • cmhaltnode is used to manually stop a running node. (This command is also used by shutdown(1m).) A non-root with the role of Full Admin can run this command from a cluster node or through Serviceguard Manager. • cmruncl is used to manually start a stopped cluster. A non-root user with Full Admin access can run this command from a cluster node, or through Serviceguard Manager. • cmhaltcl is used to manually stop a cluster.
Building an HA Cluster Configuration Managing the Running Cluster • To verify that the node has returned to operation, check the Serviceguard Manager map or tree, or use the cmviewcl command again. 4. Bring down the cluster. In Serviceguard Manager, use the Halt Cluster command. On the command line, use the cmhaltcl -v -f command. Additional cluster testing is described in the “Troubleshooting” chapter. Refer to Appendix A for a complete list of Serviceguard commands.
Building an HA Cluster Configuration Managing the Running Cluster Setting up Autostart Features Automatic startup is the process in which each node individually joins a cluster; Serviceguard provides a startup script to control the startup process. If a cluster already exists, the node attempts to join it; if no cluster is running, the node attempts to form a cluster consisting of all configured nodes. Automatic cluster start is the preferred way to start a cluster.
Building an HA Cluster Configuration Managing the Running Cluster Changing the System Message You may find it useful to modify the system's login message to include a statement such as the following: This system is a node in a high availability cluster. Halting this system may cause applications and services to start up on another node in the cluster. You might wish to include a list of all cluster nodes in this message, together with additional cluster-specific information.
Building an HA Cluster Configuration Managing the Running Cluster currently available for package switching. However, you should not try to restart HP Serviceguard, since data corruption might occur if another node were to attempt to start up a new instance of the application that is still running on the single node.
Building an HA Cluster Configuration Managing the Running Cluster 178 Chapter 5
Configuring Packages and Their Services 6 Configuring Packages and Their Services In addition to configuring the cluster, you need to identify the applications and services that you wish to group into packages.
Configuring Packages and Their Services Using Serviceguard Manager to Configure a Package Using Serviceguard Manager to Configure a Package To configure a high availability package use the following steps on the configuration node (ftsys9). Serviceguard Manager configuration is available in Serviceguard Version A.11.16 and later. 1. Connect to a session server node that has Serviceguard version A.11.16 installed. Discover a cluster that has version A.11.16 or later.
Configuring Packages and Their Services Creating the Package Configuration Creating the Package Configuration The package configuration process defines a set of application services that are run by the package manager when a package starts up on a node in the cluster. The configuration also includes a prioritized list of cluster nodes on which the package can run together with definitions of the acceptable types of failover allowed for the package.
Configuring Packages and Their Services Creating the Package Configuration Configuring in Stages It is recommended to configure packages on the cluster in stages, as follows: 1. Configure volume groups and mount points only. 2. Apply the configuration. 3. Distribute the control script to all nodes. 4. Run the package and ensure that it can be moved from node to node. 5. Halt the package. 6. Configure package IP addresses and application services in the control script. 7.
Configuring Packages and Their Services Creating the Package Configuration # PACKAGE_TYPE FAILOVER # # # # # Enter the failover policy for this package. This policy will be used to select an adoptive node whenever the package needs to be started. The default policy unless otherwise specified is CONFIGURED_NODE. This policy will select nodes in priority order from the list of NODE_NAME entries specified below. # # # # The alternative policy is MIN_PACKAGE_NODE.
Configuring Packages and Their Services Creating the Package Configuration NODE_NAME # # # # # # Enter the value The default for package will be package will be ftsys10 for AUTO_RUN. Possible values are YES and NO. AUTO_RUN is YES. When the cluster is started the automaticaly started. In the event of a failure the started on an adoptive node. Adjust as necessary. AUTO_RUN replaces obsolete PKG_SWITCHING_ENABLED. AUTO_RUN # # # # # # Enter the value for NODE_FAIL_FAST_ENABLED.
Configuring Packages and Their Services Creating the Package Configuration # default will be NO. # # SERVICE_HALT_TIMEOUT is represented as a number of seconds. # This timeout is used to determine the length of time (in # seconds) the cluster software will wait for the service to # halt before a SIGKILL signal is sent to force the termination # of the service. In the event of a service halt, the cluster # software will first send a SIGTERM signal to terminate the # service.
Configuring Packages and Their Services Creating the Package Configuration # # # # # # # # # # # # # # # # # # # # 1. USER_NAME can either be ANY_USER, or a maximum of 8 login names from the /etc/passwd file on user host. 2. USER_HOST is where the user can issue Serviceguard commands. If using Serviceguard Manager, it is the COM server. Choose one of these three values: ANY_SERVICEGUARD_NODE, or (any) CLUSTER_MEMBER_NODE, or a specific node.
Configuring Packages and Their Services Creating the Package Configuration • If the package depends on monitored devices, be sure to include separate service entries (SERVICE_NAME DiskMond) for disk monitoring and include the resclient command as the SERVICE_CMD in the control script. • If your package has IP addresses associated with it, enter the SUBNET. This must be a subnet that is already specified in the cluster configuration. • NODE_FAIL_FAST_ENABLED parameter. Enter YES or NO.
Configuring Packages and Their Services Writing the Package Control Script Writing the Package Control Script The package control script contains all the information necessary to run all the services in the package, monitor them during operation, react to a failure, and halt the package when necessary. Each package must have a separate control script, which must be executable.
Configuring Packages and Their Services Writing the Package Control Script Customizing the Package Control Script Check the definitions and declarations at the beginning of the control script using the information in the Package Configuration worksheet. For more information see “Understanding the Location of Serviceguard Files” on page 122. You need to customize as follows: NOTE • Update the PATH statement to reflect any required paths needed to start your services.
Configuring Packages and Their Services Writing the Package Control Script Optimizing for Large Numbers of Storage Units Two variables are provided to allow performance improvement when employing a large number of file systems or storage groups. For more detail, see the comments in the control script template. They are: • CONCURRENT_FSCK_OPERATIONS—defines a number of parallel fsck operations that will be carried out at package startup.
Configuring Packages and Their Services Writing the Package Control Script Package Control Script Template File The following is an excerpt from the package control script template file. Comment out all references to RAIDSTART, RAIDSTOP, and MD. They are deprecated in Serviceguard for Linux but the package control script has not yet been updated to reflect this. Do not make any changes in the SOFTWARE RAID DATA REPLICATION section, as this is reserved for a possible future enhancement.
Configuring Packages and Their Services Writing the Package Control Script # arrays and is included as /bin/bash2. We will first check to see of # the shell that invoked us (/bin/bash) will work (in case someone # changed it, if not we will use /bin/bash2. # # At SG installation time we checked to make sure # that either /bin/bash will work with this control script or # that /bin/bash2 is installed. The SG rpm would not install unless # one of these conditions are true. On RH7.
Configuring Packages and Their Services Writing the Package Control Script # the variable DATA_REP to the data replication method. The current supported # methods are "clx", "clxeva" and "xdcmd". # DATA_REP="none" # SOFTWARE RAID DATA REPLICATION # Specify the location of the Software RAID configuration file if "DATA_REP" # is set to "xdcmd" or "XDCMD". This is the Extended Distance cluster Configuration file. # A separate copy of this file will be used by each package.
Configuring Packages and Their Services Writing the Package Control Script # Serviceguard and may still be used by some storage system/HBA # combinations. For that reason there are references to MD in the template # files, worksheets, and other areas. Only use MD if your storage systems # specifically calls out its use for multipath. # If some other multipath mechanism is used (e.g. one built # into an HBA driver), then references to MD, RAIDTAB, RAIDSTART, etc. # should be commented out.
Configuring Packages and Their Services Writing the Package Control Script # VOLUME GROUPS # Specify which volume groups are used by this package. Uncomment VG[0]="" # and fill in the name of your first volume group. You must begin with # VG[0], and increment the list in sequence. # # For example, if this package uses your volume groups vg01 and vg02, enter: # VG[0]=vg01 # VG[1]=vg02 # # The volume group activation method is defined above.
Configuring Packages and Their Services Writing the Package Control Script # The filesystems are defined as entries specifying the logical # volume, the mount point, the file system type, the mount, # umount and fsck options. # Each filesystem will be fsck'd prior to being mounted. # The filesystems will be mounted in the order specified during package # startup and will be unmounted in reverse order during package # shutdown.
Configuring Packages and Their Services Writing the Package Control Script # LV[1]=/dev/vg01/lvol2; FS[1]=/pkg1b; FS_TYPE[1]="reiserfs"; # FS_MOUNT_OPT[1]="-o rw"; FS_UMOUNT_OPT[1]=""; FS_FSCK_OPT[1]=""; # #LV[0]=""; FS[0]=""; FS_TYPE[0]=""; FS_MOUNT_OPT[0]="" #FS_UMOUNT_OPT[0]=""; FS_FSCK_OPT[0]="" # FILESYSTEM UNMOUNT COUNT # Specify the number of unmount attempts for each filesystem during package # shutdown. The default is set to 1. # FS_UMOUNT_COUNT=1 # FILESYSTEM MOUNT RETRY COUNT.
Configuring Packages and Their Services Writing the Package Control Script # performance for starting up or halting a package. # each concurrent operation parameter is 1024. The maximum value for Set these values carefully. # The performance could actually decrease if the values are set too high # for the system resources available on your cluster nodes.
Configuring Packages and Their Services Writing the Package Control Script # package startup or shutdown. # Setting this value to an appropriate number may improve the performance # while mounting or un-mounting a large number of file systems in the package. # If the specified value is less than 1, the script defaults it to 1 and # proceeds with a warning message in the package control script logfile.
Configuring Packages and Their Services Writing the Package Control Script # this package. Some examples of the HA Servers are Network File System # (NFS), Apache Web Server, and SAMBA (CIFS) Server. # # If you plan to use one of the HA server toolkits to run an application server, # you need to set the HA_APP_SERVER value to either "pre-IP" or "post-IP" in # order to enable this control script to check and run the Toolkit Interface # Script (toolkit.sh) in the package directory.
Configuring Packages and Their Services Writing the Package Control Script # SERVICE_RESTART[0]="" and fill in the name of the first service, command, # and restart parameters. You must begin with SERVICE_NAME[0], SERVICE_CMD[0], # and SERVICE_RESTART[0] and increment the list in sequence. # # For example: # SERVICE_NAME[0]=cmresserviced_pkg1 # SERVICE_CMD[0]="/usr/local/cmcluster/bin/cmresserviced /dev/md0" # SERVICE_RESTART[0]="" # Will not restart the service.
Configuring Packages and Their Services Writing the Package Control Script Linux commands, and Serviceguard commands including cmrunserv, cmmodnet, and cmhaltserv. Examine a copy of the control script template to see the flow of logic. Use the following command: # cmmakepkg -s | more The main function appears at the end of the script. Note that individual variables are optional; you should include only as many as you need for proper package operation.
Configuring Packages and Their Services Writing the Package Control Script # You should define all actions you want to happen here, before the service is # halted. function customer_defined_halt_cmds { # ADD customer defined halt commands. : # do nothing instruction, because a function must contain some command. date >> /tmp/pkg1.datelog echo 'Halting pkg1' >> /tmp/pkg1.
Configuring Packages and Their Services Verifying the Package Configuration Verifying the Package Configuration If you have edited an ASCII package configuration file, use the following command to verify the content of the file: # cmcheckconf -v -P $SGCONF/pkg1/pkg1.config Errors are displayed on the standard output. If necessary, edit the file to correct any errors, then run the command again until it completes without errors.
Configuring Packages and Their Services Applying and Distributing the Configuration Applying and Distributing the Configuration Use the cmapplyconf command to apply and distribute a binary cluster configuration file containing the package configuration among the nodes of the cluster. Example: # cmapplyconf -v -C $SGCONF/cmcl.config -P \ $SGCONF/pkg1/pkg1.config The cmapplyconf command creates a binary cluster configuration database file and distributes it to all nodes in the cluster.
Configuring Packages and Their Services Creating a Disk Monitor Configuration Creating a Disk Monitor Configuration HP Serviceguard provides disk monitoring for the shared storage that is activated by packages in the cluster. The monitor daemon on each node tracks the status of all the disks that are flagged for monitoring on that node in the package configuration files and control scripts.
Configuring Packages and Their Services Creating a Disk Monitor Configuration NOTE The SERVICE_CMD must include the string “cmresserviced” or the cmconfigres will fail to find the disks to be monitored. cmconfigres looks for SERVICE_CMD entries including the text “cmresserviced” to determine which disks to monitor. It is important to set SERVICE_RESTART to an empty string (““).
Configuring Packages and Their Services Creating a Disk Monitor Configuration normal 1 Pkg1 /usr/local/cmcluster/bin/cmcheckdisk 60 60 /dev/sdd1 /dev/sde1
Configuring Packages and Their Services Creating a Disk Monitor Configuration is already running, cmresmond --restart should be run to apply this new configuration. Use the following commands to complete the change and active the new configuration: # cmconfigres --delete # cmconfigres --create # cmresmond --restart Configuring Disks on a Single Node for Monitoring An alternate method for configuring monitoring is to use the sync option to configure monitor requests for a single node.
Configuring Packages and Their Services Creating a Disk Monitor Configuration See “Viewing Status of Monitored Disks” on page 240 for information about viewing the status of disks in the cluster. More information about disk monitoring is provided in “Resource Monitor Daemon: cmresmond” on page 35, “Disk Monitor Services” on page 62, and “Monitoring Disks” on page 80.
Cluster and Package Maintenance 7 Cluster and Package Maintenance This chapter describes the cmviewcl command, then shows how to start and halt a cluster or an individual node, how to perform permanent reconfiguration, and how to start, halt, move, and modify packages during routine maintenance of the cluster.
Cluster and Package Maintenance Reviewing Cluster and Package States with the cmviewcl Command Reviewing Cluster and Package States with the cmviewcl Command Information about cluster status is stored in the status database, which is maintained on each individual node in the cluster. You can display information contained in this database by issuing the cmviewcl command: # cmviewcl -v You can issue the cmviewcl command with non-root access: In clusters with Serviceguard version A.11.
Cluster and Package Maintenance Reviewing Cluster and Package States with the cmviewcl Command Types of Cluster and Package States A cluster or its component nodes may be in several different states at different points in time. The following sections describe many of the common conditions the cluster or package may be in. Cluster Status The status of a cluster may be one of the following: • Up. At least one node has a running cluster daemon, and reconfiguration is not taking place. • Down.
Cluster and Package Maintenance Reviewing Cluster and Package States with the cmviewcl Command • Unknown. A node never sees itself in this state. Other nodes assign a node this state if it has never been an active cluster member. Quorum Server Status and State The status of the quorum server can be one of the following: • Up. The quorum server is active. • Down. The quorum server is not active. The state of the quorum server can be one of the following: • Running.
Cluster and Package Maintenance Reviewing Cluster and Package States with the cmviewcl Command Package Switching Attributes Packages also have the following switching attributes: • AUTO_RUN. Enabled means that the package can switch to another node in the event of failure. • Node Switching. Enabled means that the package can switch to the referenced node. Disabled means that the package cannot switch to the specified node until the node is enabled for the package using the cmmodpkg command.
Cluster and Package Maintenance Reviewing Cluster and Package States with the cmviewcl Command Failover and Failback Policies Packages can be configured with one of two values for the FAILOVER_POLICY parameter: • CONFIGURED_NODE. The package fails over to the next node in the node list in the package configuration file. • MIN_PACKAGE_NODE. The package fails over to the node in the cluster with the fewest running packages on it.
Cluster and Package Maintenance Reviewing Cluster and Package States with the cmviewcl Command Examples of Cluster and Package States The following sample output from the cmviewcl -v command shows status for the cluster in the sample configuration. Normal Running Status Everything is running normally; both nodes in the cluster are running, and the packages are in their primary locations.
Cluster and Package Maintenance Reviewing Cluster and Package States with the cmviewcl Command ITEM Service Subnet STATUS up up MAX_RESTARTS 0 0 Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING Primary up enabled Alternate up enabled RESTARTS 0 0 NAME ftsys10 ftsys9 NAME service2 15.13.168.
Cluster and Package Maintenance Reviewing Cluster and Package States with the cmviewcl Command Status After Halting a Package After halting pkg2 with the cmhaltpkg command, the output of cmviewcl-v is as follows: CLUSTER example NODE ftsys9 STATUS up STATUS up STATE running Network_Parameters: INTERFACE STATUS PRIMARY up PRIMARY up NAME eth0 eth1 PACKAGE pkg1 STATE running STATUS up AUTO_RUN enabled NODE ftsys9 Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manua
Cluster and Package Maintenance Reviewing Cluster and Package States with the cmviewcl Command Primary Alternate up up enabled enabled ftsys10 ftsys9 Pkg2 now has the status “down”, and it is shown as in the unowned state, with package switching disabled. Note that switching is enabled for both nodes, however. This means that once global switching is re-enabled for the package, it will attempt to start up on the primary node.
Cluster and Package Maintenance Reviewing Cluster and Package States with the cmviewcl Command Subnet up 0 Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING Primary up enabled Alternate up enabled NODE ftsys10 STATUS up 0 NAME ftsys10 ftsys9 15.13.168.0 (current) STATE running Network_Parameters: INTERFACE STATUS PRIMARY up PRIMARY up NAME eth0 eth1 Now pkg2 is running on node ftsys9. Note that it is still disabled from switching.
Cluster and Package Maintenance Reviewing Cluster and Package States with the cmviewcl Command Status After Halting a Node After halting ftsys10, with the following command: # cmhaltnode ftsys10 the output of cmviewcl is as follows on ftsys9: CLUSTER example NODE ftsys9 PACKAGE pkg1 pkg2 NODE ftsys10 STATUS up STATUS up STATUS up up STATUS down STATE running STATE running running AUTO_RUN enabled enabled NODE ftsys9 ftsys9 STATE halted This output is seen on both ftsys9 and ftsys10.
Cluster and Package Maintenance Reviewing Cluster and Package States with the cmviewcl Command Viewing Data on Unowned Packages The following example shows packages that are currently unowned, that is, not running on any configured node.
Cluster and Package Maintenance Using Serviceguard Manager Using Serviceguard Manager Serviceguard Manager is a graphical user interface for managing Serviceguard clusters. Use Serviceguard Manager to display data from running clusters, or optionally to view saved data files. Serviceguard Manager runs on HP-UX, Linux, and Windows systems. From there, you can view Serviceguard clusters on HP-UX or Linux. Options can be selected through dialog boxes or at the command line.
Cluster and Package Maintenance Using Serviceguard Manager Running Serviceguard Manager with a Command To start Serviceguard Manager from the command line, either in HP-UX, Linux, or from a Windows NT DOS window, use the following syntax: sgmgr [-s server [-l username [-p password]] | -f filename ] [-c clustername...] Table 7-1 shows the use of the various options.
Cluster and Package Maintenance Using Serviceguard Manager Starting with a Specific Cluster Use the following procedures to start Serviceguard Manager to display a specific cluster.
Cluster and Package Maintenance Using Serviceguard Manager When started, the Serviceguard Manager splash screen shows program initialization. Then you will see the Object Manager login screen displayed in Figure 7-1. Figure 7-1 Object Manager Login Screen Proceed to the section below entitled “Connecting to an Object Manager.
Cluster and Package Maintenance Using Serviceguard Manager Starting Serviceguard Manager without a Specific Cluster Use the following procedures to start Serviceguard Manager without specifying a particular cluster. This approach will search for clusters throughout your subnet; if many clusters are configured, the search can be time-consuming. • From the HP-UX management station, enter the sgmgr command from a terminal window in the install directory, typically: # export DISPLAY=mydisplay:0.
Cluster and Package Maintenance Using Serviceguard Manager When started, the Serviceguard Manager splash screen appears briefly. Then you see the opening screen displayed in Figure 7-2. Figure 7-2 Serviceguard Manager Opening Screen This screen lets you choose one of the following: Chapter 7 • to connect to Serviceguard Object Manager to view data from a running cluster • to open a saved file to view data previously saved.
Cluster and Package Maintenance Using Serviceguard Manager Connecting to an Object Manager The Serviceguard Object Manager daemon on a cluster node provides information about running clusters to Serviceguard Manager. If you want to view data about running clusters, you need to establish a connection to an Object Manager by logging on to a node running HP Serviceguard 11.12 or later with the Object Manager daemon running. The login screen is shown in Figure 7-3.
Cluster and Package Maintenance Using Serviceguard Manager Opening a Saved File with Cluster Data Choosing the second option from the opening screen lets you open a file of saved data. You will see a list of saved files from which to choose, as shown in Figure 7-4. Figure 7-4 Serviceguard Manager Open File Screen Choose the file you wish, then click “Open.
Cluster and Package Maintenance Using Serviceguard Manager Viewing Cluster Data Whether you are connecting to an Object Manager node or displaying data from a saved file, you will see an initial map and hierarchical tree display. This screen is shown in Figure 7-5.
Cluster and Package Maintenance Using Serviceguard Manager To display the objects within a specific cluster, double click the cluster icon, which will display a screen like the one in Figure 7-6.
Cluster and Package Maintenance Using Serviceguard Manager For a display of an object’s property sheet, click on the object, then click on “Properties” in the tool bar at the top of the screen. A cluster property sheet is shown in Figure 7-7.
Cluster and Package Maintenance Using Serviceguard Manager Obtaining Help To obtain online help, click on the “Help” menu item at the top of the screen. The top level help screen is shown in Figure 7-8.
Cluster and Package Maintenance Using Serviceguard Manager Managing Cluster Objects You can use Serviceguard Manager to manipulate clusters, nodes, and packages. It is possible to start and halt the cluster (including all packages), bring individual nodes in and out of cluster operation, start or stop individual packages, and move packages from one node to another.
Cluster and Package Maintenance Using Serviceguard Manager The GUI displays a progress box that shows the Serviceguard commands that are being run to start the cluster (Figure 7-10).
Cluster and Package Maintenance Using Serviceguard Manager Figure 7-11 shows the outcome—a running cluster.
Cluster and Package Maintenance Using Serviceguard Manager CAUTION Chapter 7 In order to carry out administrative operations, the user of a Serviceguard Manager client must log into the Object Manager system as root, using the root password. This effectively grants to the client the ability to manage all clusters that are serviced by that Object Manager. If it is not necessary for some users to administer the clusters, you can have them log in as ordinary users on the Object Manager system.
Cluster and Package Maintenance Viewing Status of Monitored Disks Viewing Status of Monitored Disks You can view the status of monitored disks by accessing the monitor server through the following URL: http://hostname:3542 Supported web browsers include Internet Explorer 6.0 on Windows and Mozilla 1.0 on Linux.
Cluster and Package Maintenance Viewing Status of Monitored Disks The following is a sample output, with resource /dev/sdb1 specified is: # cmviewres /dev/sdb1 Resource: /dev/sdb1 Category: pkg23837_1 Status: down Timeout (seconds): 60.00 Polling Interval (seconds): 60.
Cluster and Package Maintenance Managing the Cluster and Nodes Managing the Cluster and Nodes Managing the cluster involves the following tasks: • Starting the Cluster When All Nodes are Down • Adding Previously Configured Nodes to a Running Cluster • Removing Nodes from Operation in a Running Cluster • Halting the Entire Cluster • Reconfiguring a Halted Cluster Starting the cluster means running the cluster daemon on one or more of the nodes in a cluster.
Cluster and Package Maintenance Managing the Cluster and Nodes Starting the Cluster When all Nodes are Down Using Serviceguard Manager to Start the Cluster Select the cluster icon, then right-click to display the action menu. Select “Start the Cluster.” The progress window shows messages as the action takes place. This will include messages for starting each node and package. Click OK on the progress window when the operation is complete.
Cluster and Package Maintenance Managing the Cluster and Nodes Adding Previously Configured Nodes to a Running Cluster You can use Serviceguard Manager or HP Serviceguard commands to bring a configured node up within a running cluster. Using Serviceguard Manager to Add a Node to the Cluster Select the node icon, then right-click to display the action menu. Select “Start the Node.” The progress window shows messages as the action takes place.
Cluster and Package Maintenance Managing the Cluster and Nodes Removing Nodes from Operation in a Running Cluster You can use Serviceguard Manager or HP Serviceguard commands to remove nodes from operation in a cluster. This operation removes the node from cluster operation by halting the cluster daemon, but it does not modify the cluster configuration. To remove a node from the cluster configuration permanently, you must recreate the cluster configuration file. See the next section.
Cluster and Package Maintenance Managing the Cluster and Nodes NOTE It is recommended to run cmhaltnode prior to running the HP-UX shutdown command, especially for cases where a packaged application might have trouble during shutdown and not halt cleanly. Halting the Entire Cluster You can use Serviceguard Manager or HP Serviceguard commands to halt a running cluster. Using Serviceguard Manager to Halt the Cluster Select the cluster, then right-click to display the action menu. Select “Halt the Cluster.
Cluster and Package Maintenance Managing the Cluster and Nodes Reconfiguring a Halted Cluster You can also make a permanent change in cluster configuration when the cluster is halted. This procedure must be used for changes to the quorum server configuration, changes in timing parameters, and changes to the Maximum Number of Packages parameter, but it can be used for any other cluster configuration changes as well. Use the following steps: 1. Halt the cluster on all nodes. 2.
Cluster and Package Maintenance Reconfiguring a Running Cluster Reconfiguring a Running Cluster You can add new nodes to the cluster configuration or delete nodes from the cluster configuration while the cluster is up and running. Note the following, however: • You cannot remove an active node from the cluster. You must halt the node first. • You cannot change cluster timing parameters.
Cluster and Package Maintenance Reconfiguring a Running Cluster Table 7-2 Types of Changes to Permanent Cluster Configuration Change to the Cluster Configuration Change IP addresses for heartbeats or monitored subnets Required Cluster State Cluster must not be running. The following sections describe how to perform dynamic reconfiguration tasks. Adding Nodes to the Configuration While the Cluster is Running Use the following procedure to add a node.
Cluster and Package Maintenance Reconfiguring a Running Cluster Deleting Nodes from the Configuration While the Cluster is Running Use the following procedure to delete a node. For this example, nodes ftsys8, ftsys9 and ftsys10 are already configured in a running cluster named cluster1, and you are deleting node ftsys10. 1. Halt the node, which you are planning to remove by using the following command: # cmhaltnode -f ftsys10 2.
Cluster and Package Maintenance Managing Packages and Services Managing Packages and Services Managing packages and services involves the following tasks: • Starting a Package • Halting a Package • Moving a Package • Reconfiguring a Package on a Halted Cluster Starting a Package Ordinarily, a package configured as part of the cluster will start up on its primary node when the cluster starts up. You may need to start a package manually after it has been halted manually.
Cluster and Package Maintenance Managing Packages and Services Halting a Package You halt an HP Serviceguard package when you wish to bring the package out of use but wish the node to continue in operation. You can halt a package using Serviceguard Manager or HP Serviceguard commands. Halting a package has a different effect than halting the node.
Cluster and Package Maintenance Managing Packages and Services Moving a Package You can use Serviceguard Manager or HP Serviceguard commands to move a package from one node to another. Using Serviceguard Manager to Move a Package Select the icon of the package you wish to halt, and right-click to display the action list. Select “Move package to node.” The package must be running. You will see a list of possible destinations. Click on the node where you want the package to run.
Cluster and Package Maintenance Managing Packages and Services Reconfiguring a Package on a Halted Cluster You can also make permanent changes in package configuration while the cluster is not running. Use the following steps: 254 • On one node, reconfigure the package as described earlier in chapter 6. You can do this by editing the package ASCII file. • Edit the package control script. Any changes in service names will also require changes in the package configuration file.
Cluster and Package Maintenance Reconfiguring a Package on a Running Cluster Reconfiguring a Package on a Running Cluster You can reconfigure a package while the cluster is running, and in some cases you can reconfigure the package while the package itself is running. Only certain changes may be made while the package is running. To modify the package, use the following procedure (pkg1 is used as an example): 1.
Cluster and Package Maintenance Reconfiguring a Package on a Running Cluster Adding a Package to a Running Cluster You can create a new package and add it to the cluster configuration while the cluster is up and while other packages are running. The number of packages you can add is subject to the value of Maximum Configured Packages as defined in the cluster configuration file.
Cluster and Package Maintenance Reconfiguring a Package on a Running Cluster Changing Package Switching Behavior You can change package switching behavior either temporarily or permanently. To temporarily disable switching to other nodes for a running package, use the cmmodpkg command.
Cluster and Package Maintenance Reconfiguring a Package on a Running Cluster The current value of the restart counter may be seen in the output of the cmviewcl -v command. Allowable Package States During Reconfiguration All nodes in the cluster must be powered up and accessible when making configuration changes. Refer to Table 7-3 to determine whether or not the package may be running while you implement a particular kind of change.
Cluster and Package Maintenance Reconfiguring a Package on a Running Cluster Table 7-3 Types of Changes to Packages (Continued) Change to the Package Chapter 7 Required Package State Change run script contents It is recommended that the package be halted. If the run script for the package is modified while the package is running, timing may cause problems. Change halt script contents It is recommended that the package be halted.
Cluster and Package Maintenance Responding to Cluster Events Responding to Cluster Events Serviceguard does not require much ongoing system administration intervention. As long as there are no failures, your cluster will be monitored and protected. In the event of a failure, those packages that you have designated to be transferred to another node will be transferred automatically.
Cluster and Package Maintenance Single-Node Operation Single-Node Operation The number of nodes you will need for your Serviceguard cluster depends on the processing requirements of the applications you want to protect. In a multi-node cluster, you could have a situation where all but one node has failed, or where you have shut down all but one node, leaving your cluster in single-node operation. This remaining node will probably have applications running on it.
Cluster and Package Maintenance Removing Serviceguard from a System Removing Serviceguard from a System If you wish to remove a node from Serviceguard use, use the rpm -e command to delete the software. Note the following: 262 • The cluster should not be running on the node from which you will be deleting Serviceguard. • The node from which you are deleting Serviceguard should not be in the cluster configuration.
Troubleshooting Your Cluster 8 Troubleshooting Your Cluster This chapter describes how to verify cluster operation, how to review cluster status, how to add and replace hardware, and how to solve some typical cluster problems.
Troubleshooting Your Cluster Testing Cluster Operation Testing Cluster Operation Once you have configured your Serviceguard cluster, you should verify that the various components of the cluster behave correctly in case of a failure. In this section, the following procedures test that the cluster responds properly in the event of a package failure, a node failure, or a LAN failure.
Troubleshooting Your Cluster Testing Cluster Operation Depending on the specific databases you are running, perform the appropriate database recovery. Testing the Cluster Manager To test that the cluster manager is operating correctly, perform the following steps for each node on the cluster: 1. Turn off the power to the node. 2.
Troubleshooting Your Cluster Testing Cluster Operation Testing the Network Manager To test that the Network Manager is operating correctly, do the following for each node in the cluster: 1. Identify the LAN cards on the node: # ifconfig and then # cmviewcl -v 2. Detach the LAN connection from one card. 3. Use cmviewcl to verify that the network is still functioning through the other cards: # cmviewcl -v 4.
Troubleshooting Your Cluster Monitoring Hardware Monitoring Hardware Good standard practice in handling a high availability system includes careful fault monitoring so as to prevent failures if possible or at least to react to them swiftly when they occur. Disks can be monitored using the Disk Monitor daemon, which is described in Chapter 5.
Troubleshooting Your Cluster Replacing Disks Replacing Disks The procedure for replacing a faulty disk mechanism depends on the type of disk configuration you are using. Refer to your Smart Array documentation for issues related to your Smart Array. Replacing a Faulty Mechanism in a Disk Array You can replace a failed disk mechanism by simply removing it from the array and replacing it with a new mechanism of the same type. The resynchronization is handled by the array itself.
Troubleshooting Your Cluster Replacement of LAN Cards Replacement of LAN Cards If you need to replace a LAN card, use the following steps. It is not necessary to bring the cluster down to do this. Step 1. Halt the node using the cmhaltnode command. Step 2. Shut down the system: shutdown -h Then power off the system. Step 3. Remove the defective LAN card. Step 4. Install the new LAN card. The new card must be exactly the same card type, and it must be installed in the same slot as the card you removed.
Troubleshooting Your Cluster Replacement of LAN Cards 1. Use the cmgetconf command to obtain a fresh ASCII configuration file, as follows: # cmgetconf config.ascii 2. Use the cmapplyconf command to apply the configuration and copy the new binary file to all cluster nodes: # cmapplyconf -C config.ascii This procedure updates the binary file with the new MAC address and thus avoids data inconsistency between the outputs of the cmviewconf and ifconfig commands.
Troubleshooting Your Cluster Replacing a Failed Quorum Server System Replacing a Failed Quorum Server System When a quorum server fails or becomes unavailable to the clusters it is providing quorum services for, this will not cause a failure on any cluster. However, the loss of the quorum server does increase the vulnerability of the clusters in case there is an additional failure. Use the following procedure to replace a defective quorum server system.
Troubleshooting Your Cluster Replacing a Failed Quorum Server System Refer to the qs(1) man page for more details. 5. All nodes in all clusters that were using the old quorum server will connect to the new quorum server. Use the cmviewcl -v command from any cluster that is using the quorum server to verify that the nodes in that cluster have connected to the QS. 6.
Troubleshooting Your Cluster Troubleshooting Approaches Troubleshooting Approaches The following sections offer a few suggestions for troubleshooting by reviewing the state of the running system and by examining cluster status data, log files, and configuration files.
Troubleshooting Your Cluster Troubleshooting Approaches Reviewing Package IP Addresses The ifconfig command can be used to examine the LAN configuration. The command, if executed on ftsys9 after the halting of node ftsys10, shows that the package IP addresses are assigned to eth1:1 and eth1:2 along with the heartbeat IP address on eth1. 274 eth0 Link encap:Ethernet HWaddr 00:01:02:77:82:75 inet addr:15.13.169.106 Bcast:15.13.175.255 Mask:255.255.248.
Troubleshooting Your Cluster Troubleshooting Approaches Reviewing the System Log File Messages from the Cluster Manager and Package Manager are written to the system log file. The default location of the log file may vary according to Linux distribution; the Red Hat default is /var/log/messages. You can use a text editor, such as vi, or the more command to view the log file for historical information on your cluster.
Troubleshooting Your Cluster Troubleshooting Approaches Sample System Log Entries The following entries from the syslog file show a package that failed to run due to a problem in the pkg5_run script. You would look at the pkg5_run.log for details. Refer to “Understanding the Location of Serviceguard Files” on page 122 for references to $SGCONF. Dec Dec Dec Dec 14 14:33:48 star04 cmcld[2048]: Starting cluster management protocols.
Troubleshooting Your Cluster Troubleshooting Approaches Reviewing Object Manager Log Files The Serviceguard Object Manager daemon cmomd logs messages to the file /usr/local/cmom/cmomd.log on Red Hat and /var/log/cmmomcmomd.log on SuSE. You can review these messages using the cmreadlog command, for example: # /usr/local/cmom/bin/cmreadlog /usr/local/cmom/log/cmomd.
Troubleshooting Your Cluster Troubleshooting Approaches Using the cmquerycl and cmcheckconf Commands In addition, cmquerycl and cmcheckconf can be used to troubleshoot your cluster just as they were used to verify its configuration. The following example shows the commands used to verify the existing cluster configuration on ftsys9 and ftsys10: # cmquerycl -v -C $SGCONF/verify.ascii -n ftsys9 -n ftsys10 # cmcheckconf -v -C $SGCONF/verify.
Troubleshooting Your Cluster Solving Problems Solving Problems Problems with Serviceguard may be of several types. The following is a list of common categories of problem: • Serviceguard Command Hangs. • Cluster Re-formations. • System Administration Errors. • Package Control Script Hangs. • Package Movement Errors. • Node and Network Failures. • Quorum Server Messages.
Troubleshooting Your Cluster Solving Problems Cluster Re-formations Cluster re-formations may occur from time to time due to current cluster conditions. Some of the causes are as follows: • local switch on an Ethernet LAN if the switch takes longer than the cluster NODE_TIMEOUT value. To prevent this problem, you can increase the cluster NODE_TIMEOUT value. • excessive network traffic on heartbeat LANs. To prevent this, you can use dedicated heartbeat LANs, or LANs with less traffic on them.
Troubleshooting Your Cluster Solving Problems • fdisk -v /dev/sdx - to display information about a disk. Package Control Script Hangs or Failures When a RUN_SCRIPT_TIMEOUT or HALT_SCRIPT_TIMEOUT value is set, and the control script hangs, causing the timeout to be exceeded, Serviceguard kills the script and marks the package “Halted.” Similarly, when a package control script fails, Serviceguard kills the script and marks the package “Halted.
Troubleshooting Your Cluster Solving Problems where is the address indicated above and is the result of masking the with the mask found in the same line as the inet address in the ifconfig output. 3. Ensure that package volume groups are deactivated. First unmount any package logical volumes which are being used for file systems. This is determined by inspecting the output resulting from running the command df -l.
Troubleshooting Your Cluster Solving Problems Package Movement Errors These errors are similar to the system administration errors except they are caused specifically by errors in the package control script. The best way to prevent these errors is to test your package control script before putting your high availability application on line. Adding a “set -x” statement in the second line of your control script will give you details on where your script may be failing.
Troubleshooting Your Cluster Solving Problems Quorum Server Messages The coordinator node in Serviceguard sometimes sends a request to the quorum server to set the lock state. (This is different from a request to obtain the lock in tie-breaking.
Serviceguard Commands A Serviceguard Commands The following is an alphabetical list of commands used for ServiceGuard cluster configuration and maintenance. Man pages for these commands are available on your system after installation. Table A-1 Serviceguard Commands Command cmapplyconf Description Verify and apply ServiceGuard cluster configuration and package configuration files.
Serviceguard Commands Table A-1 Serviceguard Commands (Continued) Command cmapplyconf (continued) Description It is recommended that the user run the cmgetconf command to get either the cluster ASCII configuration file or package ASCII configuration file whenever changes to the existing configuration are required. Note that cmapplyconf will verify and distribute cluster configuration or package files. It will not cause the cluster daemon to start or removed from the cluster configuration.
Serviceguard Commands Table A-1 Serviceguard Commands (Continued) Command cmdeleteconf Description Delete either the cluster or the package configuration. cmdeleteconf deletes either the entire cluster configuration, including all its packages, or only the specified package configuration. If neither cluster_name nor package_name is specified, cmdeleteconf will delete the local cluster’s configuration and all its packages.
Serviceguard Commands Table A-1 Serviceguard Commands (Continued) Command cmhaltcl Description Halt a high availability cluster. cmhaltcl causes all nodes in a configured cluster to stop their cluster daemons, optionally halting all packages or applications in the process. This command will halt all the daemons on all currently running systems. If the user only wants to shutdown a subset of daemons, the cmhaltnode command should be used instead. cmhaltnode Halt a node in a high availability cluster.
Serviceguard Commands Table A-1 Serviceguard Commands (Continued) Command cmhaltserv Description Halt a service from the high availability package halt script. This is not a command line executable command, it runs only from within the package control script. cmhaltserv is used in the high availability package halt script to halt a service. If any part of package is marked down, the package halt script is executed as part of the recovery process.
Serviceguard Commands Table A-1 Serviceguard Commands (Continued) Command cmmodnet Description Add or remove an address from a high availability cluster. cmmodnet is used in the high availability package control scripts to add or remove an IP_address from the current network interface running the given subnet_name. Extreme caution should be exercised when executing this command outside the context of the package control script.
Serviceguard Commands Table A-1 Serviceguard Commands (Continued) Command cmquerycl Description Query cluster or node configuration information. cmquerycl searches all specified nodes for cluster configuration, cluster lock, and Logical Volume Manager (LVM) information. Cluster configuration information includes network information such as LAN interface, IP addresses, bridged networks and possible heartbeat networks.
Serviceguard Commands Table A-1 Serviceguard Commands (Continued) Command cmrunnode Description Run a node in a high availability cluster. cmrunnode causes a node to start its cluster daemon to join the existing cluster Starting a node will not cause any active packages to be moved to the new node. However, if a package is DOWN, has its switching enabled, and is able to run on the new node, that package will automatically run there. cmrunpkg Run a high availability package.
Serviceguard Commands Table A-1 Serviceguard Commands (Continued) Command cmrunserv Description Run a service from the high availability package run script. This is not a command line executable command, it runs only from within the package control script. cmrunserv is used in the high availability package run script to run a service. If the service process dies, cmrunserv updates the status of the service to down.
Serviceguard Commands Table A-1 Serviceguard Commands (Continued) Command cmscancl Description Gather system configuration information from nodes with ServiceGuard installed. cmscancl is a configuration report and diagnostic tool which gathers system software and hardware configuration information from a list of nodes, or from all the nodes in a cluster.
Serviceguard Commands Table A-1 Serviceguard Commands (Continued) Command cmviewconf Description View Serviceguard or ServiceGuard cluster configuration information. cmviewconf collects and displays the cluster configuration information, in ASCII format, from the binary configuration file for an existing cluster. Optionally, the output can be written to a file. This command can be used as a troubleshooting tool to identify the configuration of a cluster.
Serviceguard Commands 296 Appendix A
Designing Highly Available Cluster Applications B Designing Highly Available Cluster Applications This appendix describes how to create or port applications for high availability, with emphasis on the following topics: • Automating Application Operation • Controlling the Speed of Application Failover • Designing Applications to Run on Multiple Systems • Restoring Client Connections • Handling Application Failures • Minimizing Planned Downtime Designing for high availability means reducing the
Designing Highly Available Cluster Applications Automating Application Operation Automating Application Operation Can the application be started and stopped automatically or does it require operator intervention? This section describes how to automate application operations to avoid the need for user intervention. One of the first rules of high availability is to avoid manual intervention.
Designing Highly Available Cluster Applications Automating Application Operation Define Application Startup and Shutdown Applications must be restartable without manual intervention. If the application requires a switch to be flipped on a piece of hardware, then automated restart is impossible. Procedures for application startup, shutdown and monitoring must be created so that the HA software can perform these functions automatically.
Designing Highly Available Cluster Applications Controlling the Speed of Application Failover Controlling the Speed of Application Failover What steps can be taken to ensure the fastest failover? If a failure does occur causing the application to be moved (failed over) to another node, there are many things the application can do to reduce the amount of time it takes to get the application back up and running.
Designing Highly Available Cluster Applications Controlling the Speed of Application Failover Evaluate the Use of a Journaled Filesystem (JFS) If a file system must be used, a JFS offers significantly faster file system recovery than an HFS. However, performance of the JFS may vary with the application. An example of an appropriate JFS is the Reiser FS or ext3. Minimize Data Loss Minimize the amount of data that might be lost at the time of an unplanned outage.
Designing Highly Available Cluster Applications Controlling the Speed of Application Failover Keep Logs Small Some databases permit logs to be buffered in memory to increase online performance. Of course, when a failure occurs, any in-flight transaction will be lost. However, minimizing the size of this in-memory log will reduce the amount of completed transaction data that would be lost in case of failure.
Designing Highly Available Cluster Applications Controlling the Speed of Application Failover Another example is an application where a clerk is entering data about a new employee. Suppose this application requires that employee numbers be unique, and that after the name and number of the new employee is entered, a failure occurs.
Designing Highly Available Cluster Applications Controlling the Speed of Application Failover Design for Multiple Servers If you use multiple active servers, multiple service points can provide relatively transparent service to a client. However, this capability requires that the client be smart enough to have knowledge about the multiple servers and the priority for addressing them. It also requires access to the data of the failed server or replicated data.
Designing Highly Available Cluster Applications Designing Applications to Run on Multiple Systems Designing Applications to Run on Multiple Systems If an application can be failed to a backup node, how will it work on that different system? The previous sections discussed methods to ensure that an application can be automatically restarted. This section will discuss some ways to ensure the application can run on multiple systems.
Designing Highly Available Cluster Applications Designing Applications to Run on Multiple Systems Each application or package should be given a unique name as well as a relocatable IP address. Following this rule separates the application from the system on which it runs, thus removing the need for user knowledge of which system the application runs on. It also makes it easier to move the application among different systems in a cluster for for load balancing or other reasons.
Designing Highly Available Cluster Applications Designing Applications to Run on Multiple Systems Avoid Using SPU IDs or MAC Addresses Design the application so that it does not rely on the SPU ID or MAC (link-level) addresses. The SPU ID is a unique hardware ID contained in non-volatile memory, which cannot be changed. A MAC address (also known as a NIC id) is a link-specific address associated with the LAN hardware.
Designing Highly Available Cluster Applications Designing Applications to Run on Multiple Systems Assign Unique Names to Applications A unique name should be assigned to each application. This name should then be configured in DNS so that the name can be used as input to gethostbyname(3), as described in the following discussion. Use DNS DNS provides an API which can be used to map hostnames to IP addresses and vice versa.
Designing Highly Available Cluster Applications Designing Applications to Run on Multiple Systems Use uname(2) With Care Related to the hostname issue discussed in the previous section is the application's use of uname(2), which returns the official system name. The system name is unique to a given system whatever the number of LAN cards in the system. By convention, the uname and hostname are the same, but they do not have to be.
Designing Highly Available Cluster Applications Designing Applications to Run on Multiple Systems Bind to a Fixed Port When binding a socket, a port address can be specified or one can be assigned dynamically. One issue with binding to random ports is that a different port may be assigned if the application is later restarted on another cluster node. This may be confusing to clients accessing the application.
Designing Highly Available Cluster Applications Designing Applications to Run on Multiple Systems With UDP datagram sockets, however, there is a problem. The client may connect to multiple servers utilizing the relocatable IP address and sort out the replies based on the source IP address in the server’s response message. However, the source IP address given in this response will be the stationary IP address rather than the relocatable application IP address.
Designing Highly Available Cluster Applications Designing Applications to Run on Multiple Systems Give Each Application its Own Volume Group Use separate volume groups for each application that uses data. If the application doesn't use disk, it is not necessary to assign it a separate volume group. A volume group (group of disks) is the unit of storage that can move between nodes. The greatest flexibility for load balancing exists when each application is confined to its own volume group, i.e.
Designing Highly Available Cluster Applications Designing Applications to Run on Multiple Systems Avoid File Locking In an NFS environment, applications should avoid using file-locking mechanisms, where the file to be locked is on an NFS Server. File locking should be avoided in an application both on local and remote systems. If local file locking is employed and the system fails, the system acting as the backup system will not have any knowledge of the locks maintained by the failed system.
Designing Highly Available Cluster Applications Restoring Client Connections Restoring Client Connections How does a client reconnect to the server after a failure? It is important to write client applications to specifically differentiate between the loss of a connection to the server and other application-oriented errors that might be returned. The application should take special action in case of connection loss.
Designing Highly Available Cluster Applications Restoring Client Connections the retry to the current server should continue for the amount of time it takes to restart the server locally. This will keep the client from having to switch to the second server in the event of a application failure. • Use a transaction processing monitor or message queueing software to increase robustness.
Designing Highly Available Cluster Applications Handling Application Failures Handling Application Failures What happens if part or all of an application fails? All of the preceding sections have assumed the failure in question was not a failure of the application, but of another component of the cluster. This section deals specifically with application problems.
Designing Highly Available Cluster Applications Handling Application Failures Be Able to Monitor Applications All components in a system, including applications, should be able to be monitored for their health. A monitor might be as simple as a display command or as complicated as a SQL query. There must be a way to ensure that the application is behaving correctly.
Designing Highly Available Cluster Applications Minimizing Planned Downtime Minimizing Planned Downtime Planned downtime (as opposed to unplanned downtime) is scheduled; examples include backups, systems upgrades to new operating system revisions, or hardware replacements. For planned downtime, application designers should consider: • Reducing the time needed for application upgrades/patches.
Designing Highly Available Cluster Applications Minimizing Planned Downtime Reducing Time Needed for Application Upgrades and Patches Once a year or so, a new revision of an application is released. How long does it take for the end-user to upgrade to this new revision? This answer is the amount of planned downtime a user must take to upgrade their application. The following guidelines reduce this time. Provide for Rolling Upgrades Provide for a “rolling upgrade” in a client/server environment.
Designing Highly Available Cluster Applications Minimizing Planned Downtime Do Not Change the Data Layout Between Releases Migration of the data to a new format can be very time intensive. It also almost guarantees that rolling upgrade will not be possible. For example, if a database is running on the first node, ideally, the second node could be upgraded to the new revision of the database.
Integrating HA Applications with Serviceguard C Integrating HA Applications with Serviceguard The following is a summary of the steps you should follow to integrate an application into the Serviceguard environment: 1. Read the rest of this book, including the chapters on cluster and package configuration, and the appendix “Designing Highly Available Cluster Applications.” 2.
Integrating HA Applications with Serviceguard Checklist for Integrating HA Applications Checklist for Integrating HA Applications This section contains a checklist for integrating HA applications in both single and multiple systems. Defining Baseline Application Behavior on a Single System 1. Define a baseline behavior for the application on a standalone system: • Install the application, database, and other required resources on one of the systems.
Integrating HA Applications with Serviceguard Checklist for Integrating HA Applications Integrating HA Applications in Multiple Systems 1. Install the application on a second system. • Create the LVM infrastructure on the second system. • Add the appropriate users to the system. • Install the appropriate executables. • With the application not running on the first system, try to bring it up on the second system. You might use the script you created in the step above.
Integrating HA Applications with Serviceguard Checklist for Integrating HA Applications Testing the Cluster 1. Test the cluster: • Have clients connect. • Provide a normal system load. • Halt the package on the first node and move it to the second node: # cmhaltpkg pkg1 # cmrunpkg -n node2 pkg1 # cmmodpkg -e pkg1 • Move it back. # cmhaltpkg pkg1 # cmrunpkg -n node1 pkg1 # cmmodpkg -e pkg1 • Fail one of the systems. For example, turn off the power on node 1.
Blank Planning Worksheets D Blank Planning Worksheets This appendix reprints blank versions of the planning worksheets described in the “Planning” chapter. You can duplicate any of these worksheets that you find useful and fill them in as a part of the planning process.
Blank Planning Worksheets Hardware Worksheet Hardware Worksheet ============================================================================= SPU Information: Host Name ____________________ Server Series____________ Memory Capacity ____________ Number of I/O Slots ____________ ============================================================================= LAN Information: Name of Master _________ Name of Node IP Traffic Interface __________ Addr________________ Type ________ Name of Name of Node IP Traff
Blank Planning Worksheets Power Supply Worksheet Power Supply Worksheet ============================================================================ SPU Power: Host Name ____________________ Power Supply _____________________ Host Name ____________________ Power Supply _____________________ ============================================================================ Disk Power: Disk Unit __________________________ Power Supply _______________________ Disk Unit __________________________ Power Supply
Blank Planning Worksheets Quorum Server Worksheet Quorum Server Worksheet Quorum Server Data: ============================================================================== QS Hostname: _________________IP Address: ______________________ OR Cluster Name: _________________ Package Name: ____________ Package IP Address: ___________________ Hostname Given to Package by Network Administrator: _________________ ============================================================================== Quorum Services are Pr
Blank Planning Worksheets Volume Group and Physical Volume Worksheet Volume Group and Physical Volume Worksheet ============================================================================== Volume Group Name: ___________________________________ Physical Volume Name: _________________ Physical Volume Name: _________________ Physical Volume Name: _________________ ============================================================================= Volume Group Name: ___________________________________ Physical Vol
Blank Planning Worksheets Cluster Configuration Worksheet Cluster Configuration Worksheet =============================================================================== Name and Nodes: =============================================================================== Cluster Name: ______________________________ Node Names: ________________________________________________ Maximum Configured Packages: ______________ =============================================================================== Cluster Lock Da
Blank Planning Worksheets Cluster Configuration Worksheet Access Policies User: ________ Host: ________ Role: ________ User: _________ Host: _________ Role: __________ Appendix D 331
Blank Planning Worksheets Package Configuration Worksheet Package Configuration Worksheet ============================================================================= Package Configuration File Data: ============================================================================= Package Name: ____________________________ Failover Policy: ________________ Failback Policy: ________________ Primary Node: ____________________________________ First Failover Node:________________________________ Additional Fail
Blank Planning Worksheets Package Control Script Worksheet Package Control Script Worksheet PACKAGE CONTROL SCRIPT WORKSHEET Page ___ of ___ ================================================================================ Package Control Script Data: ================================================================================ PATH______________________________________________________________ VGCHANGE_________________________________ VG[0]__________________LV[0]______________________FS[0]_______________
Blank Planning Worksheets Package Control Script Worksheet 334 Appendix D
Index A active node, 20 adding a package to a running cluster, 256 adding nodes to a running cluster, 244 adding packages on a running cluster, 203 administration adding nodes to a running cluster, 244 cluster and package states, 213 halting a package, 252 halting the entire cluster, 246 moving a package, 253 of packages and services, 251 of the cluster, 242 reconfiguring a package while the cluster is running, 255 reconfiguring a package with the cluster offline, 254 reconfiguring the cluster, 247 removing
Index cluster coordinator defined, 38 cluster lock 4 or more nodes, 43 and power supplies, 30 storing configuration data, 160 two nodes, 41, 42 use in re-forming a cluster, 41, 42 cluster manager automatic restart of cluster, 40 blank planning worksheet, 330 cluster node parameter, 101, 102 defined, 38 dynamic re-formation, 40 heartbeat interval parameter, 103 heartbeat subnet parameter, 102 initial configuration of the cluster, 38 main functions, 38 maximum configured packages parameter, 105 monitored non
Index disk layout planning, 98 disk logical units hardware planning, 93 disks in Serviceguard, 28 replacing, 268 supported types in Serviceguard, 28 distributing the cluster and package configuration, 204, 205 down time minimizing planned, 318 dynamic cluster re-formation, 40 E enclosure for disks replacing a faulty mechanism , 268 Ethernet redundant configuration, 27 expanding the cluster planning ahead, 88 expansion planning for, 109 F failback policy package configuration file parameter, 110 used by pack
Index in sample ASCII package configuration file, 182 parameter in package configuration, 112 halting a cluster, 246 halting a package, 252 halting the entire cluster, 246 handling application failures, 316 hardware monitoring, 267 power supplies, 30 hardware failures response to, 82 hardware planning blank planning worksheet, 325 Disk I/O Bus Type, 93 disk I/O information for shared disks, 93 host IP address, 90, 97 host name, 90 I/O bus addresses, 93 I/O slot numbers, 93 LAN interface name, 90, 97 LAN tr
Index traffic type, 91 link-level addresses, 307 load sharing with IP addresses, 69 local switching, 70 lock cluster locks and power supplies, 30 use of the cluster lock, 42 use of the cluster lock disk, 41 lock volume group, reconfiguring, 247 logical volume parameter in package control script, 115 logical volumes creating the infrastructure, 147 planning, 98 LV, 115 in sample package control script, 191 LVM commands for cluster use, 147 creating file systems, 107 creating logical volumes, 107 creating vol
Index in Serviceguard cluster, 18 IP addresses, 68 node types active, 20 primary, 20 NODE_FAIL_FAST_ENABLED in sample ASCII package configuration file, 182 parameter in package configuration, 111 NODE_NAME in sample ASCII package configuration file, 182 parameter in cluster manager configuration, 101, 102 NODE_TIMEOUT (heartbeat timeout) in sample configuration file, 165 NODE_TIMEOUT (node timeout) parameter in cluster manager configuration, 104 nodetypes primary, 20 O Object Manager, 277 optimizing packa
Index package manager blank planning worksheet, 332, 333 testing, 264 package name parameter in package configuration, 109 package switching behavior changing, 257 PACKAGE_NAME in sample ASCII package configuration file, worksheet, 96, 97 , 328 power supplies blank planning worksheet, 327 power supply and cluster lock, 30 UPS, 30 primary LAN interfaces defined, 27 primary node, 20 182 packages deciding where and when to run, 46 parameters for failover, 55 parameters for cluster manager initial configurat
Index removing nodes from operation in a running cluster, 245 removing packages on a running cluster, 203 removing Serviceguard from a system, 262 replacing disks, 268 resources disks, 28 responses to cluster events, 260 to package and service failures, 83 responses to failures, 81 responses to hardware failures, 82 restart automatic restart of cluster, 40 following failure, 83 SERVICE_RESTART variable in package control script, 118 restartable transactions, 302 restarting the cluster automatically, 247 res
Index shared disks planning, 93 shutdown and startup defined for applications, 299 single point of failure avoiding, 18 single-node operation, 176, 261 SNA applications, 312 software failure Serviceguard behavior, 26 software planning LVM, 98 solving problems, 279 SPU information planning, 90 standby LAN interfaces defined, 27 starting a package, 251 startup and shutdown defined for applications, 299 startup of cluster manual, 39 state of cluster and package, 213 stationary IP addresses, 68 STATIONARY_IP pa
Index VGChange, 114 volume group for cluster lock, 41, 42 planning, 98 volume group and physical volume planning, 98 Volume groups in control script, 114 W What is Serviceguard?, 18 worksheet blanks, 325 cluster configuration, 106, 331 hardware configuration, 95, 325 package configuration, 332, 333 package control script, 118 power supply configuration, 96, 97, 327, 328 use in planning, 85 344
