Managing HP Serviceguard for Linux, Tenth Edition HP Part Number: B9903-90073 Published: November 2012
Legal Notices The information in this document is subject to change without notice. Hewlett-Packard makes no warranty of any kind with regard to this manual, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. Hewlett-Packard shall not be held liable for errors contained herein or direct, indirect, special, incidental or consequential damages in connection with the furnishing, performance, or use of this material. Warranty.
Contents Printing History .......................................................................................................................16 Preface...................................................................................................................................17 1 Serviceguard for Linux at a Glance..........................................................................................19 What is Serviceguard for Linux? .............................................................
Cluster WBEM Agent Daemon: cmwbemd.......................................................37 Proxy Daemon: cmproxyd..............................................................................37 How the Cluster Manager Works ............................................................................37 Configuration of the Cluster ................................................................................37 Heartbeat Messages .............................................................................
Bonding of LAN Interfaces .................................................................................71 Bonding for Load Balancing................................................................................74 Monitoring LAN Interfaces and Detecting Failure: Link Level.....................................75 Monitoring LAN Interfaces and Detecting Failure: IP Level.......................................75 Reasons To Use IP Monitoring...................................................................
Power Supply Planning ...........................................................................................95 Power Supply Configuration Worksheet ...............................................................96 Cluster Lock Planning..............................................................................................96 Cluster Lock Requirements...................................................................................96 Planning for Expansion..........................................
Configuring Weights and Capacities.............................................................139 Simple Method...........................................................................................139 Example 1.............................................................................................139 Points to Keep in Mind............................................................................140 Comprehensive Method.............................................................................
Setting Up and Running the Quorum Server........................................................169 Creating the Logical Volume Infrastructure ..........................................................169 Using the Generic Resources Disk Monitor......................................................170 Displaying Disk Information..........................................................................170 Creating Partitions.....................................................................................
Deleting the Cluster Configuration .....................................................................198 6 Configuring Packages and Their Services ...............................................................................199 Choosing Package Modules...................................................................................200 Types of Package: Failover, Multi-Node, System Multi-Node...................................200 Package Modules and Parameters.........................................
File system parameters.................................................................................220 concurrent_fsck_operations...........................................................................220 concurrent_mount_and_umount_operations.....................................................220 fs_mount_retry_count...................................................................................221 fs_umount_retry_count ............................................................................
Status After Package Switching is Enabled......................................................242 Status After Halting a Node.........................................................................242 Viewing Information about Unowned Packages...............................................243 Managing the Cluster and Nodes .........................................................................243 Starting the Cluster When all Nodes are Down....................................................
Reconfiguring a Halted Cluster .........................................................................264 Reconfiguring a Running Cluster........................................................................265 Adding Nodes to the Configuration While the Cluster is Running .....................265 Removing Nodes from the Cluster while the Cluster Is Running .........................266 Changing the Cluster Networking Configuration while the Cluster Is Running...........267 What You Can Do...............
Removing Serviceguard from a System....................................................................288 8 Troubleshooting Your Cluster.................................................................................................289 Testing Cluster Operation .....................................................................................289 Testing the Package Manager ...........................................................................289 Testing the Cluster Manager .......................
Minimize Data Loss .........................................................................................308 Minimize the Use and Amount of Memory-Based Data ....................................309 Keep Logs Small ........................................................................................309 Eliminate Need for Local Data .....................................................................309 Use Restartable Transactions .......................................................................
Cluster Configuration Worksheet ...........................................................................325 Package Configuration Worksheet .........................................................................326 Package Control Script Worksheet (Legacy)..............................................................327 D IPv6 Network Support..........................................................................................................328 IPv6 Address Types..................................
Printing History Table 1 Printing Date Part Number Edition November 2001 B9903-90005 First November 2002 B9903-90012 First December 2002 B9903-90012 Second November 2003 B9903-90033 Third February 2005 B9903-90043 Fourth June 2005 B9903-90046 Fifth August 2006 B9903-90050 Sixth July 2007 B9903-90054 Seventh March 2008 B9903-90060 Eighth April 2009 B9903-90068 Ninth July 2009 B9903-90073 Tenth The last printing date and part number indicate the current edition, which appli
Preface This guide describes how to configure and manage Serviceguard for Linux on HP ProLiant and HP Integrity servers under the Linux operating system. It is intended for experienced Linux system administrators. (For Linux system administration tasks that are not specific to Serviceguard, use the system administration documentation and manpages for your distribution of Linux.) The contents are as follows: • Chapter 1 (page 19) describes a Serviceguard cluster and provides a roadmap for using this guide.
Related Publications The following documents contain additional useful information: • HP Serviceguard for Linux Version A.11.19 Release Notes • HP Serviceguard Quorum Server Version A.04.00 Release Notes • Clusters for High Availability: a Primer of HP Solutions. Second Edition. HP Press, 2001 (ISBN 0-13-089355-2) Use the following URL to access HP’s high availability documentation web page: http://docs.hp.
1 Serviceguard for Linux at a Glance This chapter introduces Serviceguard for Linux and shows where to find different kinds of information in this book. It includes the following topics: • What is Serviceguard for Linux? • Using Serviceguard Manager (page 22) • Configuration Roadmap (page 23) If you are ready to start setting up Serviceguard clusters, skip ahead to Chapter 4 (page 91). Specific steps for setup are in Chapter 5 (page 156).
Figure 1 Typical Cluster Configuration In the figure, node 1 (one of two SPU's) is running package A, and node 2 is running package B. Each package has a separate group of disks associated with it, containing data needed by the package's applications, and a copy of the data. Note that both nodes are physically connected to disk arrays. However, only one node at a time may access the data for a given group of disks.
Failover Under normal conditions, a fully operating Serviceguard cluster simply monitors the health of the cluster's components while the packages are running on individual nodes. Any host system running in the Serviceguard cluster is called an active node. When you create the package, you specify a primary node and one or more adoptive nodes.When a node or its network communications fails, Serviceguard can transfer control of the package to the next available adoptive node.
Serviceguard is designed to work in conjunction with other high availability products, such as disk arrays, which use various RAID levels for data protection; and HP-supported uninterruptible power supplies (UPS), which eliminate failures related to power outage. HP recommends these products; in conjunction with Serviceguard they provide the highest degree of availability.
Configuring Clusters with Serviceguard Manager You can configure clusters and packages in Serviceguard Manager. You must have root (UID=0) access to the cluster nodes. Starting Serviceguard Manager Follow the directions in the Release Notes for your version of Serviceguard for Linux to start the Serviceguard Manager. Then select a cluster, node, or package, and use the drop-down menus below the “Serviceguard Manager” banner to navigate to the task you need to do.
2 Understanding Hardware Configurations for Serviceguard for Linux This chapter gives a broad overview of how the server hardware components operate with Serviceguard for Linux. The following topics are presented: • Redundant Cluster Components • Redundant Network Components (page 25) • Redundant Disk Storage (page 29) • Redundant Power Supplies (page 31) Refer to the next chapter for information about Serviceguard software components.
Redundant Network Components To eliminate single points of failure for networking, each subnet accessed by a cluster node is required to have redundant network interfaces. Redundant cables are also needed to protect against cable failures. Each interface card is connected to a different cable and hub or switch. Network interfaces are allowed to share IP addresses through a process known as channel bonding.
unless those IP addresses themselves will be immediately configured into the cluster as stationary IP addresses. CAUTION: If you configure any address other than a stationary IP address on a Serviceguard network interface, it could collide with a relocatable package IP address assigned by Serviceguard. See “Stationary and Relocatable IP Addresses and Monitored Subnets” (page 68).
Figure 4 Redundant LANs In Linux configurations, the use of symmetrical LAN configurations is strongly recommended, with the use of redundant hubs or switches to connect Ethernet segments. The software bonding configuration should be identical on each node, with the active interfaces connected to the same hub or switch. Cross-Subnet Configurations As of Serviceguard A.11.
Configuration Tasks Cluster and package configuration tasks are affected as follows: • You must use the -w full option to cmquerycl to discover actual or potential nodes and subnets across routers.
• Deploying applications in this environment requires careful consideration; see “Implications for Application Deployment” (page 152). • cmrunnode will fail if the “hostname LAN” is down on the node in question. (“Hostname LAN” refers to the public LAN on which the IP address that the node’s hostname resolves to is configured).
NOTE: As of release A.11.16.07, Serviceguard for Linux provides functionality similar to HP-UX exclusive activation. This feature is based on LVM2 hosttags, and is available only for Linux distributions that officially support LVM2. All of the disks in the volume group owned by a package must be connected to the original node and to all possible adoptive nodes for that package.
Sample Disk Configurations Figure 5 shows a two node cluster. Each node has one root disk which is mirrored and one package for which it is the primary node. Resources have been allocated to each node so that each node can adopt the package from the other node. Each package has one disk volume group assigned to it and the logical volumes in that volume group are mirrored.
3 Understanding Serviceguard Software Components This chapter gives a broad overview of how the Serviceguard software components work.
Figure 6 Serviceguard Software Components on Linux Serviceguard Daemons Serviceguard for Linux uses the following daemons: • cmclconfd—configuration daemon • cmcld—cluster daemon • cmnetd—Network Manager daemon • cmlogd—cluster system log daemon • cmdisklockd—cluster lock LUN daemon • cmomd—Cluster Object Manager daemon • cmresourced—Serviceguard Generic Resource Assistant Daemon • cmserviced—Service Assistant daemon • qs—Quorum Server daemon • cmlockd—utility daemon • cmsnmpd—cluster
• cmwbemd—WBEM daemon • cmproxyd—proxy daemon Each of these daemons logs to the Linux system logging files. The quorum server daemon logs to the user specified log file, such as, /usr/local/qs/log/qs.log file on Red Hat or /var/log/qs/sq.log on SUSE and cmomd logs to /usr/local/cmom/log/ cmomd.log on Red Hat or /var/log/cmom/log/cmomd.log on SUSE. NOTE: The file cmcluster.
NOTE: Two of the central components of Serviceguard—Package Manager, and Cluster Manager—run as parts of the cmcld daemon. This daemon runs at priority 94 and is in the SCHED_RR class. No other process is allowed a higher real-time priority. Log Daemon: cmlogd cmlogd is used by cmcld to write messages to the system log file. Any message written to the system log by cmcld it written through cmlogd. This is to prevent any delays in writing to syslog from impacting the timing of cmcld.
the package to halt and moves the package to an available alternate node. The path for this daemon is: $SGLBIN/cmserviced. Generic Resource Assistant Daemon: cmresourced This daemon is responsible to set and get the status/value of generic resources configured as part of the package and influence the availability of the package based on the availability of the resource. Generic resources allows custom defined monitors to be integrated.
Cluster SNMP Agent Daemon: cmsnmpd This daemon collaborates with the SNMP Master Agent to provide instrumentation for the cluster Management Information Base (MIB). The SNMP Master Agent and the cmsnmpd provide notification (traps) for cluster-related events. For example, a trap is sent when the cluster configuration changes, or when a Serviceguard package has failed. To configure the agent to send traps to one or more specific destinations, add the trap destinations to /etc/snmp/snmptrapd.
networking parameters for the cluster heartbeat, cluster lock information, and timing parameters (discussed in detail in Chapter 4 (page 91) ). Cluster parameters are entered by editing the cluster configuration file (see “Configuring the Cluster” (page 179)). The parameters you enter are used to build a binary configuration file which is propagated to all nodes in the cluster. This binary cluster configuration file must be the same on all the nodes in the cluster.
Manual Startup of Entire Cluster A manual startup forms a cluster out of all the nodes in the cluster configuration. Manual startup is normally done the first time you bring up the cluster, after cluster-wide maintenance or upgrade, or after reconfiguration. Before startup, the same binary cluster configuration file must exist on all nodes in the cluster. The system administrator starts the cluster with the cmruncl command issued from one node.
Typically, re-formation results in a cluster with a different composition. The new cluster may contain fewer or more nodes than in the previous incarnation of the cluster. Cluster Quorum to Prevent Split-Brain Syndrome In general, the algorithm for cluster re-formation requires a cluster quorum of a strict majority (that is, more than 50%) of the nodes previously running.
NOTE: The lock LUN is dedicated for use as the cluster lock, and, in addition, HP recommends that this LUN comprise the entire disk; that is, the partition should take up the entire disk. The complete path name of the lock LUN is identified in the cluster configuration file. The operation of the lock LUN is shown in Figure 7. Figure 7 Lock LUN Operation Serviceguard periodically checks the health of the lock LUN and writes messages to the syslog file if the disk fails the health check.
Figure 8 Quorum Server Operation A quorum server can provide quorum services for multiple clusters. Figure 9 illustrates quorum server use across four clusters.
IMPORTANT: For more information about the quorum server, see the latest version of the HP Serviceguard Quorum Server release notes at http://docs.hp.com -> High Availability -> Quorum Server. No Cluster Lock Normally, you should not configure a cluster of three or fewer nodes without a cluster lock. In two-node clusters, a cluster lock is required.
IMPORTANT: During Step 1, while the nodes are using a strict majority quorum, node failures can cause the cluster to go down unexpectedly if the cluster has been using a quorum device before the configuration change. For example, suppose you change the quorum server polling interval while a two-node cluster is running. If a node fails during Step 1, the cluster will lose quorum and go down, because a strict majority of prior cluster members (two out of two in this case) is required.
Failover Packages A failover package starts up on an appropriate node (see node_name (page 206)) when the cluster starts. In the case of a service, network, or other resource or dependency failure, package failover takes place. A package failover involves both halting the existing package and starting the new instance of the package on a new node. Failover is shown in the following figure: Figure 10 Package Moving During Failover Configuring Failover Packages You configure each package separately.
managed by a master control script that is installed with Serviceguard; see Chapter 6: “Configuring Packages and Their Services ” (page 199), for instructions for creating modular packages. Deciding When and Where to Run and Halt Failover Packages The package configuration file assigns a name to the package and includes a list of the nodes on which the package can run. Failover packages list the nodes in order of priority (i.e., the first node in the list is the highest priority node).
The switching of relocatable IP addresses is shown in the figures that follow. Users connect to each node with the IP address of the package they wish to use. Each node has a stationary IP address associated with it, and each package has an IP address associated with it. Figure 11 Before Package Switching In Figure 12, node1 has failed and pkg1 has been transferred to node2. pkg1's IP address was transferred to node2 along with the package. pkg1 continues to be available and is now running on node2.
Figure 12 After Package Switching Failover Policy The Package Manager selects a node for a failover package to run on based on the priority list included in the package configuration file together with the failover_policy parameter, also in the configuration file. The failover policy governs how the package manager selects which node to run a package on when a specific node has not been identified and the package needs to be started.
If you use min_package_node as the value for the failover policy, the package will start up on the node that is currently running the fewest other packages. (Note that this does not mean the lightest load; the only thing that is checked is the number of packages currently running on the node.) Automatic Rotating Standby Using the min_package_node failover policy, it is possible to configure a cluster that lets you use one node as an automatic rotating standby node for the cluster.
Figure 13 Rotating Standby Configuration before Failover If a failure occurs, the failing package would fail over to the node containing fewest running packages: 50 Understanding Serviceguard Software Components
Figure 14 Rotating Standby Configuration after Failover NOTE: Under the min_package_node policy, when node2 is repaired and brought back into the cluster, it will then be running the fewest packages, and thus will become the new standby node. If these packages had been set up using the configured_node failover policy, they would start initially as in Figure 13, but the failure of node2 would cause the package to start on node3, as shown in Figure 15.
Figure 15 configured_node Policy Packages after Failover If you use configured_node as the failover policy, the package will start up on the highest-priority eligible node in its node list. When a failover occurs, the package will move to the next eligible node in the list, in the configured order of priority.
Figure 16 Automatic Failback Configuration before Failover Table 3 Node Lists in Sample Cluster Package Name NODE_NAME List FAILOVER POLICY FAILBACK POLICY pkgA node1, node4 configured_node automatic pkgB node2, node4 configured_node automatic pkgC node3, node4 configured_node automatic node1 panics, and after the cluster reforms, pkgA starts running on node4: How the Package Manager Works 53
Figure 17 Automatic Failback Configuration After Failover After rebooting, node1 rejoins the cluster. At that point, pkgA will be automatically stopped on node4 and restarted on node1.
Figure 18 Automatic Failback Configuration After Restart of node1 NOTE: Setting the failback_policy to automatic can result in a package failback and application outage during a critical production period. If you are using automatic failback, you may want to wait to add the package’s primary node back into the cluster until you can allow the package to be taken out of service temporarily while it switches back to the primary node.
provides integration of custom, user-defined monitors in Serviceguard by configuring generic resources as part of package configuration. Generic resources can also be used as an alternative approach to the Event Monitoring Service Hardware Monitors. With generic resources different kind of monitoring mechanisms, such as EMS monitors, SFM/WBEM monitors, Custom monitors, can be used and these can co-exist in a single package.
monitoring script fails or is halted, then all the other packages that are using this common resource also fail. See the recommendation from HP and an example under “Launching Monitoring Scripts” (page 341). Generic resources can be of two types - Simple and Extended. A given generic resource is considered to be a simple generic resource when the up criteria parameter is not specified. • For a simple resource, the monitoring mechanism is based on the status of the resource.
was made. For example, the default for failover_policy is now configured_node and the default for failback_policy is now manual. For full details of the current parameters and their default values, see Chapter 6: “Configuring Packages and Their Services ” (page 199), and the package configuration file template itself. How Packages Run Packages are the means by which Serviceguard starts and halts configured applications. Failover packages are also units of failover behavior in Serviceguard.
A failover package starts on the first available node in its configuration file; by default, it fails over to the next available one in the list. Note that you do not necessarily have to use a cmrunpkg command to restart a failed failover package; in many cases, the best way is to enable package and/or node switching with the cmmodpkg command. When you create the package, you indicate the list of nodes on which it is allowed to run. System multi-node packages must list all cluster nodes in their cluster.
Figure 19 Legacy Package Time Line Showing Important Events The following are the most important moments in a package’s life: 1. Before the control script starts. (For modular packages, this is the master control script.) 2. During run script execution. (For modular packages, during control script execution to start the package.) 3. While services are running 4. If there is a generic resource configured and it fails, then the package will be halted. 5.
available on the chosen node. One resource is the subnet that is monitored for the package. If the subnet is not available, the package cannot start on this node. Another type of resource is a dependency on another package. If monitoring shows a value for a configured resource that is outside the permitted range, the package cannot start. Once a node is selected, a check is then done to make sure the node allows the package to start on it.
Figure 20 Legacy Package Time Line At any step along the way, an error will result in the script exiting abnormally (with an exit code of 1). For example, if a package service is unable to be started, the control script will exit with an error. NOTE: This diagram is specific to legacy packages. Modular packages also run external scripts and “pre-scripts” as explained above.
NOTE: After the package run script has finished its work, it exits, which means that the script is no longer executing once the package is running normally. After the script exits, the PIDs of the services started by the script are monitored by the package manager directly. If the service dies, the package manager will then run the package halt script or, if service_fail_fast_enabled (page 217) is set to yes, it will halt the node on which the package is running.
NOTE: If you set restarts and also set service_fail_fast_enabled to yes, the failfast will take place after restart attempts have failed. It does not make sense to set service_restart to “-R” for a service and also set service_fail_fast_enabled to yes.
graceful shutdown of the package that is followed by disabling automatic package startup (see “auto_run” (page 206)). You cannot halt a multi-node or system multi-node package unless all the packages that have a configured dependency on it are down. Use cmviewcl to check the status of dependents. For example, if pkg1 and pkg2 depend on PKGa, both pkg1 and pkg2 must be halted before you can halt PKGa.
Figure 21 Legacy Package Time Line for Halt Script Execution At any step along the way, an error will result in the script exiting abnormally (with an exit code of 1). If the halt script execution is not complete before the time specified in the halt_script_timeout (page 208), the package manager will kill the script. During halt script execution, messages are written to a log file.
Normal and Abnormal Exits from the Halt Script The package’s ability to move to other nodes is affected by the exit conditions on leaving the halt script. The following are the possible exit codes: • 0—normal exit. The package halted normally, so all services are down on this node. • 1—abnormal exit, also known as no_restart exit. The package did not halt normally. Services are killed, and the package is disabled globally. It is not disabled on the current node, however.
Table 4 Error Conditions and Package Movement for Failover Packages (continued) Package Error Condition Results Error or Exit Code Node Failfast Enabled Service Failfast Enabled Linux Status Halt script on Primary runs after after Error Error or Exit Package Allowed Package to Run on Primary Allowed to Node after Error Run on Alternate Node Halt Script Timeout YES Either Setting system reset N/A N/A (system reset) Yes, unless the timeout happened after the cmhaltpkg command was executed.
/etc/sysconfig/network/ifcfg- on SUSE. The stationary IP address is not associated with packages, and it is not transferable to another node. Stationary IP addresses are used to transmit data, heartbeat messages (described under “How the Cluster Manager Works ” (page 37)), or both. They are configured into the cluster via the cluster configuration file; see the entries for HEARTBEAT_IP and STATIONARY_IP under “Cluster Configuration Parameters ” (page 103).
IMPORTANT: Any subnet that is used by a package for relocatable addresses should be configured into the cluster via NETWORK_INTERFACE and either STATIONARY_IP or HEARTBEAT_IP in the cluster configuration file. For more information about those parameters, see “Cluster Configuration Parameters ” (page 103). For more information about configuring relocatable addresses, see the descriptions of the package ip_ parameters (page 214).
loaded system when necessary) you can do so by putting each service in its own package and giving it a unique IP address. Bonding of LAN Interfaces Several LAN interfaces on a node can be grouped together in a process known in Linux as channel bonding. In the bonded group, typically one interface is used to transmit and receive data, while the others are available as backups. If one interface fails, another interface in the bonded group takes over.
Figure 22 Bonded Network Interfaces The LANs in the non-bonded configuration have four LAN cards, each associated with a separate non-aggregated IP address and MAC address, and each with its own LAN name (eth1, eth2, eth3, eth4). When these ports are aggregated, all four ports are associated with a single IP address and MAC address. In this example, the aggregated ports are collectively known as bond0, and this is the name by which the bond is known during cluster configuration.
Figure 23 Bonded NICs Node2 Node1 bond0: bond0: eth0 eth1 eth0 eth1 active active Hub Crossover cable Hub In the bonding model, individual Ethernet interfaces are slaves, and the bond is the master. In the basic high availability configuration (mode 1), one slave in a bond assumes an active role, while the others remain inactive until a failure is detected. (In Figure 3-18, both eth0 slave interfaces are active.
Figure 24 Bonded NICs After Failure Various combinations of Ethernet card types (single or dual-ported) and bond groups are possible, but it is vitally important to remember that at least two physical cards (or physically separate on-board LAN interfaces) must be used in any combination of channel bonds to avoid a single point of failure for heartbeat connections.
Figure 25 Bonded NICs Configured for Load Balancing Monitoring LAN Interfaces and Detecting Failure: Link Level At regular intervals, determined by the NETWORK_POLLING_INTERVAL (see “Cluster Configuration Parameters ” (page 103)), Serviceguard polls all the network interface cards specified in the cluster configuration file (both bonded and non-bonded).
The IP Monitor: • Detects when a network interface fails to send or receive IP messages, even though it is still up at the link level. • Handles the failure, failover, recovery, and failback.
Target polling enables monitoring beyond the first level of switches, allowing you to detect if the route is broken anywhere between each monitored IP address and the target. NOTE: In a cross-subnet configuration, nodes can configure peer interfaces on nodes on the other routed subnet as polling targets. HP recommends that you configure target polling if the subnet is not private to the cluster.
NOTE: This is not the default. If cmquerycl does not detect a gateway for the subnet in question, it sets IP_MONITOR to OFF, disabling IP-level polling for this subnet; if it does detect a gateway, it populates POLLING_TARGET, enabling target polling. See SUBNET under “Cluster Configuration Parameters ” (page 103) for more information. The IP Monitor section of the cluster configuration file will look similar to the following in the case of a subnet on which IP monitoring is disabled: SUBNET 192.168.3.
Reporting Link-Level and IP-Level Failures Any given failure may occur at the link level or the IP level; a failure is reported slightly differently in the output of cmviewcl (1m) depending on whether link-level or IP monitoring detects the failure.
that node, and identified as monitored subnets in the package configuration file, must be available.) The switching of relocatable IP addresses is shown in Figure 11 and Figure 12 . Address Resolution Messages after Switching on the Same Subnet When a relocatable IP address is moved to a new interface, either locally or remotely, an ARP message is broadcast to indicate the new mapping between IP address and link layer address. An ARP message is sent for each IP address that has been moved.
cluster node. To prevent this and other problems, Serviceguard imposes the following restrictions: • A maximum of 30 network interfaces per node is supported. The interfaces can be physical NIC ports, VLAN interfaces, Channel Bonds, or any combination of these. • Only port-based and IP-subnet-based VLANs are supported. Protocol-based VLAN is not supported because Serviceguard does not support any transport protocols other than TCP/IP.
sizes and types. File systems or databases used by the applications in the cluster are mounted on these logical volumes. In Serviceguard clusters, volume groups are activated by package control scripts when an application starts up, and they are deactivated by package control scripts when the application halts. Storage on Arrays Figure 26 shows LUNs configured on a storage array. Physical disks are configured by an array utility program into logical units, or LUNs, which are seen by the operating system.
Monitoring Disks Each package configuration includes information about the disks that are to be activated by the package at startup. If monitoring is used, the health of the disks is checked at package startup. The package will fail if the disks are not available. When this happens, the package may be restarted on another node. If auto_run is set to yes, the package will start up on another eligible node, if it meets all the requirements for startup.
Advantages of PR are: • Consistent behavior. Whereas different volume managers may implement exclusive activation differently (or not at all) PR is implemented at the device level and does not depend on volume-manager support for exclusive activation. • Packages can control access to LUN devices independently of a volume manager.
NOTE: • PR is turned off at the cluster level if any node is an HPVM guest. Clusters that have nodes that are VMware guests can use PR, with the following restrictions: ◦ Two or more VMware guests acting as nodes in the same cluster cannot run on the same host. (A cluster can have multiple VMware guests if each is on a separate host; and a host can have multiple guests if each is in a different cluster.
revokes them during package shutdown, using the sg_persist command. This command is available, and has a manpage, on both Red Hat 5 and SUSE SLES 10/11. Serviceguard makes a PR of type Write Exclusive Registrants Only (WERO) on the package's LUN devices. This gives read access to any initiator regardless of whether the initiator is registered or not, but grants write access only to those initiators who are registered. (WERO is defined in the SPC-3 standard.
A reboot is also initiated by Serviceguard itself under specific circumstances; see “Responses to Package and Service Failures ” (page 89). What Happens when a Node Times Out Each node sends a heartbeat message to all other nodes at an interval equal to one-fourth of the value of the configured MEMBER_TIMEOUT or 1 second, whichever is less. You configure MEMBER_TIMEOUT in the cluster configuration file; see “Cluster Configuration Parameters ” (page 103). The heartbeat interval is not directly configurable.
group vg02 and the Package2 IP address) as quickly as possible, SystemB halts (system reset). NOTE: If AUTOSTART_CMCLD in /etc/rc.config.d/cmcluster ($SGAUTOSTART) is set to zero, the node will not attempt to join the cluster when it comes back up. For more information on cluster failover, see the white paper Optimizing Failover Time in a Serviceguard Environment (version A.11.19 and later) at http:// www.docs.hp.com -> High Availability -> Serviceguard -> White Papers.
Responses to Package and Service Failures In the default case, the failure of a package, a generic resource or service of the package or of a service within a package causes the package to shut down by running the control script with the stop parameter, and then restarting the package on an alternate node. A package will also fail if it is configured to have a dependency on another package, and that package fails.
generic_resource_up_criteria, then the generic resource is marked as 'down' on that node. NOTE: If a simple resource is down on a particular node, it is down on that node for all the packages using it whereas, in case of an extended resource the resource may be up on a node for a particular package and down for another package, since it is dependent on the generic_resource_up_criteria.
4 Planning and Documenting an HA Cluster Building a Serviceguard cluster begins with a planning phase in which you gather and record information about all the hardware and software components of the configuration.
• electrical points of failure. • application points of failure. Serviceguard Memory Requirements Serviceguard requires approximately 15.5 MB of lockable memory. Planning for Expansion When you first set up the cluster, you indicate a set of nodes and define a group of packages for the initial configuration. At a later time, you may wish to add additional nodes and packages, or you may wish to use additional disk hardware for shared data storage.
LAN Information While a minimum of one LAN interface per subnet is required, at least two LAN interfaces are needed to eliminate single points of network failure. HP recommends that you configure heartbeats on all subnets, including those to be used for client data. Collect the following information for each LAN interface: Subnet Name The IP address for the subnet. Note that heartbeat IP addresses must be on the same subnet on each node.
NOTE: Multipath capabilities are supported by FibreChannel HBA device drivers and the Linux Device Mapper. Check with the storage device documentation for details. See also “Multipath for Storage ”. You can use the worksheet to record the names of the device files that correspond to each LUN for the Fibre-Channel-attached storage unit.
This information is needed when you create the mirrored disk configuration using LVM. In addition, it is useful to gather as much information as possible about your disk configuration. You can obtain information about available disks by using the following commands; your system may provide other utilities as well.
separate from that of every cluster it serves. If you use software mirroring, make sure power supplies are not shared among different physical volume groups; this allows you to set up mirroring between physical disks that are not only on different I/O buses, but also connected to different power supplies. To prevent confusion, label each hardware unit and power supply unit clearly with a different unit number.
Planning for Expansion Bear in mind that a cluster with more than 4 nodes cannot use a lock LUN. So if you plan to add enough nodes to bring the total to more than 4, you should use a quorum server. Using a Quorum Server The Quorum Server is described under “Use of the Quorum Server as a Cluster Lock” (page 41). See also “Cluster Lock” (page 40). A quorum server: • Can be used with up to 150 clusters, not exceeding 300 nodes total. • Can support a cluster with any supported number of nodes.
• You must not group two different high availability applications, services, or data, whose control needs to be transferred independently, on the same volume group. • Your root disk must not belong to a volume group that can be activated on another node. Using Generic Resources to Monitor Volume Groups You can monitor a particular disk that is a part of an LVM volume group used by packages.
Volume Groups and Physical Volume Worksheet You can organize and record your physical disk configuration by identifying which physical disks, LUNs, or disk array groups will be used in building each volume group for use with high availability applications. Use the Volume Group and Physical Volume worksheet (page 325). NOTE: HP recommends that you use volume group names other than the default volume group names (vg01, vg02, etc.).
If there is more than one heartbeat subnet, and there is a failure on one of them, heartbeats will go through another, so you can configure a smaller MEMBER_TIMEOUT value. NOTE: For heartbeat configuration requirements, see the discussion of the HEARTBEAT_IP parameter later in this chapter.
NOTE: This applies only to hostname resolution. You can have IPv6 heartbeat and data LANs no matter what the HOSTNAME_ADDRESS_FAMILY parameter is set to. (IPv4 heartbeat and data LANs are allowed in IPv4 and mixed mode.
• You must use $SGCONF/cmclnodelist, not ~/.rhosts or /etc/hosts.equiv, to provide root access to an unconfigured node. NOTE: This also applies if HOSTNAME_ADDRESS_FAMILY is set to ANY. See “Allowing Root Access to an Unconfigured Node” (page 158) for more information. • If you use a Quorum Server, you must make sure that the Quorum Server hostname (and the alternate Quorum Server address specified by QS_ADDR, if any) resolve to IPv6 addresses, and you must use Quorum Server version A.04.00 or later.
Rules and Restrictions for Mixed Mode IMPORTANT: See the latest version of the Serviceguard release notes for the most current information on these and other restrictions. • Red Hat 5 clusters are not supported. NOTE: This also applies if HOSTNAME_ADDRESS_FAMILY is set to IPv6; Red Hat 5 supports only IPv4-only clusters.
NOTE: See “Reconfiguring a Cluster” (page 260) for a summary of changes you can make while the cluster is running. The following parameters must be configured: CLUSTER_NAME The name of the cluster as it will appear in the output of cmviewcl and other commands, and as it appears in the cluster configuration file. The cluster name must not contain any of the following characters: space, slash (/), backslash (\), and asterisk (*).
values are IPV4, IPV6, and ANY. The default is IPV4. • IPV4 means Serviceguard will try to resolve the names to IPv4 addresses only. • IPV6 means Serviceguard will try to resolve the names to IPv6 addresses only. • ANY means Serviceguard will try to resolve the names to both IPv4 and IPv6 addresses. IMPORTANT: See “About Hostname Address Families: IPv4-Only, IPv6-Only, and Mixed Mode” (page 100) for important information. See also the latest Serviceguard release notes at docs.hp.
IMPORTANT: See also“About Hostname Address Families: IPv4-Only, IPv6-Only, and Mixed Mode” (page 100) for important information about requirements and restrictions in an IPv6–only cluster. Can be changed while the cluster is running; see “What Happens when You Change the Quorum Configuration Online” (page 43) for important information. QS_ADDR An alternate fully-qualified hostname or IP address for the quorum server. It must be (or resolve to) an IPv4 address on Red Hat 5.
(5 minutes). Minimum is 10,000,000 (10 seconds). Maximum is 2,147,483,647 (approximately 35 minutes). Can be changed while the cluster is running; see “What Happens when You Change the Quorum Configuration Online” (page 43) for important information.
IMPORTANT: Node names must be 39 characters or less, and are case-sensitive; for each node, the node_name in the cluster configuration file must exactly match the corresponding node_name in the package configuration file (see Chapter 6: “Configuring Packages and Their Services ” (page 199)) and these in turn must exactly match the hostname portion of the name specified in the node’s networking configuration.
NOTE: Any subnet that is configured in this cluster configuration file as a SUBNET for IP monitoring purposes, or as a monitored_subnet in a package configuration file (or SUBNET in a legacy package; see “Package Configuration Planning ” (page 120)) must be specified in the cluster configuration file via NETWORK_INTERFACE and either STATIONARY_IP or HEARTBEAT_IP.
NOTE: Any subnet that is configured in this cluster configuration file as a SUBNET for IP monitoring purposes, or as a monitored_subnet in a package configuration file (or SUBNET in a legacy package; see “Package Configuration Planning ” (page 120)) must be specified in the cluster configuration file via NETWORK_INTERFACE and either STATIONARY_IP or HEARTBEAT_IP.
or • one heartbeat subnet using bonding in high availability mode (or mode 1) with two slaves. You cannot configure more than one heartbeat IP address on an interface; only one HEARTBEAT_IP is allowed for each NETWORK_INTERFACE. NOTE: The Serviceguard cmapplyconf, cmcheckconf, and cmquerycl commands check that these minimum requirements are met, and produce a warning if they are not met at the immediate network level.
NOTE: IPv6 heartbeat subnets are not supported in a cross-subnet configuration. NOTE: The use of a private heartbeat network is not advisable if you plan to use Remote Procedure Call (RPC) protocols and services. RPC assumes that each network adapter device or I/O card is connected to a route-able network. An isolated or private heartbeat LAN is not route-able, and could cause an RPC request-reply, directed to that LAN, to time out without being serviced.
below. If HOSTNAME_ADDRESS_FAMILY is set to IPV6, all the IP addresses used by the cluster must be IPv6 addresses. If you want to separate application data from heartbeat messages, define one or more monitored non-heartbeat subnets here. You can identify any number of subnets to be monitored. A stationary IP address can be either an IPv4 or an IPv6 address. For more information about IPv6 addresses, see “IPv6 Address Types” (page 328).
NOTE: cmapplyconf will fail if any node defines a capacity and any package has min_package_node as its failover_policy (page 209) or automatic as its failback_policy (page 210). To specify more than one capacity for a node, repeat these parameters for each capacity. You can specify a maximum of four capacities per cluster, unless you use the reserved CAPACITY_NAME package_limit; in that case, you can use only that capacity throughout the cluster.
If you enter a value greater than 60 seconds (60,000,000 microseconds), cmcheckconf and cmapplyconf will note the fact, as confirmation that you intend to use a large value. Minimum supported values: • 3 seconds for a cluster with more than one heartbeat subnet. • 14 seconds for a cluster that has only one heartbeat LAN With the lowest supported value of 3 seconds, a failover time of 4 to 5 seconds can be achieved.
See “Cluster Re-formations Caused by MEMBER_TIMEOUT Being Set too Low” (page 300) for troubleshooting information. • For fewer re-formations, use a setting in the range of 10 to 25 seconds (10,000,000 to 25,000,000 microseconds), keeping in mind that a value larger than the default will lead to slower re-formations than the default.
IMPORTANT: HP strongly recommends using the default. Changing this value can affect how quickly the link-level and IP-level monitors detect a network failure. See “Monitoring LAN Interfaces and Detecting Failure: Link Level” (page 75). Can be changed while the cluster is running. SUBNET IP address of a cluster subnet for which IP Monitoring can be turned on or off (see IP_MONITOR). The subnet must be configured into the cluster, via NETWORK_INTERFACE and either HEARTBEAT_IP or STATIONARY_IP.
HP recommends you use target polling because it enables monitoring beyond the first level of switches, but if you want to use peer polling instead, set IP_MONITOR to ON for this SUBNET, but do not use POLLING_TARGET (comment out or delete any POLLING_TARGET entries that are already there). If a network interface in this subnet fails at the IP level and IP_MONITOR is set toON, the interface will be marked down. If it is set to OFF, failures that occur only at the IP-level will not be detected.
(page 138). WEIGHT_NAME specifies a name for a weight that exactly corresponds to a CAPACITY_NAME specified earlier in the cluster configuration file. (A package has weight; a node has capacity.) The rules for forming WEIGHT_NAME are the same as those spelled out for CAPACITY_NAME earlier in this list. These parameters are optional, but if they are defined, WEIGHT_DEFAULT must follow WEIGHT_NAME, and must be set to a floating-point value between 0 and 1000000.
(Access Control Policies) Specify three things for each policy: USER_NAME, USER_HOST, and USER_ROLE. Policies set in the configuration file of a cluster and its packages must not be conflicting or redundant. For more information, see “Controlling Access to the Cluster” (page 186). MAX_CONFIGURED_PACKAGES This parameter sets the maximum number of packages that can be configured in the cluster. The minimum value is 0, and the maximum value, which is also the default, is 300.
Logical Volume and File System Planning Use logical volumes in volume groups as the storage infrastructure for package operations on a cluster. When the package moves from one node to another, it must still be able to access the same data on the same disk as it did when it was running on the previous node. This is accomplished by activating the volume group and mounting the file system that resides on it.
that represent the high availability applications that they are associated with (for example, lvoldatabase) will simplify cluster administration. To further document your package-related volume groups, logical volumes, and file systems on each node, you can add commented lines to the /etc/fstab file.
IMPORTANT: Find out the MBTD value for each affected router and switch from the vendors' documentation; determine all of the possible paths; find the worst case sum of the MBTD values on these paths; and use the resulting value to set the Serviceguard CONFIGURED_IO_TIMEOUT_EXTENSION parameter. For instructions, see the discussion of this parameter under “Cluster Configuration Parameters ” (page 103). Switches and routers that do not support MBTD value must not be used in a Serviceguard NFS configuration.
In addition, you should observe the following guidelines. • CacheFS and AutoFS should be disabled on all nodes configured to run a package that uses NFS mounts. For more information, see the NFS Services Administrator's Guide HP-UX 11i version 3 at http://www.hp.com/go/hpux-networking-docs. • HP recommends that you avoid a single point of failure by ensuring that the NFS server is highly available.
The following table describes different types of failover behavior and the settings in the package configuration file that determine each behavior. See “Package Parameter Explanations” (page 205) for more information. Table 5 Package Failover Behavior Switching Behavior Parameters in Configuration File Package switches normally after detection of service or network failure, generic resource failure or when a configured dependency is not met. Halt script runs before switch takes place.
before_package_start. If not specified, default during_package_start is considered. • ◦ during_package_start means the status of generic resources are evaluated during the course of start of the package. ◦ before_package_start means resource monitoring must be started before the package start and all the configured resources have to be UP on a given node for the package to be started on that node. generic_resource_up_criteria: defines a criterion to determine the 'up' condition for a generic resource.
generic_resource_name generic_resource_evaluation_type sfm_disk during_package_start NOTE: Generic resources must be configured to use the monitoring script. It is the monitoring script that contains the logic to monitor the resource and set the status of a generic resource accordingly by using cmsetresource(1m). These scripts must be written by the end-users according to their requirements.
pkg1 down halted disabled unowned Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM STATUS Generic Resource unknown Generic Resource unknown NODE_NAME node1 node2 Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING Primary up enabled Alternate up enabled Other_Attributes: ATTRIBUTE_NAME Style Priority NAME sfm_disk sfm_disk NAME node1 node2 ATTRIBUTE_VALUE modular no_priority The cmviewcl -v -f line output will be as follows: cmview
Using Serviceguard Command to Get the Status/Value of a Simple/Extended Generic Resource Use the cmgetresource command to get the status of a simple generic resource or the value of an extended generic resource. For example: cmgetresource -r sfm_disk This retrieves the status of the generic resource sfm_disk if it is configured as a simple resource. If configured as an extended resource, the value is returned.
the new up criteria does not cause the resource status to evaluate to 'down' (i.e., the current_value of the resource still satisfies the new up_criteria). • Modification of resource type from a simple resource to an extended resource is allowed only if the generic_resource_evaluation_type is during_package_start in all the running packages that currently use the resource.
NOTE: pkg1 can depend on more than one other package, and pkg2 can depend on another package or packages; we are assuming only two packages in order to make the rules as clear as possible. • pkg1 will not start on any node unless pkg2 is running on that node. • pkg1’s package_type (page 206) and failover_policy (page 209) constrain the type and characteristics of pkg2, as follows: ◦ If pkg1 is a multi-node package, pkg2 must be a multi-node or system multi-node package.
• In the case of failover packages with a configured_node failover_policy, a set of rules governs under what circumstances pkg1 can force pkg2 to start on a given node. This is called dragging and is determined by each package’s priority (page 210). See “Dragging Rules for Simple Dependencies” (page 132). • If pkg2 fails, Serviceguard will halt pkg1 and any other packages that depend directly or indirectly on pkg2.
NOTE: Keep the following in mind when reading the examples that follow, and when actually configuring priorities: 1. auto_run (page 206) should be set to yes for all the packages involved; the examples assume that it is. 2. Priorities express a ranking order, so a lower number means a higher priority (10 is a higher priority than 30, for example).
If pkg1 depends on pkg2, and pkg1’s priority is lower than or equal to pkg2’s, pkg2’s node order dominates. Assuming pkg2’s node order is node1, node2, node3, then: • On startup: ◦ pkg2 will start on node1, or node2 if node1 is not available or does not at present meet all of its dependencies, etc. – pkg1 will start on whatever node pkg2 has started on (no matter where that node appears on pkg1’s node_name list) provided all of pkg1’s other dependencies are met there.
• On startup: ◦ pkg1 will select node1 to start on. ◦ pkg2 will start on node1, provided it can run there (no matter where node1 appears on pkg2’s node_name list). – If pkg2 is already running on another node, it will be dragged to node1, provided it can run there. ◦ If pkg2 cannot start on node1, then both packages will attempt to start on node2 (and so on).
You also need to think about what happens when a package fails. If other packages depend on it, Serviceguard will halt those packages (and any packages that depend on them, etc.) This happens regardless of the priority of the failed package. By default the packages are halted in the reverse of the order in which they were started; and if the halt script for any of the dependent packages hangs, the failed package will wait indefinitely to complete its own halt process.
The interaction of the legal values of dependency_location and dependency_condition creates the following possibilities: • Same-node dependency: a package can require that another package be UP on the same node. This is the case covered in the section on Simple Dependencies (page 130). • Different-node dependency: a package can require that another package be UP on a different node. • Any-node dependency: a package can require that another package be UP on any node in the cluster.
dependencies, see Simple Dependencies (page 130); for exclusionary dependencies, see “Rules for Exclusionary Dependencies” (page 137). • Both packages must be failover packages whose failover_policy (page 209) is configured_node. • The priority (page 210) of the package depended on must be higher than or equal to the priority of the dependent package and the priorities of that package's dependents.
Configuring Weights and Capacities You can configure multiple capacities for nodes, and multiple corresponding weights for packages, up to four capacity/weight pairs per cluster. This allows you considerable flexibility in managing package use of each node's resources — but it may be more flexibility than you need. For this reason Serviceguard provides two methods for configuring capacities and weights: a simple method and a comprehensive method. The subsections that follow explain each of these methods.
NOTE: Serviceguard does not require you to define a capacity for each node. If you define the CAPACITY_NAME and CAPACITY_VALUE parameters for some nodes but not for others, the nodes for which these parameters are not defined are assumed to have limitless capacity; in this case, those nodes would be able to run any number of eligible packages at any given time.
• If you use the reserved CAPACITY_NAME package_limit, weight_name, if used, must also be package_limit. • You do not have to define a capacity for every node; if you don't, the node is assumed to have unlimited capacity and will be able to run any number of eligible packages at the same time. • If you want to define only a single capacity, but you want the default weight to be zero rather than 1, do not use the reserved name package_limit.
there is no mapping to actual processor and memory usage and you would get exactly the same results if you used the names apples and oranges, for example. Suppose you have the following configuration: • A two node cluster running four packages. These packages contend for resource we'll simply call A and B. • node1 has a capacity of 80 for A and capacity of 50 for B. • node2 has a capacity of 60 for A and capacity of 70 for B. • pkg1 uses 60 of the A capacity and 15 of the B capacity.
CAPACITY_VALUE 70 ... NOTE: You do not have to define capacities for every node in the cluster. If any capacity is not defined for any node, Serviceguard assumes that node has an infinite amount of that capacity.
This would leave 20 units of spare A capacity on this node, and 5 units of spare B capacity. Defining Weights for Individual Packages For each capacity you define in the cluster configuration file (see “Defining Capacities”) you have the following choices when it comes to assigning a corresponding weight to a given package: 1. Configure a cluster-wide default weight and let the package use that default. 2.
weight_value 40 IMPORTANT: weight_name in the package configuration file must exactly match the corresponding CAPACITY_NAME in the cluster configuration file. This applies to case as well as spelling: weight_name a would not match CAPACITY_NAME A.
• Capacities can be added, changed, and deleted while the cluster is running. This can cause some packages to be moved, or even halted and not restarted. • Package weight can be defined in cluster configuration file, via the WEIGHT_NAME and WEIGHT_DEFAULT parameters, or in the package configuration file, via the weight_name and weight_value parameters, or both.
packages without priority, Serviceguard will decide which package to start if it cannot start them both because there is not enough node capacity to support their weight. Example 1 • pkg1 is configured to run on nodes turkey and griffon. It has a weight of 1 and a priority of 10. It is down and has switching disabled. • pkg2 is configured to run on nodes turkey and griffon. It has a weight of 1 and a priority of 20. It is running on node turkey and has switching enabled.
The scripts are also run when the package is validated by cmcheckconf and cmapplyconf. A package can make use of both kinds of script, and can launch more than one of each kind; in that case the scripts will be executed in the order they are listed in the package configuration file (and in the reverse order when the package shuts down). Each external script must have three entry points: start, stop, and validate, and should exit with one of the following values: • 0 - indicating success.
. $SGCONF.conf SG_UTILS=$SGCONF/scripts/mscripts/utils.sh fi if [[ -f ${SG_UTILS} ]]; then . ${SG_UTILS} if (( $? != 0 )) then echo "ERROR: Unable to source package utility functions file: ${SG_UTILS}" exit 1 fi else echo "ERROR: Unable to find package utility functions file: ${SG_UTILS}" exit 1 fi # Get the environment for this package through utility function # sg_source_pkg_env().
sg_log 5 "stop_command" # log current PEV_MONITORING_INTERVAL value, PEV_ attribute can be changed # while the package is running sg_log 0 "PEV_MONITORING_INTERVAL for $SG_PACKAGE_NAME is $PEV_MONITORING_INTERVAL" return 0 } typeset -i exit_val=0 case ${1} in start) start_command $* exit_val=$? ;; stop) stop_command $* exit_val=$? ;; validate) validate_command $* exit_val=$? ;; *) sg_log 0 "Unknown entry point $1" ;; esac exit $exit_val Using Serviceguard Commands in an External Script You can use Serviceg
Serviceguard sets the environment variable SG_HALT_REASON in the package control script to one of the following values when the package halts: • failure - set if the package halts because of the failure of a subnet, resource, or service it depends on • user_halt - set if the package is halted by a cmhaltpkg or cmhaltnode command, or by corresponding actions in Serviceguard Manager • automatic_halt - set if the package is failed over automatically because of the failure of a package it depends on, or is
The implications for configuring a package for cross-subnet failover are as follows: • For modular packages, you must configure two new parameters in the package configuration file to allow packages to fail over across subnets: ◦ ip_subnet_node (page 215) - to indicate which nodes a subnet is configured on ◦ monitored_subnet_access (page 213) - to indicate whether a monitored subnet is configured on all nodes ( FULL) or only some (PARTIAL).
For more information, see the white paper Technical Considerations for Creating a Serviceguard Cluster that Spans Multiple IP Subnets, at http://docs.hp.com -> High Availability. Configuring a Package to Fail Over across Subnets: Example To configure a package to fail over across subnets, you need to make some additional edits to the package configuration file. NOTE: This section provides an example for a modular package; for legacy packages, see “Configuring Cross-Subnet Failover” (page 279).
NOTE: Configuring monitored_subnet_access as FULL (or not configuring monitored_subnet_access) for either of these subnets will cause the package configuration to fail, because neither subnet is available on all the nodes. Configuring ip_subnet_node Now you need to specify which subnet is configured on which nodes. In our example, you would do this by means of entries such as the following in the package configuration file: ip_subnet 15.244.65.0 ip_subnet_node nodeA ip_subnet_node nodeB ip_address 15.244.
If you are planning to add a node online, and a package will run on the new node, ensure that any existing cluster-bound volume groups for the package have been imported to the new node. Also, ensure that the MAX_CONFIGURED_PACKAGES parameter is set high enough to accommodate the total number of packages you will be using; see “Cluster Configuration Parameters ” (page 103).
5 Building an HA Cluster Configuration This chapter and the next take you through the configuration tasks required to set up a Serviceguard cluster. You carry out these procedures on one node, called the configuration node, and Serviceguard distributes the resulting binary file to all the nodes in the cluster. In the examples in this chapter, the configuration node is named ftsys9, and the sample target node is called ftsys10.
The following are example locations for a SUSE distribution: ############################## cmcluster.
Configuring Root-Level Access The subsections that follow explain how to set up root access between the nodes in the prospective cluster. (When you proceed to configuring the cluster, you will define various levels of non-root access as well; see “Controlling Access to the Cluster” (page 186).) NOTE: For more information and advice, see the white paper Securing Serviceguard at http://docs.hp.com -> High Availability -> Serviceguard -> White Papers.
NOTE: When you upgrade a cluster from Version A.11.15 or earlier, entries in $SGCONF/cmclnodelist are automatically updated to Access Control Policies in the cluster configuration file. All non-root user-hostname pairs are assigned the role of Monitor. Ensuring that the Root User on Another Node Is Recognized The Linux root user on any cluster node can configure the cluster. This requires that Serviceguard on one node be able to recognize the root user on another.
NOTE: If you are using private IP addresses for communication within the cluster, and these addresses are not known to DNS (or the name resolution service you use) these addresses must be listed in /etc/hosts. For requirements and restrictions that apply to IPv6–only clusters and mixed-mode clusters, see “Rules and Restrictions for IPv6-Only Mode” (page 101) and “Rules and Restrictions for Mixed Mode” (page 103), respectively, and the latest version of the Serviceguard release notes.
Safeguarding against Loss of Name Resolution Services When you employ any user-level Serviceguard command (including cmviewcl), the command uses the name service you have configured (such as DNS) to obtain the addresses of all the cluster nodes. If the name service is not available, the command could hang or return an unexpected networking error message. NOTE: If such a hang or error occurs, Serviceguard and all protected applications will continue working even though the command you issued does not.
4. Create a $SGCONF/cmclnodelist file on all nodes that you intend to configure into the cluster, and allow access by all cluster nodes. See “Allowing Root Access to an Unconfigured Node” (page 158). NOTE: HP recommends that you also make the name service itself highly available, either by using multiple name servers or by configuring the name service into a Serviceguard package.
For more information on networking bonding, make sure you have installed the kernel-doc rpm, and see: /usr/share/doc/kernel-doc-/Documentation/networking/bonding.txt NOTE: HP recommends that you do the bonding configuration from the system console, because you will need to restart networking from the console when the configuration is done. Sample Configuration Configure the following files to support LAN redundancy. For a single failover only one bond is needed. 1.
For Red Hat 5 only, add a line containing the hardware (MAC) address of the interface to the corresponding ifcfg-ethn slave file, for example: HWADDR=00:12:79:43:5b:f4 3. Add the following lines to /etc/modprobe.
collisions:0 txqueuelen:100 Interrupt:10 Base address:0x1080 eth1 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4 inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.
BONDING_SLAVE0='eth0' BONDING_SLAVE1='eth1' The above example configures bond0 with mii monitor equal to 100 and active-backup mode. Adjust the IP, BROADCAST, NETMASK, and NETWORK parameters to correspond to your configuration. As you can see, you are adding the configuration options BONDING_MASTER, BONDING-MODULE_OPTS, and BONDING_SLAVE. BONDING-MODULE_OPTS are the additional options you want to pass to the bonding module.
Patches can be downloaded from HP Support Centre at http://www.hp.com/go/hpsc. You will need the pathnames for the lock LUN as it is seen on each cluster node. On one node, use the fdisk command to define a partition of 1 cylinder, type 83, on this LUN. The following example illustrates the creation of a lock LUN on a partitioned disk.
NOTE: Follow these rules: • Do not try to use LVM to configure the lock LUN. • The partition type must be 83. • Do not create any filesystem on the partition used for the lock LUN. • Do not use md to configure multiple paths to the lock LUN. • HP recommends that you can configure the same multipath device name for all the LUNs on various nodes of a cluster, so that the administration of the cluster can be made easier.
Setting Up and Running the Quorum Server If you will be using a quorum server rather than a lock LUN, the Quorum Server software must be installed on a system other than the nodes on which your cluster will be running, and must be running during cluster configuration. For detailed discussion, recommendations, and instructions for installing, updating, configuring, and running the Quorum Server, see the HP Serviceguard Quorum Server Version A.04.00 Release Notes at http://www.docs.hp.
• Storing Volume Group Configuration Data (page 178) • Setting up Disk Monitoring (page 179) CAUTION: The minor numbers used by the LVM volume groups must be the same on all cluster nodes. This means that if there are any non-shared volume groups in the cluster, create the same number of them on all nodes, and create them before you define the shared storage. If possible, avoid using private volume groups, especially LVM boot volumes.
fdisk -l You will see output such as the following: Disk /dev/sda: 64 heads, 32 sectors, 8678 cylinders Units = cylinders of 2048 * 512 bytes Device Boot /dev/sda1 * /dev/sda2 /dev/sda5 /dev/sda6 /dev/sda7 Start 1 1002 1002 4003 5004 End 1001 8678 4002 5003 8678 Blocks 1025008 7861248 3073008 1025008 3763184 Id 83 5 83 82 83 System Linux Extended Linux Linux swap Linux Disk /dev/sdb: 64 heads, 32 sectors, 8678 cylinders Units = cylinders of 2048 * 512 bytes Device Boot Start End Blocks Id System
Prompt Response Action Performed Command (m for help): p Display partition data Command (m for help): w Write data to the partition table The following example of the fdisk dialog shows that the disk on the device file /dev/sdc is configured as one partition, and appears as follows: fdisk /dev/sdc Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-4067, default 1): Enter Using default value 1Last cylinder or +size or +sizeM or +
Units = cylinders of 2048 * 512 bytes Device Boot /dev/sdc Start 1 End 4067 Blocks Id System 4164592 8e Linux LVM Command (m for help): w The partition table has been altered! 3. Repeat this process for each device file that you will use for shared storage. fdisk /dev/sdd fdisk /dev/sdf fdisk /dev/sdg 4. If you will be creating volume groups for internal storage, make sure to create those partitions as well, and create those volume groups before you define the shared storage.
5. Run vgscan: vgscan NOTE: At this point, the setup for volume-group activation protection is complete. Serviceguard adds a tag matching the uname -n value of the owning node to each volume group defined for a package when the package runs and deletes the tag when the package halts. The command vgs -o +tags vgname will display any tags that are set for a volume group.
vgdisplay 4. Create separate volume groups for each Serviceguard package you will define. In the following example, we add the LUNs /dev/sda1 and /dev/sdb1 to volume group vgpkgA, and /dev/sdc1 to vgpkgB: vgcreate --addtag $(uname -n) /dev/vgpkgA /dev/sda1 /dev/sdb1 vgcreate --addtag $(uname -n) /dev/vgpkgB /dev/sdc1 NOTE: Use vgchange --addtag only if you are implementing volume-group activation protection. Remember that volume-group activation protection, if used, must be implemented on each node.
5. To test that the file system /extra was created correctly and with high availability, you can create a file on it, and read it. echo "Test of LVM" >> /extra/LVM-test.conf cat /extra/LVM-test.conf NOTE: Be careful if you use YAST or YAST2 to configure volume groups, as that may cause all volume groups on that system to be activated.
NOTE: 3. You must reboot at this time. Run vgscan to make the LVM configuration visible on the new node and to create the LVM database on/etc/lvmtab and /etc/lvmtab.d. For example, on ftsys10: vgscan Testing the Shared Configuration When you have finished the shared volume group configuration, you can test that the storage is correctly sharable as follows: 1.
2. On ftsys10, activate the volume group, mount the file system, write a date stamp on to the shared file, and then look at the content of the file: vgchange --addtag $(uname -n) vgpkgB vgchange -a y vgpkgB mount /dev/vgpkgB/lvol1 /extra echo ‘Written by’ ‘hostname‘ ‘on’ ‘date‘ >> /extra/datestamp cat /extra/datestamp You should see something like the following, including the date stamp written by the other node: Written by ftsys9.mydomain on Mon Jan 22 14:23:44 PST 2006 Written by ftsys10.
cause problems for volumes used in a Serviceguard environment (for example, a volume group for a Serviceguard package that is not currently running may be activated). To prevent such problems, proceed as follows on the various Linux versions. NOTE: You do not need to perform these actions if you have implemented volume-group activation protection as described under “Enabling Volume Group Activation Protection” (page 173). SUSE Linux Enterprise Server Prevent a vgscan at boot time by removing the /etc/rc.
Use the cmquerycl command to specify a set of nodes to be included in the cluster and to generate a template for the cluster configuration file. IMPORTANT: See NODE_NAME under “Cluster Configuration Parameters ” (page 103) for important information about restrictions on the node name. Here is an example of the command (enter it all one line): cmquerycl -v -C $SGCONF/clust1.conf -n ftsys9 -n ftsys10 This creates a template file, by default /etc/cmcluster/clust1.conf.
IMPORTANT: See “About Hostname Address Families: IPv4-Only, IPv6-Only, and Mixed Mode” (page 100) for a full discussion, including important restrictions for IPv6–only and mixed modes.
Full Network Probing -w full lets you specify full network probing, in which actual connectivity is verified among all LAN interfaces on all nodes in the cluster, whether or not they are all on the same subnet. NOTE: This option must be used to discover actual or potential nodes and subnets in a cross-subnet configuration. See “Obtaining Cross-Subnet Information” (page 183).
Enter the QS_HOST (IPv4 or IPv6 on SLES 10 and 11; IPv4 only on Red Hat 5), optional QS_ADDR (IPv4 or IPv6 on SLES 10 and 11; IPv4 only on Red Hat 5), QS_POLLING_INTERVAL, and optionally a QS_TIMEOUT_EXTENSION; and also check the HOSTNAME_ADDRESS_FAMILY setting, which defaults to IPv4. See the parameter descriptions under Cluster Configuration Parameters (page 103).
IP subnets: IPv4: 15.13.164.0 15.13.172.0 15.13.165.0 15.13.182.0 15.244.65.0 15.244.56.0 lan1 lan1 lan1 lan1 lan2 lan2 lan2 lan2 lan3 lan3 lan4 lan4 (nodeA) (nodeB) (nodeC) (nodeD) (nodeA) (nodeB) (nodeC) (nodeD) (nodeA) (nodeB) (nodeC) (nodeD) lan3 lan3 lan3 lan3 (nodeA) (nodeB) (nodeC) (nodeD) IPv6: 3ffe:1111::/64 3ffe:2222::/64 Possible Heartbeat IPs: 15.13.164.0 15.13.164.1 15.13.164.2 15.13.172.0 15.13.172.158 15.13.172.159 15.13.165.0 15.13.165.1 15.13.165.2 15.13.182.0 15.13.182.158 15.13.182.
IMPORTANT: Note that in this example subnet 15.244.65.0, used by NodeA and NodeB, is not routed to 15.244.56.0, used by NodeC and NodeD. But subnets 15.13.164.0 and 15.13.165.0, used by NodeA and NodeB, are routed respectively to subnets 15.13.172.0 and 15.13.182.0, used by NodeC and NodeD. At least one such routing among all the nodes must exist for cmquerycl to succeed.
Controlling Access to the Cluster Serviceguard access-control policies define cluster users’ administrative or monitoring capabilities. A Note about Terminology Although you will also sometimes see the term role-based access (RBA) in the output of Serviceguard commands, the preferred set of terms, always used in this manual, is as follows: • Access-control policies - the set of rules defining user access to the cluster.
Figure 27 Access Roles Levels of Access Serviceguard recognizes two levels of access, root and non-root: • Root access: Full capabilities; only role allowed to configure the cluster. As Figure 27 shows, users with root access have complete control over the configuration of the cluster and its packages. This is the only role allowed to use the cmcheckconf, cmapplyconf, cmdeleteconf, and cmmodnet -a commands.
IMPORTANT: Users on systems outside the cluster can gain Serviceguard root access privileges to configure the cluster only via a secure connection (rsh or ssh). • Non-root access: Other users can be assigned one of four roles: ◦ Full Admin: Allowed to perform cluster administration, package administration, and cluster and package view operations. These users can administer the cluster, but cannot configure or create a cluster. Full Admin includes the privileges of the Package Admin role.
NOTE: For more information and advice, see the white paper Securing Serviceguard at http://docs.hp.com -> High Availability -> Serviceguard -> White Papers. Define access-control policies for a cluster in the cluster configuration file; see “Cluster Configuration Parameters ” (page 103). To define access control for a specific package, use user_host (page 224) and related parameters in the package configuration file. You can define up to 200 access policies for each cluster.
NOTE: If you set USER_HOST to ANY_SERVICEGUARD_NODE, set USER_ROLE to MONITOR; users connecting from outside the cluster cannot have any higher privileges (unless they are connecting via rsh or ssh; this is treated as a local connection). Depending on your network configuration, ANY_SERVICEGUARD_NODE can provide wide-ranging read-only access to the cluster.
defined in the package configuration file for PackageA, then user john on node bit has the PACKAGE_ADMIN role only for PackageA. Plan the cluster’s roles and validate them as soon as possible. If your organization’s security policies allow it, you may find it easiest to create group logins. For example, you could create a MONITOR role for user operator1 from CLUSTER_MEMBER_NODE (that is, from any node in the cluster).
NOTE: Check spelling especially carefully when typing wildcards, such as ANY_USER and ANY_SERVICEGUARD_NODE. If they are misspelled, Serviceguard will assume they are specific users or nodes.
• The value for AUTO_START_TIMEOUT variables is greater than zero. • Heartbeat network minimum requirement. See HEARTBEAT_IP under “Cluster Configuration Parameters ” (page 103). • At least one NODE_NAME is specified. • Each node is connected to each heartbeat network. • All heartbeat networks are of the same type of LAN. • The network interface device files specified are valid LAN device files. • Other configuration parameters for the cluster and packages are valid.
Managing the Running Cluster This section describes some approaches to routine management of the cluster. For more information, see Chapter 7: “Cluster and Package Maintenance” (page 233). You can manage the cluster from Serviceguard Manager, or by means of Serviceguard commands as described below. Checking Cluster Operation with Serviceguard Commands • cmviewcl checks the status of the cluster and many of its components.
4. • Start the node. You can use Serviceguard Manager or the cmrunnode command. • Verify that the node has returned to operation. You can use Serviceguard Manager or the cmviewcl command again. Bring down the cluster. You can use Serviceguard Manager or the cmhaltcl -v -f command. See the manpages for more information about these commands. See Chapter 8: “Troubleshooting Your Cluster” (page 289) for more information about cluster testing.
# # # # # # a cluster consisting of all configured nodes. Automatic cluster start is the preferred way to start a cluster. No action is required by the system administrator. If set to 1, the node will attempt to join/form its CM cluster automatically as described above. If set to 0, the node will not attempt to join its CM cluster. AUTOSTART_CMCLD=1 NOTE: The /sbin/init.d/cmcluster file may call files that Serviceguard stores in$SGCONF/rc.
cluster, which causes the node to halt with a reboot, and packages to be switched to adoptive nodes.) It is not necessary to halt the single node in this case, since the applications are still running, and no other node is currently available for package switching. CAUTION: But you should not try to restart Serviceguard; data corruption might occur if another node were to attempt to start up a new instance of an application that is still running on the single node.
Deleting the Cluster Configuration You can delete a cluster configuration by means of the cmdeleteconf command. The command prompts for a verification before deleting the files unless you use the -f option. You can delete the configuration only when the cluster is down. The action removes the binary configuration file from all the nodes in the cluster and resets all cluster-aware volume groups to be no longer cluster-aware.
6 Configuring Packages and Their Services Serviceguard packages group together applications and the services and resources they depend on. The typical Serviceguard package is a failover package that starts on one node but can be moved (“failed over”) to another if necessary. For more information, see “What is Serviceguard for Linux? ” (page 19), “How the Package Manager Works” (page 44), and“Package Configuration Planning ” (page 120).
Choosing Package Modules IMPORTANT: Before you start, you need to do the package-planning tasks described under “Package Configuration Planning ” (page 120). To choose the right package modules, you need to decide the following things about the package you are creating: • What type of package it is; see “Types of Package: Failover, Multi-Node, System Multi-Node” (page 200).
NOTE: The following parameters cannot be configured for multi-node packages: • failover_policy • failback_policy • ip_subnet • ip_address Volume groups configured for packages of this type must be activated in shared mode. For more information about types of packages and how they work, see “How the Package Manager Works” (page 44). For information on planning a package, see “Package Configuration Planning ” (page 120).
Table 7 Base Modules Module Name Parameters (page) Comments failover package_name (page 205) * module_name (page 205) * module_version (page 206) * package_type (page 206) package_description (page 206) * node_name (page 206) auto_run (page 206) node_fail_fast_enabled (page 207) run_script_timeout (page 207) halt_script_timeout (page 208) successor_halt_script_timeout (page 208) * script_log_file (page 208) operation_sequence (page 209) * log_level (page 209) * failover_policy (page 209) failback_policy
Optional Package Modules Add optional modules to a base module if you need to configure the functions in question. Parameters marked with an asterisk (*) are new or changed as of Serviceguard A.11.18 or A.11.19. (S) indicates that the parameter (or its equivalent) has moved from the package control script to the package configuration file for modular packages. See the “Package Parameter Explanations” (page 205) for more information.
Table 8 Optional Modules (continued) Module Name Parameters (page) Comments * pev pev_ (page 223) Add to a base module to configure environment variables to be passed to an external script. external_pre external_pre_script (page 223) external external_script (page 223) acp user_name (page 224) user_host (page 224) user_role (page 224) Add to a base module to configure Access Control Policies for the package.
Package Parameter Explanations Brief descriptions of the package configuration parameters follow. NOTE: For more information, see the comments in the editable configuration file output by the cmmakepkg command, and the cmmakepkg (1m) manpage.
module_version The module version. Do not change it. New for modular packages. package_type The type can be failover, multi_node, or system multi_node. You can configure only failover or multi-node packages; see “Types of Package: Failover, Multi-Node, System Multi-Node” (page 200). package_description The application that the package runs. This is a descriptive parameter that can be set to any value you choose, up to a maximum of 80 characters. Default value is Serviceguard Package. New for 11.
adoptive node if it fails. no prevents Serviceguard from automatically starting the package, and from restarting it on another node. This is also referred to as package switching, and can be enabled or disabled while the package is running, by means of the cmmodpkg command. auto_run should be set to yes if the package depends on another package, or is depended on; see “About Package Dependencies” (page 130). For system multi-node packages, auto_run must be set to yes.
If a timeout occurs: • Switching will be disabled. • The current node will be disabled from running the package. NOTE: If no_timeout is specified, and the script hangs, or takes a very long time to complete, during the validation step (cmcheckconf (1m)), cmcheckconf will wait 20 minutes to allow the validation to complete before giving up. halt_script_timeout The amount of time, in seconds, allowed for the package to halt; or no_timeout. The default is no_timeout. The maximum is 4294.
operation_sequence Defines the order in which the scripts defined by the package’s component modules will start up. See the package configuration file for details. This parameter is not configurable; do not change the entries in the configuration file. New for modular packages. log_level Determines the amount of information printed to stdout when the package is validated, and to the script_log_file when the package is started and halted.
failback_policy Specifies whether or not Serviceguard will automatically move a package that is not running on its primary node (the first node on its node_name list) when the primary node is once again available. Can be set to automatic or manual. The default is manual. • manual means the package will continue to run on the current node.
Configure this parameter, along with dependency_condition and dependency_location, and optionally priority (page 210), if this package depends on another package; for example, if this package depends on a package named pkg2: dependency_name pkg2dep dependency_condition pkg2 = UP dependency_location same_node For more information about package dependencies, see “About Package Dependencies” (page 130). dependency_condition The condition that must be met for this dependency to be satisfied.
NOTE: But if weight_name is package_limit, you can use only that one weight and capacity throughout the cluster. package_limit is a reserved value, which, if used, must be entered exactly in that form. It provides the simplest way of managing weights and capacities; see “Simple Method” (page 139) for more information. The rules for forming weight_name are the same as those for forming package_name (page 205). weight_name must exactly match the corresponding CAPACITY_NAME.
If any monitored_subnet fails, Serviceguard will switch the package to any other node specified by node_name (page 206) which can communicate on all the monitored_subnets defined for this package. See the comments in the configuration file for more information and examples. monitored_subnet_access In cross-subnet configurations, specifies whether each monitored_subnet is accessible on all nodes in the package’s node_name list (page 206), or only some.
ip_subnet Specifies an IP subnet used by the package. Replaces SUBNET, which is still supported in the package control script for legacy packages. CAUTION: HP recommends that this subnet be configured into the cluster. You do this in the cluster configuration file by specifying a HEARTBEAT_IP or STATIONARY_IP under a NETWORK_INTERFACE on the same subnet, for each node in this package's NODE_NAME list. For example, an entry such as the following in the cluster configuration file configures subnet 192.10.25.
This parameter can be set for failover packages only. ip_subnet_node In a cross-subnet configuration, specifies which nodes an ip_subnet is configured on. If no ip_subnet_nodes are listed under an ip_subnet, it is assumed to be configured on all nodes in this package’s node_name list (page 206). Can be added or deleted while the package is running, with these restrictions: • The package must not be running on the node that is being added or deleted.
service_name service_cmd service_restart service_fail_fast_enabled service_halt_timeout patricks-package4-ping] "/usr/sbin/ping hasupt22" unlimited no 300 See the package configuration template file for more examples. For legacy packages, this parameter is in the package control script as well as the package configuration file. service_cmd The command that runs the program or function for this service_name, for example, /usr/bin/X11/xclock -display 15.244.58.
service_fail_fast_enabled Specifies whether or not Serviceguard will halt the node (reboot) on which the package is running if the service identified by service_name fails. Valid values are yes and no. Default is no, meaning that failure of this service will not cause the node to halt. service_halt_timeout The length of time, in seconds, Serviceguard will wait for the service to halt before forcing termination of the service’s process. The maximum value is 4294.
Valid values are during_package_start and before_package_start. The default is during_package_start. The resources that will be available during the course of start of the package should be configured with an evaluation_type as during_package_start. Monitoring for these generic resources can be started and stopped as a part of the package, and the monitoring script can be configured as a service.
NOTE: Operators other than the ones mentioned above are not supported. This attribute does not accept more than one up criterion. For e.g., >> 10, << 100 are not valid.
File system parameters A package can activate one or more storage groups on startup, and to mount logical volumes to file systems. At halt time, the package script unmounts the file systems and deactivates each storage group. All storage groups must be accessible on each target node.
fs_mount_retry_count The number of mount retries for each file system. Legal value is zero or any greater number. The default is zero. The only valid value for Red Hat GFS (see fs_type) is zero. If the mount point is busy at package startup and fs_mount_retry_count is set to zero, package startup will fail.
fs_directory The root of the file system specified by fs_name. Replaces FS, which is still supported in the package control script for legacy packages; see “Configuring a Legacy Package” (page 271). See the mount manpage and the comments in the configuration file for more information. fs_type The type of the file system specified by fs_name. This parameter is in the package control script for legacy packages. Supported types are ext2, ext3, reiserfs, and gfs.
IMPORTANT: This parameter is for use only by HP partners, who should follow the instructions in the package configuration file. For information about Serviceguard's implementation of PR, see “About Persistent Reservations” (page 83). pev_ Specifies a package environment variable that can be passed to external_pre_script, external_script, or both, by means of the cmgetpkgenv command. New for modular packages.
See “About External Scripts” (page 147), as well as the comments in the configuration file, for more information and examples. See also service_cmd (page 216). user_host The system from which a user specified by user_name (page 224) can execute package-administration commands. Legal values are any_serviceguard_node, or cluster_member_node, or a specific cluster node.
These two parameters allow you to separate package run instructions and package halt instructions for legacy packages into separate scripts if you need to. In this case, make sure you include identical configuration information (such as node names, IP addresses, etc.) in both scripts. In most cases, though, HP recommends that you use the same script for both run and halt instructions. (When the package starts, the script is passed the parameter start; when it halts, it is passed the parameter stop.
NOTE: If you do not include a base module (or default or all) on the cmmakepkg command line, cmmakepkg will ignore the modules you specify and generate a default configuration file containing all the parameters. For a complex package, or if you are not yet sure which parameters you will need to set, the default may be the best choice; see the first example below. You can use the-v option with cmmakepkg to control how much information is displayed online or included in the configuration file.
the file to set the package parameters to the values that will make the package function as you intend. It is a good idea to configure complex failover packages in stages, as follows: 1. Configure volume groups and mount points only. 2. Check and apply the configuration; see “Verifying and Applying the Package Configuration” (page 230). 3. Run the package and ensure that it can be moved from node to node. NOTE: cmcheckconf and cmapplyconf check for missing mount points, volume groups, etc. 4. 5. 6.
restart it later if it fails. Enter no to keep Serviceguard from automatically starting the package. • node_fail_fast_enabled. Enter yes to cause the node to be halted (system halt) if the package fails; otherwise enter no. • run_script_timeout and halt_script_timeout. Enter the number of seconds Serviceguard should wait for package startup or shutdown, respectively, to complete; or leave the default, no_timeout. See (page 207). • successor_halt_timeout.
• If your package will use relocatable IP addresses, enter the ip_subnet and ip_address addresses. See the parameter descriptions (page 214) for rules and restrictions. In a cross-subnet configuration, configure the additional ip_subnet_node parameter for each ip_subnet as necessary; see “About Cross-Subnet Failover” (page 151) for more information.
• If your package uses a large number of volume groups or disk groups, or mounts a large number of file systems, consider increasing the values of the following parameters: ◦ concurrent_fsck_operations—specifies the number of parallel fsck operations that will be allowed at package startup (not used for Red Hat GFS). ◦ concurrent_mount_and_umount_operations—specifies the number of parallel mount operations allowed during package startup and unmount operations during package shutdown.
The following items are checked: • Package name is valid, and at least one node_name entry is included. • There are no duplicate parameter entries (except as permitted for multiple volume groups, etc). • Values for all parameters are within permitted ranges. • Configured resources are available on cluster nodes. • File systems and volume groups are valid. • Services are executable. • Any package that this package depends on is already be part of the cluster configuration.
use the package name in combination with the string cmresserviced. The following shows an entry in the package configuration file for pkg1: service_name service_fail_fast_enabled service_halt_timeout service_cmd service_restart cmresserviced_pkg1 yes 300 "cmresserviced /dev/sdd1 /dsv/sde1" none CAUTION: Because of a limitation in LVM, service_fail_fast_enabled must be set to yes, forcing the package to fail over to another node if it loses its storage.
7 Cluster and Package Maintenance This chapter describes the cmviewcl command, then shows how to start and halt a cluster or an individual node, how to perform permanent reconfiguration, and how to start, halt, move, and modify packages during routine maintenance of the cluster.
Viewing Package Dependencies The cmviewcl -v command output lists dependencies throughout the cluster. For a specific package’s dependencies, use the -p option. Cluster Status The status of a cluster, as shown by cmviewcl, can be one of the following: • up - At least one node has a running cluster daemon, and reconfiguration is not taking place. • down - No cluster daemons are running on any cluster node. • starting - The cluster is in the process of determining its active membership.
• start_wait - A cmrunpkg command is in progress for this package. The package is waiting for packages it depends on (predecessors) to start before it can start. • starting - The package is starting. The package master control script is running. • halting - A cmhaltpkg command is in progress for this package and the halt script is running. • halt_wait -A cmhaltpkg command is in progress for this package.
• halted - The package is down and halted. • failing - The package is halting because it, or a package it depends on, has failed. • fail_wait - The package is waiting to be halted because the package or a package it depends on has failed, but must wait for a package it depends on to halt before it can halt. • failed - The package is down and failed. • relocate_wait - The package’s halt script has completed or Serviceguard is still trying to place the package.
switch to the specified node until the node is enabled to run the package via the cmmodpkg command. Every failover package is marked enabled or disabled for each node that is either a primary or adoptive node for the package. For multi-node packages, node switching disabled means the package cannot start on that node. Service Status Services have only status, as follows: • Up. The service is being monitored. • Down. The service is not running. It may not have started, or have halted or failed.
Failover packages can also be configured with one of two values for the failback_policy parameter (page 210), and these are also displayed in the output of cmviewcl -v: • automatic: Following a failover, a package returns to its primary node when the primary node becomes available again. • manual: Following a failover, a package will run on the adoptive node until moved back to its original node by a system administrator.
PACKAGE pkg2 STATUS up STATE running AUTO_RUN enabled NODE ftsys10 Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM STATUS Service up Subnet up MAX_RESTARTS 0 0 Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING Primary up enabled Alternate up enabled RESTARTS 0 0 NAME ftsys10 ftsys9 NAME service2 15.13.168.
NODE ftsys9 STATUS up STATE running Network_Parameters: INTERFACE STATUS PRIMARY up PRIMARY up NAME eth0 eth1 PACKAGE pkg1 STATE running STATUS up AUTO_RUN enabled NODE ftsys9 Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM STATUS Service up Subnet up MAX_RESTARTS 0 0 Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING Primary up enabled Alternate up enabled NODE ftsys10 STATUS up Network_Parameters: INTERFACE STATUS PRIMARY u
Primary Alternate up up enabled enabled ftsys10 ftsys9 pkg2 now has the status down, and it is shown as unowned, with package switching disabled. Note that switching is enabled for both nodes, however. This means that once global switching is re-enabled for the package, it will attempt to start up on the primary node.
Subnet up 0 Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING Primary up enabled Alternate up enabled NODE ftsys10 STATUS up 0 NAME ftsys10 ftsys9 15.13.168.
NODE ftsys10 STATUS down STATE halted This output can be seen on both ftsys9 and ftsys10. Viewing Information about Unowned Packages The following example shows packages that are currently unowned, that is, not running on any configured node.
start the cluster depending on whether all nodes are currently down (that is, no cluster daemons are running), or whether you are starting the cluster daemon on an individual node. Note the distinction that is made in this chapter between adding an already configured node to the cluster and adding a new node to the cluster configuration. An already configured node is one that is already entered in the cluster configuration file; a new node is added to the cluster by modifying the cluster configuration file.
Use the cmrunnode command to add one or more nodes to an already running cluster. Any node you add must already be a part of the cluster configuration. The following example adds node ftsys8 to the cluster that was just started with only nodes ftsys9 and ftsys10. The-v (verbose) option prints out all the messages cmrunnode -v ftsys8 By default, cmrunnode will do network validation, making sure the actual network setup matches the configured network setup. This is the recommended method.
This halts any packages running on the node ftsys9 by executing the halt instructions in each package's master control script. ftsys9 is halted and the packages start on the adoptive node, ftsys10. Halting the Entire Cluster You can use Serviceguard Manager, or Serviceguard commands as shown below, to halt a running cluster. The cmhaltcl command can be used to halt the entire cluster. This command causes all nodes in a configured cluster to halt their HP Serviceguard daemons.
What You Can Do You can do the following. • Halt a node (cmhaltnode (1m) with the -d option) without causing its running packages to halt or fail over. Until you restart the node (cmrunnode (1m)) these packages are detached — not being monitored by Serviceguard. • Halt the cluster (cmhaltcl (1m) with the -d option) without causing its running packages to halt. Until you restart the cluster (cmruncl (1m)) these packages are detached — not being monitored by Serviceguard.
• You cannot detach a package that is in maintenance mode, and you cannot place a package into maintenance mode if any of its dependent packages are detached. For more information about maintenance mode, see“Maintaining a Package: Maintenance Mode” (page 255). For more information about dependencies, see “About Package Dependencies” (page 130). • You cannot make configuration changes to a cluster in which any packages are detached. cmapplyconf (1m) will fail.
• You cannot detach a package that has relocatable IP addresses on a subnet that is not configured into the cluster. Such addresses will not be handled properly when the package is attached or re-attached. IMPORTANT: (page 214). HP does not recommend such configurations. See “ip_subnet” • You cannot detach packages when the HP-UX Clustered I/O Congestion Control feature is enabled. • As of the date of this manual, you cannot detach ECMT-based packages.
in fact failed while it was detached, Serviceguard will halt it and restart it on another eligible node. CAUTION: Serviceguard does not check LVM volume groups and mount points when re-attaching packages. • The detached state and status could appear to persist across a reboot. If a node reboots while packages are detached (or detaching, or re-attaching), and package components are left in an inconsistent state, the appropriate package halt scripts will run to clean things up when the node comes back up.
3. Halt the node with the -d (detach) option: cmhaltnode -d node1 NOTE: -d and -f are mutually exclusive. See cmhaltnode (1m) for more information. To re-attach the packages, restart the node: cmrunnode -n node1 You cannot halt or detach a node if any package on the given node is in the halt_aborted state; cmhaltnode will fail. However, you can forcefully halt the node using cmhaltnode (1m) with the -f option. The node will then be halted irrespective of the package state.
In this example we'll assume that packages pkg1 through pkg5 are unsupported for Live Application Detach, and pkg6 through pkgn are supported. Proceed as follows: 1. Halt all the unsupported packages: cmhaltpkg pkg1 pkg2 pkg3 pkg4 pkg5 2. Halt the cluster, detaching the remaining packages: cmhaltcl -d 3. 4. Upgrade the heartbeat networks as needed.
Use the cmrunpkg command to run the package on a particular node, then use the cmmodpkg command to enable switching for the package; for example: cmrunpkg -n ftsys9 pkg1 cmmodpkg -e pkg1 This starts up the package on ftsys9, then enables package switching. This sequence is necessary when a package has previously been halted on some node, since halting the package disables switching.
cmviewcl to be sure that no other running package has a dependency on any of the packages you are halting. Moving a Failover Package You can use Serviceguard Manager to move a failover package from one node to another, or Serviceguard commands as shown below. Before you move a failover package to a new node, it is a good idea to run cmviewcl -v -l package and look at dependencies. If the package has dependencies, be sure they can be met on the new node.
You can disable package switching to particular nodes by using the -n option of the cmmodpkg command. The following prevents pkg1 from switching to node lptest3: cmmodpkg -d -n lptest3 pkg1 To permanently disable switching so that the next time the cluster restarts, the change you made in package switching is still in effect, change the auto_run flag in the package configuration file, then re-apply the configuration. (See “Reconfiguring a Package on a Running Cluster ” (page 281).
NOTE: But a failure in the package control script will cause the package to fail. The package will also fail if an external script (or pre-script) cannot be executed or does not exist. • The package will not be automatically failed over, halted, or started. • A package in maintenance mode still has its configured (or default) weight, meaning that its weight, if any, is counted against the node's capacity; this applies whether the package is up or down.
◦ If the package is running, you can put it into maintenance only on the node on which it is running. ◦ While the package is in maintenance mode on a node, you can run the package only on that node. • You cannot put a package in maintenance mode, or take it out maintenance mode, if doing so will cause another running package to halt. • Since package failures are ignored while in maintenance mode, you can take a running package out of maintenance mode only if the package is healthy.
exclusionary dependencies) as described under “About Package Dependencies” (page 130). ◦ You cannot enable a package that depends on pkgA. ◦ You cannot run a package that depends on pkgA, unless the dependent package itself is in maintenance mode. • Dependency rules governing packages that pkgA depends on to be UP are bypassed so that these packages can halt and fail over as necessary while pkgA is in maintenance mode.
Performing Maintenance Using Partial-Startup Maintenance Mode To put a package in partial-startup maintenance mode, you put it in maintenance mode, then restart it, running only those modules that you will not be working on. Procedure Follow this procedure to perform maintenance on a package. In this example, we'll assume a package pkg1 is running on node1, and that we want to do maintenance on the package's services. 1. Halt the package: cmhaltpkg pkg1 2.
8. Restart the package: cmrunpkg pkg1 Excluding Modules in Partial-Startup Maintenance Mode In the example above, we used cmrunpkg -m to run all the modules up to and including package_ip, but none of those after it. But you might want to run the entire package apart from the module whose components you are going to work on. In this case you can use the -e option: cmrunpkg -e sg/service pkg1 This runs all the package's modules except the services module. You can also use -e in combination with -m.
Table 9 Types of Changes to the Cluster Configuration (continued) Change to the Cluster Configuration Required Cluster State Change Cluster Lock Configuration (lock LUN) Cluster can be running. See “Updating the Cluster Lock LUN Configuration Online” (page 270) and“What Happens when You Change the Quorum Configuration Online” (page 43). Add NICs and their IP addresses to the cluster configuration Cluster can be running.
package halts and cannot restart because none of the nodes on its node_list is available. Serviceguard provides two ways to do this: you can use the preview mode of Serviceguard commands, or you can use the cmeval (1m) command to simulate different cluster states. Alternatively, you might want to model changes to the cluster as a whole; cmeval allows you to do this; see “Using cmeval” (page 263).
pkg2 and pkg3 to run on the same node. These are lower-priority packages which are currently running on node2.
lower-priority-packages are currently running on node2. pkg1 is down and disabled, and you want to see the effect of enabling it. In the output of cmviewcl -v -f line, you would find the line package:pkg1|autorun=disabled and change it to package:pkg1|autorun=enabled. You should also make sure that the nodes the package is configured to run on are shown as available; for example: package:pkg1|node:node1|available=yes. Then save the file (for example as newstate.in) and run cmeval: cmeval -v newstate.
Reconfiguring a Running Cluster You can add new nodes to the cluster configuration or delete nodes from the cluster configuration while the cluster is up and running. Note the following, however: • You cannot remove an active node from the cluster. You must halt the node first. • The only configuration change allowed while a node is unreachable (for example, completely disconnected from the network) is to delete the unreachable node from the cluster configuration.
Removing Nodes from the Cluster while the Cluster Is Running You can use Serviceguard Manager to delete nodes, or Serviceguard commands as shown below. The following restrictions apply: • The node must be halted. See “Removing Nodes from Participation in a Running Cluster” (page 245). • If the node you want to delete is unreachable (disconnected from the LAN, for example), you can delete the node only if there are no packages which specify the unreachable node.
NOTE: If you are trying to remove an unreachable node on which many packages are configured to run, you may see the following message: The configuration change is too large to process while the cluster is running. Split the configuration change into multiple requests or halt the cluster. In this situation, you must halt the cluster to remove the node.
• You cannot change the designation of an existing interface from HEARTBEAT_IP to STATIONARY_IP, or vice versa, without also making the same change to all peer network interfaces on the same subnet on all other nodes in the cluster.
1. Run cmquerycl to get a cluster configuration template file that includes networking information for interfaces that are available to be added to the cluster configuration: cmquerycl -c cluster1 -C clconfig.conf NOTE: As of Serviceguard A.11.18, cmquerycl -c produces output that includes commented-out entries for interfaces that are not currently part of the cluster configuration, but are available. The networking portion of the resulting clconfig.
1. 2. 3. 4. Halt the package Add the new networking information to the package configuration file In the case of a legacy package, add the new networking information to the package control script if necessary Apply the new package configuration, and redistribute the control script if necessary. For more information, see “Reconfiguring a Package on a Running Cluster ” (page 281). Example: Deleting a Subnet Used by a Package In this example, we are deleting subnet 15.13.170.0 (lan0). Proceed as follows. 1.
1. 2. 3. In the cluster configuration file, modify the value of CLUSTER_LOCK_LUN for each node. Run cmcheckconf to check the configuration. Run cmapplyconf to apply the configuration. If you need to replace the physical device, see “Replacing a Lock LUN” (page 291). Changing MAX_CONFIGURED_PACKAGES As of Serviceguard A.11.18, you can change MAX_CONFIGURED_PACKAGES while the cluster is running. The default for MAX_CONFIGURED_PACKAGES is the maximum number allowed in the cluster.
5. 6. Distribute the control script to the cluster nodes Apply the package configuration file Each of these tasks is described in the sub-sections that follow. Creating the Legacy Package Configuration The package configuration process defines a set of application services that are run by the package manager when a package starts up on a node in the cluster.
7. 8. Distribute the control script to all nodes. Run the package and ensure that applications run as expected and that the package fails over correctly when services are disrupted. Editing the Package Configuration File Edit the file you generated in step 2 of “Using Serviceguard Commands to Configure a Package ” (page 272). Use the bullet points that follow as a checklist. • PACKAGE_TYPE.
• If your package has relocatable IP addresses, enter the SUBNET if you want it to be monitored (this means the package will stop if the subnet fails). This must be a subnet that is already specified in the cluster configuration, and it can be either an IPv4 or an IPv6 subnet. It must not be a link-local subnet (link-local package IPs are not allowed). See monitored_subnet (page 212).
For security reasons, the control script must reside in a directory with the string cmcluster in the path. The control script is placed in the package directory and is given the same name as specified in the RUN_SCRIPT and HALT_SCRIPT parameters in the package configuration file. The package control script template contains both the run instructions and the halt instructions for the package. You can use a single script for both run and halt operations, or, if you wish, you can create separate scripts.
• Add service command(s) • Add a service restart parameter, if you so decide. For more information about services, see the discussion of the service_ parameters starting with service_name (page 215). Adding Customer Defined Functions to the Package Control Script You can add additional shell commands to the package control script to be executed whenever the package starts or stops. Enter these commands in the CUSTOMER DEFINED FUNCTIONS area of the script.
If a Serviceguard command interacts with another package, be careful to avoid command loops. For instance, a command loop might occur under the following circumstances. Suppose pkg1 does a cmmodpkg -d of pkg2, and pkg2 does a cmmodpkg -d of pkg1. If both pkg1 and pkg2 start at the same time, pkg1 tries to cmmodpkg pkg2. However, that cmmodpkg command has to wait for pkg2 startup to complete. pkg2 tries to cmmodpkg pkg1, but pkg2 has to wait for pkg1 startup to complete, thereby causing a command loop.
• Configured resources are available on cluster nodes. • If a dependency is configured, the dependency package must already be configured in the cluster. Distributing the Configuration You can use Serviceguard Manager or Linux commands to distribute the binary cluster configuration file among the nodes of the cluster. Distributing the Configuration And Control Script with Serviceguard Manager When you have finished creating a package in Serviceguard Manager, click Apply Configuration.
Configuring Cross-Subnet Failover To configure a legacy package to fail over across subnets (see “Cross-Subnet Configurations” (page 27)), you need to do some additional configuration. NOTE: You cannot use Serviceguard Manager to configure cross-subnet packages. Suppose that you want to configure a package, pkg1, so that it can fail over among all the nodes in a cluster comprising NodeA, NodeB, NodeC, and NodeD. NodeA and NodeB use subnet 15.244.65.
IMPORTANT: In a cross-subnet configuration, you cannot share a single package control script among nodes on different subnets if you are using relocatable IP addresses. In this case you will need to create a separate control script to be used by the nodes on each subnet. In our example, you would create two copies of pkg1’s package control script, add entries to customize it for subnet 15.244.65.0 or 15.244.56.0, and copy one of the resulting scripts to each node, as follows.
Migrating a Legacy Package to a Modular Package The Serviceguard command cmmigratepkg automates the process of migrating legacy packages to modular packages as far as possible. Many, but not all, packages can be migrated in this way; for details, see the white paper Package Migration from Legacy Style to Modular Style at http://docs.hp.com -> High Availability -> Serviceguard -> White papers. Do not attempt to convert Serviceguard Toolkit packages. NOTE: The cmmigratepkg command requires Perl version 5.8.
5. Distribute your changes to all nodes: cmapplyconf -v -P pkg1.conf 6. If this is a legacy package, copy the package control script to all nodes that can run the package. Reconfiguring a Package on a Halted Cluster You can also make permanent changes in the package configuration while the cluster is not running. Use the same steps as in “Reconfiguring a Package on a Running Cluster ”.
Resetting the Service Restart Counter The service restart counter tracks the number of times a package service has been automatically restarted. This value is used to determine when the package service has exceeded its maximum number of allowable automatic restarts. When a package service successfully restarts after several attempts, the package manager does not automatically reset the restart count.
Table 10 Types of Changes to Packages (continued) Change to the Package Required Package State NOTE: on it. You cannot delete a package if another package has a dependency Change package type Package must not be running. Add or delete a module: modular package Package can be running. Change run script contents: Package can be running, but should not be starting. legacy package Timing problems may occur if the script is changed while the package is starting.
Table 10 Types of Changes to Packages (continued) Change to the Package Required Package State Add or remove an IP (in control script): legacy package Package must not be running. (Also applies to cross-subnet configurations.) Add or delete nodes from Package can be running. package’s Serviceguard will reject the change if you are trying to add a node on which ip_subnet_node the specified ip_subnet is not configured.
Table 10 Types of Changes to Packages (continued) Change to the Package Required Package State Add or change a file system: legacy package Package must not be running. Remove a file system: modular package Package should not be running. Remove a file system: legacy package Package must not be running. CAUTION: Removing a file system may cause problems if the file system cannot be unmounted because it's in use by a running process.
Changes that Will Trigger Warnings Changes to the following will trigger warnings, giving you a chance to cancel, if the change would cause the package to fail. NOTE: You will not be able to cancel if you use cmapplyconf -f.
It is not necessary to halt the single node in this scenario, since the application is still running, and no other node is currently available for package switching. (This is different from the loss of the Serviceguard daemon in a multi-node cluster, which halts the node (system reset), and causes packages to be switched to adoptive nodes.
8 Troubleshooting Your Cluster This chapter describes how to verify cluster operation, how to review cluster status, how to add and replace hardware, and how to solve some typical cluster problems.
The package should be running on the specified adoptive node. 4. Halt the package, then move it back to the primary node using the cmhaltpkg, cmmodpkg, and cmrunpkg commands: cmhaltpkg cmmodpkg -e cmrunpkg -v Depending on the specific databases you are running, perform the appropriate database recovery. Testing the Cluster Manager To test that the cluster manager is operating correctly, perform the following steps for each node on the cluster: 1.
Configuration” (page 231). In addition, the following should be monitored for errors or warnings of all kinds: • CPUs • Memory • LAN cards • Power sources • All cables • Disk interface cards Some monitoring can be done through simple physical inspection, but for the most comprehensive monitoring, you should examine the system log file (/var/log/messages) periodically for reports on all configured HA devices. The presence of errors relating to a device will show the need for maintenance.
If you need to use a different devicefile, you must change the name of the devicefile in the cluster configuration file; see “Updating the Cluster Lock LUN Configuration Online” (page 270). CAUTION: Before you start, make sure that all nodes have logged a message such as the following in syslog: WARNING: Cluster lock LUN /dev/sda1 is corrupt: bad label. Until this situation is corrected, a single failure could cause all nodes in the cluster to crash.
pr_cleanup lun -v -k [-f | ] • lun, if used, specifies that a LUN, rather than a volume group, is to be operated on. • -v, if used, specifies verbose output detailing the actions the script performs and their status. • -k , if used, specifies the key to be used in the clear operation. • -f , if used, specifies that the name of the DSFs to be operated on are listed in the file specified by .
5. 6. 7. Power up the system. As the system comes up, the kudzu program on Red Hat systems will detect and report the hardware changes. Accept the changes and add any information needed for the new LAN card. On SUSE systems, run YAST2 after the system boots and make adjustments to the NIC setting of the new LAN card. If the old LAN card was part of a “bond”, the new LAN card needs to be made part of the bond.
3. Install and configure the quorum server software on the new system. Be sure to include in the new QS authorization file (for example, /usr/local/qs/conf/ qs_authfile) on all of the nodes that were configured for the old quorum server. Refer to the qs(1) man page for details about configuring the QS authorization file. NOTE: The quorum server reads the authorization file at startup. Whenever you modify the file qs_authfile, run the following command to force a re-read of the file.
NOTE: While the old quorum server is down and the new one is being set up: • The cmquerycl, cmcheckconf and cmapplyconf commands will not work • The cmruncl, cmhaltcl, cmrunnode, and cmhaltnode commands will work • If there is a node or network failure that creates a 50-50 membership split, the quorum server will not be available as a tie-breaker, and the cluster will fail.
Interrupt:9 Base address:0xda00 eth1:2 Link encap:Ethernet HWaddr 00:50:DA:64:8A:7C inet addr:192.168.1.201 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:9 Base address:0xda00 lo Link encap:Local Loopback inet addr:127.0.0.1 Bcast:192.168.1.255 Mask:255.255.255.
Dec Dec Dec Dec Dec Dec 14 14:39:27 star04 CM-CMD[2096]: cmruncl 14 14:39:27 star04 cmcld[2098]: Starting cluster management protocols. 14 14:39:27 star04 cmcld[2098]: Attempting to form a new cluster 14 14:39:27 star04 cmclconfd[2097]: Command execution message 14 14:39:33 star04 cmcld[2098]: 3 nodes have formed a new cluster 14 14:39:33 star04 cmcld[2098]: The new active cluster membership is: star04(id=1), star05(id=2), star06(id=3) Dec 14 17:39:33 star04 cmlvmd[2099]: Clvmd initialized successfully.
Using the cmquerycl and cmcheckconf Commands In addition, cmquerycl and cmcheckconf can be used to troubleshoot your cluster just as they were used to verify its configuration. The following example shows the commands used to verify the existing cluster configuration on ftsys9 and ftsys10: cmquerycl -v -C $SGCONF/verify.conf -n ftsys9 -n ftsys10 cmcheckconf -v -C $SGCONF/verify.conf cmcheckconf checks: • The network addresses and connections.
• Package Movement Errors. • Node and Network Failures. • Quorum Server Messages. Name Resolution Problems Many Serviceguard commands, including cmviewcl, depend on name resolution services to look up the addresses of cluster nodes. When name services are not available (for example, if a name server is down), Serviceguard commands may hang, or may return a network-related error message. If this happens, use the host command on each cluster node to see whether name resolution is correct.
1. Warning: cmcld was unable to run for the last seconds. Consult the Managing Serviceguard manual for guidance on setting MEMBER_TIMEOUT, and information on cmcld. This means that cmcld was unable to get access to a CPU for a significant amount of time. If this occurred while the cluster was re-forming, one or more nodes could have failed.
These are errors caused specifically by errors in the cluster configuration file and package configuration scripts. Examples of these errors include: • Volume groups not defined on adoptive node. • Mount point does not exist on adoptive node. • Network errors on adoptive node (configuration errors). • User information not correct on adoptive node. You can use the following commands to check the status of your disks: • df - to see if your package’s volume group is mounted.
2. on an alternate node. This might include such things as shutting down application processes, removing lock files, and removing temporary files. Ensure that package IP addresses are removed from the system. This step is accomplished via the cmmodnet(1m) command. First determine which package IP addresses are installed by inspecting the output resulting from running the ifconfig command.
the package will be automatically restarted on any available alternate node for which it is configured. Package Movement Errors These errors are similar to the system administration errors except they are caused specifically by errors in the package control script. The best way to prevent these errors is to test your package control script before putting your high availability application on line.
The reason may be that you have not updated the authorization file. Verify that the node is included in the file, and try using /usr/lbin/qs -update to re-read the quorum server authorization file. Timeout Problems The following kinds of message in a Serviceguard node’s syslog file may indicate timeout problems: Unable to set client version at quorum server 192.6.7.2: reply timed out Probe of quorum server 192.6.7.
A Designing Highly Available Cluster Applications This appendix describes how to create or port applications for high availability, with emphasis on the following topics: • Automating Application Operation • Controlling the Speed of Application Failover (page 308) • Designing Applications to Run on Multiple Systems (page 311) • Restoring Client Connections (page 316) • Handling Application Failures (page 317) • Minimizing Planned Downtime (page 318) Designing for high availability means reducing
There are two principles to keep in mind for automating application relocation: • Insulate users from outages. • Applications must have defined startup and shutdown procedures. You need to be aware of what happens currently when the system your application is running on is rebooted, and whether changes need to be made in the application's response for high availability. Insulate Users from Outages Wherever possible, insulate your end users from outages.
issues the command ps -ef | grep xxx for all the processes belonging to the application. To reduce the impact on users, the application should not simply abort in case of error, since aborting would cause an unneeded failover to a backup system. Applications should determine the exact error and take specific action to recover from the error rather than, for example, aborting upon receipt of any error.
advisable to take certain actions to minimize the amount of data that will be lost, as explained in the following discussion. Minimize the Use and Amount of Memory-Based Data Any in-memory data (the in-memory context) will be lost when a failure occurs. The application should be designed to minimize the amount of in-memory data that exists unless this data can be easily recalculated.
the system comes back up again? Does the job restart from the beginning, reprinting all the paychecks, does the job start from where it left off, or does the scheduler assume that the job was done and not print the last hours worth of paychecks? The correct behavior in a highly available environment is to restart where it left off, ensuring that everyone gets one and only one paycheck. Another example is an application where a clerk is entering data about a new employee.
there are also many issues with this sort of design. This discussion will not go into details other than to give a few examples. The simplest method is to have two applications running in a master/slave relationship where the slave is simply a hot standby application for the master. When the master fails, the slave on the second system would still need to figure out what state the data was in (i.e., data recovery would still take place).
The use of packages containing highly available applications adds the requirement for an additional set of IP addresses, which are assigned to the applications themselves. These are known as relocatable application IP addresses. Serviceguard’s network sensor monitors the node’s access to the subnet on which these relocatable application IP addresses reside.
with the LAN hardware. The use of these addresses is a common problem for license servers, since for security reasons they want to use hardware-specific identification to ensure the license isn't copied to multiple nodes. One workaround is to have multiple licenses; one for each node the application will run on. Another way is to have a cluster-wide mechanism that lists a set of SPU IDs or node names. If your application is running on a system in the specified set, then the license is approved.
the hostname. gethostbyname(3) will pass in the IP address of the application. This IP address will move with the application to the new node. However, gethostbyname(3) should be used to locate the IP address of an application only if the application name is configured in DNS. It is probably best to associate a different application name with each independent HA service. This allows each application and its IP address to be moved to another node without affecting other applications.
IP address is most appropriate for responses, so it will always pick the stationary IP address. For TCP stream sockets, the TCP level of the protocol stack resolves this problem for the client since it is a connection-based protocol. On the client, TCP ignores the stationary IP address and continues to use the previously bound relocatable IP address originally used by the client. With UDP datagram sockets, however, there is a problem.
Avoid File Locking In an NFS environment, applications should avoid using file-locking mechanisms, where the file to be locked is on an NFS Server. File locking should be avoided in an application both on local and remote systems. If local file locking is employed and the system fails, the system acting as the backup system will not have any knowledge of the locks maintained by the failed system. This may or may not cause problems when the application restarts.
server locally. This will keep the client from having to switch to the second server in the event of a application failure. • Use a transaction processing monitor or message queueing software to increase robustness. Use transaction processing monitors such as Tuxedo or DCE/Encina, which provide an interface between the server and the client. Transaction processing monitors (TPMs) can be useful in creating a more highly available application.
Another alternative is for the failure of one component to still allow bringing down the other components cleanly. If a database SQL server fails, the database should still be able to be brought down cleanly so that no database recovery is necessary. The worse case is for a failure of one component to cause the entire system to fail. If one component fails and all other components need to be restarted, the downtime will be high.
to the new version of the software, and then restart the application on all the affected nodes. For large systems, this could result in a long downtime. An alternative is to provide for a rolling upgrade. A rolling upgrade rolls out the new software in a phased approach by upgrading only one component at a time. For example, the database server is upgraded on Monday, causing a 15 minute downtime.
B Integrating HA Applications with Serviceguard The following is a summary of the steps you should follow to integrate an application into the Serviceguard environment: 1. Read the rest of this book, including the chapters on cluster and package configuration, and the appendix “Designing Highly Available Cluster Applications.” 2.
Defining Baseline Application Behavior on a Single System 1. Define a baseline behavior for the application on a standalone system: • Install the application, database, and other required resources on one of the systems. Be sure to follow Serviceguard rules in doing this: ◦ Install all shared data on separate external volume groups. ◦ Use a journaled filesystem (JFS) as appropriate. • Perform some sort of standard test to ensure the application is running correctly.
Testing the Cluster 1. Test the cluster: • Have clients connect. • Provide a normal system load. • Halt the package on the first node and move it to the second node: # cmhaltpkg pkg1 # cmrunpkg -n node2 pkg1 # cmmodpkg -e pkg1 • Move it back. # cmhaltpkg pkg1 # cmrunpkg -n node1 pkg1 # cmmodpkg -e pkg1 2. 3. • Fail one of the systems. For example, turn off the power on node 1. Make sure the package starts up on node 2. • Repeat failover from node 2 back to node 1.
C Blank Planning Worksheets This appendix reprints blank versions of the planning worksheets described in the “Planning” chapter. You can duplicate any of these worksheets that you find useful and fill them in as a part of the planning process.
Power Supply Worksheet ============================================================================ SPU Power: Host Name ____________________ Power Supply _____________________ Host Name ____________________ Power Supply _____________________ ============================================================================ Disk Power: Disk Unit __________________________ Power Supply _______________________ Disk Unit __________________________ Power Supply _______________________ Disk Unit ______________
Cluster Name: ___________________________________________________________ Host Names ____________________________________________ Host Names ____________________________________________ Cluster Name: ___________________________________________________________ Host Names ____________________________________________ Host Names ____________________________________________ Volume Group and Physical Volume Worksheet ============================================================================== Volume Group Nam
=============================================================================== Heartbeat Subnet: __________________________ Monitored Non-heartbeat Subnet: __________________ Monitored Non-heartbeat Subnet: ___________________ =============================================================================== Timing Parameters: =============================================================================== Heartbeat Interval: __________ ==========================================================================
Package environment variable:________________________________________________ Package environment variable:________________________________________________ External pre-script:_________________________________________________________ External script:_____________________________________________________________================================================================================ Package Control Script Worksheet (Legacy) PACKAGE CONTROL SCRIPT WORKSHEET Page ___ of ___ ============================
D IPv6 Network Support This appendix describes some of the characteristics of IPv6 network addresses, specifically: • IPv6 Address Types • Network Configuration Restrictions (page 332) • Configuring IPv6 on Linux (page 332) IPv6 Address Types Several IPv6 types of addressing schemes are specified in the RFC 2373 (IPv6 Addressing Architecture). IPv6 addresses are 128-bit identifiers for interfaces and sets of interfaces. There are various address formats for IPv6 defined by the RFC 2373.
of the 32-bit lower order bits. Typically IPv4 Mapped IPv6 addresses and IPv4 Compatible IPv6 addresses will be represented in this notation. These addresses are discussed in later sections. Examples: 0:0:0:0:0:0:10.1.2.3 and ::10.11.3.123 IPv6 Address Prefix IPv6 Address Prefix is similar to CIDR in IPv4 and is written in CIDR notation.
IPv4 Compatible IPv6 Addresses The IPv6 transition mechanisms use a technique for tunneling IPv6 packets over the existing IPv4 infrastructure. IPv6 nodes that support such mechanisms use a special kind of IPv6 addresses that carry IPv4 addresses in their lower order 32-bits. These addresses are called IPv4 Compatible IPv6 addresses. They are represented as follows: Table 13 80 bits 16 bits 32 bits zeros 0000 IPv4 address Example: ::192.168.0.
Interface ID = Interface Identifier. Link-Local Addresses Link-local addresses have the following format: Table 16 10 bits 54 bits 64 bits 1111111010 0 interface ID Link-local address are supposed to be used for addressing nodes on a single link. Packets originating from or destined to a link-local address will not be forwarded by a router.
Network Configuration Restrictions Serviceguard supports IPv6 for data and heartbeat IP. The restrictions on support for IPv6 in Serviceguard for Linux are: • Auto-configured IPv6 addresses are not supported in Serviceguard. as HEARTBEAT_IP or STATIONARY_IP addresses. IPv6 addresses that are part of a Serviceguard cluster configuration must not be auto-configured through router advertisements, for example.
Enabling IPv6 on Red Hat Linux Add the following lines to /etc/sysconfig/network: NETWORKING_IPV6=yes IPV6FORWARDING=no IPV6_AUTOCONF=no IPV6_AUTOTUNNEL=no # Enable global IPv6 initialization # Disable global IPv6 forwarding # Disable global IPv6 autoconfiguration # Disable automatic IPv6 tunneling Adding persistent IPv6 Addresses on Red Hat Linux This can be done by modifying the system configuration script, for example, /etc/sysconfig/network-scripts/ifcfg-eth1: DEVICE=eth1BOOTPROTO=static BROADCAST=192
BOOTPROTO=static BROADCAST=10.10.18.255 IPADDR=10.10.18.18 MTU="" NETMASK=255.255.255.0 NETWORK=10.10.18.0 REMOTE_IPADDR="" STARTMODE=onboot IPADDR1=3ffe::f101:10/64 IPADDR2=fec0:0:0:1::10/64 Configuring a Channel Bonding Interface with Persistent IPv6 Addresses on SUSE Configure the following parameters in /etc/sysconfig/network/ifcfg-bond0: BOOTPROTO=static BROADCAST=10.0.2.255 IPADDR=10.0.2.10 NETMASK=255.255.0.0 NETWORK=0.0.2.
E Using Serviceguard Manager HP Serviceguard Manager is a web-based, HP System Management Homepage (HP SMH) tool that replaces the functionality of the earlier Serviceguard management tools. Serviceguard Manager allows you to monitor, administer and configure a Serviceguard cluster from any system with a supported web browser. The Serviceguard Manager Main Page provides you with a summary of the health of the cluster including the status of each node and its packages.
NOTE: SMH access roles constrain a user's cluster-management capabilities: • A user with HP SMH Administrator access has full cluster management capabilities. • A user with HP SMH Operator access can monitor the cluster and has restricted cluster management capabilities as defined by the user’s Serviceguard role-based access configuration. • A user with HP SMH User access does not have any cluster management capabilities. 1.
Figure 28 System Management Homepage with Serviceguard Manager Number What is it? Description 1 Cluster and Displays information about the Cluster status, alerts and general information. Overall status NOTE: The System Tools menu item is not available in this version of and alerts Serviceguard Manager. 2 Menu tool bar The menu tool bar is available from the HP Serviceguard Manager Homepage, and from any cluster, node or package view-only property page.
Scenario 2 applies if you have: • One or more clusters with Serviceguard version A.11.15.01 through A.11.19 • Serviceguard Manager version A.05.01 with HP SIM 5.10 or later installed on a server NOTE: Serviceguard Manager can be launched by HP Systems Insight Manager version 5.10 or later if Serviceguard Manager is installed on an HP Systems Insight Manager Central Management Server. For a Serviceguard A.11.19 cluster, Systems Insight Manager will attempt to launch Serviceguard Manager B.02.
Figure 29 Cluster by Type 4. Expand HP Serviceguard, and click on a Serviceguard cluster. NOTE: If you click on a cluster running an earlier Serviceguard release, the page will display a link that will launch Serviceguard Manager A.05.01 (if installed) via Java Webstart.
F Monitoring Script for Generic Resources The monitoring scripts are the scripts written by an end-user and must contain the core logic to monitor a resource and set the status of a generic resource. These scripts are started as a part of the package start. • You can set the status/value of a simple/extended resource respectively using the cmsetresource(1m) command. • You can define the monitoring interval in the script.
Launching Monitoring Scripts Monitoring scripts can be launched in the following ways: For resources of evaluation_type: during_package_start • Monitoring scripts can be launched through the services functionality that is available in packages, as indicated by service_name, service_cmd, and service_halt_timeout. This makes the monitoring scripts highly available, since Serviceguard monitors them and is the recommended approach.
service_name service_cmd lan1_monitor $SGCONF/generic_resource_monitors/lan1.sh service_name service_cmd cpu_monitor $SGCONF/generic_resource_monitors/cpu_monitor.sh The above example shows a sample multi-node package named generic_resource_monitors and has two monitoring scripts configured — one each to monitor a LAN and CPU. These monitoring scripts will monitor the LAN interface, CPU and sets the status of the generic resources defined in them accordingly.
Figure 30 depicts a multi-node package containing two monitoring scripts configured — one to monitor a lan and other to monitor a CPU. The two packages are configured with the generic resource names and are dependent on the multi-node package. Figure 30 Multi-node package configured with all the monitoring scripts for generic resources of type before_package_start Template of a Monitoring Script Monitoring Script Template — /etc/cmcluster/examples/generic_resource_monitor.
# * configured in the package configuration file, then log * # * messages that have a log level that is equal to or * # * greater than the configured log level will be output. * # * * # * In addition, the format of the time stamp is prefixed in * # * front of the log message. * # * * # * * # ********************************************************************** # ################################################################### ########################### # Source utility functions.
sg_log 5 "stop_command" # ADD your halt steps here exit 1 } ################ # main routine ################ sg_log 5 "customer defined monitor script" ######################################################################### # # Customer defined monitor script should be doing following # functionality. # # When the package is halting, cmhaltpkg will issue a SIGTERM signal to # the service(s) configured in package. Use SIGTERM handler to stop # the monitor script.
Before using this script, copy it to the package specific directory. This script expects the following parameters when configured as a service: • $1 : Generic resource name: The name of the generic resource configured in the package that corresponds to this monitoring script. • $2 : LVM volume group name: The name of the LVM logical volume group whose physical disks need to be monitored. • $3 : Monitor interval: The time interval between successive monitors.
Index A Access Control Policies, 186 active node, 21 adding a package to a running cluster, 282 adding cluster nodes advance planning, 154 adding nodes to a running cluster, 244 adding packages on a running cluster, 231 administration adding nodes to a running cluster, 244 halting a package, 253 halting the entire cluster, 246 moving a package, 254 of packages and services, 252 of the cluster, 243 reconfiguring a package while the cluster is running, 281 reconfiguring a package with the cluster offline, 282
defined, 37 dynamic re-formation, 39 heartbeat subnet parameter, 109 initial configuration of the cluster, 37 main functions, 37 maximum configured packages parameter, 120 member timeout parameter, 115 monitored non-heartbeat subnet, 112 network polling interval parameter, 116, 120 planning the configuration, 103 quorum server parameter, 106 testing, 290 cluster node parameter in cluster manager configuration, 104, 106, 107 cluster parameters initial configuration, 37 cluster re-formation scenario, 87 clust
explanations package parameters, 205 F failback policy used by package manager, 52 FAILBACK_POLICY parameter used by package manager, 52 failover controlling the speed in applications, 308 defined, 21 failover behavior in packages, 125 failover package, 44, 200 failover policy used by package manager, 48 FAILOVER_POLICY parameter used by package manager, 48 failure kinds of responses, 86 network communication, 90 response to hardware failures, 88 responses to package and service failures, 89 restarting a s
hardware planning, 94 I/O slots hardware planning, 92, 94 identifying cluster-aware volume groups, 185 Installing Serviceguard, 156 installing software quorum server, 169 integrating HA applications with Serviceguard, 320 introduction Serviceguard at a glance, 19 understanding Serviceguard hardware, 24 understanding Serviceguard software, 32 IP in sample package control script, 275 IP address adding and deleting in packages, 70 for nodes and packages, 68 hardware planning, 93, 97 portable, 69 reviewing for
network communication failure, 90 network components in Serviceguard, 25 network manager adding and deleting package IP addresses, 70 main functions, 68 network planning subnet, 93, 97 network polling interval (NETWORK_POLLING_INTERVAL) parameter in cluster manager configuration, 116, 120 network time protocol (NTP) for clusters, 162 networking redundant subnets, 93 networks binding to IP addresses, 314 binding to port addresses, 314 IP addresses and naming, 311 node and package IP addresses, 68 packages us
for failover, 125 pacakge configuration, 205 parameters for cluster manager initial configuration, 37 PATH, 224 physical volume for cluster lock, 40, 41 physical volumes blank planning worksheet, 325 planning, 97 planning cluster configuration, 99 cluster lock and cluster expansion, 97 cluster manager configuration, 103 disk I/O information, 94 for expansion, 124 hardware configuration, 92 high availability objectives, 91 overview, 91 package configuration, 120 power, 95 quorum server, 97 SPU information, 9
and syslog, 34 duration, 34 sample disk configurations, 31 Sample monitoring script for generic resources, 343 service administration, 252 service configuration step by step, 199 service failures responses, 89 service restarts, 90 SERVICE_CMD in sample package control script, 275 SERVICE_FAIL_FAST_ENABLED and node TOC, 89 SERVICE_NAME in sample package control script, 275 SERVICE_RESTART in sample package control script, 275 Serviceguard install, 156 introduction, 19 Serviceguard at a Glance, 19 Serviceguar
figure, 21 typical cluster configuration figure, 20 U uname(2), 314 understanding network components in Serviceguard, 25 UPS in power planning, 95 power supply, 31 use of the cluster lock, 40, 41 V verifying cluster configuration, 192 verifying the cluster and package configuration, 230, 277, 278 VG in sample package control script, 275 vgcfgbackup using to back up volume group configuration, 178 VGCHANGE in package control script, 275 VGChange, 225 volume group for cluster lock, 40, 41 planning, 97 volum
