Managing HP Serviceguard for Linux, Ninth Edition HP Part Number: B9903-90068 Published: April 2009
Legal Notices The information in this document is subject to change without notice. Hewlett-Packard makes no warranty of any kind with regard to this manual, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. Hewlett-Packard shall not be held liable for errors contained herein or direct, indirect, special, incidental or consequential damages in connection with the furnishing, performance, or use of this material. Warranty.
Table of Contents Printing History ...........................................................................................................................19 Preface.......................................................................................................................................21 1 Serviceguard for Linux at a Glance............................................................................................23 What is Serviceguard for Linux? ..........................................
Service Assistant Daemon: cmserviced...................................................................40 Quorum Server Daemon: qs....................................................................................41 Utility Daemon: cmlockd.........................................................................................41 Cluster SNMP Agent Daemon: cmsnmpd...............................................................41 Cluster WBEM Agent Daemon: cmwbemd..............................................
Types of IP Addresses...................................................................................................73 Adding and Deleting Relocatable IP Addresses ..........................................................73 Load Sharing ...........................................................................................................73 Bonding of LAN Interfaces ...........................................................................................74 Bonding for Load Balancing..............
Power Supply Configuration Worksheet .....................................................................98 Cluster Lock Planning........................................................................................................98 Cluster Lock Requirements...........................................................................................98 Planning for Expansion.................................................................................................99 Using a Quorum Server.............
Configuring node_name...................................................................................143 Configuring monitored_subnet_access............................................................143 Configuring ip_subnet_node............................................................................144 Configuring a Package: Next Steps.............................................................................144 Planning for Changes in Cluster Size.................................................
Specifying the Address Family for the Heartbeat .................................................172 Full Network Probing............................................................................................172 Specifying a Lock LUN................................................................................................172 Specifying a Quorum Server.......................................................................................173 Obtaining Cross-Subnet Information.......................
halt_script_timeout...................................................................................................199 successor_halt_timeout.............................................................................................200 script_log_file...........................................................................................................200 operation_sequence...................................................................................................200 log_level...............
Before You Start...........................................................................................................214 cmmakepkg Examples.................................................................................................214 Next Step.....................................................................................................................215 Editing the Configuration File..........................................................................................
Performing Maintenance Using Partial-Startup Maintenance Mode..........................237 Excluding Modules................................................................................................238 Characteristics of a Package Running in Partial-Startup Maintenance Mode............239 Dependency Rules for a Package in Partial-Startup Maintenance Mode...................240 Reconfiguring a Cluster....................................................................................................
Migrating a Legacy Package to a Modular Package....................................................262 Reconfiguring a Package on a Running Cluster .........................................................262 Reconfiguring a Package on a Halted Cluster ............................................................263 Adding a Package to a Running Cluster.....................................................................263 Deleting a Package from a Running Cluster ...........................................
Authorization File Problems..................................................................................286 Timeout Problems..................................................................................................287 Messages................................................................................................................287 Lock LUN Messages....................................................................................................
Providing Online Application Reconfiguration .........................................................302 Documenting Maintenance Operations .....................................................................302 B Integrating HA Applications with Serviceguard..........................................................................305 Checklist for Integrating HA Applications ......................................................................
Launching Serviceguard Manager....................................................................................325 Scenario 1 - Single cluster management......................................................................325 Scenario 2- Multi-Cluster Management......................................................................327 Index........................................................................................................................................
List of Figures 1-1 1-2 1-3 2-1 2-2 3-1 3-2 3-3 3-4 3-5 3-6 3-7 3-8 3-9 3-10 3-11 3-12 3-13 3-14 3-15 3-16 3-17 3-18 3-19 3-20 3-21 5-1 E-1 E-2 16 Typical Cluster Configuration ....................................................................................24 Typical Cluster After Failover ....................................................................................25 Tasks in Configuring a Serviceguard Cluster ............................................................28 Redundant LANs ......
List of Tables 1 3-1 3-2 3-3 4-1 5-1 6-1 6-2 7-1 7-2 D-1 D-2 D-3 D-4 D-5 D-6 D-7 D-8 .....................................................................................................................................19 Package Configuration Data.......................................................................................54 Node Lists in Sample Cluster......................................................................................
Printing History Table 1 Printing Date Part Number Edition November 2001 B9903-90005 First November 2002 B9903-90012 First December 2002 B9903-90012 Second November 2003 B9903-90033 Third February 2005 B9903-90043 Fourth June 2005 B9903-90046 Fifth August 2006 B9903-90050 Sixth July 2007 B9903-90054 Seventh March 2008 B9903-90060 Eighth April 2009 B9903-90068 Ninth The last printing date and part number indicate the current edition, which applies to the A.11.
Preface This guide describes how to configure and manage Serviceguard for Linux on HP ProLiant and HP Integrity servers under the Linux operating system. It is intended for experienced Linux system administrators. (For Linux system administration tasks that are not specific to Serviceguard, use the system administration documentation and manpages for your distribution of Linux.
distributions refer to the HP Serviceguard for Linux Certification Matrix. Both documents are available at: http://www.hp.com/info/sglx Problem Reporting If you have any problems with the software or documentation, please contact your local Hewlett-Packard Sales Office or Customer Service Center.
1 Serviceguard for Linux at a Glance This chapter introduces Serviceguard for Linux and shows where to find different kinds of information in this book. It includes the following topics: • What is Serviceguard for Linux? • Using Serviceguard Manager (page 26) • Configuration Roadmap (page 27) If you are ready to start setting up Serviceguard clusters, skip ahead to Chapter 4 (page 93). Specific steps for setup are in Chapter 5 (page 147).
Figure 1-1 Typical Cluster Configuration In the figure, node 1 (one of two SPU's) is running package A, and node 2 is running package B. Each package has a separate group of disks associated with it, containing data needed by the package's applications, and a copy of the data. Note that both nodes are physically connected to disk arrays. However, only one node at a time may access the data for a given group of disks.
Failover Under normal conditions, a fully operating Serviceguard cluster simply monitors the health of the cluster's components while the packages are running on individual nodes. Any host system running in the Serviceguard cluster is called an active node. When you create the package, you specify a primary node and one or more adoptive nodes. When a node or its network communications fails, Serviceguard can transfer control of the package to the next available adoptive node.
uninterruptible power source. For more details, refer to the section on “Power Supply Planning” in Chapter 4, “Planning and Documenting an HA Cluster.” Serviceguard is designed to work in conjunction with other high availability products, such as disk arrays, which use various RAID levels for data protection; and HP-supported uninterruptible power supplies (UPS), which eliminate failures related to power outage.
Administering Clusters with Serviceguard Manager You can administer clusters, nodes, and packages if access control policies permit: • • • Cluster: halt, run Cluster nodes: halt, run Package: halt, run, move from one node to another, reset node- and package-switching flags Configuring Clusters with Serviceguard Manager You can configure clusters and packages in Serviceguard Manager. You must have root (UID=0) access to the cluster nodes.
Figure 1-3 Tasks in Configuring a Serviceguard Cluster HP recommends that you gather all the data that is needed for configuration before you start. See Chapter 4 (page 93) for tips on gathering data.
2 Understanding Hardware Configurations for Serviceguard for Linux This chapter gives a broad overview of how the server hardware components operate with Serviceguard for Linux. The following topics are presented: • Redundant Cluster Components • Redundant Network Components (page 30) • Redundant Disk Storage (page 34) • Redundant Power Supplies (page 36) Refer to the next chapter for information about Serviceguard software components.
Redundant Network Components To eliminate single points of failure for networking, each subnet accessed by a cluster node is required to have redundant network interfaces. Redundant cables are also needed to protect against cable failures. Each interface card is connected to a different cable and hub or switch. Network interfaces are allowed to share IP addresses through a process known as channel bonding.
unless those IP addresses themselves will be immediately configured into the cluster as stationary IP addresses. CAUTION: If you configure any address other than a stationary IP address on a Serviceguard network interface, it could collide with a relocatable package IP address assigned by Serviceguard. See “Stationary and Relocatable IP Addresses and Monitored Subnets” (page 71).
Figure 2-1 Redundant LANs In Linux configurations, the use of symmetrical LAN configurations is strongly recommended, with the use of redundant hubs or switches to connect Ethernet segments. The software bonding configuration should be identical on each node, with the active interfaces connected to the same hub or switch. Cross-Subnet Configurations As of Serviceguard A.11.
Configuration Tasks Cluster and package configuration tasks are affected as follows: • • You must use the -w full option to cmquerycl to discover actual or potential nodes and subnets across routers.
the node_name list for that package. Conversely, if all of the subnets that are being monitored for this package are configured for PARTIAL access, each node on the node_name list must have at least one of these subnets configured. — As in other configurations, a package will not start on a node unless the subnets configured on that node, and specified in the package configuration file as monitored subnets, are up.
Supported Disk Interfaces The following interfaces are supported by Serviceguard for disks that are connected to two or more nodes (shared data disks): • • MSA (Modular Smart Array) 2000 family FibreChannel. For information on configuring multipathing, see “Multipath for Storage ” (page 96). Disk Monitoring You can configure monitoring for disks and configure packages to be dependent on the monitor. For each package, you define a package service that monitors the disks that are activated by that package.
Figure 2-2 Mirrored Disks Connected for High Availability Redundant Power Supplies You can extend the availability of your hardware by providing battery backup to your nodes and disks. HP-supported uninterruptible power supplies (UPS) can provide this protection from momentary power loss. Disks should be attached to power circuits in such a way that disk array copies are attached to different power sources. The boot disk should be powered from the same circuit as its corresponding node.
3 Understanding Serviceguard Software Components This chapter gives a broad overview of how the Serviceguard software components work.
Figure 3-1 Serviceguard Software Components on Linux Serviceguard Daemons Serviceguard for Linux uses the following daemons: • cmclconfd—configuration daemon • cmcld—cluster daemon • cmnetd—Network Manager daemon • cmlogd—cluster system log daemon • cmdisklockd—cluster lock LUN daemon • cmomd—Cluster Object Manager daemon • cmserviced—Service Assistant daemon • qs—Quorum Server daemon • cmlockd—utility daemon • cmsnmpd—cluster SNMP subagent (optionally running) • cmwbemd—WBEM daemon • cmproxyd—proxy daemon
on Red Hat or /var/log/qs/sq.log on SUSE and cmomd logs to /usr/local/cmom/log/cmomd.log on Red Hat or /var/log/cmom/log/cmomd.log on SUSE. NOTE: The file cmcluster.conf contains the mappings that resolve symbolic references to $SGCONF, $SGROOT, $SGLBIN, etc, used in the pathnames in the subsections that follow. See “Understanding the Location of Serviceguard Files” (page 147) for details.
NOTE: Two of the central components of Serviceguard—Package Manager, and Cluster Manager—run as parts of the cmcld daemon. This daemon runs at priority 94 and is in the SCHED_RR class. No other process is allowed a higher real-time priority. Log Daemon: cmlogd cmlogd is used by cmcld to write messages to the system log file. Any message written to the system log by cmcld it written through cmlogd. This is to prevent any delays in writing to syslog from impacting the timing of cmcld.
For services, cmcld monitors the service process and, depending on the number of service retries, cmcld either restarts the service through cmsrvassistd or it causes the package to halt and moves the package to an available alternate node. The path for this daemon is: $SGLBIN/cmserviced. Quorum Server Daemon: qs Using a quorum server is one way to break a tie and establish a quorum when the cluster is re-forming; the other way is to use a Lock LUN.
The installation of the cmsnmpd rpm configures snmpd and cmsnmpd to start up automatically. Their startup scripts are in /etc/init.d/. The scripts can be run manually to start and stop the daemons. For more information, see the cmsnmpd (1)manpage.
Heartbeat Messages Central to the operation of the cluster manager is the sending and receiving of heartbeat messages among the nodes in the cluster. Each node in the cluster exchanges UDP heartbeat messages with every other node over each IP network configured as a heartbeat device. If a cluster node does not receive heartbeat messages from all other cluster nodes within the prescribed time, a cluster re-formation is initiated; see “What Happens when a Node Times Out” (page 88).
During startup, the cluster manager software checks to see if all nodes specified in the startup command are valid members of the cluster, are up and running, are attempting to form a cluster, and can communicate with each other. If they can, then the cluster manager forms the cluster. Automatic Cluster Startup An automatic cluster startup occurs any time a node reboots and joins the cluster.
Cluster Lock Although a cluster quorum of more than 50% is generally required, exactly 50% of the previously running nodes may re-form as a new cluster provided that the other 50% of the previously running nodes do not also re-form. This is guaranteed by the use of a tie-breaker to choose between the two equal-sized node groups, allowing one group to form the cluster and forcing the other group to shut down. This tie-breaker is known as a cluster lock.
Figure 3-2 Lock LUN Operation Serviceguard periodically checks the health of the lock LUN and writes messages to the syslog file if the disk fails the health check. This file should be monitored for early detection of lock disk problems. Use of the Quorum Server as a Cluster Lock The cluster lock in Linux can also be implemented by means of a quorum server. A quorum server can be used in clusters of any size.
Figure 3-3 Quorum Server Operation A quorum server can provide quorum services for multiple clusters. Figure 3-4 illustrates quorum server use across four clusters.
IMPORTANT: For more information about the quorum server, see the latest version of the HP Serviceguard Quorum Server release notes at http://docs.hp.com -> High Availability -> Quorum Server. No Cluster Lock Normally, you should not configure a cluster of three or fewer nodes without a cluster lock. In two-node clusters, a cluster lock is required.
IMPORTANT: During Step 1, while the nodes are using a strict majority quorum, node failures can cause the cluster to go down unexpectedly if the cluster has been using a quorum device before the configuration change. For example, suppose you change the quorum server polling interval while a two-node cluster is running. If a node fails during Step 1, the cluster will lose quorum and go down, because a strict majority of prior cluster members (two out of two in this case) is required.
Failover Packages A failover package starts up on an appropriate node (see node_name (page 197)) when the cluster starts. In the case of a service, network, or other resource or dependency failure, package failover takes place. A package failover involves both halting the existing package and starting the new instance of the package on a new node. Failover is shown in the following figure: Figure 3-5 Package Moving During Failover Configuring Failover Packages You configure each package separately.
by a master control script that is installed with Serviceguard; see Chapter 6: “Configuring Packages and Their Services ” (page 189), for instructions for creating modular packages. Deciding When and Where to Run and Halt Failover Packages The package configuration file assigns a name to the package and includes a list of the nodes on which the package can run. Failover packages list the nodes in order of priority (i.e., the first node in the list is the highest priority node).
The switching of relocatable IP addresses is shown in the figures that follow. Users connect to each node with the IP address of the package they wish to use. Each node has a stationary IP address associated with it, and each package has an IP address associated with it. Figure 3-6 Before Package Switching In Figure 3-7, node1 has failed and pkg1 has been transferred to node2. pkg1's IP address was transferred to node2 along with the package. pkg1 continues to be available and is now running on node2.
NOTE: For design and configuration information about clusters that span subnets, see the documents listed under “Cross-Subnet Configurations” (page 32). Figure 3-7 After Package Switching Failover Policy The Package Manager selects a node for a failover package to run on based on the priority list included in the package configuration file together with the failover_policy parameter, also in the configuration file.
If you use configured_node as the value for the failover policy, the package will start up on the highest priority node available in the node list. When a failover occurs, the package will move to the next highest priority node in the list that is available. If you use min_package_node as the value for the failover policy, the package will start up on the node that is currently running the fewest other packages.
Figure 3-8 Rotating Standby Configuration before Failover If a failure occurs, the failing package would fail over to the node containing fewest running packages: How the Package Manager Works 55
Figure 3-9 Rotating Standby Configuration after Failover NOTE: Under the min_package_node policy, when node2 is repaired and brought back into the cluster, it will then be running the fewest packages, and thus will become the new standby node. If these packages had been set up using the configured_node failover policy, they would start initially as in Figure 3-8, but the failure of node2 would cause the package to start on node3, as shown in Figure 3-10.
Figure 3-10 configured_node Policy Packages after Failover If you use configured_node as the failover policy, the package will start up on the highest-priority eligible node in its node list. When a failover occurs, the package will move to the next eligible node in the list, in the configured order of priority.
Figure 3-11 Automatic Failback Configuration before Failover Table 3-2 Node Lists in Sample Cluster Package Name NODE_NAME List FAILOVER POLICY FAILBACK POLICY pkgA node1, node4 configured_node automatic pkgB node2, node4 configured_node automatic pkgC node3, node4 configured_node automatic node1 panics, and after the cluster reforms, pkgA starts running on node4: 58 Understanding Serviceguard Software Components
Figure 3-12 Automatic Failback Configuration After Failover After rebooting, node1 rejoins the cluster. At that point, pkgA will be automatically stopped on node4 and restarted on node1.
Figure 3-13 Automatic Failback Configuration After Restart of node1 NOTE: Setting the failback_policy to automatic can result in a package failback and application outage during a critical production period. If you are using automatic failback, you may want to wait to add the package’s primary node back into the cluster until you can allow the package to be taken out of service temporarily while it switches back to the primary node.
a new template, and then copy the parameter values into it. In the new template, read the descriptions and defaults of the choices that did not exist when the original configuration was made. For example, the default for failover_policy is now configured_node and the default for failback_policy is now manual. For full details of the current parameters and their default values, see Chapter 6: “Configuring Packages and Their Services ” (page 189), and the package configuration file template itself.
A failover package starts on the first available node in its configuration file; by default, it fails over to the next available one in the list. Note that you do not necessarily have to use a cmrunpkg command to restart a failed failover package; in many cases, the best way is to enable package and/or node switching with the cmmodpkg command. When you create the package, you indicate the list of nodes on which it is allowed to run. System multi-node packages must list all cluster nodes in their cluster.
NOTE: This diagram applies specifically to legacy packages. Differences for modular scripts are called out below. Figure 3-14 Legacy Package Time Line Showing Important Events The following are the most important moments in a package’s life: 1. Before the control script starts. (For modular packages, this is the master control script.) 2. During run script execution. (For modular packages, during control script execution to start the package.) 3. While services are running 4.
Before the Control Script Starts First, a node is selected. This node must be in the package’s node list, it must conform to the package’s failover policy, and any resources required by the package must be available on the chosen node. One resource is the subnet that is monitored for the package. If the subnet is not available, the package cannot start on this node. Another type of resource is a dependency on another package.
Figure 3-15 Legacy Package Time Line At any step along the way, an error will result in the script exiting abnormally (with an exit code of 1). For example, if a package service is unable to be started, the control script will exit with an error. NOTE: This diagram is specific to legacy packages. Modular packages also run external scripts and “pre-scripts” as explained above.
NOTE: After the package run script has finished its work, it exits, which means that the script is no longer executing once the package is running normally. After the script exits, the PIDs of the services started by the script are monitored by the package manager directly. If the service dies, the package manager will then run the package halt script or, if service_fail_fast_enabled (page 208) is set to yes, it will halt the node on which the package is running.
NOTE: If you set restarts and also set service_fail_fast_enabled to yes, the failfast will take place after restart attempts have failed. It does not make sense to set service_restart to “-R” for a service and also set service_fail_fast_enabled to yes.
a graceful shutdown of the package that is followed by disabling automatic package startup (see “auto_run” (page 198)). You cannot halt a multi-node or system multi-node package unless all the packages that have a configured dependency on it are down. Use cmviewcl to check the status of dependents. For example, if pkg1 and pkg2 depend on PKGa, both pkg1 and pkg2 must be halted before you can halt PKGa.
Figure 3-16 Legacy Package Time Line for Halt Script Execution At any step along the way, an error will result in the script exiting abnormally (with an exit code of 1). If the halt script execution is not complete before the time specified in the halt_script_timeout (page 199), the package manager will kill the script. During halt script execution, messages are written to a log file.
• • • 0—normal exit. The package halted normally, so all services are down on this node. 1—abnormal exit, also known as no_restart exit. The package did not halt normally. Services are killed, and the package is disabled globally. It is not disabled on the current node, however. Timeout—Another type of exit occurs when the halt_script_timeout is exceeded. In this scenario, the package is killed and disabled globally. It is not disabled on the current node, however.
Table 3-3 Error Conditions and Package Movement for Failover Packages (continued) Package Error Condition Results Error or Exit Code Node Failfast Enabled Service Failfast Enabled Linux Status Halt script on Primary runs after after Error Error or Exit Package Allowed Package to Run on Primary Allowed to Node after Error Run on Alternate Node Halt Script Timeout YES Either Setting system reset N/A N/A (system reset) Yes, unless the timeout happened after the cmhaltpkg command was executed.
/etc/sysconfig/network-scripts/ifcfg- on Red Hat or /etc/sysconfig/network/ifcfg- on SUSE. The stationary IP address is not associated with packages, and it is not transferable to another node. Stationary IP addresses are used to transmit data, heartbeat messages (described under “How the Cluster Manager Works ” (page 42)), or both.
IMPORTANT: Any subnet that is used by a package for relocatable addresses should be configured into the cluster via NETWORK_INTERFACE and either STATIONARY_IP or HEARTBEAT_IP in the cluster configuration file. For more information about those parameters, see “Cluster Configuration Parameters ” (page 100). For more information about configuring relocatable addresses, see the descriptions of the package ip_ parameters (page 205).
loaded system when necessary) you can do so by putting each service in its own package and giving it a unique IP address. Bonding of LAN Interfaces Several LAN interfaces on a node can be grouped together in a process known in Linux as channel bonding. In the bonded group, typically one interface is used to transmit and receive data, while the others are available as backups. If one interface fails, another interface in the bonded group takes over.
Figure 3-17 Bonded Network Interfaces The LANs in the non-bonded configuration have four LAN cards, each associated with a separate non-aggregated IP address and MAC address, and each with its own LAN name (eth1, eth2, eth3, eth4). When these ports are aggregated, all four ports are associated with a single IP address and MAC address. In this example, the aggregated ports are collectively known as bond0, and this is the name by which the bond is known during cluster configuration.
Figure 3-18 Bonded NICs Node2 Node1 bond0: bond0: eth0 eth1 eth0 eth1 active active Hub Crossover cable Hub In the bonding model, individual Ethernet interfaces are slaves, and the bond is the master. In the basic high availability configuration (mode 1), one slave in a bond assumes an active role, while the others remain inactive until a failure is detected. (In Figure 3-18, both eth0 slave interfaces are active.
Figure 3-19 Bonded NICs After Failure Various combinations of Ethernet card types (single or dual-ported) and bond groups are possible, but it is vitally important to remember that at least two physical cards (or physically separate on-board LAN interfaces) must be used in any combination of channel bonds to avoid a single point of failure for heartbeat connections.
Figure 3-20 Bonded NICs Configured for Load Balancing Monitoring LAN Interfaces and Detecting Failure: Link Level At regular intervals, determined by the NETWORK_POLLING_INTERVAL (see “Cluster Configuration Parameters ” (page 100)), Serviceguard polls all the network interface cards specified in the cluster configuration file (both bonded and non-bonded).
Reasons To Use IP Monitoring Beyond the capabilities already provided by link-level monitoring, IP monitoring can: • Monitor network status beyond the first level of switches; see “How the IP Monitor Works” (page 79) • Detect and handle errors such as: — IP packet corruption on the router or switch — Link failure between switches and a first-level router — Inbound failures — Errors that prevent packets from being received but do not affect the link-level health of an interface IMPORTANT: You should configur
The IP Monitor section of the cmquerycl output looks similar to this: … Route Connectivity (no probing was performed): IPv4: 1 16.89.143.192 16.89.120.0 … Possible IP Monitor Subnets: IPv4: 16.89.112.0 Polling Target 16.89.112.1 IPv6: 3ffe:1000:0:a801:: Polling Target 3ffe:1000:0:a801::254 … The IP Monitor section of the cluster configuration file will look similar to the following for a subnet on which IP monitoring is configured with target polling.
NOTE: This is not the default. If cmquerycl does not detect a gateway for the subnet in question, it sets IP_MONITOR to OFF, disabling IP-level polling for this subnet; if it does detect a gateway, it populates POLLING_TARGET, enabling target polling. See SUBNET under “Cluster Configuration Parameters ” (page 100) for more information. The IP Monitor section of the cluster configuration file will look similar to the following in the case of a subnet on which IP monitoring is disabled: SUBNET 192.168.3.
Reporting Link-Level and IP-Level Failures Any given failure may occur at the link level or the IP level; a failure is reported slightly differently in the output of cmviewcl (1m) depending on whether link-level or IP monitoring detects the failure.
before the package will be started. (In a cross-subnet configuration, all subnets configured on that node, and identified as monitored subnets in the package configuration file, must be available.) The switching of relocatable IP addresses is shown in Figure 3-6 and Figure 3-7 .
Storage on Arrays Figure 3-21 shows LUNs configured on a storage array. Physical disks are configured by an array utility program into logical units, or LUNs, which are seen by the operating system. Figure 3-21 Physical Disks Combined into LUNs NOTE: LUN definition is normally done using utility programs provided by the disk array manufacturer. Since arrays vary considerably, you should refer to the documentation that accompanies your storage unit.
for startup. If auto_run is set to no, then the package simply halts without starting up anywhere else. The process for configuring disk monitoring is described in “Creating a Disk Monitor Configuration” (page 220). More Information on LVM Refer to the section “Creating the Logical Volume Infrastructure” in Chapter 5 for details about configuring volume groups, logical volumes, and file systems for use in Serviceguard packages.
Rules and Limitations Serviceguard automatically implements PR for packages that use LUN storage, subject to the following constraints: • • The LUN device must support PR and be consistent with the SPC-3 specification PR is not available in legacy multi-node packages. PR is available in modular multi-node packages, and in both modular and legacy failover packages. — All instances of a modular multi-node package must be able to use PR; otherwise it will be turned off for all instances.
• ENABLED means that packages can in principle use PR, but in practice will do so only if they meet the conditions spelled out under “Rules and Limitations”. • DISABLED means that no packages can use PR because at least one node is an HPVM guest. You can see the setting of cluster_pr_mode in the output of cmviewcl -f line; for example: ... cluster_pr_mode: pr_enabled NOTE: You cannot change the setting of cluster_pr_mode.
Responses to Failures Serviceguard responds to different kinds of failures in specific ways. For most hardware failures, the response is not user-configurable, but for package and service failures, you can choose the system’s response, within limits. Reboot When a Node Fails The most dramatic response to a failure in a Serviceguard cluster is a system reboot. This allows packages to move quickly to another node, protecting the integrity of the data.
SystemA; volume group vg02is exclusively activated on SystemB. Package IP addresses are assigned to SystemA and SystemB respectively. Failure. Only one LAN has been configured for both heartbeat and data traffic. During the course of operations, heavy application traffic monopolizes the bandwidth of the network, preventing heartbeat packets from getting through. Since SystemA does not receive heartbeat messages from SystemB, SystemA attempts to re-form as a one-node cluster.
written so that they can detect such a restart. This is the same application design required for restart after a normal system crash. In the event of a LAN interface failure, bonding provides a backup path for IP messages. If a heartbeat LAN interface fails and no redundant heartbeat is configured, the node fails with a reboot.
Service Restarts You can allow a service to restart locally following a failure. To do this, you indicate a number of restarts for each service in the package control script. When a service starts, the variable service_restart is set in the service’s environment. The service, as it executes, can examine this variable to see whether it has been restarted after a failure, and if so, it can take appropriate action such as cleanup.
4 Planning and Documenting an HA Cluster Building a Serviceguard cluster begins with a planning phase in which you gather and record information about all the hardware and software components of the configuration.
• • electrical points of failure. application points of failure. Serviceguard Memory Requirements Serviceguard requires approximately 15.5 MB of lockable memory. Planning for Expansion When you first set up the cluster, you indicate a set of nodes and define a group of packages for the initial configuration. At a later time, you may wish to add additional nodes and packages, or you may wish to use additional disk hardware for shared data storage.
LAN Information While a minimum of one LAN interface per subnet is required, at least two LAN interfaces are needed to eliminate single points of network failure. HP recommends that you configure heartbeats on all subnets, including those to be used for client data. Collect the following information for each LAN interface: Subnet Name The IP address for the subnet. Note that heartbeat IP addresses must be on the same subnet on each node.
NOTE: Multipath capabilities are supported by FibreChannel HBA device drivers and the Linux Device Mapper. Check with the storage device documentation for details. See also “Multipath for Storage ”. You can use the worksheet to record the names of the device files that correspond to each LUN for the Fibre-Channel-attached storage unit.
Disk Device File Enter the disk device file name for each SCSI disk or LUN. This information is needed when you create the mirrored disk configuration using LVM. In addition, it is useful to gather as much information as possible about your disk configuration. You can obtain information about available disks by using the following commands; your system may provide other utilities as well.
quorum server, or quorum server cluster, make sure each quorum server node has a power source separate from that of every cluster it serves. If you use software mirroring, make sure power supplies are not shared among different physical volume groups; this allows you to set up mirroring between physical disks that are not only on different I/O buses, but also connected to different power supplies. To prevent confusion, label each hardware unit and power supply unit clearly with a different unit number.
Planning for Expansion Bear in mind that a cluster with more than 4 nodes cannot use a lock LUN. So if you plan to add enough nodes to bring the total to more than 4, you should use a quorum server. Using a Quorum Server The Quorum Server is described under “Use of the Quorum Server as a Cluster Lock” (page 46). See also “Cluster Lock” (page 45). A quorum server: • • • • Can be used with up to 150 clusters, not exceeding 300 nodes total. Can support a cluster with any supported number of nodes.
• • • You must group high availability applications, services, and data, whose control needs to be transferred together, on a single volume group or a series of volume groups. You must not group two different high availability applications, services, or data, whose control needs to be transferred independently, on the same volume group. Your root disk must not belong to a volume group that can be activated on another node.
parameters by editing the cluster configuration template file created by means of the cmquerycl command, as described under “Configuring the Cluster” (page 171). NOTE: See “Reconfiguring a Cluster” (page 240) for a summary of changes you can make while the cluster is running. The following parameters must be configured: CLUSTER_NAME The name of the cluster as it will appear in the output of cmviewcl and other commands, and as it appears in the cluster configuration file.
NOTE: In addition, the following characters must not be used in the cluster name if you are using the Quorum Server: at-sign (@), equal-sign (=), or-sign (|), semicolon (;). These characters are deprecated, meaning that you should not use them, even if you are not using the Quorum Server. All other characters are legal. The cluster name can contain up to 39 characters.
hosts should contain the following entry: ::1 ipv6-localhost ipv6-loopback For more information and recommendations about hostname resolution, see “Configuring Name Resolution” (page 150). NOTE: — ANY is not supported on Red Hat 5. — ANY is not supported in a cluster that includes Virtual Machines (VMs). QS_HOST The fully-qualified hostname or IP address of a host system outside the current cluster that is providing quorum server functionality.
IMPORTANT: For special instructions that may apply to your version of Serviceguard and the Quorum Server see “Configuring Serviceguard to Use the Quorum Server” in the latest version HP Serviceguard Quorum Server Version A.04.00 Release Notes, at http://www.docs.hp.com -> High Availability -> Quorum Server. Can be changed while the cluster is running; see “What Happens when You Change the Quorum Configuration Online” (page 48) for important information.
NODE_NAME The hostname of each system that will be a node in the cluster. CAUTION: Make sure that the node name is unique within the subnets configured on the cluster nodes; under some circumstances Serviceguard may not be able to detect a duplicate name and unexpected problems may result. Do not use the full domain name. For example, enter ftsys9, not ftsys9.cup.hp.com. A cluster can contain up to 16 nodes.
Online” (page 251). See also “What Happens when You Change the Quorum Configuration Online” (page 48) for important information. NETWORK_INTERFACE The name of each LAN that will be used for heartbeats or for user data on the node identified by the preceding NODE_NAME. An example is eth0.
NOTE: Any subnet that is configured in this cluster configuration file as a SUBNET for IP monitoring purposes, or as a monitored_subnet in a package configuration file (or SUBNET in a legacy package; see “Package Configuration Planning ” (page 118)) must be specified in the cluster configuration file via NETWORK_INTERFACE and either STATIONARY_IP or HEARTBEAT_IP.
You cannot configure more than one heartbeat IP address on an interface; only one HEARTBEAT_IP is allowed for each NETWORK_INTERFACE. NOTE: The Serviceguard cmapplyconf, cmcheckconf, and cmquerycl commands check that these minimum requirements are met, and produce a warning if they are not met at the immediate network level. If you see this warning, you need to check that the requirements are met in your overall network configuration.
NOTE: IPv6 heartbeat subnets are not supported in a cross-subnet configuration. NOTE: The use of a private heartbeat network is not advisable if you plan to use Remote Procedure Call (RPC) protocols and services. RPC assumes that each network adapter device or I/O card is connected to a route-able network. An isolated or private heartbeat LAN is not route-able, and could cause an RPC request-reply, directed to that LAN, to time out without being serviced.
NOTE: Any subnet that is configured in this cluster configuration file as a SUBNET for IP monitoring purposes, or as a monitored_subnet in a package configuration file (or SUBNET in a legacy package; see “Package Configuration Planning ” (page 118)) must be specified in the cluster configuration file via NETWORK_INTERFACE and either STATIONARY_IP or HEARTBEAT_IP.
characters, dot (.), dash (-), or underscore (_). Maximum length is 39 characters. CAPACITY_NAME must be unique in the cluster. CAPACITY_VALUE specifies a value for the CAPACITY_NAME that precedes it. It must be a floating-point value between 0 and 1000000. Capacity values are arbitrary as far as Serviceguard is concerned; they have meaning only in relation to the corresponding package weights.
and begins re-forming the cluster without this node. Default value: 14 seconds (14,000,000 microseconds). This value leads to a failover time of between approximately 18 and 22 seconds, if you are using a quorum server, or a Fiber Channel cluster lock, or no cluster lock. Increasing the value to 25 seconds increases the failover time to between approximately 29 and 39 seconds. The time will increase by between 5 and 13 seconds if you are you using a SCSI cluster lock or dual Fibre Channel cluster lock).
• To ensure the fastest cluster re-formations, use the minimum value applicable to your cluster. But keep in mind that this setting will lead to a cluster re-formation, and to the node being removed from the cluster and rebooted, if a system hang or network load spike prevents the node from sending a heartbeat signal within the MEMBER_TIMEOUT value.
the fastest booting node plus 600 seconds (ten minutes). Default is 600,000,000 microseconds. Can be changed while the cluster is running. NETWORK_POLLING_INTERVAL Specifies how frequently the networks configured for Serviceguard are checked. Default is 2,000,000 microseconds (2 seconds). This means that the network manager will poll each network interface every 2 seconds, to make sure it can still send and receive information.
SUBNET IP address of a cluster subnet for which IP Monitoring can be turned on or off (see IP_MONITOR). The subnet must be configured into the cluster, via NETWORK_INTERFACE and either HEARTBEAT_IP or STATIONARY_IP. All entries for IP_MONITOR and POLLING_TARGET apply to this subnet until the next SUBNET entry; SUBNET must be the first of each trio.
IP level and IP_MONITOR is set toON, the interface will be marked down. If it is set to OFF, failures that occur only at the IP-level will not be detected. Can be changed while the cluster is running; must be removed if the preceding SUBNET entry is removed. POLLING_TARGET The IP address to which polling messages will be sent from all network interfaces on the subnet specified in the preceding SUBNET entry, if IP_MONITOR is set to ON. This is called target polling.
WEIGHT_NAME are the same as those spelled out for CAPACITY_NAME earlier in this list. These parameters are optional, but if they are defined, WEIGHT_DEFAULT must follow WEIGHT_NAME, and must be set to a floating-point value between 0 and 1000000. If they are not specified for a given weight, Serviceguard will assume a default value of zero for that weight.
redundant. For more information, see “Controlling Access to the Cluster” (page 176). MAX_CONFIGURED_PACKAGES This parameter sets the maximum number of packages that can be configured in the cluster. The minimum value is 0, and the maximum value, which is also the default, is 300. Can be changed while the cluster is running. Cluster Configuration: Next Step When you are ready to configure the cluster, proceed to “Configuring the Cluster” (page 171).
the previous node. This is accomplished by activating the volume group and mounting the file system that resides on it. In Serviceguard, high availability applications, services, and data are located in volume groups that are on a shared bus. When a node fails, the volume groups containing the applications, services, and data of the failed node are deactivated on the failed node and activated on the adoptive node (the node the packages move to).
Create an entry for each logical volume, indicating its use for a file system or for a raw device. CAUTION: Do not use /etc/fstab to mount file systems that are used by Serviceguard packages. For information about creating, exporting, and importing volume groups, see “Creating the Logical Volume Infrastructure ” (page 159). Planning for Expansion You can add packages to a running cluster. This process is described in Chapter 7: “Cluster and Package Maintenance” (page 221).
Table 4-1 Package Failover Behavior Switching Behavior Parameters in Configuration File Package switches normally after • node_fail_fast_enabled set to no. (Default) detection of service or network, failure, • service_fail_fast_enabled set to no for all services. (Default) or when a configured dependency is not • auto_run set to yes for the package. (Default) met. Halt script runs before switch takes place. (Default) Package fails over to the node with the • failover_policy set to min_package_node.
detailed configuration information, see the package parameter definitions starting with “dependency_name” (page 202). For a discussion of complex dependencies, see Make a package dependent on another package if the first package cannot (or should not) function without the services provided by the second. For example, pkg1 might run a real-time web interface to a database managed by pkg2. In this case it might make sense to make pkg1 dependent on pkg2.
• • node_name list on which pkg2 is running (and any other dependencies, such as resource dependencies or a dependency on a third package, are met). In the case of failover packages with a configured_node failover_policy, a set of rules governs under what circumstances pkg1 can force pkg2 to start on a given node. This is called dragging and is determined by each package’s priority (page 201). See “Dragging Rules for Simple Dependencies” (page 123).
NOTE: Keep the following in mind when reading the examples that follow, and when actually configuring priorities: 1. auto_run (page 198) should be set to yes for all the packages involved; the examples assume that it is. 2. Priorities express a ranking order, so a lower number means a higher priority (10 is a higher priority than 30, for example).
— If both packages have moved from node1 to node2 and node1 becomes available, pkg2 will fail back to node1 only if pkg2’s priority is higher than pkg1’s: ◦ If the priorities are equal, neither package will fail back (unless pkg1 is not running; in that case pkg2 can fail back).
But you also need to weigh the relative importance of the packages. If pkg2 runs a database that is central to your business, you probably want it to run undisturbed, no matter what happens to application packages that depend on it. In this case, the database package should have the highest priority. Note that, if no priorities are set, the dragging rules favor a package that is depended on over a package that depends on it.
different_node and any_node are allowed only if dependency_condition is UP. all_nodes is allowed only if dependency_condition is DOWN. See “Rules for different_node and any_node Dependencies” (page 128). For more information about the dependency_ parameters, see the definitions starting with “dependency_name” (page 202), and the cmmakepkg (1m) manpage. IMPORTANT: If you have not already done so, read the discussion of Simple Dependencies (page 121) before you go on.
• • dependency_location must be either same_node or all_nodes, and must be the same for both packages. Both packages must be failover packages whose failover_policy (page 201) is configured_node. Rules for different_node and any_node Dependencies These rules apply to packages whose dependency_condition is UP and whose dependency_location is different_node or any_node.
on it; if a node's available capacity will be exceeded by a package that wants to run on that node, the package will not run there.
in each case. You may want to set CAPACITY_VALUE to different values for different nodes. A ten-package capacity might represent the most powerful node, for example, while the least powerful has a capacity of only two or three. NOTE: Serviceguard does not require you to define a capacity for each node.
• • If you use the reserved CAPACITY_NAME package_limit, then this is the only type of capacity and weight you can define in this cluster. If you use the reserved CAPACITY_NAME package_limit, the default weight for all packages is 1. You can override this default in the package configuration file, via the weight_name and weight_value parameters, as in the example above. (The default weight remains 1 for any package to which you do not explicitly assign a different weight in the package configuration file.
ensures that for each capacity configured for a node, the combined weight of packages currently running on that node does not exceed that capacity.
CAPACITY_VALUE 50 NODE_NAME node2 CAPACITY_NAME A CAPACITY_VALUE 60 CAPACITY_NAME B CAPACITY_VALUE 70 ... NOTE: You do not have to define capacities for every node in the cluster. If any capacity is not defined for any node, Serviceguard assumes that node has an infinite amount of that capacity.
WEIGHT_NAME B WEIGHT_DEFAULT 15 This means that any package for which weight A is not defined in its package configuration file will have a weight A of 20, and any package for which weight B is not defined in its package configuration file will have a weight B of 15. Given the capacities we defined in the cluster configuration file (see “Defining Capacities”), node1 can run any three packages that use the default for both A and B.
weight_name B weight_value 35 weight_name A weight_value 0 In pkg4's package configuration file: weight_name B weight_value 40 IMPORTANT: weight_name in the package configuration file must exactly match the corresponding CAPACITY_NAME in the cluster configuration file. This applies to case as well as spelling: weight_name a would not match CAPACITY_NAME A.
Rules and Guidelines The following rules and guidelines apply to both the Simple Method (page 129) and the Comprehensive Method (page 131) of configuring capacities and weights. • You can define a maximum of four capacities, and corresponding weights, throughout the cluster. NOTE: But if you use the reserved CAPACITY_NAME package_limit, you can define only that single capacity and corresponding weight. See “Simple Method” (page 129).
For further discussion and use cases, see the white paper Using Serviceguard’s Node Capacity and Package Weight Feature on docs.hp.com under High Availability —> Serviceguard —> White Papers. How Package Weights Interact with Package Priorities and Dependencies If necessary, Serviceguard will halt a running lower-priority package that has weight to make room for a higher-priority package that has weight.
• On package startup and shutdown, as essentially the first and last functions the package performs. These scripts are invoked by means of the parameter external_pre_script (page 212); or • During package execution, after volume-groups and file systems are activated, and IP addresses are assigned, and before the service and resource functions are executed; and again, in the reverse order, on package shutdown. These scripts are invoked by means of the parameter external_script (page 212).
PEV_MONITORING_INTERVAL 60 At validation time, the sample script makes sure the PEV_MONITORING_INTERVAL and the monitoring service are configured properly; at start and stop time it prints out the interval to the log file. #!/bin/sh # Source utility functions. if [[ -z $SG_UTILS ]] then . $SGCONF.conf SG_UTILS=$SGCONF/scripts/mscripts/utils.sh fi if [[ -f ${SG_UTILS} ]]; then .
function start_command { sg_log 5 "start_command" # log current PEV_MONITORING_INTERVAL value, PEV_ attribute can be changed # while the package is running sg_log 0 "PEV_MONITORING_INTERVAL for $SG_PACKAGE_NAME is $PEV_MONITORING_INTERVAL" return 0 } function stop_command { sg_log 5 "stop_command" # log current PEV_MONITORING_INTERVAL value, PEV_ attribute can be changed # while the package is running sg_log 0 "PEV_MONITORING_INTERVAL for $SG_PACKAGE_NAME is $PEV_MONITORING_INTERVAL" return 0 } typeset -i e
Determining Why a Package Has Shut Down You can use an external script (or CUSTOMER DEFINED FUNCTIONS area of a legacy package control script) to find out why a package has shut down.
The implications for configuring a package for cross-subnet failover are as follows: • For modular packages, you must configure two new parameters in the package configuration file to allow packages to fail over across subnets: — ip_subnet_node (page 206) - to indicate which nodes a subnet is configured on — monitored_subnet_access (page 204) - to indicate whether a monitored subnet is configured on all nodes ( FULL) or only some (PARTIAL).
Configuring a Package to Fail Over across Subnets: Example To configure a package to fail over across subnets, you need to make some additional edits to the package configuration file. NOTE: This section provides an example for a modular package; for legacy packages, see “Configuring Cross-Subnet Failover” (page 259). Suppose that you want to configure a package, pkg1, so that it can fail over among all the nodes in a cluster comprising NodeA, NodeB, NodeC, and NodeD. NodeA and NodeB use subnet 15.244.65.
NOTE: Configuring monitored_subnet_access as FULL (or not configuring monitored_subnet_access) for either of these subnets will cause the package configuration to fail, because neither subnet is available on all the nodes. Configuring ip_subnet_node Now you need to specify which subnet is configured on which nodes. In our example, you would do this by means of entries such as the following in the package configuration file: ip_subnet 15.244.65.0 ip_subnet_node nodeA ip_subnet_node nodeB ip_address 15.244.
for cluster locks described above. See “Cluster Lock Planning” (page 98) for more information. If you are planning to add a node online, and a package will run on the new node, ensure that any existing cluster-bound volume groups for the package have been imported to the new node. Also, ensure that the MAX_CONFIGURED_PACKAGES parameter is set high enough to accommodate the total number of packages you will be using; see “Cluster Configuration Parameters ” (page 100).
5 Building an HA Cluster Configuration This chapter and the next take you through the configuration tasks required to set up a Serviceguard cluster. You carry out these procedures on one node, called the configuration node, and Serviceguard distributes the resulting binary file to all the nodes in the cluster. In the examples in this chapter, the configuration node is named ftsys9, and the sample target node is called ftsys10.
The following are example locations for a SUSE distribution: ############################## cmcluster.
Configuring Root-Level Access The subsections that follow explain how to set up root access between the nodes in the prospective cluster. (When you proceed to configuring the cluster, you will define various levels of non-root access as well; see “Controlling Access to the Cluster” (page 176).) NOTE: For more information and advice, see the white paper Securing Serviceguard at http://docs.hp.com -> High Availability -> Serviceguard -> White Papers.
IMPORTANT: If $SGCONF/cmclnodelist does not exist, Serviceguard will look at ~/.rhosts. HP strongly recommends that you use cmclnodelist. NOTE: When you upgrade a cluster from Version A.11.15 or earlier, entries in $SGCONF/cmclnodelist are automatically updated to Access Control Policies in the cluster configuration file. All non-root user-hostname pairs are assigned the role of Monitor.
NOTE: If you are using private IP addresses for communication within the cluster, and these addresses are not known to DNS (or the name resolution service you use) these addresses must be listed in /etc/hosts. For example, consider a two node cluster (gryf and sly) with two private subnets and a public subnet. These nodes will be granting access by a non-cluster node (bit) which does not share the private subnets. The /etc/hosts file on both cluster nodes should contain: 15.145.162.131 10.8.0.131 10.8.1.
NOTE: If such a hang or error occurs, Serviceguard and all protected applications will continue working even though the command you issued does not. That is, only the Serviceguard configuration commands (and corresponding Serviceguard Manager functions) are affected, not the cluster daemon or package services. The procedure that follows shows how to create a robust name-resolution configuration that will allow cluster nodes to continue communicating with one another if a name service fails. 1.
NOTE: HP recommends that you also make the name service itself highly available, either by using multiple name servers or by configuring the name service into a Serviceguard package. Ensuring Consistency of Kernel Configuration Make sure that the kernel configurations of all cluster nodes are consistent with the expected behavior of the cluster during failover.
NOTE: HP recommends that you do the bonding configuration from the system console, because you will need to restart networking from the console when the configuration is done. Sample Configuration Configure the following files to support LAN redundancy. For a single failover only one bond is needed. 1. Create a bond0 file, ifcfg-bond0. Create the configuration in the /etc/sysconfig/network-scripts directory.
HWADDR=00:12:79:43:5b:f4 3. Add the following lines to /etc/modprobe.conf: alias bond0 bonding options bond0 miimon=100 mode=1 Use MASTER=bond1 for bond1 if you have configured a second bonding interface, then add the following after the first bond (bond0): options bond1 -o bonding1 miimon=100 mode=1 NOTE: During configuration, you need to make sure that the active slaves for the same bond on each node are connected the same hub or switch.
eth1 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4 inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:3651769 errors:0 dropped:0 overruns:0 frame:0 TX packets:1643480 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 Interrupt:9 Base address:0x1400 Implementing Channel Bonding (SUSE) If you are using a Red Hat distribution, use the procedures described in the previous section.
BONDING_SLAVE0='eth0' BONDING_SLAVE1='eth1' The above example configures bond0 with mii monitor equal to 100 and active-backup mode. Adjust the IP, BROADCAST, NETMASK, and NETWORK parameters to correspond to your configuration. As you can see, you are adding the configuration options BONDING_MASTER, BONDING-MODULE_OPTS, and BONDING_SLAVE. BONDING-MODULE_OPTS are the additional options you want to pass to the bonding module.
Table 5-1 Changing Linux Partition Types Prompt Response Action Performed 1. Command (m for help): n Create new partition 2. Partition number (1-4): 1 Partition affected 3.
sfdisk -R /dev/sdc You can check the partition table by using the command: fdisk -l /dev/sdc NOTE: fdisk may not be available for SUSE on all platforms. In this case, using YAST2 to set up the partitions is acceptable. Setting Up and Running the Quorum Server If you will be using a quorum server rather than a lock LUN, the Quorum Server software must be installed on a system other than the nodes on which your cluster will be running, and must be running during cluster configuration.
• • • • Distributing the Shared Configuration to all Nodes (page 166) Testing the Shared Configuration (page 167) Storing Volume Group Configuration Data (page 169) Setting up Disk Monitoring (page 170) CAUTION: The minor numbers used by the LVM volume groups must be the same on all cluster nodes. This means that if there are any non-shared volume groups in the cluster, create the same number of them on all nodes, and create them before you define the shared storage.
NOTE: fdisk may not be available for SUSE on all platforms. In this case, using YAST2 to set up the partitions is acceptable. Creating Partitions You must define a partition on each disk device (individual disk or LUN in an array) that you want to use for your shared storage. Use the fdisk command for this. The following steps create the new partition: 1.
Command (m for help): w The partition table has been altered! 2. Respond to the prompts as shown in the following table to set a partition type: Prompt Response Action Performed 1. Command (m for help): t Set the partition type 2. Partition number (1-4): 1 Partition affected 3.
NOTE: fdisk may not be available for SUSE on all platforms. In this case, using YAST2 to set up the partitions is acceptable. Enabling Volume Group Activation Protection As of Serviceguard for Linux A.11.16.07, you can enable activation protection for logical volume groups, preventing the volume group from being activated by more than one node at the same time. Activation protection, if used, must be enabled on each cluster node.
5. Run vgscan: vgscan NOTE: At this point, the setup for volume-group activation protection is complete. Serviceguard adds a tag matching the uname -n value of the owning node to each volume group defined for a package when the package runs and deletes the tag when the package halts. The command vgs -o +tags vgname will display any tags that are set for a volume group.
3. Check whether there are already volume groups defined on this node. Be sure to give each volume group a unique name. vgdisplay 4. Create separate volume groups for each Serviceguard package you will define.
NOTE: For information about supported filesystem types, see the fs_type discussion on (page 210). 5. To test that the file system /extra was created correctly and with high availability, you can create a file on it, and read it. echo "Test of LVM" >> /extra/LVM-test.conf cat /extra/LVM-test.conf NOTE: Be careful if you use YAST or YAST2 to configure volume groups, as that may cause all volume groups on that system to be activated.
The partition table on the rebooted node is then rebuilt using the information placed on the disks when they were partitioned on the other node. NOTE: 3. You must reboot at this time. Run vgscan to make the LVM configuration visible on the new node and to create the LVM database on/etc/lvmtab and /etc/lvmtab.d.
1. On ftsys9, activate the volume group, mount the file system that was built on it, write a file in the shared file system and look at the result: vgchange --addtag $(uname -n) vgpkgB NOTE: If you are using the volume-group activation protection feature of Serviceguard for Linux, you must use vgchange --addtag to add a tag when you manually activate a volume group. Similarly, you must remove the tag when you deactivate a volume group that will be used in a package (as shown at the end of each step).
2. On ftsys10, activate the volume group, mount the file system, write a date stamp on to the shared file, and then look at the content of the file: vgchange --addtag $(uname -n) vgpkgB vgchange -a y vgpkgB mount /dev/vgpkgB/lvol1 /extra echo ‘Written by’ ‘hostname‘ ‘on’ ‘date‘ >> /extra/datestamp cat /extra/datestamp You should see something like the following, including the date stamp written by the other node: Written by ftsys9.mydomain on Mon Jan 22 14:23:44 PST 2006 Written by ftsys10.
Preventing Boot-Time vgscan and Ensuring Serviceguard Volume Groups Are Deactivated By default, Linux will perform LVM startup actions whenever the system is rebooted. These include a vgscan (on some Linux distributions) and volume group activation. This can cause problems for volumes used in a Serviceguard environment (for example, a volume group for a Serviceguard package that is not currently running may be activated). To prevent such problems, proceed as follows on the various Linux versions.
Configuring the Cluster This section describes how to define the basic cluster configuration. This must be done on a system that is not part of a Serviceguard cluster (that is, on which Serviceguard is installed but not configured). You can do this in Serviceguard Manager, or from the command line as described below. Use the cmquerycl command to specify a set of nodes to be included in the cluster and to generate a template for the cluster configuration file.
Specifying the Address Family for the Heartbeat To tell Serviceguard to use only IPv4, or only IPv6, addresses for the heartbeat, use the -h option. For example, to use only IPv6 addresses: cmquerycl -v -h ipv6 -C $SGCONF/clust1.conf -n ftsys9 -n ftsys10 • • • • -h ipv4 tells Serviceguard to discover and configure only IPv4 subnets. If it does not find any eligible subnets, the command will fail. -h ipv6 tells Serviceguard to discover and configure only IPv6 subnets.
Specifying a Quorum Server IMPORTANT: The following are standard instructions. For special instructions that may apply to your version of Serviceguard and the Quorum Server see “Configuring Serviceguard to Use the Quorum Server” in the latest version HP Serviceguard Quorum Server Version A.04.00 Release Notes, at http://www.docs.hp.com -> High Availability -> Quorum Server. A cluster lock LUN or quorum server, is required for two-node clusters.
1 2 3 4 5 6 lan3 lan4 lan3 lan4 lan1 lan1 lan2 lan2 lan3 lan4 lan3 lan4 (nodeA) (nodeA) (nodeB) (nodeB) (nodeA) (nodeB) (nodeA) (nodeB) (nodeC) (nodeC) (nodeD) (nodeD) lan1 lan1 lan2 lan2 (nodeC) (nodeD) (nodeC) (nodeD) IP subnets: IPv4: 15.13.164.0 15.13.172.0 15.13.165.0 15.13.182.0 15.244.65.0 15.244.56.
15.13.165.2 (nodeB) 15.13.182.158 (nodeC) 15.13.182.159 (nodeD) Route connectivity(full probing performed): 15.13.182.0 1 2 3 4 15.13.164.0 15.13.172.0 15.13.165.0 15.13.182.0 15.244.65.0 15.244.56.0 In the Route connectivity section, the numbers on the left (1-4) identify which subnets are routed to each other (for example 15.13.164.0 and 15.13.172.0). IMPORTANT: Note that in this example subnet 15.244.65.0, used by NodeA and NodeB, is not routed to 15.244.56.0, used by NodeC and NodeD. But subnets 15.
NOTE: Remember to tune kernel parameters on each node to ensure that they are set high enough for the largest number of packages that will ever run concurrently on that node. Modifying the MEMBER_TIMEOUT Parameter The cmquerycl command supplies a default value of 14 seconds for the MEMBER_TIMEOUT parameter. Changing this value will directly affect the cluster’s re-formation and failover times.
its own functions plus all the functions of Full Admin, Package Admin and Monitor; Full Admin can perform its own functions plus the functions of Package Admin and Monitor; and so on. Figure 5-1 Access Roles Levels of Access Serviceguard recognizes two levels of access, root and non-root: • Root access: Full capabilities; only role allowed to configure the cluster. As Figure 5-1 shows, users with root access have complete control over the configuration of the cluster and its packages.
user on any node in the cluster always has full Serviceguard root access privileges for that cluster; no additional Serviceguard configuration is needed to grant these privileges. IMPORTANT: Users on systems outside the cluster can gain Serviceguard root access privileges to configure the cluster only via a secure connection (rsh or ssh).
NOTE: For more information and advice, see the white paper Securing Serviceguard at http://docs.hp.com -> High Availability -> Serviceguard -> White Papers. Define access-control policies for a cluster in the cluster configuration file; see “Cluster Configuration Parameters ” (page 100). To define access control for a specific package, use user_host (page 212) and related parameters in the package configuration file. You can define up to 200 access policies for each cluster.
NOTE: If you set USER_HOST to ANY_SERVICEGUARD_NODE, set USER_ROLE to MONITOR; users connecting from outside the cluster cannot have any higher privileges (unless they are connecting via rsh or ssh; this is treated as a local connection). Depending on your network configuration, ANY_SERVICEGUARD_NODE can provide wide-ranging read-only access to the cluster.
Plan the cluster’s roles and validate them as soon as possible. If your organization’s security policies allow it, you may find it easiest to create group logins. For example, you could create a MONITOR role for user operator1 from CLUSTER_MEMBER_NODE (that is, from any node in the cluster). Then you could give this login name and password to everyone who will need to monitor your clusters.
NOTE: Check spelling especially carefully when typing wildcards, such as ANY_USER and ANY_SERVICEGUARD_NODE. If they are misspelled, Serviceguard will assume they are specific users or nodes.
• • • • • • Heartbeat network minimum requirement. See HEARTBEAT_IP under “Cluster Configuration Parameters ” (page 100). At least one NODE_NAME is specified. Each node is connected to each heartbeat network. All heartbeat networks are of the same type of LAN. The network interface device files specified are valid LAN device files. Other configuration parameters for the cluster and packages are valid.
Managing the Running Cluster This section describes some approaches to routine management of the cluster. For more information, see Chapter 7: “Cluster and Package Maintenance” (page 221). You can manage the cluster from Serviceguard Manager, or by means of Serviceguard commands as described below. Checking Cluster Operation with Serviceguard Commands • • • • • cmviewcl checks the status of the cluster and many of its components.
• • 4. Start the node. You can use Serviceguard Manager or the cmrunnode command. Verify that the node has returned to operation. You can use Serviceguard Manager or the cmviewcl command again. Bring down the cluster. You can use Serviceguard Manager or the cmhaltcl -v -f command. See the manpages for more information about these commands. See Chapter 8: “Troubleshooting Your Cluster” (page 271) for more information about cluster testing.
# # # # # start is the preferred way to start a cluster. No action is required by the system administrator. If set to 1, the node will attempt to join/form its CM cluster automatically as described above. If set to 0, the node will not attempt to join its CM cluster. AUTOSTART_CMCLD=1 Changing the System Message You may find it useful to modify the system's login message to include a statement such as the following: This system is a node in a high availability cluster.
It is not necessary to halt the single node in this case, since the applications are still running, and no other node is currently available for package switching. CAUTION: But you should not try to restart Serviceguard; data corruption might occur if another node were to attempt to start up a new instance of an application that is still running on the single node. Instead, choose an appropriate time to shut down and reboot the node.
Deleting the Cluster Configuration You can delete a cluster configuration by means of the cmdeleteconf command. The command prompts for a verification before deleting the files unless you use the -f option. You can delete the configuration only when the cluster is down. The action removes the binary configuration file from all the nodes in the cluster and resets all cluster-aware volume groups to be no longer cluster-aware.
6 Configuring Packages and Their Services Serviceguard packages group together applications and the services and resources they depend on. The typical Serviceguard package is a failover package that starts on one node but can be moved (“failed over”) to another if necessary. For more information, see “What is Serviceguard for Linux? ” (page 23), “How the Package Manager Works” (page 49), and“Package Configuration Planning ” (page 118).
NOTE: This is a new process for configuring packages, as of Serviceguard A.11.18. This manual refers to packages created by this method as modular packages, and assumes that you will use it to create new packages. Packages created using Serviceguard A.11.16 or earlier are referred to as legacy packages. If you need to reconfigure a legacy package (rather than create a new package), see “Configuring a Legacy Package” (page 252).
them, and then start them up on another node selected from the package’s configuration list; see node_name (page 197). To generate a package configuration file that creates a failover package, include-m sg/failover on the cmmakepkg command line. See “Generating the Package Configuration File” (page 214). • Multi-node packages. These packages run simultaneously on more than one node in the cluster.
notice that parameters from the package control script (or their equivalents) are now in the package configuration file; these parameters are marked (S) in the table. You can use cmmakepkg -l (letter “l”) to see a list of all available modules, including non-Serviceguard modules such as those supplied in the HP Toolkits.
Table 6-1 Base Modules Module Name Parameters (page) Comments failover package_name (page 196) * module_name (page 197) * module_version (page 197) * package_type (page 197) package_description (page 197) * node_name (page 197) auto_run (page 198) node_fail_fast_enabled (page 198) run_script_timeout (page 199) halt_script_timeout (page 199) successor_halt_script_timeout (page 200) * script_log_file (page 200) operation_sequence (page 200) * log_level (page 200) * failover_policy (page 201) failback_poli
Optional Package Modules Add optional modules to a base module if you need to configure the functions in question. Parameters marked with an asterisk (*) are new or changed as of Serviceguard A.11.18 or A.11.19. (S) indicates that the parameter (or its equivalent) has moved from the package control script to the package configuration file for modular packages. See the “Package Parameter Explanations” (page 196) for more information.
Table 6-2 Optional Modules (continued) Module Name Parameters (page) Comments pev pev_ (page 211) * Add to a base module to configure environment variables to be passed to an external script. external_pre external_pre_script (page 212) * Add to a base module to specify additional programs to be run before volume groups are activated while the package is starting and after they are deactivated while the package is halting.
NOTE: The default form for parameter names in the modular package configuration file is lower case; for legacy packages the default is upper case. There are no compatibility issues; Serviceguard is case-insensitive as far as the parameter names are concerned. This manual uses lower case, unless the parameter in question is used only in legacy packages, or the context refers exclusively to such a package. Package Parameter Explanations Brief descriptions of the package configuration parameters follow.
IMPORTANT: Restrictions on package names in previous Serviceguard releases were less stringent. Packages whose names do not conform to the above rules will continue to run, but if you reconfigure them, you will need to change the name; cmcheckconf and cmapplyconf will enforce the new rules. module_name The module name. Do not change it. Used in the form of a relative path (for example sg/failover) as a parameter to cmmakepkg to specify modules to be used in configuring the package.
adoptive node name (the best candidate for failover), then the second adoptive node name, followed by additional node names in order of preference. In case of a failover, control of the package will be transferred to the next adoptive node name listed in the package configuration file, or (if that node is not available or cannot run the package at that time) to the next node in the list, and so on. IMPORTANT: See “Cluster Configuration Parameters ” (page 100) for important information about node names.
NOTE: If the package halt function fails with “exit 1”, Serviceguard does not halt the node, but sets no_restart for the package, which disables package switching, setting auto_run (page 198) to no and thereby preventing the package from starting on any adoptive node. Setting node_fail_fast_enabled to yes prevents Serviceguard from repeatedly trying (and failing) to start the package on the same node. For system multi-node packages, node_fail_fast_enabled must be set to yes.
successor_halt_timeout Specifies how long, in seconds, Serviceguard will wait for packages that depend on this package to halt, before halting this package. Can be 0 through 4294, or no_timeout. The default is no_timeout. • • no_timeout means that Serviceguard will wait indefinitely for the dependent packages to halt. 0 means Serviceguard will not wait for the dependent packages to halt before halting this package. New as of A.11.18 (for both modular and legacy packages).
failover_policy Specifies how Serviceguard decides where to start the package, or restart it if it fails. Can be set to configured_node or min_package_node. The default is configured_node. • configured_node means Serviceguard will attempt to start the package on the first available node in the list you provide under node_name (page 197). • min_package_node means Serviceguard will start the package on whichever node in the node_name list has the fewest packages running at the time.
IMPORTANT: Because priority is a matter of ranking, a lower number indicates a higher priority (20 is a higher priority than 40). A numerical priority is higher than no_priority. New as of A.11.18 (for both modular and legacy packages). See “About Package Dependencies” (page 121) for more information. dependency_name A unique identifier for a particular dependency (see dependency_condition) that must be met in order for this package to run (or keep running).
• • • If the current package is a multi-node package, must identify a multi-node or system multi-node package. If the current package is a failover package and its failover_policy (page 201) is min_package_node, must identify a multi-node or system multi-node package.
For more information, see “About Package Weights” (page 128). See also the discussion of the relevant parameters under “Cluster Configuration Parameters ” (page 100), in the cmmakepkg (1m) and cmquerycl (1m) manpages, and in the cluster configuration and package configuration template files. New for 11.19. monitored_subnet The LAN subnet that is to be monitored for this package.
ip_subnet Specifies an IP subnet used by the package. Replaces SUBNET, which is still supported in the package control script for legacy packages. CAUTION: HP recommends that this subnet be configured into the cluster. You do this in the cluster configuration file by specifying a HEARTBEAT_IP or STATIONARY_IP under a NETWORK_INTERFACE on the same subnet, for each node in this package's NODE_NAME list. For example, an entry such as the following in the cluster configuration file configures subnet 192.10.25.
This parameter can be set for failover packages only. ip_subnet_node In a cross-subnet configuration, specifies which nodes an ip_subnet is configured on. If no ip_subnet_nodes are listed under an ip_subnet, it is assumed to be configured on all nodes in this package’s node_name list (page 197). Can be added or deleted while the package is running, with these restrictions: • • The package must not be running on the node that is being added or deleted.
service_fail_fast_enabled service_halt_timeout no 300 See the package configuration template file for more examples. For legacy packages, this parameter is in the package control script as well as the package configuration file. service_cmd The command that runs the program or function for this service_name, for example, /usr/bin/X11/xclock -display 15.244.58.208:0 An absolute pathname is required; neither the PATH variable nor any other environment variable is passed to the command.
service_fail_fast_enabled Specifies whether or not Serviceguard will halt the node (reboot) on which the package is running if the service identified by service_name fails. Valid values are yes and no. Default is no, meaning that failure of this service will not cause the node to halt. service_halt_timeout The length of time, in seconds, Serviceguard will wait for the service to halt before forcing termination of the service’s process. The maximum value is 4294.
fs_type "ext2" A logical volume must be built on an LVM volume group. Logical volumes can be entered in any order. A gfs file system can be configured using only the fs_name, fs_directory, and fs_mount_opt parameters; see the configuration file for an example. Additional rules apply for gfs as explained under fs_type. The parameter explanations that follow provide more detail. concurrent_fsck_operations The number of concurrent fsck operations allowed on file systems being mounted during package startup.
fs_umount_retry_count The number of umount retries for each file system. Replaces FS_UMOUNT_COUNT, which is still supported in the package control script for legacy packages; see “Configuring a Legacy Package” (page 252). Legal value is 1 or (for filesystem types other than Red Hat GFS) any greater number. The default is 1. Operates in the same way as fs_mount_retry_count.
NOTE: A package using gfs (Red Hat Global File System, or GFS) cannot use any other file systems of a different type. vg and vgchange_cmd (page 208) are not valid for GFS file systems. For more information about using GFS with Serviceguard, see Clustering Linux Servers with the Concurrent Deployment of HP Serviceguard for Linux and Red Hat Global File Systems for RHEL5 on docs.hp.
The variable name must be in the form pev_ and contain only alphanumeric characters and underscores. The letters pev (upper-case or lower-case) followed by the underscore (_) are required. The variable name and value can each consist of a maximum of MAXPATHLEN characters (4096 on Linux systems). You can define more than one variable. See “About External Scripts” (page 137), as well as the comments in the configuration file, for more information.
user_name Specifies the name of a user who has permission to administer this package. See also user_host (page 212) and user_role; these three parameters together define the access control policy for this package (see “Controlling Access to the Cluster” (page 176)). These parameters must be defined in this order: user_name, user_host, user_role. Legal values for user_name are any_user or a maximum of eight login names from /etc/passwd on user_host.
is passed the parameter start; when it halts, it is passed the parameter stop.) LV The name of a logical volume hosting a file system that will be mounted by the package. FS The name of the mount point for a file system to be mounted by the package. VGCHANGE As vgchange_cmd (page 208).
NOTE: If you do not include a base module (or default or all) on the cmmakepkg command line, cmmakepkg will ignore the modules you specify and generate a default configuration file containing all the parameters. For a complex package, or if you are not yet sure which parameters you will need to set, the default may be the best choice; see the first example below. You can use the-v option with cmmakepkg to control how much information is displayed online or included in the configuration file.
the file to set the package parameters to the values that will make the package function as you intend. It is a good idea to configure complex failover packages in stages, as follows: 1. 2. 3. Configure volume groups and mount points only. Check and apply the configuration; see “Verifying and Applying the Package Configuration” (page 219). Run the package and ensure that it can be moved from node to node. NOTE: cmcheckconf and cmapplyconf check for missing mount points, volume groups, etc. 4. 5. 6.
• • • • • • later if it fails. Enter no to keep Serviceguard from automatically starting the package. node_fail_fast_enabled. Enter yes to cause the node to be halted (system halt) if the package fails; otherwise enter no. run_script_timeout and halt_script_timeout. Enter the number of seconds Serviceguard should wait for package startup or shutdown, respectively, to complete; or leave the default, no_timeout. See (page 199). successor_halt_timeout.
• For each service the package will run: — enter the service_name (for example, a daemon or long-running process) — enter the service_cmd (for example, the command that starts the process) — enter values for service_fail_fast_enabled and service_halt_timeout if you need to change them from their defaults. — service_restart if you want the package to restart the service if it exits. (A value of unlimited can be useful if you want the service to execute in a loop, rather than exit and halt the package.
• • If you want the package to run an external “pre-script” during startup and shutdown, use the external_pre_script parameter (see (page 212)) to specify the full pathname of the script, for example $SGCONF/pkg1/pre_script1. If the package will run an external script, use the external_script parameter (see (page 212)) to specify the full pathname of the script, for example $SGCONF/pkg1/ script1. See “About External Scripts” (page 137), and the comments in the configuration file, for more information.
NOTE: For modular packages, you now need to distribute any external scripts identified by the external_pre_script and external_script parameters. But, if you are accustomed to configuring legacy packages, note that you do not have to create a separate package control script for a modular package, or distribute it manually. (You do still have to do this for legacy packages; see “Configuring a Legacy Package” (page 252).
7 Cluster and Package Maintenance This chapter describes the cmviewcl command, then shows how to start and halt a cluster or an individual node, how to perform permanent reconfiguration, and how to start, halt, move, and modify packages during routine maintenance of the cluster.
Viewing Package Dependencies The cmviewcl -v command output lists dependencies throughout the cluster. For a specific package’s dependencies, use the -p option. Cluster Status The status of a cluster, as shown by cmviewcl, can be one of the following: • • • • up - At least one node has a running cluster daemon, and reconfiguration is not taking place. down - No cluster daemons are running on any cluster node. starting - The cluster is in the process of determining its active membership.
• • • • • • • • • • start_wait - A cmrunpkg command is in progress for this package. The package is waiting for packages it depends on (predecessors) to start before it can start. starting - The package is starting. The package master control script is running. halting - A cmhaltpkg command is in progress for this package and the halt script is running. halt_wait -A cmhaltpkg command is in progress for this package.
• • • • • • • fail_wait - The package is waiting to be halted because the package or a package it depends on has failed, but must wait for a package it depends on to halt before it can halt. failed - The package is down and failed. relocate_wait - The package’s halt script has completed or Serviceguard is still trying to place the package. maintenance — The package is in maintenance mode; see “Maintaining a Package: Partial-Startup Maintenance Mode” (page 237).
Service Status Services have only status, as follows: • • • Up. The service is being monitored. Down. The service is not running. It may not have started, or have halted or failed. Unknown. Serviceguard cannot determine the status. Network Status The network interfaces have only status, as follows: • • • Up. Down. Unknown. Serviceguard cannot determine whether the interface is up or down.
PRIMARY up eth1 PACKAGE pkg1 STATUS up STATE running AUTO_RUN enabled NODE ftsys9 Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM STATUS Service up Subnet up MAX_RESTARTS 0 0 Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING Primary up enabled Alternate up enabled NODE ftsys10 STATUS up NAME ftsys9 ftsys10 NAME service1 15.13.168.
NOTE: The Script_Parameters section of the PACKAGE output of cmviewcl shows the Subnet status only for the node that the package is running on. In a cross-subnet configuration, in which the package may be able to fail over to a node on another subnet, that other subnet is not shown (see “Cross-Subnet Configurations” (page 32)).
POLICY_NAME Failover Failback CONFIGURED_VALUE configured_node manual Script_Parameters: ITEM STATUS Service up Subnet up MAX_RESTARTS 0 0 Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING Primary up enabled Alternate up enabled NODE ftsys10 STATUS up Network_Parameters: INTERFACE STATUS PRIMARY up PRIMARY up RESTARTS 0 0 NAME ftsys9 ftsys10 NAME service1 15.13.168.
the output of the cmviewcl -v command is as follows: CLUSTER example NODE ftsys9 STATUS up STATUS up STATE running Network_Parameters: INTERFACE STATUS PRIMARY up PRIMARY up NAME eth0 eth1 PACKAGE pkg1 STATE running STATUS up AUTO_RUN enabled NODE ftsys9 Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM STATUS Service up Subnet up MAX_RESTARTS 0 0 Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING Primary up enabled Alternate up e
Status After Package Switching is Enabled The following command changes package status back to Auto Run Enabled: cmmodpkg -e pkg2 The output of the cmviewcl command is now as follows: CLUSTER example NODE ftsys9 STATUS up STATUS up PACKAGE pkg1 pkg2 NODE ftsys10 STATUS up up STATUS up STATE running STATE running running AUTO_RUN enabled enabled NODE ftsys9 ftsys9 STATE running Both packages are now running on ftsys9 and pkg2 is enabled for switching.
Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover min_package_node Failback automatic Script_Parameters: ITEM STATUS Subnet up Subnet up Subnet up Subnet up NODE_NAME manx burmese tabby persian NAME 192.8.15.0 192.8.15.0 192.8.15.0 192.8.15.
NOTE: Manually starting or halting the cluster or individual nodes does not require access to the quorum server, if one is configured. The quorum server is only used when tie-breaking is needed following a cluster partition. Starting the Cluster When all Nodes are Down You can use Serviceguard Manager, or the cmruncl command as described in this section, to start the cluster when all cluster nodes are down. Particular command options can be used to start the cluster under specific circumstances.
By default, cmrunnode will do network validation, making sure the actual network setup matches the configured network setup. This is the recommended method. If you have recently checked the network and find the check takes a very long time, you can use the -w none option to bypass the validation. Since the node's cluster is already running, the node joins the cluster and packages may be started, depending on the package configuration (see node_name (page 197)).
Halting the Entire Cluster You can use Serviceguard Manager, or Serviceguard commands as shown below, to halt a running cluster. The cmhaltcl command can be used to halt the entire cluster. This command causes all nodes in a configured cluster to halt their HP Serviceguard daemons. You can use the -f option to force the cluster to halt even when packages are running. This command can be issued from any running node. Example: cmhaltcl -f -v This halts all the cluster nodes.
Use the cmrunpkg command to run the package on a particular node, then use the cmmodpkg command to enable switching for the package; for example: cmrunpkg -n ftsys9 pkg1 cmmodpkg -e pkg1 This starts up the package on ftsys9, then enables package switching. This sequence is necessary when a package has previously been halted on some node, since halting the package disables switching.
cmviewcl to be sure that no other running package has a dependency on any of the packages you are halting. Moving a Failover Package You can use Serviceguard Manager to move a failover package from one node to another, or Serviceguard commands as shown below. Before you move a failover package to a new node, it is a good idea to run cmviewcl -v -l package and look at dependencies. If the package has dependencies, be sure they can be met on the new node.
You can disable package switching to particular nodes by using the -n option of the cmmodpkg command. The following prevents pkg1 from switching to node lptest3: cmmodpkg -d -n lptest3 pkg1 To permanently disable switching so that the next time the cluster restarts, the change you made in package switching is still in effect, change the auto_run flag in the package configuration file, then re-apply the configuration. (See “Reconfiguring a Package on a Running Cluster ” (page 262).
modules used by a package are started in the order shown near the top of its package configuration file.) cmrunpkg -m sg/package_ip pkg1 4. Perform maintenance on the services and test manually that they are working correctly. NOTE: If you now run cmviewcl, you'll see that the STATUS of pkg1 is up and its STATE is MAINTENANCE. 5.
You can also use -e in combination with -m. This has the effect of starting all modules up to and including the module identified by -m, except the module identified by -e. In this case the excluded (-e) module must be earlier in the execution sequence (as listed near the top of the package's configuration file) than the -m module. For example: cmrunpkg -m sg/services -e sg/package_ip pkg1 NOTE: The full execution sequence for starting a package is: 1. The master control script itself 2.
• Node-wide and cluster-wide events affect the package as follows: — If the node the package is running on is halted, the package will also be halted, and will remain in maintenance mode; it will not be automatically re-started. — If the node crashes, the package will remain in maintenance mode and will not be automatically re-started. — If the cluster is halted or crashes, the package will not be in maintenance mode when the cluster comes back up.
Table 7-1 Types of Changes to the Cluster Configuration Change to the Cluster Configuration Required Cluster State Add a new node All cluster nodes must be running. Delete a node A node can be deleted even though it is unavailable or unreachable. Change Maximum Configured Packages Cluster can be running. Change Quorum Server Configuration Cluster can be running; see“What Happens when You Change the Quorum Configuration Online” (page 48).
them. You can preview the effect on packages of certain actions or events before they actually occur.
NOTE: You cannot use the -t option with any command operating on a package in maintenance mode; see “Maintaining a Package: Partial-Startup Maintenance Mode” (page 237). For more information about these commands, see their respective manpages. You can also perform these preview functions in Serviceguard Manager: check the Preview [...] box for the action in question. When you use the -t option, the command, rather than executing as usual, predicts the results that would occur, sending a summary to $stdout.
Use cmeval rather than command preview mode when you want to see more than the effect of a single command, and especially when you want to see the results of large-scale changes, or changes that may interact in complex ways, such as changes to package priorities, node order, dependencies and so on. Using cmeval involves three major steps: 1. Use cmviewcl -v -f line to write the current cluster configuration out to a file. 2. Edit the file to include the events or changes you want to preview 3.
IMPORTANT: For detailed information and examples, see the cmeval (1m) manpage. Reconfiguring a Halted Cluster You can make a permanent change in cluster configuration when the cluster is halted. This procedure must be used for changes marked “Cluster must not be running” in Table 7-1, but it can be used for any other cluster configuration changes as well. Use the following steps: 1. 2. 3. 4. Halt the cluster on all nodes.
NOTE: Before you start, make sure you have configured access to ftsys10 as described under “Configuring Root-Level Access” (page 149). 1. Use the following command to store a current copy of the existing cluster configuration in a temporary file in case you need to revert to it: cmgetconf -C temp.conf 2. Specify a new set of nodes to be configured and generate a template of the new configuration (all on one line): cmquerycl -C clconfig.conf -c cluster1 -n ftsys8 -n ftsys9 -n ftsys10 3. 4.
NOTE: If you want to remove a node from the cluster, run the cmapplyconf command from another node in the same cluster. If you try to issue the command on the node you want removed, you will get an error message. 1. Use the following command to store a current copy of the existing cluster configuration in a temporary file: cmgetconf -c cluster1 temp.conf 2. Specify the new set of nodes to be configured (omitting ftsys10) and generate a template of the new configuration: cmquerycl -C clconfig.
• • Change IP Monitor parameters: SUBNET, IP_MONITOR, POLLING TARGET; see the entries for these parameters under“Cluster Configuration Parameters ” (page 100) for more information. A combination of any of these in one transaction (cmapplyconf), given the restrictions below. What You Must Keep in Mind The following restrictions apply: • You must not change the configuration of all heartbeats at one time, or change or delete the only configured heartbeat.
Examples of when you must do this include: — moving a NIC from one subnet to another — adding an IP address to a NIC — removing an IP address from a NIC CAUTION: Do not add IP addresses to network interfaces that are configured into the Serviceguard cluster, unless those IP addresses themselves will be immediately configured into the cluster as stationary IP addresses.
NODE_NAME NETWORK_INTERFACE HEARTBEAT_IP NETWORK_INTERFACE HEARTBEAT_IP NETWORK_INTERFACE NODE_NAME NETWORK_INTERFACE HEARTBEAT_IP NETWORK_INTERFACE HEARTBEAT_IP NETWORK_INTERFACE 3. ftsys9 lan1 192.3.17.18 lan0 15.13.170.18 lan3 ftsys10 lan1 192.3.17.19 lan0 15.13.170.19 lan3 Verify the new configuration: cmcheckconf -C clconfig.conf 4. Apply the changes to the configuration and distribute the new binary configuration file to all cluster nodes.: cmapplyconf -C clconfig.
NODE_NAME NETWORK_INTERFACE HEARTBEAT_IP # NETWORK_INTERFACE # STATIONARY_IP # NETWORK_INTERFACE NODE_NAME NETWORK_INTERFACE HEARTBEAT_IP # NETWORK_INTERFACE # STATIONARY_IP # NETWORK_INTERFACE 4. ftsys9 lan1 192.3.17.18 lan0 15.13.170.18 lan3 ftsys10 lan1 192.3.17.19 lan0 15.13.170.19 lan3 Verify the new configuration: cmcheckconf -C clconfig.conf 5. Apply the changes to the configuration and distribute the new binary configuration file to all cluster nodes.: cmapplyconf -C clconfig.
Use the cmapplyconf command to apply the changes to the configuration and send the new configuration file to all cluster nodes. Using -k or -K can significantly reduce the response time. Configuring a Legacy Package IMPORTANT: You can still create a new legacy package. If you are using a Serviceguard Toolkit such as Serviceguard NFS Toolkit, consult the documentation for that product. Otherwise, use this section to maintain and re-work existing legacy packages rather than to create new ones.
1. Create a subdirectory for each package you are configuring in the $SGCONF directory: mkdir $SGCONF/pkg1 You can use any directory names you like. (See “Understanding the Location of Serviceguard Files” (page 147) for the name of Serviceguard directories on your version of Linux.) 2. Generate a package configuration file for each package, for example: cmmakepkg -p $SGCONF/pkg1/pkg1.conf You can use any file name you like for the configuration file. 3.
NOTE: For modular packages, the default form for parameter names and literal values in the package configuration file is lower case; for legacy packages the default is upper case. There are no compatibility issues; Serviceguard is case-insensitive as far as the parameter names are concerned. Because this section is intended to be used primarily when you reconfiguring an existing legacy package, we are using the legacy parameter names (in upper case) for sake of continuity.
IMPORTANT: Each subnet specified here must already be specified in the cluster configuration file via the NETWORK_INTERFACE parameter and either the HEARTBEAT_IP or STATIONARY_IP parameter. See “Cluster Configuration Parameters ” (page 100) for more information. See also “Stationary and Relocatable IP Addresses and Monitored Subnets” (page 71) and monitored_subnet (page 204). IMPORTANT: For cross-subnet configurations, see “Configuring Cross-Subnet Failover” (page 259).
Use cmmakepkg to create the control script, then edit the control script. Use the following procedure to create the template for the sample failover package pkg1. First, generate a control script template, for example: cmmakepkg -s $SGCONF/pkg1/pkg1.sh Next, customize the script; see “Customizing the Package Control Script ”. Customizing the Package Control Script You need to customize as follows; see the relevant entries under “Package Parameter Explanations” (page 196) for more discussion.
Adding Customer Defined Functions to the Package Control Script You can add additional shell commands to the package control script to be executed whenever the package starts or stops. Enter these commands in the CUSTOMER DEFINED FUNCTIONS area of the script. If your package needs to run short-lived processes, such as commands to initialize or halt a packaged application, you can also run these from the CUSTOMER DEFINED FUNCTIONS.
cmmodpkg pkg1, but pkg2 has to wait for pkg1 startup to complete, thereby causing a command loop. To avoid this situation, it is a good idea to always specify a RUN_SCRIPT_TIMEOUT and a HALT_SCRIPT_TIMEOUT for all packages, especially packages that use Serviceguard commands in their control scripts. If a timeout is not specified and your configuration has a command loop as described above, inconsistent results can occur, including a hung cluster.
Distributing the Configuration And Control Script with Serviceguard Manager When you have finished creating a package in Serviceguard Manager, click Apply Configuration. If the package configuration has no errors, it is converted to a binary file and distributed to the cluster nodes. Copying Package Control Scripts with Linux commands IMPORTANT: In a cross-subnet configuration, you cannot use the same package control script on all nodes if the package uses relocatable IP addresses.
NodeA and NodeB use subnet 15.244.65.0, which is not used by NodeC and NodeD; and NodeC and NodeD use subnet 15.244.56.0, which is not used by NodeA and NodeB. (See “Obtaining Cross-Subnet Information” (page 173) for sample cmquerycl output). Configuring node_name First you need to make sure that pkg1 will fail over to a node on another subnet only if it has to.
IMPORTANT: In a cross-subnet configuration, you cannot share a single package control script among nodes on different subnets if you are using relocatable IP addresses. In this case you will need to create a separate control script to be used by the nodes on each subnet. In our example, you would create two copies of pkg1’s package control script, add entries to customize it for subnet 15.244.65.0 or 15.244.56.0, and copy one of the resulting scripts to each node, as follows.
Migrating a Legacy Package to a Modular Package The Serviceguard command cmmigratepkg automates the process of migrating legacy packages to modular packages as far as possible. Many, but not all, packages can be migrated in this way; for details, see the white paper Package Migration from Legacy Style to Modular Style at http://docs.hp.com -> High Availability -> Serviceguard -> White papers. Do not attempt to convert Serviceguard Toolkit packages. NOTE: The cmmigratepkg command requires Perl version 5.8.
5. Distribute your changes to all nodes: cmapplyconf -v -P pkg1.conf 6. If this is a legacy package, copy the package control script to all nodes that can run the package. Reconfiguring a Package on a Halted Cluster You can also make permanent changes in the package configuration while the cluster is not running. Use the same steps as in “Reconfiguring a Package on a Running Cluster ”.
Resetting the Service Restart Counter The service restart counter tracks the number of times a package service has been automatically restarted. This value is used to determine when the package service has exceeded its maximum number of allowable automatic restarts. When a package service successfully restarts after several attempts, the package manager does not automatically reset the restart count.
NOTE: All the nodes in the cluster must be powered up and accessible when you make package configuration changes. Table 7-2 Types of Changes to Packages Change to the Package Required Package State Delete a package Package must not be running. NOTE: on it. You cannot delete a package if another package has a dependency Change package type Package must not be running. Add or delete a module: modular package Package can be running.
Table 7-2 Types of Changes to Packages (continued) Change to the Package Required Package State Add or remove an Package can be running. ip_address: modular package See “ip_subnet” (page 205) and “ip_address” (page 206) for important information. Serviceguard will reject the change if you are trying to add an ip_address that cannot be configured on the specified ip_subnet, or is on a subnet that is not configured on all the nodes on the package's node_name list.
Table 7-2 Types of Changes to Packages (continued) Change to the Package Required Package State Change a file system: modular package Package should not be running (unless you are only changing fs_umount_opt). Changing file-system options other than fs_umount_opt may cause problems because the file system must be unmounted (using the existing fs_umount_opt) and remounted with the new options; the CAUTION under “Remove a file system: modular package” applies in this case as well.
Table 7-2 Types of Changes to Packages (continued) Change to the Package Required Package State Change package auto_run Package can be either running or halted. See “Choosing Switching and Failover Behavior” (page 120). Add or delete a configured Both packages can be either running or halted. dependency Special rules apply to packages in maintenance mode; see “Dependency Rules for a Package in Partial-Startup Maintenance Mode” (page 240).
The typical corrective actions to take in the event of a transfer of package include: • • • • • • • Determining when a transfer has occurred. Determining the cause of a transfer. Repairing any hardware failures. Correcting any software problems. Restarting nodes. Transferring packages back to their original nodes. Enabling package switching.
1. 2. 3. 270 If the node is an active member of a cluster, halt the node first. If the node is included in a cluster configuration, remove the node from the configuration. If you are removing Serviceguard from more than one node, run rpm -eon one node at a time.
8 Troubleshooting Your Cluster This chapter describes how to verify cluster operation, how to review cluster status, how to add and replace hardware, and how to solve some typical cluster problems.
The package should be running on the specified adoptive node. 4. Halt the package, then move it back to the primary node using the cmhaltpkg, cmmodpkg, and cmrunpkg commands: cmhaltpkg cmmodpkg -e cmrunpkg -v Depending on the specific databases you are running, perform the appropriate database recovery. Testing the Cluster Manager To test that the cluster manager is operating correctly, perform the following steps for each node on the cluster: 1. 2.
Configuration” (page 220). In addition, the following should be monitored for errors or warnings of all kinds: • • • • • • CPUs Memory LAN cards Power sources All cables Disk interface cards Some monitoring can be done through simple physical inspection, but for the most comprehensive monitoring, you should examine the system log file (/var/log/messages) periodically for reports on all configured HA devices. The presence of errors relating to a device will show the need for maintenance.
If you need to use a different devicefile, you must change the name of the devicefile in the cluster configuration file; see “Updating the Cluster Lock LUN Configuration Online” (page 251). CAUTION: Before you start, make sure that all nodes have logged a message such as the following in syslog: WARNING: Cluster lock LUN /dev/sda1 is corrupt: bad label. Until this situation is corrected, a single failure could cause all nodes in the cluster to crash.
pr_cleanup lun -v -k [-f | ] • • • • • lun, if used, specifies that a LUN, rather than a volume group, is to be operated on. -v, if used, specifies verbose output detailing the actions the script performs and their status. -k , if used, specifies the key to be used in the clear operation. -f , if used, specifies that the name of the DSFs to be operated on are listed in the file specified by . Each DSF must be listed on a separate line.
5. 6. 7. Power up the system. As the system comes up, the kudzu program on Red Hat systems will detect and report the hardware changes. Accept the changes and add any information needed for the new LAN card. On SUSE systems, run YAST2 after the system boots and make adjustments to the NIC setting of the new LAN card. If the old LAN card was part of a “bond”, the new LAN card needs to be made part of the bond.
IMPORTANT: Make sure you read the latest version of the HP Serviceguard Quorum Server Release Notes before you proceed. You can find them at: http://www.docs.hp.com -> High Availability -> Quorum Server. You should also consult the Quorum Server white papers at the same location. 1. 2. 3. Remove the old quorum server system from the network. Set up the new system and configure it with the old quorum server’s IP address and hostname. Install and configure the quorum server software on the new system.
cmquerycl -q -n -n ... The command will output an error message if the specified nodes cannot communicate with the quorum server. CAUTION: Make sure that the old system does not rejoin the network with the old IP address.
Interrupt:9 Base address:0xda00 eth1:1 Link encap:Ethernet HWaddr 00:50:DA:64:8A:7C inet addr:192.168.1.200 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:9 Base address:0xda00 eth1:2 Link encap:Ethernet HWaddr 00:50:DA:64:8A:7C inet addr:192.168.1.201 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:9 Base address:0xda00 lo Link encap:Local Loopback inet addr:127.0.0.1 Bcast:192.168.1.255 Mask:255.255.
NO_RESTART. Dec 14 14:34:45 star04 cmcld[2048]: Examine the file /usr/local/cmcluster/pkg5/pkg5_run.log for more details. The following is an example of a successful package starting: Dec Dec Dec Dec Dec Dec 14 14:39:27 star04 CM-CMD[2096]: cmruncl 14 14:39:27 star04 cmcld[2098]: Starting cluster management protocols.
Using the cmquerycl and cmcheckconf Commands In addition, cmquerycl and cmcheckconf can be used to troubleshoot your cluster just as they were used to verify its configuration. The following example shows the commands used to verify the existing cluster configuration on ftsys9 and ftsys10: cmquerycl -v -C $SGCONF/verify.conf -n ftsys9 -n ftsys10 cmcheckconf -v -C $SGCONF/verify.conf cmcheckconf checks: • • • • The network addresses and connections.
Name Resolution Problems Many Serviceguard commands, including cmviewcl, depend on name resolution services to look up the addresses of cluster nodes. When name services are not available (for example, if a name server is down), Serviceguard commands may hang, or may return a network-related error message. If this happens, use the host command on each cluster node to see whether name resolution is correct. For example: host ftsys9 ftsys9.cup.hp.com has address 15.13.172.
1. Warning: cmcld was unable to run for the last seconds. Consult the Managing Serviceguard manual for guidance on setting MEMBER_TIMEOUT, and information on cmcld. This means that cmcld was unable to get access to a CPU for a significant amount of time. If this occurred while the cluster was re-forming, one or more nodes could have failed.
These are errors caused specifically by errors in the cluster configuration file and package configuration scripts. Examples of these errors include: • • • • Volume groups not defined on adoptive node. Mount point does not exist on adoptive node. Network errors on adoptive node (configuration errors). User information not correct on adoptive node. You can use the following commands to check the status of your disks: • • • • df - to see if your package’s volume group is mounted.
script appear in the ifconfig output under the inet addr: in the ethX:Y block, use cmmodnet to remove them: cmmodnet -r -i where is the address indicated above and is the result of masking the with the mask found in the same line as the inet address in the ifconfig output. 3. Ensure that package volume groups are deactivated. First unmount any package logical volumes which are being used for file systems.
is to test your package control script before putting your high availability application on line. Adding a set -x statement in the second line of your control script will give you details on where your script may be failing. Node and Network Failures These failures cause Serviceguard to transfer control of a package to another node.
Timeout Problems The following kinds of message in a Serviceguard node’s syslog file may indicate timeout problems: Unable to set client version at quorum server 192.6.7.2: reply timed out Probe of quorum server 192.6.7.2 timed out These messages could be an indication of an intermittent network problem; or the default quorum server timeout may not be sufficient. You can set the QS_TIMEOUT_EXTENSION to increase the timeout, or you can increase the MEMBER_TIMEOUT value.
A Designing Highly Available Cluster Applications This appendix describes how to create or port applications for high availability, with emphasis on the following topics: • Automating Application Operation • Controlling the Speed of Application Failover (page 291) • Designing Applications to Run on Multiple Systems (page 294) • Restoring Client Connections (page 299) • Handling Application Failures (page 300) • Minimizing Planned Downtime (page 301) Designing for high availability means reducing the amount
There are two principles to keep in mind for automating application relocation: • Insulate users from outages. • Applications must have defined startup and shutdown procedures. You need to be aware of what happens currently when the system your application is running on is rebooted, and whether changes need to be made in the application's response for high availability. Insulate Users from Outages Wherever possible, insulate your end users from outages.
To reduce the impact on users, the application should not simply abort in case of error, since aborting would cause an unneeded failover to a backup system. Applications should determine the exact error and take specific action to recover from the error rather than, for example, aborting upon receipt of any error.
Minimize the Use and Amount of Memory-Based Data Any in-memory data (the in-memory context) will be lost when a failure occurs. The application should be designed to minimize the amount of in-memory data that exists unless this data can be easily recalculated. When the application restarts on the standby node, it must recalculate or reread from disk any information it needs to have in memory.
reprinting all the paychecks, does the job start from where it left off, or does the scheduler assume that the job was done and not print the last hours worth of paychecks? The correct behavior in a highly available environment is to restart where it left off, ensuring that everyone gets one and only one paycheck. Another example is an application where a clerk is entering data about a new employee.
The simplest method is to have two applications running in a master/slave relationship where the slave is simply a hot standby application for the master. When the master fails, the slave on the second system would still need to figure out what state the data was in (i.e., data recovery would still take place). However, the time to fork the application and do the initial startup is saved. Another possibility is having two applications that are both active.
sensor monitors the node’s access to the subnet on which these relocatable application IP addresses reside. When packages are configured in Serviceguard, the associated subnetwork address is specified as a package dependency, and a list of nodes on which the package can run is also provided. When failing a package over to a remote node, the subnetwork must already be active on the target node. Each application or package should be given a unique name as well as a relocatable IP address.
to have multiple licenses; one for each node the application will run on. Another way is to have a cluster-wide mechanism that lists a set of SPU IDs or node names. If your application is running on a system in the specified set, then the license is approved. Previous generation HA software would move the MAC address of the network card along with the IP address when services were moved to a backup system. This is no longer allowed in Serviceguard.
However, gethostbyname(3) should be used to locate the IP address of an application only if the application name is configured in DNS. It is probably best to associate a different application name with each independent HA service. This allows each application and its IP address to be moved to another node without affecting other applications. Only the stationary IP addresses should be associated with the hostname in DNS.
IP address is most appropriate for responses, so it will always pick the stationary IP address. For TCP stream sockets, the TCP level of the protocol stack resolves this problem for the client since it is a connection-based protocol. On the client, TCP ignores the stationary IP address and continues to use the previously bound relocatable IP address originally used by the client. With UDP datagram sockets, however, there is a problem.
Avoid File Locking In an NFS environment, applications should avoid using file-locking mechanisms, where the file to be locked is on an NFS Server. File locking should be avoided in an application both on local and remote systems. If local file locking is employed and the system fails, the system acting as the backup system will not have any knowledge of the locks maintained by the failed system. This may or may not cause problems when the application restarts.
restart the server locally. This will keep the client from having to switch to the second server in the event of a application failure. • Use a transaction processing monitor or message queueing software to increase robustness. Use transaction processing monitors such as Tuxedo or DCE/Encina, which provide an interface between the server and the client. Transaction processing monitors (TPMs) can be useful in creating a more highly available application.
Another alternative is for the failure of one component to still allow bringing down the other components cleanly. If a database SQL server fails, the database should still be able to be brought down cleanly so that no database recovery is necessary. The worse case is for a failure of one component to cause the entire system to fail. If one component fails and all other components need to be restarted, the downtime will be high.
Provide for Rolling Upgrades Provide for a “rolling upgrade” in a client/server environment. For a system with many components, the typical scenario is to bring down the entire system, upgrade every node to the new version of the software, and then restart the application on all the affected nodes. For large systems, this could result in a long downtime. An alternative is to provide for a rolling upgrade.
after a failure, he or she will continue to do so even if the application has been redesigned to handle a single failure. It is important that application documentation discuss alternatives with regards to high availability for typical maintenance operations.
B Integrating HA Applications with Serviceguard The following is a summary of the steps you should follow to integrate an application into the Serviceguard environment: 1. Read the rest of this book, including the chapters on cluster and package configuration, and the appendix “Designing Highly Available Cluster Applications.” 2.
Defining Baseline Application Behavior on a Single System 1. Define a baseline behavior for the application on a standalone system: • Install the application, database, and other required resources on one of the systems. Be sure to follow Serviceguard rules in doing this: — Install all shared data on separate external volume groups. — Use a journaled filesystem (JFS) as appropriate. • Perform some sort of standard test to ensure the application is running correctly.
• Halt the package on the first node and move it to the second node: # cmhaltpkg pkg1 # cmrunpkg -n node2 pkg1 # cmmodpkg -e pkg1 • Move it back. # cmhaltpkg pkg1 # cmrunpkg -n node1 pkg1 # cmmodpkg -e pkg1 • 2. 3. Fail one of the systems. For example, turn off the power on node 1. Make sure the package starts up on node 2. • Repeat failover from node 2 back to node 1. Be sure to test all combinations of application load during the testing.
C Blank Planning Worksheets This appendix reprints blank versions of the planning worksheets described in the “Planning” chapter. You can duplicate any of these worksheets that you find useful and fill them in as a part of the planning process.
Hardware Worksheet ============================================================================= SPU Information: Host Name ____________________ Server Series____________ Memory Capacity ____________ Number of I/O Slots ____________ ============================================================================= LAN Information: Name of Master _________ Name of Node IP Traffic Interface __________ Addr________________ Type ________ Name of Name of Node IP Traffic Master __________ Interface __________ Addr__
Power Supply Worksheet ============================================================================ SPU Power: Host Name ____________________ Power Supply _____________________ Host Name ____________________ Power Supply _____________________ ============================================================================ Disk Power: Disk Unit __________________________ Power Supply _______________________ Disk Unit __________________________ Power Supply _______________________ Disk Unit ______________
Quorum Server Worksheet Quorum Server Data: ============================================================================== QS Hostname: _________________IP Address: ______________________ OR Cluster Name: _________________ Package Name: ____________ Package IP Address: ___________________ Hostname Given to Package by Network Administrator: _________________ ============================================================================== Quorum Services are Provided for: Cluster Name: ________________________
Volume Group and Physical Volume Worksheet ============================================================================== Volume Group Name: ___________________________________ Physical Volume Name: _________________ Physical Volume Name: _________________ Physical Volume Name: _________________ ============================================================================= Volume Group Name: ___________________________________ Physical Volume Name: _________________ Physical Volume Name: _________________ Ph
Cluster Configuration Worksheet =============================================================================== Name and Nodes: =============================================================================== Cluster Name: ______________________________ Node Names: ________________________________________________ Maximum Configured Packages: ______________ =============================================================================== Cluster Lock Data: =======================================================
Package Configuration Worksheet ============================================================================= Package Configuration File Data: ========================================================================== Package Name: __________________Package Type:______________ Primary Node: ____________________ First Failover Node:__________________ Additional Failover Nodes:__________________________________ Run Script Timeout: _____ Halt Script Timeout: _____________ Package AutoRun Enabled? ______ Node F
Package Control Script Worksheet (Legacy) PACKAGE CONTROL SCRIPT WORKSHEET Page ___ of ___ ================================================================================ Package Control Script Data: ================================================================================ PATH______________________________________________________________ VGCHANGE_________________________________ VG[0]__________________LV[0]______________________FS[0]____________________ VG[1]__________________LV[1]_________________
D IPv6 Network Support This appendix describes some of the characteristics of IPv6 network addresses, specifically: • IPv6 Address Types • Network Configuration Restrictions (page 321) • Configuring IPv6 on Linux (page 322) IPv6 Address Types Several IPv6 types of addressing schemes are specified in the RFC 2373 (IPv6 Addressing Architecture). IPv6 addresses are 128-bit identifiers for interfaces and sets of interfaces. There are various address formats for IPv6 defined by the RFC 2373.
values of higher order 96 bits of IPv6 address and the d’s are the decimal values of the 32-bit lower order bits. Typically IPv4 Mapped IPv6 addresses and IPv4 Compatible IPv6 addresses will be represented in this notation. These addresses are discussed in later sections. Examples: 0:0:0:0:0:0:10.1.2.3 and ::10.11.3.123 IPv6 Address Prefix IPv6 Address Prefix is similar to CIDR in IPv4 and is written in CIDR notation.
IPv4 Compatible IPv6 Addresses The IPv6 transition mechanisms use a technique for tunneling IPv6 packets over the existing IPv4 infrastructure. IPv6 nodes that support such mechanisms use a special kind of IPv6 addresses that carry IPv4 addresses in their lower order 32-bits. These addresses are called IPv4 Compatible IPv6 addresses. They are represented as follows: Table D-3 80 bits 16 bits 32 bits zeros 0000 IPv4 address Example: ::192.168.0.
SLA ID = Site-Level Aggregation Identifier. Interface ID = Interface Identifier. Link-Local Addresses Link-local addresses have the following format: Table D-6 10 bits 54 bits 64 bits 1111111010 0 interface ID Link-local address are supposed to be used for addressing nodes on a single link. Packets originating from or destined to a link-local address will not be forwarded by a router.
The “group ID” field identifies the multicast group. Some frequently used multicast groups are the following: All Node Addresses = FF02:0:0:0:0:0:0:1 (link-local) All Router Addresses = FF02:0:0:0:0:0:0:2 (link-local) All Router Addresses = FF05:0:0:0:0:0:0:2 (site-local) Network Configuration Restrictions Serviceguard supports IPv6 for data and heartbeat IP. The restrictions on support for IPv6 in Serviceguard for Linux are: • Auto-configured IPv6 addresses are not supported in Serviceguard.
IMPORTANT: For important information, see also “Cross-Subnet Configurations” (page 32), the description of the HOSTNAME_ADDRESS_FAMILY, QS_HOST and QS_ADDR parameters under Cluster Configuration Parameters (page 100), “Configuring Name Resolution” (page 150), and the Release Notes for your version of Serviceguard for Linux.
NETMASK=255.255.255.0 NETWORK=12.12.12.0 BROADCAST=12.12.12.255 IPV6INIT=yes IPV6ADDR=3ffe:ffff:0000:f101::10/64 IPV6ADDR_SECONDARIES=fec0:0:0:1::10/64 IPV6_MTU=1280 ONBOOT=yes BOOTPROTO=none USERCTL=no Add the following two lines to /etc/modprobe.
E Using Serviceguard Manager HP Serviceguard Manager is a web-based, HP System Management Homepage (HP SMH) tool that replaces the functionality of the earlier Serviceguard management tools. Serviceguard Manager allows you to monitor, administer and configure a Serviceguard cluster from any system with a supported web browser. The Serviceguard Manager Main Page provides you with a summary of the health of the cluster including the status of each node and its packages.
• • Managing a single cluster, and Have installed Serviceguard version A.11.19. NOTE: SMH access roles constrain a user's cluster-management capabilities: • A user with HP SMH Administrator access has full cluster management capabilities. • A user with HP SMH Operator access can monitor the cluster and has restricted cluster management capabilities as defined by the user’s Serviceguard role-based access configuration. • A user with HP SMH User access does not have any cluster management capabilities. 1.
Figure E-1 System Management Homepage with Serviceguard Manager Number What is it? Description 1 Cluster and Overall status and alerts Displays information about the Cluster status, alerts and general information. 2 Menu tool bar The menu tool bar is available from the HP Serviceguard Manager Homepage, and from any cluster, node or package view-only property page. Menu option availability depends on which type of property page (cluster, node or package) you are currently viewing.
• • One or more clusters with Serviceguard version A.11.15.01 through A.11.19 Serviceguard Manager version A.05.01 with HP SIM 5.10 or later installed on a server NOTE: Serviceguard Manager can be launched by HP Systems Insight Manager version 5.10 or later if Serviceguard Manager is installed on an HP Systems Insight Manager Central Management Server. For a Serviceguard A.11.19 cluster, Systems Insight Manager will attempt to launch Serviceguard Manager B.02.
Figure E-2 Cluster by Type 4. Expand HP Serviceguard, and click on a Serviceguard cluster. NOTE: If you click on a cluster running an earlier Serviceguard release, the page will display a link that will launch Serviceguard Manager A.05.01 (if installed) via Java Webstart.
Index A Access Control Policies, 176 active node, 25 adding a package to a running cluster, 263 adding cluster nodes advance planning, 144 adding nodes to a running cluster, 232 adding packages on a running cluster, 220 administration adding nodes to a running cluster, 232 halting a package, 235 halting the entire cluster, 234 moving a package, 236 of packages and services, 234 of the cluster, 231 reconfiguring a package while the cluster is running, 262 reconfiguring a package with the cluster offline, 263
defined, 42 dynamic re-formation, 44 heartbeat subnet parameter, 106 initial configuration of the cluster, 42 main functions, 42 maximum configured packages parameter, 118 member timeout parameter, 112 monitored non-heartbeat subnet, 109 network polling interval parameter, 114, 117 planning the configuration, 100 quorum server parameter, 104 testing, 272 cluster node parameter in cluster manager configuration, 101, 104 cluster parameters initial configuration, 42 cluster re-formation scenario, 88 cluster st
package parameters, 196 F failback policy used by package manager, 57 FAILBACK_POLICY parameter used by package manager, 57 failover controlling the speed in applications, 291 defined, 25 failover behavior in packages, 120 failover package, 49, 190 failover policy used by package manager, 53 FAILOVER_POLICY parameter used by package manager, 53 failure kinds of responses, 88 network communication, 91 response to hardware failures, 89 responses to package and service failures, 90 restarting a service after
IP in sample package control script, 256 IP address adding and deleting in packages, 73 for nodes and packages, 71 hardware planning, 95, 99 portable, 72 reviewing for packages, 278 switching, 52, 53, 83 IP_MONITOR defined, 115 J JFS, 291 K kernel hang, and TOC, 88 safety timer, 39 kernel consistency in cluster configuration, 153 kernel interrupts and possible TOC, 113 L LAN heartbeat, 43 interface name, 95, 99 LAN failure Serviceguard behavior, 29 LAN interfaces primary and secondary, 30 LAN planning ho
binding to IP addresses, 297 binding to port addresses, 297 IP addresses and naming, 294 node and package IP addresses, 71 packages using IP addresses, 296 supported types in Serviceguard, 30 writing network applications as HA services, 291 no cluster lock choosing, 48 node basic concepts, 29 halt (TOC), 88 in Serviceguard cluster, 23 IP addresses, 71 timeout and TOC example, 88 node types active, 25 primary, 25 NODE_FAIL_FAST_ENABLED effect of setting, 90 NODE_NAME parameter in cluster configuration, 105 p
for expansion, 120 hardware configuration, 94 high availability objectives, 93 overview, 93 package configuration, 118 power, 97 quorum server, 99 SPU information, 94 volume groups and physical volumes, 99 worksheets, 97 planning and documenting an HA cluster, 93 planning for cluster expansion, 94 planning worksheets blanks, 309 point of failure in networking, 31 POLLING_TARGET defined, 116 ports dual and single aggregated, 76 power planning power sources, 97 worksheet, 98, 312 power supplies blank planning
in sample package control script, 256 SERVICE_RESTART in sample package control script, 256 Serviceguard install, 147 introduction, 23 Serviceguard at a Glance, 23 Serviceguard behavior in LAN failure, 29 in monitored resource failure, 29 in software failure, 29 Serviceguard commands to configure a package, 252 Serviceguard Manager, 27 overview, 26 Serviceguard software components figure, 38 shared disks planning, 96 shutdown and startup defined for applications, 290 single point of failure avoiding, 23 sin
verifying the cluster and package configuration, 219, 258 VG in sample package control script, 256 vgcfgbackup using to back up volume group configuration, 169 VGCHANGE in package control script, 256 VGChange, 214 volume group for cluster lock, 45, 46 planning, 99 volume group and physical volume planning, 99 W WEIGHT_DEFAULT defined, 116 WEIGHT_NAME defined, 116 What is Serviceguard?, 23 worksheet blanks, 309 cluster configuration, 118, 314 hardware configuration, 97, 309 package configuration, 315, 316 p
