Managing HP Serviceguard for Linux, Eighth Edition Manufacturing Part Number : B9903-90060 March 2008
Legal Notices The information in this document is subject to change without notice. Hewlett-Packard makes no warranty of any kind with regard to this manual, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. Hewlett-Packard shall not be held liable for errors contained herein or direct, indirect, special, incidental or consequential damages in connection with the furnishing, performance, or use of this material. Warranty.
Trademark Notices HP Serviceguard® is a registered trademark of Hewlett-Packard Company, and is protected by copyright. NIS™ is a trademark of Sun Microsystems, Inc. UNIX® is a registered trademark of The Open Group. Linux® is a registered trademark of Linus Torvalds. Red Hat® is a registered trademark of Red Hat Software, Inc. SUSE® is a registered trademark of SUSE AG, a Novell Business.
Contents 1. Serviceguard for Linux at a Glance What is Serviceguard for Linux? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Using Serviceguard Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Monitoring Clusters with Serviceguard Manager . . . . . . . . . . . . . . . . . . . . . . . . . . .
Contents What Makes a Package Run? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Before the Control Script Starts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . During Run Script Execution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Normal and Abnormal Exits from the Run Script . . . . . . . . . . . . . . . . . . . . . . . . . . . Service Startup with cmrunserv . . . . . . . . . . . . . .
Contents Hardware Configuration Worksheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Power Supply Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Power Supply Configuration Worksheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Cluster Lock Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Cluster Lock Requirements . . . . . . . . . . . .
Contents Specifying a Lock LUN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Specifying a Quorum Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Obtaining Cross-Subnet Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Identifying Heartbeat Subnets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Specifying Maximum Number of Configured Packages . .
Contents Cluster Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Node Status and State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Package Status and State. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Package Switching Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Service Status . . . . . . . . .
Contents Responding to Cluster Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296 Single-Node Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297 Removing Serviceguard from a System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298 8. Troubleshooting Your Cluster Testing Cluster Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Contents Define Application Startup and Shutdown . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Controlling the Speed of Application Failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Replicate Non-Data File Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Evaluate the Use of a Journaled Filesystem (JFS). . . . . . . . . . . . . . . . . . . . . . . . . . Minimize Data Loss . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Contents Volume Group and Physical Volume Worksheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Cluster Configuration Worksheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Package Configuration Worksheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Package Control Script Worksheet (Legacy). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367 368 370 372 E. IPv6 Network Support IPv6 Address Types . . . . .
Printing History Table 1 Printing Date Part Number Edition November 2001 B9903-90005 First November 2002 B9903-90012 First December 2002 B9903-90012 Second November 2003 B9903-90033 Third February 2005 B9903-90043 Fourth June 2005 B9903-90046 Fifth August 2006 B9903-90050 Sixth July 2007 B9903-90054 Seventh March 2008 B9903-90060 Eighth The last printing date and part number indicate the current edition, which applies to the A.11.18 version of HP Serviceguard for Linux.
Preface This guide describes how to configure and manage Serviceguard for Linux on HP ProLiant and HP Integrity servers under the Linux operating system. It is intended for experienced Linux system administrators. (For Linux system administration tasks that are not specific to Serviceguard, use the system administration documentation and manpages for your distribution of Linux.
Related Publications • Appendix C, “Integrating HA Applications with Serviceguard,” presents suggestions for integrating your existing applications with Serviceguard for Linux. • Appendix D, “Blank Planning Worksheets,” contains a set of empty worksheets for preparing a Serviceguard configuration. The following documents contain additional useful information: • HP Serviceguard for Linux Version A.11.18 Release Notes • HP Serviceguard Quorum Server Version A.03.
Serviceguard for Linux at a Glance 1 Serviceguard for Linux at a Glance This chapter introduces Serviceguard for Linux and shows where to find different kinds of information in this book.
Serviceguard for Linux at a Glance What is Serviceguard for Linux? What is Serviceguard for Linux? Serviceguard for Linux allows you to create high availability clusters of HP ProLiant and HP Integrity servers. A high availability computer system allows application services to continue in spite of a hardware or software failure. Highly available systems protect users from software failures as well as from failure of a system processing unit (SPU), disk, or local area network (LAN) component.
Serviceguard for Linux at a Glance What is Serviceguard for Linux? In the figure, node 1 (one of two SPU's) is running package A, and node 2 is running package B. Each package has a separate group of disks associated with it, containing data needed by the package's applications, and a copy of the data. Note that both nodes are physically connected to disk arrays. However, only one node at a time may access the data for a given group of disks.
Serviceguard for Linux at a Glance What is Serviceguard for Linux? Failover Under normal conditions, a fully operating Serviceguard cluster simply monitors the health of the cluster's components while the packages are running on individual nodes. Any host system running in the Serviceguard cluster is called an active node. When you create the package, you specify a primary node and one or more adoptive nodes.
Serviceguard for Linux at a Glance What is Serviceguard for Linux? Serviceguard is designed to work in conjunction with other high availability products, such as disk arrays, which use various RAID levels for data protection; and HP-supported uninterruptible power supplies (UPS), such as HP PowerTrust, which eliminates failures related to power outage. These products are highly recommended along with Serviceguard to provide the greatest degree of availability.
Serviceguard for Linux at a Glance Using Serviceguard Manager Using Serviceguard Manager Serviceguard Manager is a “plug-in” to the web-based HP System Management Homepage (HP SMH). You can use Serviceguard Manager to monitor, administer, and configure Serviceguard clusters. • You can see properties, status, and alerts of cluster, nodes, and packages. • You can do administrative tasks such as run or halt clusters, cluster nodes, and packages.
Serviceguard for Linux at a Glance Using Serviceguard Manager Configuring Clusters with Serviceguard Manager You can configure clusters and legacy packages in Serviceguard Manager; modular packages must be configured by means of Serviceguard commands (see “How the Package Manager Works” on page 50; “Configuring Packages and Their Services” on page 199; and “Configuring a Legacy Package” on page 275). You must have root (UID=0) access to the cluster nodes.
Serviceguard for Linux at a Glance Configuration Roadmap Configuration Roadmap This manual presents the tasks you need to perform in order to create a functioning HA cluster using Serviceguard. These tasks are shown in Figure 1-3. Figure 1-3 Tasks in Configuring a Serviceguard Cluster HP recommends that you gather all the data that is needed for configuration before you start. Refer to Chapter 4, “Planning and Documenting an HA Cluster,” for tips on gathering data.
Understanding Hardware Configurations for Serviceguard for Linux 2 Understanding Hardware Configurations for Serviceguard for Linux This chapter gives a broad overview of how the server hardware components operate with Serviceguard for Linux. The following topics are presented: • Redundancy of Cluster Components • Redundant Network Components • Redundant Disk Storage • Redundant Power Supplies Refer to the next chapter for information about Serviceguard software components.
Understanding Hardware Configurations for Serviceguard for Linux Redundancy of Cluster Components Redundancy of Cluster Components In order to provide a high level of availability, a typical cluster uses redundant system components, for example two or more SPUs and two or more independent disks. Redundancy eliminates single points of failure. In general, the more redundancy, the greater your access to applications, data, and supportive services in the event of a failure.
Understanding Hardware Configurations for Serviceguard for Linux Redundant Network Components Redundant Network Components To eliminate single points of failure for networking, each subnet accessed by a cluster node is required to have redundant network interfaces. Redundant cables are also needed to protect against cable failures. Each interface card is connected to a different cable and hub or switch. Network interfaces are allowed to share IP addresses through a process known as channel bonding.
Understanding Hardware Configurations for Serviceguard for Linux Redundant Network Components Rules and Restrictions • The same subnet cannot be configured on different network interfaces (NICs) on the same node. • For IPv4 subnets, Serviceguard does not support different subnets on the same LAN interface. — For IPv6, Serviceguard supports up to two subnets per LAN interface (site-local and global).
Understanding Hardware Configurations for Serviceguard for Linux Redundant Network Components Redundant Ethernet Configuration The use of redundant network components is shown in Figure 2-1, which is an Ethernet configuration. Figure 2-1 Redundant LANs In Linux configurations, the use of symmetrical LAN configurations is strongly recommended, with the use of redundant hubs or switches to connect Ethernet segments.
Understanding Hardware Configurations for Serviceguard for Linux Redundant Network Components Cross-Subnet Configurations As of Serviceguard A.11.18 it is possible to configure multiple subnets, joined by a router, both for the cluster heartbeat and for data, with some nodes using one subnet and some another. A cross-subnet configuration allows: • Automatic package failover from a node on one subnet to a node on another • A cluster heartbeat that spans subnets.
Understanding Hardware Configurations for Serviceguard for Linux Redundant Network Components • There must be less than 200 milliseconds of latency in the heartbeat network. • Each heartbeat subnet on each node must be physically routed separately to the heartbeat subnet on another node; that is, each heartbeat path must be physically separate: — The heartbeats must be statically routed; static route entries must be configured on each node to route the heartbeats through different paths.
Understanding Hardware Configurations for Serviceguard for Linux Redundant Network Components IMPORTANT Although cross-subnet topology can be implemented on a single site, it is most commonly used by extended-distance clusters. Design and configuration of such clusters are covered in the disaster-tolerant documentation delivered with Serviceguard. For more information, see the following documents at http://www.docs.hp.
Understanding Hardware Configurations for Serviceguard for Linux Redundant Disk Storage Redundant Disk Storage Each node in a cluster has its own root disk, but each node may also be physically connected to several other disks in such a way that more than one node can obtain access to the data and programs associated with a package it is configured for. This access is provided by the Logical Volume Manager (LVM).
Understanding Hardware Configurations for Serviceguard for Linux Redundant Disk Storage disk failure occurs on one node, the monitor will cause the package to fail, with the potential to fail over to a different node on which the same disks are available. Sample Disk Configurations Figure 2-2 shows a two node cluster. Each node has one root disk which is mirrored and one package for which it is the primary node.
Understanding Hardware Configurations for Serviceguard for Linux Redundant Power Supplies Redundant Power Supplies You can extend the availability of your hardware by providing battery backup to your nodes and disks. HP-supported uninterruptible power supplies (UPS) can provide this protection from momentary power loss. Disks should be attached to power circuits in such a way that disk array copies are attached to different power sources.
Understanding Hardware Configurations for Serviceguard for Linux Redundant Power Supplies 36 Chapter 2
Understanding Serviceguard Software Components 3 Understanding Serviceguard Software Components This chapter gives a broad overview of how the Serviceguard software components work.
Understanding Serviceguard Software Components Serviceguard Architecture Serviceguard Architecture The following figure shows the main software components used by Serviceguard for Linux. This chapter discusses these components in some detail.
Understanding Serviceguard Software Components Serviceguard Architecture Each of these daemons logs to the Linux system logging files. The quorum server daemon logs to the user specified log file, such as, /usr/local/qs/log/qs.log file on Red Hat or /var/log/qs/sq.log on SUSE and cmomd logs to /usr/local/cmom/log/cmomd.log on Red Hat or /var/log/cmom/log/cmomd.log on SUSE. NOTE The file cmcluster.
Understanding Serviceguard Software Components Serviceguard Architecture NOTE Two of the central components of Serviceguard—Package Manager, and Cluster Manager—run as parts of the cmcld daemon. This daemon runs at priority 94 and is in the SCHED_RR class. No other process is allowed a higher real-time priority. Log Daemon: cmlogd cmlogd is used by cmcld to write messages to the system log file. Any message written to the system log by cmcld it written through cmlogd.
Understanding Serviceguard Software Components Serviceguard Architecture This daemon is started by xinetd. Parameters are in the /etc/xinetd.d/hacl-probe file. The path for this daemon is $SGLBIN/cmomd. This daemon may not be running on your system; it is used only by clients of the object manager. Service Assistant Daemon: cmsrvassistd This daemon forks and execs any script or processes as required by the cluster daemon, cmcld.
Understanding Serviceguard Software Components Serviceguard Architecture For more information about the Quorum Server software and how it works, including instructions for configuring the Quorum Server as a Serviceguard package, see the latest version of the HP Serviceguard Quorum Server release notes at http://docs.hp.com -> High Availability -> Quorum Server. See also “Use of the Quorum Server as a Cluster Lock” on page 47.
Understanding Serviceguard Software Components How the Cluster Manager Works How the Cluster Manager Works The cluster manager is used to initialize a cluster, to monitor the health of the cluster, to recognize node failure if it should occur, and to regulate the re-formation of the cluster when a node joins or leaves the cluster. The cluster manager operates as a daemon process that runs on each node.
Understanding Serviceguard Software Components How the Cluster Manager Works nodes as before. In such cases, packages do not halt or switch, though the application may experience a slight performance impact during the re-formation. If heartbeat and data are sent over the same LAN subnet, data congestion may cause Serviceguard to miss heartbeats and initiate a cluster re-formation that would not otherwise have been needed.
Understanding Serviceguard Software Components How the Cluster Manager Works During startup, the cluster manager software checks to see if all nodes specified in the startup command are valid members of the cluster, are up and running, are attempting to form a cluster, and can communicate with each other. If they can, then the cluster manager forms the cluster. Automatic Cluster Startup An automatic cluster startup occurs any time a node reboots and joins the cluster.
Understanding Serviceguard Software Components How the Cluster Manager Works Cluster Quorum to Prevent Split-Brain Syndrome In general, the algorithm for cluster re-formation requires a cluster quorum of a strict majority (that is, more than 50%) of the nodes previously running. If both halves (exactly 50%) of a previously running cluster were allowed to re-form, there would be a split-brain situation in which two instances of the same cluster were running.
Understanding Serviceguard Software Components How the Cluster Manager Works The operation of the lock LUN is shown in Figure 3-2. Figure 3-2 Lock LUN Operation Serviceguard periodically checks the health of the lock LUN and writes messages to the syslog file if the disk fails the health check. This file should be monitored for early detection of lock disk problems. Use of the Quorum Server as a Cluster Lock The cluster lock in Linux can also be implemented by means of a quorum server.
Understanding Serviceguard Software Components How the Cluster Manager Works The operation of the quorum server is shown in Figure 3-3. When there is a loss of communication between node 1 and node 2, the quorum server chooses one node (in this example, node 2) to continue running in the cluster. The other node halts. Figure 3-3 Quorum Server Operation A quorum server can provide quorum services for multiple clusters. Figure 3-4 illustrates quorum server use across four clusters.
Understanding Serviceguard Software Components How the Cluster Manager Works Figure 3-4 Quorum Server to Cluster Distribution HP recommends that two clusters running quorum server packages do not provide quorum services for each other. For example if qspkgA running on cluster1 is providing quorum services for cluster2, qspkgB running on cluster2 should not provide quorum services for cluster1. This recommendation is a guideline, not an absolute rule.
Understanding Serviceguard Software Components How the Package Manager Works How the Package Manager Works Packages are the means by which Serviceguard starts and halts configured applications. A package is a collection of services, disk volumes and IP addresses that are managed by Serviceguard to ensure they are available. Each node in the cluster runs an instance of the package manager; the package manager residing on the cluster coordinator is known as the package coordinator.
Understanding Serviceguard Software Components How the Package Manager Works Failover Packages A failover package starts up on an appropriate node (see node_name on page 210) when the cluster starts. In the case of a service, network, or other resource or dependency failure, package failover takes place. A package failover involves both halting the existing package and starting the new instance of the package on a new node.
Understanding Serviceguard Software Components How the Package Manager Works Deciding When and Where to Run and Halt Failover Packages The package configuration file assigns a name to the package and includes a list of the nodes on which the package can run. Failover packages list the nodes in order of priority (i.e., the first node in the list is the highest priority node). In addition, failover packages’ files contain three parameters that determine failover behavior.
Understanding Serviceguard Software Components How the Package Manager Works started. (In a cross-subnet configuration, all the monitored subnets that are specified for this package, and configured on the target node, must be up.) If the package has a dependency on a resource or another package, the dependency must be met on the target node before the package can start. The switching of relocatable IP addresses is shown in Figure 3-6 and Figure 3-7.
Understanding Serviceguard Software Components How the Package Manager Works NOTE For design and configuration information about clusters that span subnets), see the documents listed under “Cross-Subnet Configurations” on page 30. Figure 3-7 After Package Switching Failover Policy The Package Manager selects a node for a failover package to run on based on the priority list included in the package configuration file together with the failover_policy parameter, also in the configuration file.
Understanding Serviceguard Software Components How the Package Manager Works startup. The two failover policies are configured_node (the default) and min_package_node. The parameter is set in the package configuration file. If you use configured_node as the value for the failover policy, the package will start up on the highest priority node available in the node list. When a failover occurs, the package will move to the next highest priority node in the list that is available.
Understanding Serviceguard Software Components How the Package Manager Works Figure 3-8 Rotating Standby Configuration before Failover If a failure occurs, any package would fail over to the node containing fewest running packages, as in Figure 3-9, which shows a failure on node 2: Figure 3-9 Rotating Standby Configuration after Failover NOTE Using the min_package_node policy, when node 2 is repaired and brought back into the cluster, it will then be running the fewest packages, and thus will become th
Understanding Serviceguard Software Components How the Package Manager Works Figure 3-10 CONFIGURED_NODE Policy Packages after Failover If you use configured_node as the failover policy, the package will start up on the highest priority node in the node list, assuming that the node is running as a member of the cluster. When a failover occurs, the package will move to the next highest priority node in the list that is available.
Understanding Serviceguard Software Components How the Package Manager Works Figure 3-11 Automatic Failback Configuration before Failover Table 3-2 Node Lists in Sample Cluster Package Name NODE_NAME List FAILOVER POLICY FAILBACK POLICY pkgA node1, node4 CONFIGURED_NODE AUTOMATIC pkgB node2, node4 CONFIGURED_NODE AUTOMATIC pkgC node3, node4 CONFIGURED_NODE AUTOMATIC Node1 panics, and after the cluster reforms, pkgA starts running on node4: 58 Chapter 3
Understanding Serviceguard Software Components How the Package Manager Works Figure 3-12 Automatic Failback Configuration After Failover After rebooting, node 1 rejoins the cluster. At that point, pkgA will be automatically stopped on node 4 and restarted on node 1.
Understanding Serviceguard Software Components How the Package Manager Works NOTE Setting the failback_policy to automatic can result in a package failback and application outage during a critical production period. If you are using automatic failback, you may want to wait to add the package’s primary node back into the cluster until you can allow the package to be taken out of service temporarily while it switches back to the primary node.
Understanding Serviceguard Software Components How Packages Run How Packages Run Packages are the means by which Serviceguard starts and halts configured applications. Failover packages are also units of failover behavior in Serviceguard. A package is a collection of services, disk volumes and IP addresses that are managed by Serviceguard to ensure they are available. There can be a maximum of 150 packages per cluster and a total of 900 services per cluster.
Understanding Serviceguard Software Components How Packages Run nodes, or that the package has a dependency that is not being met. When a package has failed on one node and is enabled to switch to another node, it will start up automatically in a new location where its dependencies are met. This process is known as package switching, or remote switching. A failover package starts on the first available node in its configuration file; by default, it fails over to the next available one in the list.
Understanding Serviceguard Software Components How Packages Run NOTE This diagram applies specifically to legacy packages. Differences for modular scripts are called out below. Figure 3-14 Legacy Package Time Line Showing Important Events The following are the most important moments in a package’s life: 1. Before the control script starts. (For modular packages, this is the master control script.) 2. During run script execution.
Understanding Serviceguard Software Components How Packages Run Before the Control Script Starts First, a node is selected. This node must be in the package’s node list, it must conform to the package’s failover policy, and any resources required by the package must be available on the chosen node. One resource is the subnet that is monitored for the package. If the subnet is not available, the package cannot start on this node. Another type of resource is a dependency on another package.
Understanding Serviceguard Software Components How Packages Run Figure 3-15 Legacy Package Time Line At any step along the way, an error will result in the script exiting abnormally (with an exit code of 1). For example, if a package service is unable to be started, the control script will exit with an error. NOTE This diagram is specific to legacy packages. Modular packages also run external scripts and “pre-scripts” as explained above.
Understanding Serviceguard Software Components How Packages Run NOTE After the package run script has finished its work, it exits, which means that the script is no longer executing once the package is running normally. After the script exits, the PIDs of the services started by the script are monitored by the package manager directly.
Understanding Serviceguard Software Components How Packages Run Service Startup with cmrunserv Within the package control script, the cmrunserv command starts up the individual services. This command is executed once for each service that is coded in the file. You can configure a number of restarts for each service. The cmrunserv command passes this number to the package manager, which will restart the service the appropriate number of times if the service should fail.
Understanding Serviceguard Software Components How Packages Run When a Service or Subnet Fails, or a Dependency is Not Met What happens when something goes wrong? If a service fails and there are no more restarts, or if a configured dependency on another package is not met, then a failover package will halt on its current node and, depending on the setting of the package switching flags, may be restarted on another node.
Understanding Serviceguard Software Components How Packages Run NOTE If you use cmhaltpkg command with the -n option, the package is halted only if it is running on that node. The cmmodpkg command cannot be used to halt a package, but it can disable switching either on particular nodes or on all nodes. A package can continue running when its switching has been disabled, but it will not be able to start on other nodes if it stops running on its current node.
Understanding Serviceguard Software Components How Packages Run Figure 3-16 Legacy Package Time Line for Halt Script Execution At any step along the way, an error will result in the script exiting abnormally (with an exit code of 1). Also, if the halt script execution is not complete before the time specified in the halt_script_timeout, the package manager will kill the script. During halt script execution, messages are written to a log file.
Understanding Serviceguard Software Components How Packages Run • 1—abnormal exit, also known as no_restart exit. The package did not halt normally. Services are killed, and the package is disabled globally. It is not disabled on the current node, however. • Timeout—Another type of exit occurs when the halt_script_timeout is exceeded. In this scenario, the package is killed and disabled globally. It is not disabled on the current node, however.
Understanding Serviceguard Software Components How Packages Run Table 3-3 Error Conditions and Package Movement for Failover Packages Package Error Condition Results Halt script runs after Error or Exit Package Allowed to Run on Primary Node after Error Node Failfast Enabled Service Failfast Enabled Linux Status on Primary after Error Run Script Timeout YES Either Setting system reset No N/A (system reset) Yes Run Script Timeout NO Either Setting Running No Not changed No Halt Script E
Understanding Serviceguard Software Components How Packages Run Table 3-3 Error Conditions and Package Movement for Failover Packages Package Error Condition Results Linux Status on Primary after Error Halt script runs after Error or Exit Package Allowed to Run on Primary Node after Error Package Allowed to Run on Alternate Node Node Failfast Enabled Service Failfast Enabled Loss of Network NO Either Setting Running Yes Yes Yes package depended on failed Either Setting Either Setting Runn
Understanding Serviceguard Software Components How the Network Manager Works How the Network Manager Works The purpose of the network manager is to detect and recover from network card failures so that network services remain highly available to clients. In practice, this means assigning IP addresses for each package to the primary LAN interface card on the node where the package is running and monitoring the health of all interfaces, switching them when necessary.
Understanding Serviceguard Software Components How the Network Manager Works IMPORTANT Any subnet identified as a monitored_subnet in the package configuration file must be configured as a STATIONARY_IP or HEARTBEAT_IP in the cluster configuration file. See “Cluster Configuration Parameters” on page 110 and “Package Parameter Explanations” on page 209. In addition to the stationary IP address, you normally assign one or more unique IP addresses to each package.
Understanding Serviceguard Software Components How the Network Manager Works Types of IP Addresses Both IPv4 and IPv6 address types are supported in Serviceguard. IPv4 addresses are the traditional addresses of the form n.n.n.n where n is a decimal digit between 0 and 255. IPv6 addresses have the form x:x:x:x:x:x:x:x where x is the hexadecimal value of each of eight 16-bit pieces of the 128-bit address.
Understanding Serviceguard Software Components How the Network Manager Works Bonding of LAN Interfaces On the local node, several LAN interfaces can be grouped together in a process known in Linux as channel bonding. In the bonded group, one interface is used to transmit and receive data, while the others are available as backups. If one interface fails, another interface in the bonded group takes over.
Understanding Serviceguard Software Components How the Network Manager Works address and MAC address. In this example, the aggregated ports are collectively known as bond0, and this is the name by which the bond is known during cluster configuration. Figure 3-18 shows a bonded configuration using redundant hubs with a crossover cable.
Understanding Serviceguard Software Components How the Network Manager Works After the failure of a card, messages are still carried on the bonded LAN and are received on the other node, but now eth1 has become active in bond0 on node1. This situation is shown in Figure 3-19.
Understanding Serviceguard Software Components How the Network Manager Works Bonding for Load Balancing It is also possible to configure bonds in load balancing mode, which allows all slaves to transmit data in parallel, in an active/active arrangement. In this case, high availability is provided by the fact that the bond still continues to function (with less throughput) if one of the component LANs should fail.
Understanding Serviceguard Software Components How the Network Manager Works Package Switching and Relocatable IP Addresses A package switch involves moving the package to a new system.
Understanding Serviceguard Software Components How the Network Manager Works ARP request. The sender and receiver protocol address fields of the ARP request message are both set to the same relocatable IP address. This ensures that nodes receiving the message will not send replies. Unlike IPv4, IPv6 addresses use NDP messages to determine the link-layer addresses of their neighbors.
Understanding Serviceguard Software Components Volume Managers for Data Storage Volume Managers for Data Storage A volume manager is a tool that lets you create units of disk storage that are more flexible than individual disk partitions. These units can be used on single systems or in high availability clusters. HP Serviceguard for Linux uses the Linux Logical Volume Manager (LVM) which creates redundant storage groups. This section provides an overview of volume management with LVM.
Understanding Serviceguard Software Components Volume Managers for Data Storage Examples of Storage on Smart Arrays Figure 3-21 shows an illustration of storage configured on a Smart MSA500 storage. Physical disks are configured by an array utility program into logical units or LUNs which are then seen by the operating system. Figure 3-21 Physical Disks Combined into LUNs NOTE LUN definition is normally done using utility programs provided by the disk array manufacturer.
Understanding Serviceguard Software Components Volume Managers for Data Storage Figure 3-22 shows LUNs configured with a Smart Array cluster storage with single pathways to the data.
Understanding Serviceguard Software Components Volume Managers for Data Storage Finally, the Smart Array LUNs are configured into volume groups as shown in Figure 3-23. Figure 3-23 Smart Array LUNs Configured in Volume Groups For information about configuring multipathing, see “Multipath for Storage” on page 99. Monitoring Disks Each package configuration includes information about the disks that are to be activated by the package at startup.
Understanding Serviceguard Software Components Volume Managers for Data Storage More Information on LVM Refer to the section “Creating the Logical Volume Infrastructure” in Chapter 5 for details about configuring volume groups, logical volumes, and file systems for use in Serviceguard packages. Refer to the article, Logical Volume Manager HOWTO on the Linux Documentation Project page at http://www.tldp.org for a basic description of Linux LVM.
Understanding Serviceguard Software Components Responses to Failures Responses to Failures Serviceguard responds to different kinds of failures in specific ways. For most hardware failures, the response is not user-configurable, but for package and service failures, you can choose the system’s response, within limits. Reboot When a Node Fails The most dramatic response to a failure in a Serviceguard cluster is a system reboot.
Understanding Serviceguard Software Components Responses to Failures 1. The node tries to reform the cluster. 2. If the node cannot get a quorum (if it cannot get the cluster lock) then 3. The node halts (system reset). Example Situation. Assume a two-node cluster, with Package1 running on SystemA and Package2 running on SystemB. Volume group vg01 is exclusively activated on SystemA; volume group vg02 is exclusively activated on SystemB. Package IP addresses are assigned to SystemA and SystemB respectively.
Understanding Serviceguard Software Components Responses to Failures Responses to Hardware Failures If a serious system problem occurs, such as a system panic or physical disruption of the SPU's circuits, Serviceguard recognizes a node failure and transfers the packages currently running on that node to an adoptive node elsewhere in the cluster. (System multi-node and multi-node packages do not fail over.
Understanding Serviceguard Software Components Responses to Failures Responses to Package and Service Failures In the default case, the failure of the package or of a service within a package causes the package to shut down by running the control script with the 'stop' parameter, and then restarting the package on an alternate node. A package will also fail if it is configured to have a dependency on another package, and that package fails.
Understanding Serviceguard Software Components Responses to Failures Network Communication Failure An important element in the cluster is the health of the network itself. As it continuously monitors the cluster, each node listens for heartbeat messages from the other nodes confirming that all nodes are able to communicate with each other.
Planning and Documenting an HA Cluster 4 Planning and Documenting an HA Cluster Building a Serviceguard cluster begins with a planning phase in which you gather and record information about all the hardware and software components of the configuration.
Planning and Documenting an HA Cluster General Planning General Planning A clear understanding of your high availability objectives will quickly help you to define your hardware requirements and design your system. Use the following questions as a guide for general planning: 1. What applications must continue to be available in the event of a failure? 2. What system resources (processing power, networking, SPU, memory, disk space) are needed to support these applications? 3.
Planning and Documenting an HA Cluster General Planning additional disk hardware for shared data storage. If you intend to expand your cluster without the need to bring it down, careful planning of the initial configuration is required. Use the following guidelines: • Set the Maximum Configured Packages parameter (described later in this chapter under “Cluster Configuration Planning” on page 109) high enough to accommodate the additional packages you plan to add.
Planning and Documenting an HA Cluster Hardware Planning Hardware Planning Hardware planning requires examining the physical hardware itself. One useful procedure is to sketch the hardware configuration in a diagram that shows adapter cards and buses, cabling, disks and peripherals. Then record the information on the Hardware Worksheet (page 364). Indicate which device adapters occupy which slots.
Planning and Documenting an HA Cluster Hardware Planning LAN Information While a minimum of one LAN interface per subnet is required, at least two LAN interfaces are needed to eliminate single points of network failure. HP recommends that you configure heartbeats on all subnets, including those to be used for client data. Collect the following information for each LAN interface: Subnet Name Enter the IP address for the subnet. Note that heartbeat IP addresses must be on the same subnet on each node.
Planning and Documenting an HA Cluster Hardware Planning Kind of LAN Traffic Identify the purpose of the subnet. Valid types include the following: • Heartbeat • Client Traffic Label the list to show the subnets that belong to a bridged net. This information is used in creating the subnet groupings and identifying the IP addresses used in the cluster and package configuration files. Shared Storage SCSI may be used for up to four node clusters, or FibreChannel can be used for clusters of up to 16 nodes.
Planning and Documenting an HA Cluster Hardware Planning NOTE Multipath capabilities are supported by FibreChannel HBA device drivers. Check with the storage device documentation for details. See also “Multipath for Storage” on page 99. On the worksheet, enter the names of the device files that correspond to each LUN for the Fibre-Channel-attached storage unit.
Planning and Documenting an HA Cluster Hardware Planning NOTE With the rapid evolution of Linux, the multipath mechanisms may change, or new ones may be added. Serviceguard for Linux supports DeviceMapper multipath (DM-MPIO) with some restrictions; see the Serviceguard for Linux Certification Matrix at the address provided in the Preface to this manual for up-to-date information. NOTE md also supports software RAID; but this configuration is not currently supported with Serviceguard for Linux.
Planning and Documenting an HA Cluster Hardware Planning Disk Device File Enter the disk device file name for each SCSI disk or LUN. Information from this section of the worksheet is used in creating the mirrored disk configuration using LVM. In addition, it is useful to gather as much information as possible about your disk configuration. You can obtain information about available disks by using the following commands; your system may provide other utilities as well.
Planning and Documenting an HA Cluster Power Supply Planning Power Supply Planning There are two sources of power for your cluster which you will have to consider in your design: line power and uninterruptible power supplies (UPS). Loss of a power circuit should not bring down the cluster. Frequently, servers, mass storage devices, and other hardware have two or three separate power supplies, so they can survive the loss of power to one or more power supplies or power circuits.
Planning and Documenting an HA Cluster Power Supply Planning Host Name Enter the host name for each SPU. Disk Unit Enter the disk drive unit number for each disk. Tape Unit Enter the tape unit number for each backup device. Other Unit Enter the number of any other unit. Power Supply Enter the power supply unit number of the UPS to which the host or other device is connected. Be sure to follow UPS, power circuit, and cabinet power limits as well as SPU power limits.
Planning and Documenting an HA Cluster Cluster Lock Planning Cluster Lock Planning The purpose of the cluster lock is to ensure that only one new cluster is formed in the event that exactly half of the previously clustered nodes try to form a new cluster. It is critical that only one new cluster is formed and that it alone has access to the disks specified in its packages. You can specify a lock LUN or a quorum server as the cluster lock.
Planning and Documenting an HA Cluster Cluster Lock Planning Using a Quorum Server The Quorum Server is described under “Use of the Quorum Server as a Cluster Lock” on page 47. See also “Cluster Lock” on page 46. A quorum server: IMPORTANT Chapter 4 • Can be used with up to 150 clusters, not exceeding 300 nodes total. • Can support a cluster with any supported number of nodes. • Can support a cluster with any supported number of nodes. • Must be configured on an IPv4, not an IPv6, network.
Planning and Documenting an HA Cluster Cluster Lock Planning Quorum Server Worksheet Use the Quorum Server Worksheet, in Appendix D, “Blank Planning Worksheets,” on page 363, to identify a quorum server for use with one or more clusters. Make as many copies as you need. Fill out the worksheet and keep it for future reference. On the QS worksheet, enter the following: Quorum Server Host Enter the host name for the quorum server.
Planning and Documenting an HA Cluster Cluster Lock Planning Quorum Server Data: ============================================================================== QS Hostname: ___________IP Address: _____________IP Address: ____________ ============================================================================== Quorum Services are Provided for: Cluster Name: ___________________________________________________________ Host Names ____________________________________________ Host Names ______________________
Planning and Documenting an HA Cluster Volume Manager Planning Volume Manager Planning When designing your disk layout using LVM, you should consider the following: • The volume groups that contain high availability applications, services, or data must be on a bus or buses available to the primary node and all adoptive nodes. • High availability applications, services, and data should be placed in volume groups that are separate from non-high availability applications, services, and data.
Planning and Documenting an HA Cluster Cluster Configuration Planning Cluster Configuration Planning A cluster should be designed to provide the quickest possible recovery from failures. The actual time required to recover from a failure depends on several factors: • The length of the cluster heartbeat interval and node timeout. See the parameter descriptions for HEARTBEAT_INTERVAL and NODE_TIMEOUT under “Cluster Configuration Parameters” starting on page 110 for recommendations.
Planning and Documenting an HA Cluster Cluster Configuration Planning Cluster Configuration Parameters You need to define a set of cluster parameters. These are stored in the binary cluster configuration file, which is distributed to each node in the cluster. You configure these parameters by editing the cluster configuration template file created by means of the cmquerycl command, as described under “Configuring the Cluster” on page 172.
Planning and Documenting an HA Cluster Cluster Configuration Planning Make sure that the cluster name is unique within the subnets configured on the cluster nodes; under some circumstances Serviceguard may not be able to detect a duplicate name and unexpected problems may result.
Planning and Documenting an HA Cluster Cluster Configuration Planning NODE_NAME CAUTION The hostname of each system that will be a node in the cluster. Make sure that the node name is unique within the subnets configured on the cluster nodes; under some circumstances Serviceguard may not be able to detect a duplicate name and unexpected problems may result. Do not use the full domain name. For example, enter ftsys9, not ftsys9.cup.hp.com. A cluster can contain up to 16 nodes.
Planning and Documenting an HA Cluster Cluster Configuration Planning NETWORK_INTERFACE The name of each LAN that will be used for heartbeats or for user data. An example is eth0. For information about changing the configuration online, see “Changing the Cluster Networking Configuration while the Cluster Is Running” on page 269. HEARTBEAT_IP IPv4 notation indicating a subnet that will carry the cluster heartbeat. (IPv6 addresses are not supported for heartbeat).
Planning and Documenting an HA Cluster Cluster Configuration Planning Heartbeat configuration requirements: The cluster needs at least two network interfaces for the heartbeat in all cases, using one of the following minimum configurations: NOTE • two heartbeat subnets; or • one heartbeat subnet using bonding in high availability mode (or mode 1) with two slaves.
Planning and Documenting an HA Cluster Cluster Configuration Planning The use of a private heartbeat network is not advisable if you plan to use Remote Procedure Call (RPC) protocols and services. RPC assumes that each network adapter device or I/O card is connected to a route-able network. An isolated or private heartbeat LAN is not route-able, and could cause an RPC request-reply, directed to that LAN, to time-out without being serviced.
Planning and Documenting an HA Cluster Cluster Configuration Planning HEARTBEAT_INTERVAL The normal interval between the transmission of heartbeat messages from one node to the other in the cluster. Enter a number of microseconds. The default value is 1,000,000 microseconds; setting the parameter to a value less than the default is not recommended. The default should be used where possible.
Planning and Documenting an HA Cluster Cluster Configuration Planning There are more complex cases that require you to make a trade-off between fewer failovers and faster failovers. For example, a network event such as a broadcast storm may cause kernel interrupts to be turned off on some or all nodes while the packets are being processed, preventing the nodes from sending and processing heartbeat messages. This in turn could prevent the kernel’s safety timer from being reset, causing the node to halt.
Planning and Documenting an HA Cluster Cluster Configuration Planning (Access Control Policies, also known as Role Based Access) Specify three things for each policy: USER_NAME, USER_HOST, and USER_ROLE. Policies set in the configuration file of a cluster and its packages must not be conflicting or redundant. For more information, see “Controlling Access to the Cluster” on page 180. MAX_CONFIGURED_PACKAGES This parameter sets the maximum number of packages that can be configured in the cluster.
Planning and Documenting an HA Cluster Package Configuration Planning Package Configuration Planning Planning for packages involves assembling information about each group of highly available services. NOTE As of Serviceguard A.11.18, there is a new and simpler way to configure packages.
Planning and Documenting an HA Cluster Package Configuration Planning In Serviceguard, high availability applications, services, and data are located in volume groups that are on a shared bus. When a node fails, the volume groups containing the applications, services, and data of the failed node are deactivated on the failed node and activated on the adoptive node (the node the packages move to).
Planning and Documenting an HA Cluster Package Configuration Planning # # # # # # /dev/vg01/lvoldb1 /dev/vg01/lvoldb2 /dev/vg01/lvoldb3 /dev/vg01/lvoldb4 /dev/vg01/lvoldb5 /dev/vg01/lvoldb6 /applic1 ext2 defaults 0 /applic2 ext2 defaults 0 raw_tables ignore ignore /general ext2 defaults 0 raw_free ignore ignore 0 raw_free ignore ignore 0 1 # These six entries are 1 # for information purposes 0 0 # only. They record the 2 # logical volumes that 0 # exist for Serviceguard's 0 # HA package.
Planning and Documenting an HA Cluster Package Configuration Planning The following table describes different types of failover behavior and the settings in the package configuration file that determine each behavior. See “Package Parameter Explanations” on page 209 for more information. Table 4-1 Package Failover Behavior Switching Behavior 122 Parameters in Configuration File Package switches normally after detection of service or network, failure, or when a configured dependency is not met.
Planning and Documenting an HA Cluster Package Configuration Planning Table 4-1 Package Failover Behavior (Continued) Switching Behavior All packages switch following a system reboot on the node when a specific service fails. Halt scripts are not run. All packages switch following a system reboot on the node when any service fails. Parameters in Configuration File • service_fail_fast_enabled set to yes for a specific service. • auto_run set to yes for all packages.
Planning and Documenting an HA Cluster Package Configuration Planning Rules Assume that we want to make pkg1 depend on pkg2. NOTE pkg1 can depend on more than one other package, and pkg2 can depend on another package or packages; we are assuming only two packages in order to make the rules as clear as possible. • pkg1 will not start on any node unless pkg2 is running on that node.
Planning and Documenting an HA Cluster Package Configuration Planning • A package cannot depend on itself, directly or indirectly. That is, not only must pkg1 not specify itself in the dependency_condition (see page 216), but pkg1 must not specify a dependency on pkg2 if pkg2 depends on pkg1, or if pkg2 depends on pkg3 which depends on pkg1, etc.
Planning and Documenting an HA Cluster Package Configuration Planning NOTE This applies only when the packages are automatically started (package switching enabled); cmrunpkg will never force a package to halt. Keep in mind that you do not have to set priority, even when one or more packages depend on another. The default value, no_priority, may often result in the behavior you want.
Planning and Documenting an HA Cluster Package Configuration Planning If pkg1 depends on pkg2, and pkg1’s priority is lower than or equal to pkg2’s, pkg2’s node order dominates. Assuming pkg2’s node order is node1, node2, node3, then: • On startup: — pkg2 will start on node1, or node2 if node1 is not available or does not at present meet all of its dependencies, etc.
Planning and Documenting an HA Cluster Package Configuration Planning — if pkg2 has failed back to node1 and node1 does not meet all of pkg1’s dependencies, pkg1 will halt. If pkg1 depends on pkg2, and pkg1’s priority is higher than pkg2’s, pkg1’s node order dominates. Assuming pkg1’s node order is node1, node2, node3, then: • On startup: — pkg1 will select node1 to start on. — pkg2 will start on node1, provided it can run there (no matter where node1 appears on pkg2’s node_name list).
Planning and Documenting an HA Cluster Package Configuration Planning But you also need to weigh the relative importance of the packages. If pkg2 runs a database that is central to your business, you probably want it to run undisturbed, no matter what happens to application packages that depend on it. In this case, the database package should have the highest priority. Note that, if no priorities are set, the dragging rules favor a package that is depended on over a package that depends on it.
Planning and Documenting an HA Cluster Package Configuration Planning • On package startup and shutdown, as essentially the first and last functions the package performs. These scripts are invoked by means of the parameter external_pre_script (see page 226); or • During package execution, after volume-groups and file systems are activated, and IP addresses are assigned, and before the service and resource functions are executed; and again, in the reverse order, on package shutdown.
Planning and Documenting an HA Cluster Package Configuration Planning A sample script follows. It assumes there is another script called monitor.sh, which will be configured as a Serviceguard service to monitor some application. The monitor.
Planning and Documenting an HA Cluster Package Configuration Planning #!/bin/sh # Source utility functions. if [[ -z $SG_UTILS ]] then . $SGCONF.conf SG_UTILS=$SGCONF/scripts/mscripts/utils.sh fi if [[ -f ${SG_UTILS} ]]; then .
Planning and Documenting an HA Cluster Package Configuration Planning found=1 break ;; *) ;; esac (( i = i + 1 )) done if (( found == 0 )) then sg_log 0 "ERROR: monitoring service not configured!" ret=1 fi if (( ret == 1 )) then sg_log 0 "Script validation for $SG_PACKAGE_NAME failed!" fi return $ret } function start_command { sg_log 5 "start_command" # log current PEV_MONITORING_INTERVAL value, PEV_ attribute can be changed # while the package is running sg_log 0 "PEV_MONITORING_INTERVAL for $SG_PACKAGE_NA
Planning and Documenting an HA Cluster Package Configuration Planning ;; stop) stop_command $* exit_val=$? ;; validate) validate_command $* exit_val=$? ;; *) sg_log 0 "Unknown entry point $1" ;; esac exit $exit_val Using Serviceguard Commands in an External Script You can use Serviceguard commands (such as cmmodpkg) in an external script. These commands must not interact with the package itself (that is, the package that runs the external script) but can interact with other packages.
Planning and Documenting an HA Cluster Package Configuration Planning • failure - set if the package halts because of the failure of a subnet, resource, or service it depends on • user_halt - set if the package is halted by a cmhaltpkg or cmhaltnode command, or by corresponding actions in Serviceguard Manager • automatic_halt - set if the package is failed over automatically because of the failure of a package it depends on, or is failed back to its primary node automatically (failback_policy = automat
Planning and Documenting an HA Cluster Package Configuration Planning About Cross-Subnet Failover It is possible to configure a cluster that spans subnets joined by a router, with some nodes using one subnet and some another. This is known as a cross-subnet configuration (see “Cross-Subnet Configurations” on page 30). In this context, you can configure packages to fail over from a node on one subnet to a node on another.
Planning and Documenting an HA Cluster Package Configuration Planning Implications for Application Deployment Because the relocatable IP address will change when a package fails over to a node on another subnet, you need to make sure of the following: • The hostname used by the package is correctly remapped to the new relocatable IP address. • The application that the package runs must be configured so that the clients can reconnect to the package’s new relocatable IP address.
Planning and Documenting an HA Cluster Package Configuration Planning Assuming nodeA is pkg1’s primary node (where it normally starts), create node_name entries in the package configuration file as follows: node_name nodeA node_name nodeB node_name nodeC node_name nodeD Configuring monitored_subnet_access In order to monitor subnet 15.244.65.0 or 15.244.56.
Planning and Documenting an HA Cluster Package Configuration Planning Configuring ip_subnet_node Now you need to specify which subnet is configured on which nodes. In our example, you would do this by means of entries such as the following in the package configuration file: ip_subnet 15.244.65.0 ip_subnet_node nodeA ip_subnet_node nodeB ip_address 15.244.65.82 ip_address 15.244.65.83 ip_subnet 15.244.56.0 ip_subnet_node nodeC ip_subnet_node nodeD ip_address 15.244.56.100 ip_address 15.244.56.
Planning and Documenting an HA Cluster Planning for Changes in Cluster Size Planning for Changes in Cluster Size If you intend to add additional nodes to the cluster online (while it is running) ensure that they are connected to the same heartbeat subnets and to the same lock disks as the other cluster nodes. In selecting a cluster lock configuration, be careful to anticipate any potential need for additional cluster nodes.
Building an HA Cluster Configuration 5 Building an HA Cluster Configuration This chapter and the next take you through the configuration tasks required to set up a Serviceguard cluster. You carry out these procedures on one node, called the configuration node, and Serviceguard distributes the resulting binary file to all the nodes in the cluster. In the examples in this chapter, the configuration node is named ftsys9, and the sample target node is called ftsys10.
Building an HA Cluster Configuration Preparing Your Systems Preparing Your Systems Before configuring your cluster, ensure that Serviceguard is installed on all cluster nodes, and that all nodes have the appropriate security files, kernel configuration and NTP (network time protocol) configuration. Installing and Updating Serviceguard For information about installing and updating Serviceguard, see the Release Notes for your version at http://docs.hp.
Building an HA Cluster Configuration Preparing Your Systems Throughout this document, system filenames are usually given with one of these location prefixes. Thus, references to $SGCONF/ can be resolved by supplying the definition of the prefix that is found in this file. For example, if SGCONF is /usr/local/cmcluster/conf, then the complete pathname for file $SGCONF/cmclconfig would be /usr/local/cmcluster/conf/cmclconfig.
Building an HA Cluster Configuration Preparing Your Systems Configuring Root-Level Access The subsections that follow explain how to set up root access between the nodes in the prospective cluster. (When you proceed to configuring the cluster, you will define various levels of non-root access as well; see “Controlling Access to the Cluster” on page 180.) NOTE For more information and advice, see the white paper Securing Serviceguard at http://docs.hp.
Building an HA Cluster Configuration Preparing Your Systems This example grants root access to the node on which this cmclnodelist file resides to root users on the nodes gryf, sly, and bit. Serviceguard also accepts the use of a “+” in the cmclnodelist file; this indicates that the root user on any Serviceguard node can configure Serviceguard on this node. IMPORTANT If $SGCONF/cmclnodelist does not exist, Serviceguard will look at ~/.rhosts. HP strongly recommends that you use cmclnodelist.
Building an HA Cluster Configuration Preparing Your Systems Configuring Name Resolution Serviceguard uses the name resolution services built into Linux. Serviceguard nodes can communicate over any of the cluster’s shared networks, so the network resolution service you are using (such as DNS, NIS, or LDAP) must be able to resolve each of their primary addresses on each of those networks to the primary hostname of the node in question.
Building an HA Cluster Configuration Preparing Your Systems NOTE Serviceguard recognizes only the hostname (the first element) in a fully qualified domain name (a name with four elements separated by periods, like those in the example above). This means, for example, that gryf.uksr.hp.com and gryf.cup.hp.com cannot be nodes in the same cluster, as Serviceguard would see them as the same host gryf.
Building an HA Cluster Configuration Preparing Your Systems The procedure that follows shows how to create a robust name-resolution configuration that will allow cluster nodes to continue communicating with one another if a name service fails. Step 1. Edit the /etc/hosts file on all nodes in the cluster. Add name resolution for all heartbeat IP addresses, and other IP addresses from all the cluster nodes; see “Configuring Name Resolution” on page 146 for discussion and examples.
Building an HA Cluster Configuration Preparing Your Systems files [NOTFOUND=continue UNAVAIL=continue] nis [NOTFOUND=return UNAVAIL=return] This step is critical, allowing the cluster nodes to resolve hostnames to IP addresses while DNS, NIS, or the primary LAN is down. Step 4. Create a $SGCONF/cmclnodelist file on all nodes that you intend to configure into the cluster, and allow access by all cluster nodes. See “Allowing Root Access to an Unconfigured Node” on page 144.
Building an HA Cluster Configuration Preparing Your Systems Implementing Channel Bonding (Red Hat) This section applies to Red Hat installations. If you are using a SUSE distribution, skip ahead to the next section. Channel bonding of LAN interfaces is implemented by the use of the bonding driver, which is installed in the kernel at boot time. With this driver installed, the networking software recognizes bonding definitions that are created in the /etc/sysconfig/network-scripts directory for each bond.
Building an HA Cluster Configuration Preparing Your Systems Sample Configuration Configure the following files to support LAN redundancy. For a single failover only one bond is needed. 1. Create a bond0 file, ifcfg-bond0. Create the configuration in the /etc/sysconfig/network-scripts directory. For example, in the file, ifcfg-bond0, bond0 is defined as the master (for your installation, substitute the appropriate values for your network instead of 192.168.1).
Building an HA Cluster Configuration Preparing Your Systems HWADDR=00:12:79:43:5b:f4 3. Add the following lines to /etc/modprobe.conf: alias bond0 bonding options bond0 miimon=100 mode=1 Use MASTER=bond1 for bond1 if you have configured a second bonding interface, then add the following after the first bond (bond0): options bond1 -o bonding1 miimon=100 mode=1 NOTE During configuration, you need to make sure that the active slaves for the same bond on each node are connected the same hub or switch.
Building an HA Cluster Configuration Preparing Your Systems /sbin/ifconfig bond0 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4 inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:7224794 errors:0 dropped:0 overruns:0 frame:0 TX packets:3286647 errors:1 dropped:0 overruns:1 carrier:0 collisions:0 txqueuelen:0 eth0 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4 inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.
Building an HA Cluster Configuration Preparing Your Systems to: BOOTPROTO='none' STARTMODE='onboot' UNIQUE='gZD2.ZqnB7JKTdX0' _nm_name='bus-pci-0000:00:0b.0' NOTE Be sure to leave the UNIQUE and the _nm_name alone. You can also leave in the MTU and REMOTE_IPADDR as long as they are not set. Next in /etc/sysconfig/network edit your ifcfg-bond0 file so it looks like the following: BROADCAST='172.16.0.255' BOOTPROTO='static' IPADDR='172.16.0.1' MTU='' NETMASK='255.255.255.0' NETWORK='172.16.0.
Building an HA Cluster Configuration Preparing Your Systems Next is the BONDING_SLAVE. This tells ifup which ethernet devices to enslave to bond0. So if you wanted to bond four ethernet devices you would add: BONDING_SLAVE2='eth2' BONDING_SLAVE3='eth3' NOTE Use ifconfig to find the relationship of eth IDs and the MAC address. For more networking information on bonding, see /usr/src/linux/Documentation/networking/bond ing.txt. Restarting Networking Restart the networking subsystem.
Building an HA Cluster Configuration Preparing Your Systems fdisk Table 5-1 Changing Linux Partition Types Prompt Response Action Performed 1. Command (m for help): n Create new partition 2. Partition number (1-4): 1 Partition affected 3.
Building an HA Cluster Configuration Preparing Your Systems NOTE Follow these rules: • Do not try to use LVM to configure the lock LUN. • The partition type must be 83. • Do not create any filesystem on the partition used for the lock LUN. • Do not use md to configure multiple paths to the lock LUN. To transfer the disk partition format to other nodes in the cluster use the command: sfdisk -R where corresponds to the same physical device as on the first node.
Building an HA Cluster Configuration Preparing Your Systems Creating the Logical Volume Infrastructure Serviceguard makes use of shared disk storage. This is set up to provide high availability by using redundant data storage and redundant paths to the shared devices. Storage for a Serviceguard package is logically composed of LVM Volume Groups that are activated on a node as part of starting a package on that node. Storage is generally configured on logical units (LUNs).
Building an HA Cluster Configuration Preparing Your Systems CAUTION The minor numbers used by the LVM volume groups must be the same on all cluster nodes. This means that if there are any non-shared volume groups in the cluster, create the same number of them on all nodes, and create them before you define the shared storage. NOTE Except as noted in the sections that follow, you perform the LVM configuration of shared storage on only one node.
Building an HA Cluster Configuration Preparing Your Systems Disk /dev/sdd: 255 heads, 63 sectors, 1106 cylinders Units = cylinders of 16065 * 512 bytes In this example, the disk described by device file /dev/sda has already been partitioned for Linux, into partitions named /dev/sda1 /dev/sda7. The second internal device /dev/sdb and the two external devices /dev/sdc and /dev/sdd have not been partitioned. NOTE fdisk may not be available for SUSE on all platforms.
Building an HA Cluster Configuration Preparing Your Systems Prompt Response Write data to the partition table w Command (m for help): Action Performed The following example of the fdisk dialog shows that the disk on the device file /dev/sdc is configured as one partition, and appears as follows: fdisk /dev/sdc1 Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-4067, default 1): Enter Using default value 1 Last cylinder or +size o
Building an HA Cluster Configuration Preparing Your Systems Prompt 3.
Building an HA Cluster Configuration Preparing Your Systems NOTE fdisk may not be available for SUSE on all platforms. In this case, using YAST2 to set up the partitions is acceptable. Enabling VG Activation Protection Follow these steps to enable activation protection for volume groups on Red Hat and SUSE systems: NOTE Perform these steps on each node. Step 1. Edit /etc/lvm/lvm.conf and add the following line: tags { hosttags = 1 } Step 2. Create the file /etc/lvm/lvm_$(uname -n).conf Step 3.
Building an HA Cluster Configuration Preparing Your Systems vgs -o +tags vgpkgA vgchange -a y vgpkgA The VG Tags column in the output should contain the value of uname -n. Step 6. Deactivate the volume group and delete the tag: vgchange -a n vgpkgA vgchange --deltag $(uname -n) vgpkgA Building Volume Groups: Example for Smart Array Cluster Storage (MSA 500 Series) Use Logical Volume Manager (LVM) on your system to create volume groups that can be activated by Serviceguard packages.
Building an HA Cluster Configuration Preparing Your Systems 3. Check whether there are already volume groups defined on this node. Be sure to give each volume group a unique name. vgdisplay 4. Create separate volume groups for each Serviceguard package you will define.
Building an HA Cluster Configuration Preparing Your Systems Building Volume Groups and Logical Volumes Step 1. Use Logical Volume Manager (LVM) to create volume groups that can be activated by Serviceguard packages. For an example showing volume-group creation on LUNs, see “Building Volume Groups: Example for Smart Array Cluster Storage (MSA 500 Series)” on page 164. (For Fibre Channel storage you would use device-file names such as those used in the section “Creating Partitions” on page 160.) Step 2.
Building an HA Cluster Configuration Preparing Your Systems NOTE Be careful if you use YAST or YAST2 to configure volume groups, as that may cause all volume groups on that system to be activated. After running YAST or YAST2, check to make sure that volume groups for Serviceguard packages not currently running have not been activated, and use LVM commands to deactivate any that have. For example, use the command vgchange -a n /dev/sgvg00 to deactivate the volume group sgvg00.
Building an HA Cluster Configuration Preparing Your Systems The partition table on the rebooted node is then rebuilt using the information placed on the disks when they were partitioned on the other node. NOTE You must reboot at this time. 3. Run vgscan to make the LVM configuration visible on the new node and to create the LVM database on /etc/lvmtab and /etc/lvmtab.d.
Building an HA Cluster Configuration Preparing Your Systems Step 2. On ftsys10, activate the volume group, mount the file system, write a datestamp on to the shared file, and then look at the content of the file: vgchange --addtag $(uname -n) vgpkgB vgchange -a y vgpkgB mount /dev/vgpkgB/lvol1 /extra echo ‘Written by’ ‘hostname‘ ‘on’ ‘date‘ >> /extra/datestamp cat /extra/datestamp You should see something like the following, including the datestamp written by the other node: Written by ftsys9.
Building an HA Cluster Configuration Preparing Your Systems Storing Volume Group Configuration Data When you create volume groups, LVM creates a backup copy of the volume group configuration on the configuration node.
Building an HA Cluster Configuration Preparing Your Systems Red Hat It is not necessary to prevent vgscan on Red Hat. To deactivate any volume groups that will be under Serviceguard control, add vgchange commands to the end of /etc/rc.d/rc.
Building an HA Cluster Configuration Configuring the Cluster Configuring the Cluster This section describes how to define the basic cluster configuration. This must be done on a system that is not part of a Serviceguard cluster (that is, on which Serviceguard is installed but not configured). You can do this in Serviceguard Manager, or from the command line as described below.
Building an HA Cluster Configuration Configuring the Cluster cmquerycl Options Speeding up the Process In a larger or more complex cluster with many nodes, networks or disks, the cmquerycl command may take several minutes to complete. To speed up the configuration process, you can direct the command to return selected information only by using the -k and -w options: -k eliminates some disk probing, and does not return information about potential cluster lock volume groups and lock physical volumes.
Building an HA Cluster Configuration Configuring the Cluster If the name of the device is different on the different nodes, use the option to specific each device file following each node name, as in the following example: cmquerycl -v -n lp01 -L /dev/cciss/c0d0p1 \ -n lp02 -L /dev/cciss/c0d0p2 \ -C $SGCONF/lpcluster.config Specifying a Quorum Server A cluster lock LUN or quorum server, is required for two-node clusters.
Building an HA Cluster Configuration Configuring the Cluster Obtaining Cross-Subnet Information As of Serviceguard A.11.18 it is possible to configure multiple subnets, joined by a router, both for the cluster heartbeat and for data, with some nodes using one subnet and some another. See “Cross-Subnet Configurations” on page 30 for rules and definitions. You must use the -w full option to cmquerycl to discover the available subnets.
Building an HA Cluster Configuration Configuring the Cluster 5 6 lan3 (nodeD) lan4 (nodeD) lan1 (nodeC) lan1 (nodeD) lan2 (nodeC) lan2 (nodeD) IP subnets: IPv4: 15.13.164.0 15.13.172.0 15.13.165.0 15.13.182.0 15.244.65.0 15.244.56.
Building an HA Cluster Configuration Configuring the Cluster 3ffe:2222::/64 lan3 (nodeC) lan3 (nodeD) Possible Heartbeat IPs: 15.13.164.0 15.13.172.0 15.13.165.0 15.13.182.0 15.13.164.1 (nodeA) 15.13.164.2 (nodeB) 15.13.172.158 (nodeC) 15.13.172.159 (nodeD) 15.13.165.1 (nodeA) 15.13.165.2 (nodeB) 15.13.182.158 (nodeC) 15.13.182.159 (nodeD) Route connectivity(full probing performed): 1 15.13.164.0 15.13.172.0 2 15.13.165.0 15.13.182.0 3 15.244.65.0 4 15.244.56.
Building an HA Cluster Configuration Configuring the Cluster IMPORTANT Note that in this example subnet 15.244.65.0, used by NodeA and NodeB, is not routed to 15.244.56.0, used by NodeC and NodeD. But subnets 15.13.164.0 and 15.13.165.0, used by NodeA and NodeB, are routed respectively to subnets 15.13.172.0 and15.13.182.0, used by NodeC and NodeD. At least one such routing among all the nodes must exist for cmquerycl to succeed.
Building an HA Cluster Configuration Configuring the Cluster Modifying Cluster Timing Parameters The cmquerycl command supplies default cluster timing parameters for HEARTBEAT_INTERVAL and NODE_TIMEOUT. Changing these parameters will directly affect the cluster’s reformation and failover times. It is useful to modify these parameters if the cluster is re-forming occasionally because of heavy system load or heavy network traffic; you can do this while the cluster is running.
Building an HA Cluster Configuration Configuring the Cluster Controlling Access to the Cluster Serviceguard access-control policies define cluster users’ administrative or monitoring capabilities. A Note about Terminology Although you will also sometimes see the term role-based access (RBA) in the output of Serviceguard commands, the preferred set of terms, always used in this manual, is as follows: • Access-control policies - the set of rules defining user access to the cluster.
Building an HA Cluster Configuration Configuring the Cluster Figure 5-2 Chapter 5 Access Roles 181
Building an HA Cluster Configuration Configuring the Cluster Levels of Access Serviceguard recognizes two levels of access, root and non-root: • Root access: Full capabilities; only role allowed to configure the cluster. As Figure 5-2 shows, users with root access have complete control over the configuration of the cluster and its packages. This is the only role allowed to use the cmcheckconf, cmapplyconf, cmdeleteconf, and cmmodnet -a commands.
Building an HA Cluster Configuration Configuring the Cluster These users can run and halt a specified package, and change its switching behavior, but cannot configure or create packages. This is the only access role defined in the package configuration file; the others are defined in the cluster configuration file. Single-package Package Admin also includes the cluster-wide privileges of the Monitor role. — Monitor: Allowed to perform cluster and package view operations.
Building an HA Cluster Configuration Configuring the Cluster Setting up Access-Control Policies The root user on each cluster node is automatically granted the Serviceguard root access role on all nodes. (See “Configuring Root-Level Access” on page 144 for more information.) Access-control policies define non-root roles for other cluster users. NOTE For more information and advice, see the white paper Securing Serviceguard at http://docs.hp.com -> High Availability -> Serviceguard -> White Papers.
Building an HA Cluster Configuration Configuring the Cluster • USER_HOST is the node where USER_NAME will issue Serviceguard commands. The commands must be issued on USER_HOST but can take effect on other nodes; for example patrick can use bit’s command line to start a package on gryf (assuming bit and gryf are in the same cluster).
Building an HA Cluster Configuration Configuring the Cluster MONITOR and FULL_ADMIN can be set only in the cluster configuration file and they apply to the entire cluster. PACKAGE_ADMIN can be set in the cluster configuration file or a package configuration file. If it is set in the cluster configuration file, PACKAGE_ADMIN applies to all configured packages; if it is set in a package configuration file, it applies to that package only.
Building an HA Cluster Configuration Configuring the Cluster IMPORTANT Wildcards do not degrade higher-level roles that have been granted to individual members of the class specified by the wildcard.
Building an HA Cluster Configuration Configuring the Cluster Package versus Cluster Roles Package configuration will fail if there is any conflict in roles between the package configuration and the cluster configuration, so it is a good idea to have the cluster configuration file in front of you when you create roles for a package; use cmgetconf to get a listing of the cluster configuration file.
Building an HA Cluster Configuration Configuring the Cluster • The value for HEARTBEAT_INTERVAL is at least one second. • The value for NODE_TIMEOUT is at least twice the value of HEARTBEAT_INTERVAL. • The value for AUTO_START_TIMEOUT variables is greater than zero. • Heartbeat network minimum requirement. (See HEARTBEAT_IP under “Cluster Configuration Parameters” starting on page 110.) • At least one NODE_NAME is specified. • Each node is connected to each heartbeat network.
Building an HA Cluster Configuration Configuring the Cluster If you attempt to configure both a quorum server and a lock LUN, the following message appears on standard output when issuing the cmcheckconf or cmapplyconf command: Duplicate cluster lock, line 55. Quorum Server already specified. Distributing the Binary Configuration File After specifying all cluster parameters, use the cmapplyconf command to apply the configuration.
Building an HA Cluster Configuration Managing the Running Cluster Managing the Running Cluster This section describes some approaches to routine management of the cluster. Additional tools and suggestions are found in Chapter 7, “Cluster and Package Maintenance,” on page 241. You can manage the cluster from Serviceguard Manager, or by means of Serviceguard commands as described below. Checking Cluster Operation with Serviceguard Commands • cmviewcl checks status of the cluster and many of its components.
Building an HA Cluster Configuration Managing the Running Cluster 2. When the cluster has started, make sure that cluster components are operating correctly: cmviewcl -v Make sure that all nodes and networks are functioning as expected. For more information, refer to the chapter on “Cluster and Package Maintenance.” 3. Verify that nodes leave and enter the cluster as expected using the following steps: • Halt the cluster. You can use Serviceguard Manager or the cmhaltnode command.
Building an HA Cluster Configuration Managing the Running Cluster Setting up Autostart Features Automatic startup is the process in which each node individually joins a cluster; Serviceguard provides a startup script to control the startup process. If a cluster already exists, the node attempts to join it; if no cluster is running, the node attempts to form a cluster consisting of all configured nodes. Automatic cluster start is the preferred way to start a cluster.
Building an HA Cluster Configuration Managing the Running Cluster Here is an example of the $SGAUTOSTART file: SGAUTOSTART=/usr/local/cmcluster/conf/cmcluster.rc #*************************** CMCLUSTER ************************* # # # # # # # # # # # # # # # # Highly Available Cluster configuration @(#) $Revision: 82.2 $ AUTOSTART_CMCLD Automatic startup is the process in which each node individually joins a cluster.
Building an HA Cluster Configuration Managing the Running Cluster Changing the System Message You may find it useful to modify the system's login message to include a statement such as the following: This system is a node in a high availability cluster. Halting this system may cause applications and services to start up on another node in the cluster. You might wish to include a list of all cluster nodes in this message, together with additional cluster-specific information.
Building an HA Cluster Configuration Managing the Running Cluster currently available for package switching. However, you should not try to restart HP Serviceguard, since data corruption might occur if another node were to attempt to start up a new instance of the application that is still running on the single node.
Building an HA Cluster Configuration Managing the Running Cluster For Red Hat this would be changed from: server_args = -f /user/local/cmom/log/cmomd.log -r /user/local/cmom/run to server_args = -i -f /user/local/cmom/log/cmomd.log -r /user/local/cmom/run 3. Restart xinetd: /etc/init.d/xinetd restart Deleting the Cluster Configuration You can delete a cluster configuration by issuing the cmdeleteconf command. The command prompts for a verification before deleting the files unless you use the -f option.
Building an HA Cluster Configuration Managing the Running Cluster 198 Chapter 5
Configuring Packages and Their Services 6 Configuring Packages and Their Services Serviceguard packages group together applications and the services and resources they depend on. The typical Serviceguard package is a failover package that starts on one node but can be moved (“failed over”) to another if necessary. See “What is Serviceguard for Linux?” on page 18, “How the Package Manager Works” on page 50, and“Package Configuration Planning” on page 119 for more information.
Configuring Packages and Their Services NOTE This is a new process for configuring packages, as of Serviceguard A.11.18. This manual refers to packages created by this method as modular packages, and assumes that you will use it to create new packages. Packages created using Serviceguard A.11.16 or earlier are referred to as legacy packages. If you need to reconfigure a legacy package (rather than create a new package), see “Configuring a Legacy Package” on page 275.
Configuring Packages and Their Services Choosing Package Modules Choosing Package Modules IMPORTANT Before you start, you need to do the package-planning tasks described under “Package Configuration Planning” on page 119. To choose the right package modules, you need to decide the following things about the package you are creating: • What type of package it is; see “Types of Package: Failover, Multi-Node, System Multi-Node” on page 201.
Configuring Packages and Their Services Choosing Package Modules Relocatable IP addresses cannot be assigned to multi-node packages. Multi-node packages must either use a clustered file system such as Red Hat GFS, or not use shared storage. IMPORTANT To generate a package configuration file that creates a multi-node package, include -m sg/multi_node on the cmmakepkg command line. See “Generating the Package Configuration File” on page 229. • NOTE System multi-node packages.
Configuring Packages and Their Services Choosing Package Modules Package Modules and Parameters The table that follows shows the package modules and the configuration parameters each module includes. Read this section in conjunction with the discussion under “Package Configuration Planning” on page 119.
Configuring Packages and Their Services Choosing Package Modules Base Package Modules At least one base module (or default or all, which include the base module) must be specified on the cmmakepkg command line. Parameters marked with an asterisk (*) are new or changed as of Serviceguard A.11.18. (S) indicates that the parameter (or its equivalent) has moved from the package control script to the package configuration file for modular packages.
Configuring Packages and Their Services Choosing Package Modules Table 6-1 Base Modules (Continued) Module Name Parameters (page) Comments multi_node package_name (209) * module_name (210) * module_version (210) * package_type (210) node_name (210) auto_run (211) node_fail_fast_enabled (211) run_script_timeout (212) halt_script_timeout (212) successor_halt_timeout (213) * script_log_file (213) operation_sequence (213) * log_level (214) * priority (215) * Base module.
Configuring Packages and Their Services Choosing Package Modules Optional Package Modules Add optional modules to a base module if you need to configure the functions in question. Parameters marked with an asterisk (*) are new or changed as of Serviceguard A.11.18. (S) indicates that the parameter (or its equivalent) has moved from the package control script to the package configuration file for modular packages. See the “Package Parameter Explanations” on page 209 for more information.
Configuring Packages and Their Services Choosing Package Modules Table 6-2 Module Name Optional Modules (Continued) Parameters (page) Comments filesystem concurrent_fsck_operations (222) (S) concurrent_mount_and_umount_ operations (223) (S) fs_mount_retry_count (223) (S) fs_umount_retry_count (223) * (S) fs_name (224) * (S) fs_directory (224) * (S) fs_type (225) (S) fs_mount_opt (225) (S) fs_umount_opt (225) (S) fs_fsck_opt (225) (S) Add to a base module to configure filesystem options for the package.
Configuring Packages and Their Services Choosing Package Modules Table 6-2 Module Name Optional Modules (Continued) Parameters (page) Comments all all parameters Use if you are creating a complex package that requires most or all of the optional parameters; or if you want to see the specifications and comments for all available parameters.
Configuring Packages and Their Services Choosing Package Modules Package Parameter Explanations Brief descriptions of the package configuration parameters follow. NOTE For more information, see the comments in the editable configuration file output by the cmmakepkg command, and the cmmakepkg (1m) manpage.
Configuring Packages and Their Services Choosing Package Modules module_name The module name. Do not change it. Used in the form of a relative path (for example sg/failover) as a parameter to cmmakepkg to specify modules to be used in configuring the package. (The files reside in the $SGCONF/modules directory; see “Understanding the Location of Serviceguard Files” on page 142 for the location of $SGCONF on your version of Linux.) New for modular packages. module_version The module version. Do not change it.
Configuring Packages and Their Services Choosing Package Modules IMPORTANT See “Cluster Configuration Parameters” on page 110 for important information about node names. See “About Cross-Subnet Failover” on page 136 for considerations when configuring cross-subnet packages, which are further explained under “Cross-Subnet Configurations” on page 30. auto_run Can be set to yes or no. The default is yes.
Configuring Packages and Their Services Choosing Package Modules NOTE • A package subnet fails and no backup network is available • Serviceguard is unable to execute the halt function • The start or halt function times out If the package halt function fails with “exit 1”, Serviceguard does not halt the node, but sets no_restart for the package, which disables package switching (auto_run), thereby preventing the package from starting on any adoptive node.
Configuring Packages and Their Services Choosing Package Modules If a halt_script_timeout is specified, it should be greater than the sum of all the values set for service_halt_timeout (see page 221) for this package. If a timeout occurs: • • Switching will be disabled. The current node will be disabled from running the package. If a halt-script timeout occurs, you may need to perform manual cleanup. See Chapter 8, “Troubleshooting Your Cluster,” on page 299.
Configuring Packages and Their Services Choosing Package Modules log_level Determines the amount of information printed to stdout when the package is validated, and to the script_log_file (see page 213) when the package is started and halted. Valid values are 0 through 5, but you should normally use only the first two (0 or 1); the remainder (2 through 5) are intended for use by HP Support.
Configuring Packages and Their Services Choosing Package Modules • manual means the package will continue to run on the current node. • automatic means Serviceguard will move the package to the primary node as soon as that node becomes available, unless doing so would also force a package with a higher priority (see page 215) to move. This parameter can be set for failover packages only. If this package will depend on another package or vice versa, see also “About Package Dependencies” on page 123.
Configuring Packages and Their Services Choosing Package Modules IMPORTANT Restrictions on dependency names in previous Serviceguard releases were less stringent. Packages that specify dependency_names that do not conform to the above rules will continue to run, but if you reconfigure them, you will need to change the dependency_name; cmcheckconf and cmapplyconf will enforce the new rules.
Configuring Packages and Their Services Choosing Package Modules dependency_location Specifies where the dependency_condition must be met. The only legal value is same_node. monitored_subnet The LAN subnet that is to be monitored for this package. Replaces legacy SUBNET which is still supported in the package configuration file for legacy packages; see “Configuring a Legacy Package” on page 275. You can specify multiple subnets; use a separate line for each.
Configuring Packages and Their Services Choosing Package Modules New for modular packages. For legacy packages, see “Configuring Cross-Subnet Failover” on page 286. ip_subnet Specifies an IP subnet used by the package. Replaces SUBNET, which is still supported in the package control script for legacy packages. For each subnet used, specify the subnet address on one line and, on the following lines, the relocatable IP addresses that the package uses on that subnet.
Configuring Packages and Their Services Choosing Package Modules New for modular packages. For legacy packages, see “Configuring Cross-Subnet Failover” on page 286. ip_address A relocatable IP address on a specified ip_subnet (see page 218). Replaces IP, which is still supported in the package control script for legacy packages. For more information about relocatable IP addresses, see “Stationary and Relocatable IP Addresses and Monitored Subnets” on page 74.
Configuring Packages and Their Services Choosing Package Modules For legacy packages, this parameter is in the package control script as well as the package configuration file. service_cmd The command that runs the program or function for this service_name, for example, /usr/bin/X11/xclock -display 15.244.58.208:0 An absolute pathname is required; neither the PATH variable nor any other environment variable is passed to the command. The default shell is /bin/sh.
Configuring Packages and Their Services Choosing Package Modules service_restart The number of times Serviceguard will attempt to re-run the service_cmd. Valid values are unlimited, none or any positive integer value. Default is none. If the value is unlimited, the service will be restarted an infinite number of times. If the value is none, the service will not be restarted. This parameter is in the package control script for legacy packages.
Configuring Packages and Their Services Choosing Package Modules File system parameters A package can activate one or more storage groups on startup, and to mount logical volumes to file systems. At halt time, the package script unmounts the file systems and deactivates each storage group. All storage groups must be accessible on each target node.
Configuring Packages and Their Services Choosing Package Modules concurrent_mount_and_umount_operations The number of concurrent mounts and umounts to allow during package startup or shutdown. Legal value is any number greater than zero. The default is 1. If the package needs to mount and unmount a large number of file systems, you can improve performance by carefully tuning this parameter during testing (increase it a little at time and monitor performance each time).
Configuring Packages and Their Services Choosing Package Modules fs_name This parameter, in conjunction with fs_directory, fs_type, fs_mount_opt, fs_umount_opt, and fs_fsck_opt, specifies a filesystem that is to be mounted by the package. Replaces LV, which is still supported in the package control script for legacy packages. fs_name must specify the block devicefile for a logical volume. File systems are mounted in the order you specify in the package configuration file, and unmounted in the reverse order.
Configuring Packages and Their Services Choosing Package Modules fs_type The type of the file system specified by fs_name. This parameter is in the package control script for legacy packages. Supported types are ext2, ext3, reiserfs, and gfs. NOTE A package using gfs (Red Hat GFS) cannot use any other file systems of a different type. vg and vgchange_cmd (see page 221) are not valid for GFS file systems.
Configuring Packages and Their Services Choosing Package Modules The variable name must be in the form pev_ and contain only alphanumeric characters and underscores. The letters pev (upper-case or lower-case) followed by the underscore (_) are required. The variable name and value can each consist of a maximum of MAXPATHLEN characters (4096 on Linux systems). You can define more than one variable.
Configuring Packages and Their Services Choosing Package Modules user_host The system from which a user specified by user_name can execute package-administration commands. Legal values are any_serviceguard_node, or cluster_member_node, or a specific cluster node. If you specify a specific node it must be the official hostname (the hostname portion, and only the hostname portion, of the fully qualified domain name). As with user_name, be careful to spell the keywords exactly as given.
Configuring Packages and Their Services Choosing Package Modules Additional Parameters Used Only by Legacy Packages IMPORTANT The following parameters are used only by legacy packages. Do not try to use them in modular packages. See“Creating the Legacy Package Configuration” on page 275 for more information. PATH Specifies the path to be used by the script. SUBNET Specifies the IP subnets that are to be monitored for the package. RUN_SCRIPT and HALT_SCRIPT Use the full pathname of each script.
Configuring Packages and Their Services Generating the Package Configuration File Generating the Package Configuration File When you have chosen the configuration modules your package needs (see “Choosing Package Modules” on page 201), you are ready to generate a package configuration file that contains those modules. This file will consist of a base module (failover, multi-node or system multi-node) plus the modules that contain the additional parameters you have decided to include.
Configuring Packages and Their Services Generating the Package Configuration File • To generate a configuration file that contains all the optional modules: cmmakepkg $SGCONF/pkg1/pkg1.conf • To create a generic failover package (that could be applied without editing): cmmakepkg -n pkg1 -m sg/failover $SGCONF/pkg1/pkg1.
Configuring Packages and Their Services Editing the Configuration File Editing the Configuration File When you have generated the configuration file that contains the modules your package needs (see “Generating the Package Configuration File” on page 229), you need to edit the file to set the package parameters to the values that will make the package function as you intend. It is a good idea to configure complex failover packages in stages, as follows: 1. Configure volume groups and mount points only. 2.
Configuring Packages and Their Services Editing the Configuration File Use the following bullet points as a checklist, referring to the “Package Parameter Explanations” on page 209, and the comments in the configuration file itself, for detailed specifications for each parameter. NOTE Optional parameters are commented out in the configuration file (with a # at the beginning of the line).
Configuring Packages and Their Services Editing the Configuration File • successor_halt_timeout. Used if other packages depend on this package; see “About Package Dependencies” on page 123. • script_log_file. See script_log_file on page 213. • log_level. See log_level on page 214. • failover_policy. Enter configured_node or min_package_node. See page 214 for more information. (This parameter can be set for failover packages only.) • failback_policy. Enter automatic or manual.
Configuring Packages and Their Services Editing the Configuration File • For each service the package will run: — enter the service_name (for example, a daemon or long-running process) — enter the service_cmd (for example, the command that starts the process) — enter values for service_fail_fast_enabled and service_halt_timeout if you need to change them from their defaults. — service_restart if you want the package to restart the service if it exits.
Configuring Packages and Their Services Editing the Configuration File — concurrent_fsck_operations—specifies the number of parallel fsck operations that will be allowed at package startup (not used for Red Hat GFS). — concurrent_mount_and_umount_operations—specifies the number of parallel mount operations allowed during package startup and unmount operations during package shutdown. • Specify the filesystem mount and unmount retry options. For Red Hat GFS (see fs_type on page 225), use the default (zero).
Configuring Packages and Their Services Verifying and Applying the Package Configuration Verifying and Applying the Package Configuration Serviceguard checks the configuration you enter and reports any errors. Use a command such as the following to verify the content of the package configuration file you have created, for example: cmcheckconf -v -P $SGCONF/pkg1/pkg1.config Errors are displayed on the standard output.
Configuring Packages and Their Services Verifying and Applying the Package Configuration NOTE For modular packages, you now need to distribute any external scripts identified by the external_pre_script and external_script parameters. But, if you are accustomed to configuring legacy packages, note that you do not have to create a separate package control script for a modular package, or distribute it manually.
Configuring Packages and Their Services Adding the Package to the Cluster Adding the Package to the Cluster You can add the new package to the cluster while the cluster is running, subject to the value of max_configured_packages in the cluster configuration file. See “Adding a Package to a Running Cluster” on page 291.
Configuring Packages and Their Services Creating a Disk Monitor Configuration Creating a Disk Monitor Configuration Serviceguard provides disk monitoring for the shared storage that is activated by packages in the cluster. The monitor daemon on each node tracks the status of all the disks on that node that you have configured for monitoring.
Configuring Packages and Their Services Creating a Disk Monitor Configuration 240 Chapter 6
Cluster and Package Maintenance 7 Cluster and Package Maintenance This chapter describes the cmviewcl command, then shows how to start and halt a cluster or an individual node, how to perform permanent reconfiguration, and how to start, halt, move, and modify packages during routine maintenance of the cluster.
Cluster and Package Maintenance Reviewing Cluster and Package Status Reviewing Cluster and Package Status You can check status using Serviceguard Manager, or from a cluster node’s command line. Reviewing Cluster and Package Status with the cmviewcl Command Information about cluster status is stored in the status database, which is maintained on each individual node in the cluster.
Cluster and Package Maintenance Reviewing Cluster and Package Status Cluster Status The status of a cluster, as shown by cmviewcl, can be one of the following: • up - At least one node has a running cluster daemon, and reconfiguration is not taking place. • down - No cluster daemons are running on any cluster node. • starting - The cluster is in the process of determining its active membership. At least one cluster daemon is running.
Cluster and Package Maintenance Reviewing Cluster and Package Status Package Status and State The status of a package can be one of the following: • up - The package master control script is active. • down - The package master control script is not active. • start_wait - A cmrunpkg command is in progress for this package. The package is waiting for packages it depends on (predecessors) to start before it can start. • starting - The package is starting. The package master control script is running.
Cluster and Package Maintenance Reviewing Cluster and Package Status The state of a package can be one of the following: • starting - The package is starting. The package master control script is running. • start_wait - A cmrunpkg command is in progress for this package. The package is waiting for packages it depends on (predecessors) to start before it can start. • running - Services are active and being monitored.
Cluster and Package Maintenance Reviewing Cluster and Package Status Package Switching Attributes cmviewcl shows the following package switching information: • AUTO_RUN: Can be enabled or disabled. For failover packages, enabled means that the package starts when the cluster starts, and Serviceguard can switch the package to another node in the event of failure.
Cluster and Package Maintenance Reviewing Cluster and Package Status Failover and Failback Policies Failover packages can be configured with one of two values for the failover_policy parameter (see page 214), as displayed in the output of cmviewcl -v: • configured_node. The package fails over to the next node in the node_name list in the package configuration file (see page 210). • min_package_node. The package fails over to the node in the cluster with the fewest running packages on it.
Cluster and Package Maintenance Reviewing Cluster and Package Status Script_Parameters: ITEM STATUS Service up Subnet up MAX_RESTARTS 0 0 Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING Primary up enabled Alternate up enabled NODE ftsys10 STATUS up RESTARTS 0 0 NAME ftsys9 ftsys10 NAME service1 15.13.168.
Cluster and Package Maintenance Reviewing Cluster and Package Status CLUSTER example NODE ftsys9 STATUS up STATUS up STATE running Quorum Server Status: NAME STATUS lp-qs up ...
Cluster and Package Maintenance Reviewing Cluster and Package Status Status After Halting a Package After we halt pkg2 with the cmhaltpkg command, the output of cmviewcl-v is as follows: CLUSTER example NODE ftsys9 STATUS up STATUS up STATE running Network_Parameters: INTERFACE STATUS PRIMARY up PRIMARY up NAME eth0 eth1 PACKAGE pkg1 STATE running STATUS up AUTO_RUN enabled NODE ftsys9 Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM
Cluster and Package Maintenance Reviewing Cluster and Package Status Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING Primary up enabled Alternate up enabled NAME ftsys10 ftsys9 pkg2 now has the status down, and it is shown as unowned, with package switching disabled. Note that switching is enabled for both nodes, however. This means that once global switching is re-enabled for the package, it will attempt to start up on the primary node.
Cluster and Package Maintenance Reviewing Cluster and Package Status Script_Parameters: ITEM STATUS Service up Subnet up NAME Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING Primary up enabled Alternate up enabled NODE ftsys10 STATUS up MAX_RESTARTS RESTARTS 0 service2 0 15.13.168.
Cluster and Package Maintenance Reviewing Cluster and Package Status Status After Halting a Node After halting ftsys10, with the following command: cmhaltnode ftsys10 the output of cmviewcl is as follows on ftsys9: CLUSTER example NODE ftsys9 PACKAGE pkg1 pkg2 NODE ftsys10 STATUS up STATUS up STATUS up up STATUS down STATE running STATE running running AUTO_RUN enabled enabled NODE ftsys9 ftsys9 STATE halted This output can be seen on both ftsys9 and ftsys10.
Cluster and Package Maintenance Reviewing Cluster and Package Status Viewing Information about Unowned Packages The following example shows packages that are currently unowned, that is, not running on any configured node.
Cluster and Package Maintenance Managing the Cluster and Nodes Managing the Cluster and Nodes This section describes the following tasks: • “Starting the Cluster When all Nodes are Down” on page 256 • “Adding Previously Configured Nodes to a Running Cluster” on page 257 • “Removing Nodes from Participation in a Running Cluster” on page 258 • “Halting the Entire Cluster” on page 259 • “Automatically Restarting the Cluster” on page 259 In Serviceguard A.11.
Cluster and Package Maintenance Managing the Cluster and Nodes Starting the Cluster When all Nodes are Down You can use Serviceguard Manager, or the cmruncl command as described in this section, to start the cluster when all cluster nodes are down. Particular command options can be used to start the cluster under specific circumstances. The -v option produces the most informative output.
Cluster and Package Maintenance Managing the Cluster and Nodes Adding Previously Configured Nodes to a Running Cluster You can use Serviceguard Manager, or HP Serviceguard commands as shown, to bring a configured node up within a running cluster. Use the cmrunnode command to add one or more nodes to an already running cluster. Any node you add must already be a part of the cluster configuration. The following example adds node ftsys8 to the cluster that was just started with only nodes ftsys9 and ftsys10.
Cluster and Package Maintenance Managing the Cluster and Nodes Removing Nodes from Participation in a Running Cluster You can use Serviceguard Manager, or Serviceguard commands as shown below, to remove nodes from operation in a cluster. This operation removes the node from cluster operation by halting the cluster daemon, but it does not modify the cluster configuration. To remove a node from the cluster configuration permanently, you must recreate the cluster configuration file. See the next section.
Cluster and Package Maintenance Managing the Cluster and Nodes Halting the Entire Cluster You can use Serviceguard Manager, or Serviceguard commands as shown below, to halt a running cluster. The cmhaltcl command can be used to halt the entire cluster. This command causes all nodes in a configured cluster to halt their HP Serviceguard daemons. You can use the -f option to force the cluster to halt even when packages are running. This command can be issued from any running node.
Cluster and Package Maintenance Managing Packages and Services Managing Packages and Services This section describes the following tasks: • “Starting a Package” on page 260 • “Halting a Package” on page 261 • “Moving a Failover Package” on page 262 • “Changing Package Switching Behavior” on page 262 Non-root users with the appropriate privileges can perform these tasks. See “Controlling Access to the Cluster” on page 180 for information about configuring access.
Cluster and Package Maintenance Managing Packages and Services Starting a Package that Has Dependencies Before starting a package, it is a good idea to use the cmviewcl command to check for package dependencies. You cannot start a package unless all the packages that it depends on are running. If you try, you’ll see a Serviceguard message telling you why the operation failed, and the package will not start.
Cluster and Package Maintenance Managing Packages and Services If this happens, you can repeat the halt command, this time including the dependent package(s); Serviceguard will halt the all the packages in the correct order. First, use cmviewcl to be sure that no other running package has a dependency on any of the packages you are halting. Moving a Failover Package You can use Serviceguard Manager to move a failover package from one node to another, or Serviceguard commands as shown below.
Cluster and Package Maintenance Managing Packages and Services You can use Serviceguard Manager to change package switching behavior, or Serviceguard commands as shown below. You can change package switching behavior either temporarily or permanently using Serviceguard commands. To temporarily disable switching to other nodes for a running package, use the cmmodpkg command.
Cluster and Package Maintenance Reconfiguring a Cluster Reconfiguring a Cluster You can reconfigure a cluster either when it is halted or while it is still running. Some operations can only be done when the cluster is halted. Table 7-1 shows the required cluster state for many kinds of changes. Table 7-1 Types of Changes to the Cluster Configuration Change to the Cluster Configuration 264 Required Cluster State Add a new node All cluster nodes must be running.
Cluster and Package Maintenance Reconfiguring a Cluster Table 7-1 Types of Changes to the Cluster Configuration (Continued) Change to the Cluster Configuration Required Cluster State Reconfigure IP addresses for a NIC used by the cluster Must delete the interface from the cluster configuration, reconfigure it, then add it back into the cluster configuration. See “What You Must Keep in Mind” on page 269. Cluster can be running throughout Change NETWORK_POLLING_INTERVAL Cluster can be running.
Cluster and Package Maintenance Reconfiguring a Cluster Updating the Cluster Lock LUN Configuration Offline The cluster must be halted before you change the lock LUN configuration. Proceed as follows: Step 1. Halt the cluster. Step 2. In the cluster configuration file, modify the values of CLUSTER_LOCK_LUN for each node. Step 3. Run cmcheckconf to check the configuration. Step 4. Run cmapplyconf to apply the configuration. If you need to replace the physical device, see “Replacing a Lock LUN” on page 303.
Cluster and Package Maintenance Reconfiguring a Cluster NOTE Before you start, make sure you have configured access to ftsys10 as described under “Configuring Root-Level Access” on page 144. 1. Use the following command to store a current copy of the existing cluster configuration in a temporary file in case you need to revert to it: cmgetconf -C temp.ascii 2. Specify a new set of nodes to be configured and generate a template of the new configuration: cmquerycl -C clconfig.
Cluster and Package Maintenance Reconfiguring a Cluster Use the following procedure to delete a node with Serviceguard commands. In this example, nodes ftsys8, ftsys9 and ftsys10 are already configured in a running cluster named cluster1, and you are deleting node ftsys10. NOTE If you want to remove a node from the cluster, run the cmapplyconf command from another node in the same cluster. If you try to issue the command on the node you want removed, you will get an error message. Step 1.
Cluster and Package Maintenance Reconfiguring a Cluster NOTE If you are trying to remove an unreachable node on which many packages are configured to run, you may see the following message: The configuration change is too large to process while the cluster is running. Split the configuration change into multiple requests or halt the cluster. In this situation, you must halt the cluster to remove the node.
Cluster and Package Maintenance Reconfiguring a Cluster • You cannot change the designation of an existing interface from HEARTBEAT_IP to STATIONARY_IP, or vice versa, without also making the same change to all peer network interfaces on the same subnet on all other nodes in the cluster. • You cannot change the designation of an interface from STATIONARY_IP to HEARTBEAT_IP unless the subnet is common to all nodes.
Cluster and Package Maintenance Reconfiguring a Cluster Example: Adding a Heartbeat LAN Suppose that a subnet 15.13.170.0 is shared by nodes ftsys9 and ftsys10 in a two-node cluster cluster1, and you want to add it to the cluster configuration as a heartbeat subnet. Proceed as follows. Step 1. Run cmquerycl to get a cluster configuration template file that includes networking information for interfaces that are available to be added to the cluster configuration: cmquerycl -c cluster1 -C clconfig.
Cluster and Package Maintenance Reconfiguring a Cluster NODE_NAME NETWORK_INTERFACE HEARTBEAT_IP NETWORK_INTERFACE HEARTBEAT_IP NETWORK_INTERFACE NODE_NAME NETWORK_INTERFACE HEARTBEAT_IP NETWORK_INTERFACE HEARTBEAT_IP NETWORK_INTERFACE ftsys9 lan1 192.3.17.18 lan0 15.13.170.18 lan3 ftsys10 lan1 192.3.17.19 lan0 15.13.170.19 lan3 Step 3. Verify the new configuration: cmcheckconf -C clconfig.ascii Step 4.
Cluster and Package Maintenance Reconfiguring a Cluster Example: Deleting a Subnet Used by a Package In this example, we are deleting subnet 15.13.170.0 (lan0). Proceed as follows. Step 1. Halt any package that uses this subnet and delete the corresponding networking information (monitored_subnet, ip_subnet, ip_address; see page 217). See “Reconfiguring a Package on a Running Cluster” on page 290 for more information. Step 2.
Cluster and Package Maintenance Reconfiguring a Cluster Changing MAX_CONFIGURED_PACKAGES As of Serviceguard A.11.18, you can change MAX_CONFIGURED_PACKAGES while the cluster is running. The default for MAX_CONFIGURED_PACKAGES is the maximum number allowed in the cluster. You can use Serviceguard Manager to change MAX_CONFIGURED_PACKAGES, or Serviceguard commands as shown below.
Cluster and Package Maintenance Configuring a Legacy Package Configuring a Legacy Package IMPORTANT You can still create a new legacy package. If you are using a Serviceguard Toolkit such as Serviceguard NFS Toolkit, consult the documentation for that product. Otherwise, use this section to maintain and re-work existing legacy packages rather than to create new ones.
Cluster and Package Maintenance Configuring a Legacy Package Using Serviceguard Manager to Configure a Package You can create a legacy package and its control script in Serviceguard Manager; use the Help for detailed instructions. Using Serviceguard Commands to Configure a Package Use the following procedure to create a legacy package. Step 1. Create a subdirectory for each package you are configuring in the $SGCONF directory: mkdir $SGCONF/pkg1 You can use any directory names you like.
Cluster and Package Maintenance Configuring a Legacy Package Configuring a Package in Stages It is a good idea to configure failover packages in stages, as follows: 1. Configure volume groups and mount points only. 2. Distribute the control script to all nodes. 3. Apply the configuration. 4. Run the package and ensure that it can be moved from node to node. 5. Halt the package. 6. Configure package IP addresses and application services in the control script. 7. Distribute the control script to all nodes. 8.
Cluster and Package Maintenance Configuring a Legacy Package • FAILBACK_POLICY. For failover packages, enter the failback_policy (see page 214). • NODE_NAME. Enter the node or nodes on which the package can run; as described under node_name (see page 210). • AUTO_RUN. Configure the package to start up automatically or manually; as described under auto_run (see page 211). • NODE_FAIL_FAST_ENABLED. Enter the policy as described under node_fail_fast_enabled (see page 211).
Cluster and Package Maintenance Configuring a Legacy Package Note that the rules for valid SERVICE_NAMEs are more restrictive as of Serviceguard A.11.18. IMPORTANT • ACCESS_CONTROL_POLICY. You can grant a non-root user PACKAGE_ADMIN privileges for this package. See the entries for user_name, user_host, and user_role on page 227, and “Controlling Access to the Cluster” on page 180, for more information.
Cluster and Package Maintenance Configuring a Legacy Package cmmakepkg -s $SGCONF/pkg1/pkg1.sh Next, customize the script; see “Customizing the Package Control Script”. Customizing the Package Control Script You need to customize as follows; see the relevant entries under “Package Parameter Explanations” on page 209 for more discussion. • Update the PATH statement to reflect any required paths needed to start your services.
Cluster and Package Maintenance Configuring a Legacy Package For more information about services, see the discussion of the service_ parameters that starts on page 219. Adding Customer Defined Functions to the Package Control Script You can add additional shell commands to the package control script to be executed whenever the package starts or stops. Enter these commands in the CUSTOMER DEFINED FUNCTIONS area of the script.
Cluster and Package Maintenance Configuring a Legacy Package # START OF CUSTOMER DEFINED FUNCTIONS # This function is a place holder for customer defined functions. # You should define all actions you want to happen here, before the service is # started. You can create as many functions as you need. function customer_defined_run_cmds { # ADD customer defined run commands. : # do nothing instruction, because a function must contain some command. date >> /tmp/pkg1.datelog echo 'Starting pkg1' >> /tmp/pkg1.
Cluster and Package Maintenance Configuring a Legacy Package Adding Serviceguard Commands in Customer Defined Functions You can add Serviceguard commands (such as cmmodpkg) in the Customer Defined Functions section of a package control script. These commands must not interact with the package itself. If a Serviceguard command interacts with another package, be careful to avoid command loops. For instance, a command loop might occur under the following circumstances.
Cluster and Package Maintenance Configuring a Legacy Package Verifying the Package Configuration Serviceguard checks the configuration you create and reports any errors. For legacy packages, you can do this in Serviceguard Manager: click Check to verify the package configuration you have done under any package configuration tab, or to check changes you have made to the control script. Click Apply to verify the package as a whole. See the local Help for more details.
Cluster and Package Maintenance Configuring a Legacy Package Distributing the Configuration And Control Script with Serviceguard Manager When you have finished creating a package in Serviceguard Manager, click Apply Configuration. If the package configuration has no errors, it is converted to a binary file and distributed to the cluster nodes.
Cluster and Package Maintenance Configuring a Legacy Package NOTE You must use cmcheckconf and cmapplyconf again any time you make changes to the cluster and package configuration files. Configuring Cross-Subnet Failover To configure a legacy package to fail over across subnets (see “Cross-Subnet Configurations” on page 30), you need to do some additional configuration. NOTE You cannot use Serviceguard Manager to configure cross-subnet packages.
Cluster and Package Maintenance Configuring a Legacy Package Configuring monitored_subnet_access In order to monitor subnet 15.244.65.0 or 15.244.56.0, you would configure monitored_subnet and monitored_subnet_access in pkg1’s package configuration file as follows: monitored_subnet 15.244.65.0 monitored_subnet_access PARTIAL monitored_subnet 15.244.56.
Cluster and Package Maintenance Configuring a Legacy Package Control-script entries for nodeC and nodeD IP[0] = 15.244.56.100 SUBNET[0] = 15.244.56.0 IP[1] = 15.244.56.101 SUBNET[1] =15.244.56.
Cluster and Package Maintenance Reconfiguring a Package Reconfiguring a Package You reconfigure a package in much the same way as you originally configured it; for modular packages, see Chapter 6, “Configuring Packages and Their Services,” on page 199; for older packages, see “Configuring a Legacy Package” on page 275.
Cluster and Package Maintenance Reconfiguring a Package Reconfiguring a Package on a Running Cluster You can reconfigure a package while the cluster is running, and in some cases you can reconfigure the package while the package itself is running. You can do this in Serviceguard Manager (for legacy packages), or use Serviceguard commands. To modify the package with Serviceguard commands, use the following procedure (pkg1 is used as an example): 1.
Cluster and Package Maintenance Reconfiguring a Package Reconfiguring a Package on a Halted Cluster You can also make permanent changes in the package configuration while the cluster is not running. Use the same steps as in “Reconfiguring a Package on a Running Cluster” on page 290. Adding a Package to a Running Cluster You can create a new package and add it to the cluster configuration while the cluster is up and while other packages are running.
Cluster and Package Maintenance Reconfiguring a Package cmhaltpkg mypkg cmdeleteconf -p mypkg The command prompts for a verification before deleting the files unless you use the -f option. The directory $SGCONF/mypkg is not deleted by this command. Resetting the Service Restart Counter The service restart counter tracks the number of times a package service has been automatically restarted.
Cluster and Package Maintenance Reconfiguring a Package NOTE All the nodes in the cluster must be powered up and accessible when you make package configuration changes. Table 7-2 Types of Changes to Packages Change to the Package Chapter 7 Required Package State Add a new package Other packages can be in any state. Delete a package Package must not be running. You cannot delete a package if another package has a dependency on it. Change package type Package must not be running.
Cluster and Package Maintenance Reconfiguring a Package Table 7-2 Types of Changes to Packages (Continued) Change to the Package 294 Required Package State Add an IP address Package must not be running. Remove an IP address Package must not be running. Add or delete nodes from package’s ip_subnet_node list (in cross-subnet configurations; see page 218) Package can be running, with these restrictions: Add a volume group Package should not be running.
Cluster and Package Maintenance Reconfiguring a Package Table 7-2 Types of Changes to Packages (Continued) Change to the Package Change location of script log file Chapter 7 Required Package State Package must not be running.
Cluster and Package Maintenance Responding to Cluster Events Responding to Cluster Events Serviceguard does not require much ongoing system administration intervention. As long as there are no failures, your cluster will be monitored and protected. In the event of a failure, those packages that you have designated to be transferred to another node will be transferred automatically.
Cluster and Package Maintenance Single-Node Operation Single-Node Operation In a multi-node cluster, you could have a situation in which all but one node has failed, or you have shut down all but one node, leaving your cluster in single-node operation. This remaining node will probably have applications running on it. As long as the Serviceguard daemon cmcld is active, other nodes can rejoin the cluster.
Cluster and Package Maintenance Removing Serviceguard from a System Removing Serviceguard from a System If you want to disable a node permanently from Serviceguard use, use the rpm -e command to delete the software. CAUTION Remove the node from the cluster first. If you run the rpm -e command on a server that is still a member of a cluster, it will cause that cluster to halt, and the cluster to be deleted. To remove Serviceguard: 1. If the node is an active member of a cluster, halt the node first. 2.
Troubleshooting Your Cluster 8 Troubleshooting Your Cluster This chapter describes how to verify cluster operation, how to review cluster status, how to add and replace hardware, and how to solve some typical cluster problems.
Troubleshooting Your Cluster Testing Cluster Operation Testing Cluster Operation Once you have configured your Serviceguard cluster, you should verify that the various components of the cluster behave correctly in case of a failure. In this section, the following procedures test that the cluster responds properly in the event of a package failure, a node failure, or a LAN failure.
Troubleshooting Your Cluster Testing Cluster Operation Depending on the specific databases you are running, perform the appropriate database recovery. Testing the Cluster Manager To test that the cluster manager is operating correctly, perform the following steps for each node on the cluster: 1. Turn off the power to the node. 2.
Troubleshooting Your Cluster Monitoring Hardware Monitoring Hardware Good standard practice in handling a high availability system includes careful fault monitoring so as to prevent failures if possible or at least to react to them swiftly when they occur. For information about disk monitoring, see “Creating a Disk Monitor Configuration” on page 239.
Troubleshooting Your Cluster Replacing Disks Replacing Disks The procedure for replacing a faulty disk mechanism depends on the type of disk configuration you are using. Refer to your Smart Array documentation for issues related to your Smart Array. Replacing a Faulty Mechanism in a Disk Array You can replace a failed disk mechanism by simply removing it from the array and replacing it with a new mechanism of the same type. The resynchronization is handled by the array itself.
Troubleshooting Your Cluster Replacing Disks CAUTION You are responsible for determining that the device is not being used by LVM or any other subsystem on any node connected to the device before using cmdisklock. If you use cmdisklock without taking this precaution, you could lose data. NOTE cmdisklock is needed only when you are repairing or replacing a lock LUN; see the cmdisklock (1m) manpage for more information. Serviceguard checks the lock LUN every 75 seconds.
Troubleshooting Your Cluster Replacing LAN Cards Replacing LAN Cards If you need to replace a LAN card, use the following steps. It is not necessary to bring the cluster down to do this. Step 1. Halt the node using the cmhaltnode command. Step 2. Shut down the system: shutdown -h Then power off the system. Step 3. Remove the defective LAN card. Step 4. Install the new LAN card. The new card must be exactly the same card type, and it must be installed in the same slot as the card you removed. Step 5.
Troubleshooting Your Cluster Replacing LAN Cards 1. Use the cmgetconf command to obtain a fresh ASCII configuration file, as follows: cmgetconf config.ascii 2. Use the cmapplyconf command to apply the configuration and copy the new binary file to all cluster nodes: cmapplyconf -C config.ascii This procedure updates the binary file with the new MAC address and thus avoids data inconsistency between the outputs of the cmviewconf and ifconfig commands.
Troubleshooting Your Cluster Replacing a Failed Quorum Server System Replacing a Failed Quorum Server System When a quorum server fails or becomes unavailable to the clusters it is providing quorum services for, this will not cause a failure on any cluster. However, the loss of the quorum server does increase the vulnerability of the clusters in case there is an additional failure. Use the following procedure to replace a defective quorum server system.
Troubleshooting Your Cluster Replacing a Failed Quorum Server System 4. Start the quorum server as follows: • Edit the /etc/inittab file to add the quorum server entries, as shown in the latest version of the HP Serviceguard Quorum Server Version Release Notes. • Use the init q command to run the quorum server. Or • Create a package in another cluster for the Quorum Server, as described in the Release Notes for your version of Quorum Server. They can be found at http://docs.hp.
Troubleshooting Your Cluster Replacing a Failed Quorum Server System NOTE CAUTION Chapter 8 While the old quorum server is down and the new one is being set up: • The cmquerycl, cmcheckconf and cmapplyconf commands will not work • The cmruncl, cmhaltcl, cmrunnode, and cmhaltnode commands will work • If there is a node or network failure that creates a 50-50 membership split, the quorum server will not be available as a tie-breaker, and the cluster will fail.
Troubleshooting Your Cluster Troubleshooting Approaches Troubleshooting Approaches The following sections offer a few suggestions for troubleshooting by reviewing the state of the running system and by examining cluster status data, log files, and configuration files.
Troubleshooting Your Cluster Troubleshooting Approaches Reviewing Package IP Addresses The ifconfig command can be used to examine the LAN configuration. The command, if executed on ftsys9 after the halting of node ftsys10, shows that the package IP addresses are assigned to eth1:1 and eth1:2 along with the heartbeat IP address on eth1. Chapter 8 eth0 Link encap:Ethernet HWaddr 00:01:02:77:82:75 inet addr:15.13.169.106 Bcast:15.13.175.255 Mask:255.255.248.
Troubleshooting Your Cluster Troubleshooting Approaches Reviewing the System Log File Messages from the Cluster Manager and Package Manager are written to the system log file. The default location of the log file may vary according to Linux distribution; the Red Hat default is /var/log/messages. You can use a text editor, such as vi, or the more command to view the log file for historical information on your cluster.
Troubleshooting Your Cluster Troubleshooting Approaches Sample System Log Entries The following sample entries from the syslog file show a package that failed to run because of a problem in the pkg5_run script. You would look at the pkg5_run.log for details. Dec Dec Dec Dec 14 14:33:48 star04 cmcld[2048]: Starting cluster management protocols.
Troubleshooting Your Cluster Troubleshooting Approaches Reviewing Object Manager Log Files The Serviceguard Object Manager daemon cmomd logs messages to the file /usr/local/cmom/cmomd.log on Red Hat and /var/log/cmmomcmomd.log on SUSE. You can review these messages using the cmreadlog command, for example: /usr/local/cmom/bin/cmreadlog /usr/local/cmom/log/cmomd.log Messages from cmomd include information about the processes that request data from the Object Manager, including type of data, timestamp, etc.
Troubleshooting Your Cluster Troubleshooting Approaches Using the cmquerycl and cmcheckconf Commands In addition, cmquerycl and cmcheckconf can be used to troubleshoot your cluster just as they were used to verify its configuration. The following example shows the commands used to verify the existing cluster configuration on ftsys9 and ftsys10: cmquerycl -v -C $SGCONF/verify.ascii -n ftsys9 -n ftsys10 cmcheckconf -v -C $SGCONF/verify.
Troubleshooting Your Cluster Solving Problems Solving Problems Problems with Serviceguard may be of several types. The following is a list of common categories of problem: • Serviceguard Command Hangs. • Cluster Re-formations. • System Administration Errors. • Package Control Script Hangs. • Package Movement Errors. • Node and Network Failures. • Quorum Server Messages.
Troubleshooting Your Cluster Solving Problems Cluster Re-formations Cluster re-formations may occur from time to time due to current cluster conditions. Some of the causes are as follows: • local switch on an Ethernet LAN if the switch takes longer than the cluster NODE_TIMEOUT value. To prevent this problem, you can increase the cluster NODE_TIMEOUT value. • excessive network traffic on heartbeat LANs. To prevent this, you can use dedicated heartbeat LANs, or LANs with less traffic on them.
Troubleshooting Your Cluster Solving Problems • fdisk -v /dev/sdx - to display information about a disk. Package Control Script Hangs or Failures When a RUN_SCRIPT_TIMEOUT or HALT_SCRIPT_TIMEOUT value is set, and the control script hangs, causing the timeout to be exceeded, Serviceguard kills the script and marks the package “Halted.” Similarly, when a package control script fails, Serviceguard kills the script and marks the package “Halted.
Troubleshooting Your Cluster Solving Problems where is the address indicated above and is the result of masking the with the mask found in the same line as the inet address in the ifconfig output. 3. Ensure that package volume groups are deactivated. First unmount any package logical volumes which are being used for file systems. This is determined by inspecting the output resulting from running the command df -l.
Troubleshooting Your Cluster Solving Problems Package Movement Errors These errors are similar to the system administration errors except they are caused specifically by errors in the package control script. The best way to prevent these errors is to test your package control script before putting your high availability application on line. Adding a “set -x” statement in the second line of your control script will give you details on where your script may be failing.
Troubleshooting Your Cluster Solving Problems Quorum Server Messages The coordinator node in Serviceguard sometimes sends a request to the quorum server to set the lock state. (This is different from a request to obtain the lock in tie-breaking.
Troubleshooting Your Cluster Solving Problems 322 Chapter 8
Serviceguard Commands A Serviceguard Commands The following is an alphabetical list of commands used for ServiceGuard cluster configuration and maintenance. Man pages for these commands are available on your system after installation. Table A-1 Serviceguard Commands Command cmapplyconf Description Verify and apply ServiceGuard cluster configuration and package configuration files.
Serviceguard Commands Table A-1 Serviceguard Commands (Continued) Command cmapplyconf (continued) Description Run cmgetconf to get either the cluster or package configuration file whenever changes to the existing configuration are required. Note that cmapplyconf will verify and distribute cluster configuration or package files. It will not cause the cluster daemon to start or removed from the cluster configuration.
Serviceguard Commands Table A-1 Serviceguard Commands (Continued) Command cmdeleteconf Description Delete either the cluster or the package configuration. cmdeleteconf deletes either the entire cluster configuration, including all its packages, or only the specified package configuration. If neither cluster_name nor package_name is specified, cmdeleteconf will delete the local cluster’s configuration and all its packages.
Serviceguard Commands Table A-1 Serviceguard Commands (Continued) Command cmhaltcl Description Halt a high availability cluster. cmhaltcl causes all nodes in a configured cluster to stop their cluster daemons, optionally halting all packages or applications in the process. This command will halt all the daemons on all currently running systems. If the user only wants to shutdown a subset of daemons, the cmhaltnode command should be used instead. cmhaltnode Halt a node in a high availability cluster.
Serviceguard Commands Table A-1 Serviceguard Commands (Continued) Command cmhaltserv Description Halt a service from the high availability package halt script. This is not a command line executable command, it runs only from within the package control script. cmhaltserv is used in the high availability package halt script to halt a service. If any part of package is marked down, the package halt script is executed as part of the recovery process.
Serviceguard Commands Table A-1 Serviceguard Commands (Continued) Command cmmodnet Description Add or remove an address from a high availability cluster. cmmodnet is used to add or remove a relocatable package IP_address for the current network interface running the given subnet_name. cmmodnet can also be used to enable or disable a LAN_name currently configured in a cluster.
Serviceguard Commands Table A-1 Serviceguard Commands (Continued) Command cmquerycl Description Query cluster or node configuration information. cmquerycl searches all specified nodes for cluster configuration, cluster lock, and Logical Volume Manager (LVM) information. Cluster configuration information includes network information such as LAN interface, IP addresses, bridged networks and possible heartbeat networks.
Serviceguard Commands Table A-1 Serviceguard Commands (Continued) Command Description cmresserviced Request monitoring of a device. This command is used in the SERVICE_CMD parameter of the package control script to define package dependencies on monitored disks. cmruncl Run a high availability cluster. cmruncl causes all nodes in a configured cluster or all nodes specified to start their cluster daemons and form a new cluster.
Serviceguard Commands Table A-1 Serviceguard Commands (Continued) Command cmrunserv Description Run a service from the high availability package run script. This is not a command line executable command, it runs only from within the package control script. cmrunserv is used in the high availability package run script to run a service. If the service process dies, cmrunserv updates the status of the service to down.
Serviceguard Commands Table A-1 Serviceguard Commands (Continued) Command cmscancl Description Gather system configuration information from nodes with ServiceGuard installed. cmscancl is a configuration report and diagnostic tool which gathers system software and hardware configuration information from a list of nodes, or from all the nodes in a cluster.
Serviceguard Commands Table A-1 Serviceguard Commands (Continued) Command cmviewconf Description View Serviceguard or ServiceGuard cluster configuration information. cmviewconf collects and displays the cluster configuration information, in ASCII format, from the binary configuration file for an existing cluster. Optionally, the output can be written to a file. This command can be used as a troubleshooting tool to identify the configuration of a cluster.
Serviceguard Commands 334 Appendix A
Designing Highly Available Cluster Applications B Designing Highly Available Cluster Applications This appendix describes how to create or port applications for high availability, with emphasis on the following topics: • Automating Application Operation • Controlling the Speed of Application Failover • Designing Applications to Run on Multiple Systems • Restoring Client Connections • Handling Application Failures • Minimizing Planned Downtime Designing for high availability means reducing the
Designing Highly Available Cluster Applications Automating Application Operation Automating Application Operation Can the application be started and stopped automatically or does it require operator intervention? This section describes how to automate application operations to avoid the need for user intervention. One of the first rules of high availability is to avoid manual intervention.
Designing Highly Available Cluster Applications Automating Application Operation Define Application Startup and Shutdown Applications must be restartable without manual intervention. If the application requires a switch to be flipped on a piece of hardware, then automated restart is impossible. Procedures for application startup, shutdown and monitoring must be created so that the HA software can perform these functions automatically.
Designing Highly Available Cluster Applications Controlling the Speed of Application Failover Controlling the Speed of Application Failover What steps can be taken to ensure the fastest failover? If a failure does occur causing the application to be moved (failed over) to another node, there are many things the application can do to reduce the amount of time it takes to get the application back up and running.
Designing Highly Available Cluster Applications Controlling the Speed of Application Failover Evaluate the Use of a Journaled Filesystem (JFS) If a file system must be used, a JFS offers significantly faster file system recovery than an HFS. However, performance of the JFS may vary with the application. An example of an appropriate JFS is the Reiser FS or ext3. Minimize Data Loss Minimize the amount of data that might be lost at the time of an unplanned outage.
Designing Highly Available Cluster Applications Controlling the Speed of Application Failover Keep Logs Small Some databases permit logs to be buffered in memory to increase online performance. Of course, when a failure occurs, any in-flight transaction will be lost. However, minimizing the size of this in-memory log will reduce the amount of completed transaction data that would be lost in case of failure.
Designing Highly Available Cluster Applications Controlling the Speed of Application Failover Another example is an application where a clerk is entering data about a new employee. Suppose this application requires that employee numbers be unique, and that after the name and number of the new employee is entered, a failure occurs.
Designing Highly Available Cluster Applications Controlling the Speed of Application Failover Design for Multiple Servers If you use multiple active servers, multiple service points can provide relatively transparent service to a client. However, this capability requires that the client be smart enough to have knowledge about the multiple servers and the priority for addressing them. It also requires access to the data of the failed server or replicated data.
Designing Highly Available Cluster Applications Designing Applications to Run on Multiple Systems Designing Applications to Run on Multiple Systems If an application can be failed to a backup node, how will it work on that different system? The previous sections discussed methods to ensure that an application can be automatically restarted. This section will discuss some ways to ensure the application can run on multiple systems.
Designing Highly Available Cluster Applications Designing Applications to Run on Multiple Systems Each application or package should be given a unique name as well as a relocatable IP address. Following this rule separates the application from the system on which it runs, thus removing the need for user knowledge of which system the application runs on. It also makes it easier to move the application among different systems in a cluster for for load balancing or other reasons.
Designing Highly Available Cluster Applications Designing Applications to Run on Multiple Systems Avoid Using SPU IDs or MAC Addresses Design the application so that it does not rely on the SPU ID or MAC (link-level) addresses. The SPU ID is a unique hardware ID contained in non-volatile memory, which cannot be changed. A MAC address (also known as a NIC id) is a link-specific address associated with the LAN hardware.
Designing Highly Available Cluster Applications Designing Applications to Run on Multiple Systems Assign Unique Names to Applications A unique name should be assigned to each application. This name should then be configured in DNS so that the name can be used as input to gethostbyname(3), as described in the following discussion. Use DNS DNS provides an API which can be used to map hostnames to IP addresses and vice versa.
Designing Highly Available Cluster Applications Designing Applications to Run on Multiple Systems Use uname(2) With Care Related to the hostname issue discussed in the previous section is the application's use of uname(2), which returns the official system name. The system name is unique to a given system whatever the number of LAN cards in the system. By convention, the uname and hostname are the same, but they do not have to be.
Designing Highly Available Cluster Applications Designing Applications to Run on Multiple Systems Bind to a Fixed Port When binding a socket, a port address can be specified or one can be assigned dynamically. One issue with binding to random ports is that a different port may be assigned if the application is later restarted on another cluster node. This may be confusing to clients accessing the application.
Designing Highly Available Cluster Applications Designing Applications to Run on Multiple Systems With UDP datagram sockets, however, there is a problem. The client may connect to multiple servers utilizing the relocatable IP address and sort out the replies based on the source IP address in the server’s response message. However, the source IP address given in this response will be the stationary IP address rather than the relocatable application IP address.
Designing Highly Available Cluster Applications Designing Applications to Run on Multiple Systems Give Each Application its Own Volume Group Use separate volume groups for each application that uses data. If the application doesn't use disk, it is not necessary to assign it a separate volume group. A volume group (group of disks) is the unit of storage that can move between nodes. The greatest flexibility for load balancing exists when each application is confined to its own volume group, i.e.
Designing Highly Available Cluster Applications Designing Applications to Run on Multiple Systems Avoid File Locking In an NFS environment, applications should avoid using file-locking mechanisms, where the file to be locked is on an NFS Server. File locking should be avoided in an application both on local and remote systems. If local file locking is employed and the system fails, the system acting as the backup system will not have any knowledge of the locks maintained by the failed system.
Designing Highly Available Cluster Applications Restoring Client Connections Restoring Client Connections How does a client reconnect to the server after a failure? It is important to write client applications to specifically differentiate between the loss of a connection to the server and other application-oriented errors that might be returned. The application should take special action in case of connection loss.
Designing Highly Available Cluster Applications Restoring Client Connections the retry to the current server should continue for the amount of time it takes to restart the server locally. This will keep the client from having to switch to the second server in the event of a application failure. • Use a transaction processing monitor or message queueing software to increase robustness.
Designing Highly Available Cluster Applications Handling Application Failures Handling Application Failures What happens if part or all of an application fails? All of the preceding sections have assumed the failure in question was not a failure of the application, but of another component of the cluster. This section deals specifically with application problems.
Designing Highly Available Cluster Applications Handling Application Failures Be Able to Monitor Applications All components in a system, including applications, should be able to be monitored for their health. A monitor might be as simple as a display command or as complicated as a SQL query. There must be a way to ensure that the application is behaving correctly.
Designing Highly Available Cluster Applications Minimizing Planned Downtime Minimizing Planned Downtime Planned downtime (as opposed to unplanned downtime) is scheduled; examples include backups, systems upgrades to new operating system revisions, or hardware replacements. For planned downtime, application designers should consider: • Reducing the time needed for application upgrades/patches.
Designing Highly Available Cluster Applications Minimizing Planned Downtime Reducing Time Needed for Application Upgrades and Patches Once a year or so, a new revision of an application is released. How long does it take for the end-user to upgrade to this new revision? This answer is the amount of planned downtime a user must take to upgrade their application. The following guidelines reduce this time. Provide for Rolling Upgrades Provide for a “rolling upgrade” in a client/server environment.
Designing Highly Available Cluster Applications Minimizing Planned Downtime Do Not Change the Data Layout Between Releases Migration of the data to a new format can be very time intensive. It also almost guarantees that rolling upgrade will not be possible. For example, if a database is running on the first node, ideally, the second node could be upgraded to the new revision of the database.
Integrating HA Applications with Serviceguard C Integrating HA Applications with Serviceguard The following is a summary of the steps you should follow to integrate an application into the Serviceguard environment: 1. Read the rest of this book, including the chapters on cluster and package configuration, and the appendix “Designing Highly Available Cluster Applications.” 2.
Integrating HA Applications with Serviceguard Checklist for Integrating HA Applications Checklist for Integrating HA Applications This section contains a checklist for integrating HA applications in both single and multiple systems. Defining Baseline Application Behavior on a Single System 1. Define a baseline behavior for the application on a standalone system: • Install the application, database, and other required resources on one of the systems.
Integrating HA Applications with Serviceguard Checklist for Integrating HA Applications Integrating HA Applications in Multiple Systems 1. Install the application on a second system. • Create the LVM infrastructure on the second system. • Add the appropriate users to the system. • Install the appropriate executables. • With the application not running on the first system, try to bring it up on the second system. You might use the script you created in the step above.
Integrating HA Applications with Serviceguard Checklist for Integrating HA Applications Testing the Cluster 1. Test the cluster: • Have clients connect. • Provide a normal system load. • Halt the package on the first node and move it to the second node: # cmhaltpkg pkg1 # cmrunpkg -n node2 pkg1 # cmmodpkg -e pkg1 • Move it back. # cmhaltpkg pkg1 # cmrunpkg -n node1 pkg1 # cmmodpkg -e pkg1 • Fail one of the systems. For example, turn off the power on node 1.
Blank Planning Worksheets D Blank Planning Worksheets This appendix reprints blank versions of the planning worksheets described in the “Planning” chapter. You can duplicate any of these worksheets that you find useful and fill them in as a part of the planning process.
Blank Planning Worksheets Hardware Worksheet Hardware Worksheet ============================================================================= SPU Information: Host Name ____________________ Server Series____________ Memory Capacity ____________ Number of I/O Slots ____________ ============================================================================= LAN Information: Name of Master _________ Name of Node IP Traffic Interface __________ Addr________________ Type ________ Name of Name of Node IP Traff
Blank Planning Worksheets Power Supply Worksheet Power Supply Worksheet ============================================================================ SPU Power: Host Name ____________________ Power Supply _____________________ Host Name ____________________ Power Supply _____________________ ============================================================================ Disk Power: Disk Unit __________________________ Power Supply _______________________ Disk Unit __________________________ Power Supply
Blank Planning Worksheets Quorum Server Worksheet Quorum Server Worksheet Quorum Server Data: ============================================================================== QS Hostname: _________________IP Address: ______________________ OR Cluster Name: _________________ Package Name: ____________ Package IP Address: ___________________ Hostname Given to Package by Network Administrator: _________________ ============================================================================== Quorum Services are Pr
Blank Planning Worksheets Volume Group and Physical Volume Worksheet Volume Group and Physical Volume Worksheet ============================================================================== Volume Group Name: ___________________________________ Physical Volume Name: _________________ Physical Volume Name: _________________ Physical Volume Name: _________________ ============================================================================= Volume Group Name: ___________________________________ Physical Vol
Blank Planning Worksheets Cluster Configuration Worksheet Cluster Configuration Worksheet =============================================================================== Name and Nodes: =============================================================================== Cluster Name: ______________________________ Node Names: ________________________________________________ Maximum Configured Packages: ______________ =============================================================================== Cluster Lock Da
Blank Planning Worksheets Cluster Configuration Worksheet Access Policies User: ________ Host: ________ Role: ________ User: _________ Host: _________ Role: __________ Appendix D 369
Blank Planning Worksheets Package Configuration Worksheet Package Configuration Worksheet ============================================================================= Package Configuration File Data: ========================================================================== Package Name: __________________Package Type:______________ Primary Node: ____________________ First Failover Node:__________________ Additional Failover Nodes:__________________________________ Run Script Timeout: _____ Halt Script Ti
Blank Planning Worksheets Package Configuration Worksheet Logical Volumes and File Systems: fs_name___________________ fs_directory________________fs_mount_opt____________ fs_umount_opt______________fs_fsck_opt_________________fs_type_________________ fs_name____________________fs_directory________________fs_mount_opt____________ fs_umount_opt_____________ fs_fsck_opt_________________fs_type_________________ fs_name____________________fs_directory________________fs_mount_opt____________ fs_umount_opt_______
Blank Planning Worksheets Package Control Script Worksheet (Legacy) Package Control Script Worksheet (Legacy) PACKAGE CONTROL SCRIPT WORKSHEET Page ___ of ___ ================================================================================ Package Control Script Data: ================================================================================ PATH______________________________________________________________ VGCHANGE_________________________________ VG[0]__________________LV[0]______________________FS
IPv6 Network Support E IPv6 Network Support This appendix describes some of the characteristics of IPv6 network addresses, specifically: Appendix E • “IPv6 Address Types” on page 374 • “Network Configuration Restrictions” on page 380 • “Configuring IPv6 on Linux” on page 382 373
IPv6 Network Support IPv6 Address Types IPv6 Address Types Several IPv6 types of addressing schemes are specified in the RFC 2373 (IPv6 Addressing Architecture). IPv6 addresses are 128-bit identifiers for interfaces and sets of interfaces. There are various address formats for IPv6 defined by the RFC 2373. IPv6 addresses are broadly classified as unicast, anycast, and multicast. The following table explains the three types. Table E-1 IPv6 Address Types Unicast An address for a single interface.
IPv6 Network Support IPv6 Address Types multiple groups of 16-bits of zeros. The “::” can appear only once in an address and it can be used to compress the leading, trailing, or contiguous sixteen-bit zeroes in an address. Example: fec0:1:0:0:0:0:0:1234 can be represented as fec0:1::1234. • In a mixed environment of IPv4 and IPv6 nodes an alternative form of IPv6 address will be used. It is x:x:x:x:x:x:d.d.d.
IPv6 Network Support IPv6 Address Types Unicast Addresses IPv6 unicast addresses are classified into different types. They are: global aggregatable unicast address, site-local address and link-local address. Typically a unicast address is logically divided as follows: Table E-2 n bits 128-n bits Subnet prefix Interface ID Interface identifiers in a IPv6 unicast address are used to identify the interfaces on a link. Interface identifiers are required to be unique on that link.
IPv6 Network Support IPv6 Address Types IPv4 Mapped IPv6 Address There is a special type of IPv6 address that holds an embedded IPv4 address. This address is used to represent the addresses of IPv4-only nodes as IPv6 addresses. These addresses are used especially by applications that support both IPv6 and IPv4. These addresses are called as IPv4 Mapped IPv6 Addresses. The format of these address is as follows: Table E-4 80 bits 16 bits zeros 32 bits FFFF IPv4 address Example: ::ffff:192.168.0.
IPv6 Network Support IPv6 Address Types Link-Local Addresses Link-local addresses have the following format: Table E-6 10 bits 1111111010 54 bits 0 64 bits interface ID Link-local address are supposed to be used for addressing nodes on a single link. Packets originating from or destined to a link-local address will not be forwarded by a router.
IPv6 Network Support IPv6 Address Types “FF” at the beginning of the address identifies the address as a multicast address. The “flags” field is a set of 4 flags “000T”. The higher order 3 bits are reserved and must be zero. The last bit ‘T’ indicates whether it is permanently assigned or not. A value of zero indicates that it is permanently assigned otherwise it is a temporary assignment. The “scop” field is a 4-bit field which is used to limit the scope of the multicast group.
IPv6 Network Support Network Configuration Restrictions Network Configuration Restrictions Serviceguard supports IPv6 for data links only: the heartbeat IP address must be IPv4, but the package IP addresses can be IPv4 or IPv6. The restrictions for supporting IPv6 in Serviceguard for Linux are:. NOTE 380 • The heartbeat IP address must be IPv4. IPv6-only nodes are not supported in a Serviceguard environment. • The hostnames in a Serviceguard configuration must be IPv4.
IPv6 Network Support Network Configuration Restrictions Appendix E • Bonding is supported for IPv6 addresses, but only in active-backup mode. • The Quorum server, if used, must be configured on an IPv4 network. It is not IPv6-capable. A quorum server configured on an IPv4 network can still be used by Serviceguard IPv6 clusters that have IPv6 networks as a part of their cluster configuration. • Serviceguard supports IPv6 only on the Ethernet networks, including 10BT, 100BT, and Gigabit Ethernet.
IPv6 Network Support Configuring IPv6 on Linux Configuring IPv6 on Linux Red Hat Enterprise Linux and SUSE Linux Enterprise Server already have the proper IPv6 tools installed, including the /sbin/ip command. This section explains how to configure IPv6 stationary IP addresses on these systems.
IPv6 Network Support Configuring IPv6 on Linux Configuring a Channel Bonding Interface with Persistent IPv6 Addresses on Red Hat Linux Configure the following parameters in /etc/sysconfig/network-scripts/ifcfg-bond0: DEVICE=bond0 IPADDR=12.12.12.12 NETMASK=255.255.255.0 NETWORK=12.12.12.0 BROADCAST=12.12.12.255 IPV6INIT=yes IPV6ADDR=3ffe:ffff:0000:f101::10/64 IPV6ADDR_SECONDARIES=fec0:0:0:1::10/64 IPV6_MTU=1280 ONBOOT=yes BOOTPROTO=none USERCTL=no Add the following two lines to /etc/modprobe.
IPv6 Network Support Configuring IPv6 on Linux Configuring a Channel Bonding Interface with Persistent IPv6 Addresses on SUSE Configure the following parameters in /etc/sysconfig/network/ifcfg-bond0: BOOTPROTO=static BROADCAST=10.0.2.255 IPADDR=10.0.2.10 NETMASK=255.255.0.0 NETWORK=0.0.2.
Index A Access Control Policies, 180 active node, 20 adding a package to a running cluster, 291 adding cluster nodes advance planning, 140 adding nodes to a running cluster, 257 adding packages on a running cluster, 238 administration adding nodes to a running cluster, 257 halting a package, 261 halting the entire cluster, 259 moving a package, 262 of packages and services, 260 of the cluster, 255 reconfiguring a package while the cluster is running, 290 reconfiguring a package with the cluster offline, 291
Index cluster lock 4 or more nodes, 48 and cluster reformation, example, 89 and power supplies, 35 identifying in configuration file, 174 storing configuration data, 170 two nodes, 46, 47 use in re-forming a cluster, 46, 47 cluster manager automatic restart of cluster, 45 blank planning worksheet, 368 cluster node parameter, 110, 111, 112, 113 defined, 43 dynamic re-formation, 45 heartbeat interval parameter, 116 heartbeat subnet parameter, 113 initial configuration of the cluster, 43 main functions, 43 max
Index designing applications to run on multiple systems, 343 disk data, 33 interfaces, 33 root, 33 sample configurations, 34 disk I/O hardware planning, 100 disk layout planning, 108 disk logical units hardware planning, 101 disk monitoring configuring, 239 disks in Serviceguard, 33 replacing, 303 supported types in Serviceguard, 33 distributing the cluster and package configuration, 236, 284 DNS services, 147 down time minimizing planned, 356 dynamic cluster re-formation, 45 E enclosure for disks replacing
Index halting a cluster, 259 halting a package, 261 halting the entire cluster, 259 handling application failures, 354 hardware monitoring, 302 power supplies, 35 hardware failures response to, 90 hardware planning blank planning worksheet, 363 Disk I/O Bus Type, 100 disk I/O information for shared disks, 100 host IP address, 97, 106 host name, 96 I/O bus addresses, 100 I/O slot numbers, 100 LAN interface name, 97, 106 LAN traffic type, 97 memory capacity, 96 number of I/O slots, 96 planning the configurati
Index LAN planning host IP address, 97, 106 traffic type, 97 link-level addresses, 345 load sharing with IP addresses, 76 local switching, 77 lock cluster locks and power supplies, 35 use of the cluster lock, 47 use of the cluster lock disk, 46 lock volume group, reconfiguring, 265 logical volume parameter in package control script, 228 logical volumes creating the infrastructure, 158 planning, 108 LV, 228 in sample package control script, 280 LVM commands for cluster use, 158 disks, 33 planning, 108 M MAC
Index NODE_FAIL_FAST_ENABLED effect of setting, 91 NODE_NAME parameter in cluster manager configuration, 110, 111, 112, 113 NODE_TIMEOUT and HEARTBEAT_INTERVAL, 88 and node TOC, 88 NODE_TIMEOUT (node timeout) parameter in cluster manager configuration, 116 nodetypes primary, 20 NTP time protocol for clusters, 149 O Object Manager, 314 outages insulating users from, 336 P package adding and deleting package IP addresses, 76 basic concepts, 26 blank planning worksheet, 370, 372 changes allowed while the clus
Index blank planning worksheet, 367 planning, 108 planning cluster configuration, 109 cluster lock and cluster expansion, 104 cluster manager configuration, 110 disk I/O information, 100 for expansion, 121 hardware configuration, 96 high availability objectives, 94 overview, 93 package configuration, 119 power, 102 quorum server, 105 SPU information, 96 volume groups and physical volumes, 108 worksheets, 101 planning and documenting an HA cluster, 93 planning for cluster expansion, 94 planning worksheets bl
Index S S800 series number hardware planning, 96 safety timer reset failure, 117 sample disk configurations, 34 service administration, 260 service configuration step by step, 199 service failures responses, 91 service restarts, 91 SERVICE_CMD in sample package control script, 280 SERVICE_FAIL_FAST_ENABLED and node TOC, 91 SERVICE_NAME in sample package control script, 280 SERVICE_RESTART in sample package control script, 280 Serviceguard install, 142 introduction, 18 Serviceguard at a Glance, 17 Servicegua
Index and NODE_TIMEOUT, 88 and package availability, 89 and safety timer, 117 when a node fails, 88 traffic type LAN hardware planning, 97 troubleshooting approaches, 310 monitoring hardware, 302 replacing disks, 303 reviewing control scripts, 314 reviewing package IP addresses, 311 reviewing system log file, 312 using cmquerycl and cmcheckconf, 315 troubleshooting your cluster, 299 typical cluster after failover figure, 20 typical cluster configuration figure, 18 W What is Serviceguard?, 18 worksheet blan
Index 394
