Designing high availability solutions with HP Serviceguard and HP Integrity Virtual Machines Technical white paper Table of contents Executive summary .......................................................................................................................2 Integrity VM overview ...................................................................................................................2 Serviceguard for high availability .................................................................
Executive summary HP Integrity Virtual Machines (Integrity VMs) is a virtualization product in the HP Matrix Operating Environment (Matrix OE) for HP-UX to help customers maximize server resource utilization and reduce total cost of ownership by allowing individual operating system environments to share CPU and I/O resources.
Figure 1: Integrity VM components 2 app 1 app 2 HP-UX 11i v2 app 1 app 2 HP-UX 11i v3 app 1 app 1 Linux RH4 Windows app 1 app 1 Linux SLES 10 Windows HP Integrity VM Host Processors Memory I/O Integrity Hardware Integrity VM can be used with other Matrix OE components for Workload Management and HP Instant Capacity on VM hosts, including: • HP Workload Manager (WLM) • HP Global Workload Manager (gWLM) • HP Temporary Instant Capacity (TiCAP) • HP Global Instant Capacity (GiCAP) These Matr
Serviceguard uses packages to group application services (for example, individual HP-UX processes) together, and are typically configured to run on several nodes in the cluster one at a time. In the event of a service, node, network, or other monitored package resource failure on the node where the package is running, Serviceguard can automatically transfer control of the package to another node in the cluster, thus allowing the services to remain available with minimal interruption.
There are many possible ways to configure Integrity VM with Serviceguard. The following section describes these different configurations in the form of implementation models to help characterize their differences and highlight their suitability for use in various system environments. These implementation models are currently supported with several different versions of Integrity VM, Serviceguard, 3 and the Serviceguard Storage Management Suite (SMS).
• NFS file systems (with Integrity VM B.04.30, and Serviceguard A.11.20) • Files on any of the storage types listed above, including files on a Cluster File System (CFS) The VM guest backing stores must reside on shared storage so that it is accessible by all VM hosts that will run the package in the Serviceguard cluster to allow failover of the VM guests.
Example use cases The following are several example scenarios that can benefit from the Serviceguard integration with Integrity VM: HA for development and test environments: Traditional development and test environments typically are not highly available usually for cost reasons. Using Integrity VM with Serviceguard provides HA for the VMs created to consolidate the applications used in these environments. The VMs can be failed over to another node in the event of a VM host failure.
Serviceguard LANs using Accelerated Virtual I/O (AVIO) must be configured using AVIO-supported host physical point attachment (PPA) network devices. In VM as Serviceguard package configurations using AVIO network devices, the Serviceguard standby LANs must use supported host physical point attachment (PPA) devices supported by AVIO to prevent a loss of network connectivity that will occur if using non-supported devices, even if the standby LAN link is up.
Known issues when using the Integrity VM Serviceguard toolkit: • If the VM guest backing store resides on storage using a volume manager, the hpvmsg_package command expects the volume manager to be activated so that the backing store is accessible on the node before you run the command. • The hpvmsg_package command does not add appropriate entries to the package configuration and control script files for VM guests on CVM or CFS backing stores.
Advantages of the SG-IVS toolkit: • This toolkit uses Serviceguard package modules, so you can generate packages using the Serviceguard cmmakepkg command. Additionally, the Serviceguard online package configuration change feature is supported. • This toolkit will be supported with Integrity VM version B.04.30 and later. • This toolkit will be enhanced to take advantage of future Integrity VM and Serviceguard features.
There are several points to consider prior to using either toolkit to create Serviceguard packages: It is highly recommended that the VM guest to be packaged should be tested by first starting it on each VM host node in the cluster to ensure that its backing stores are accessible from all cluster nodes. The toolkit commands expect the VM guest to be running in order to identify all required backing stores and their mount points for the package configuration file it creates.
Figure 5 is an example of using VM guests as nodes within a Serviceguard cluster. VM host software running on each physical node allows operation of the VM guests on the physical nodes. Figure 5: Serviceguard cluster using VM guests as cluster nodes VM Guest 1 Serviceguard Cluster VM Guest 2 Serviceguard Package Failover VM Host 1 VM Host 2 As in the previous example, Serviceguard provides high availability in the event of a VM guest or application failure.
Figures 6 and 7 are examples showing how Integrity VM can be used to consolidate nodes within Serviceguard clusters. In figure 7, a single standby host is used to run two separate VM guests that are each part of two different Serviceguard clusters. The packages that are normally running on primary nodes 1 and 2 can failover to their corresponding VM guest failover nodes running on a single standby host configured to support the execution of multiple VM guests.
Figure 9: Serviceguard multi-cluster consolidation using VM guests Failover Physical Node SG Cluster VM Guest 1 Physical Node SG Cluster VM Guest 2 Failover Failover SG Cluster VM Guest 2 Failover VM Guest 1 Physical Node SG Cluster VM Host Physical Node Before consolidation VM Host After consolidation Storage considerations An important distinction between VM as Serviceguard package and VM as Serviceguard node configurations is that VM as Serviceguard node configurations only support whol
An example of a storage configuration used by an HP-UX VM as Serviceguard node configuration is shown in figure 10a, and a Linux VM as Serviceguard node configuration in figure 10b.
Figure 11: Dynamic memory allocation in an HP-UX VM guest Serviceguard Package Failover Target Max 3 1 Target Min Target Max 2 VM Guest VM Host In the event of a Serviceguard package failover, the external script for the Serviceguard package uses an hpvmmgmt command to dynamically allocate an amount of memory (in 64MB chunks) to a specified target value (in this example, the full amount of memory used at VM guest startup), as shown in step 2.
Note that changing the maximum memory value allocated for a VM requires the VM to be restarted with the new value. In addition, VM memory allocation may be slow or may not be able to reach the original boot size of the VM when VM host memory is tight or very fragmented. Note: Dynamic memory allocation can be done for Serviceguard Legacy packages by placing the appropriate hpvmmgmt commands in the customer_defined_run_cmds and customer_defined_halt_cmds functions.
at cluster configuration time to extend the quiescence period of the cluster reformation based on whether a VM node is present in the cluster and the I/O timeout settings on the VM host. It is important to note that: • The io_timeout_extension parameter is set internally by Serviceguard and is not configurable by the user; however, its value can be viewed using the Serviceguard cmviewconf, cmviewcl v –f line commands, or can be found in the system log file.
Serviceguard revision A.11.19 and later handle runtime delays differently than earlier revisions: • For Serviceguard A.11.19 and later revisions, an option to handle runtime delays is to increase the Serviceguard cluster MEMBER_TIMEOUT to a value larger than the run delay reported in the syslog file. The default value for this cluster parameter is 14 seconds, and the maximum recommended value is 300 seconds. For most installations, a setting of 14 seconds is usually appropriate.
This configuration provides the most flexibility in meeting recovery time objectives (RTOs) for VM guests and applications while efficiently consolidating systems on VM hosts. For example, to consolidate a system environment consisting of Windows servers and mission-critical applications running on HP-UX servers, the Windows servers with a lower RTO can be made highly available by running them as VM guests encapsulated within Serviceguard packages.
Figure 13: Example of Online VM Migration for Integrity VM Serviceguard Cluster HP-UX VM HP-UX VM HP-UX VM HP-UX VM Windows VM HP-UX VM VM Host Windows VM HP-UX VM VM Host VM Host VM Host Virtual machines can be moved from one VM host to another in a variety of ways, including: Offline: • Non-running VM configuration (hpvmmigrate command) • VM as Serviceguard package (cmhaltpkg / cmrunpkg commands) Online: • Running VM (hpvmmigrate –o command) • Running VM as a Serviceguard package (cmmovevpkg
Offline VM Migration with Serviceguard In the event of a VM host or guest failure, or if the user issues a cmhaltpkg/cmrunpkg command, this will trigger the applications, OS and VM guest to halt and the VM guest package to start on its adoptive VM host cluster node where the VM guest, its OS and applications are restarted.
Downtime experienced: • Complete VM guest downtime during the migration Advantages: • Allows copying of storage used by the VM guest from the VM host source to target • Provides movement of VM guests that are not supported using online migration Disadvantages: • Complete downtime of the VM guest during the migration Offline VM Migration with Serviceguard Used for: • Protecting VM guests from unplanned hardware and software failures (Note Serviceguard LAN failover and vswitch monitor handle network failu
Downtime experienced: • Tens of seconds for system “freeze” time during the migration (depending on VM guest activity) Advantages: • Minimal application downtime during planned maintenance periods • Protection for the VM guest against unplanned VM host hardware/software failures or failure of the VM guest itself Disadvantages: • Serviceguard does not protect the VM guest during the online migration process With the variety of migration methods available, a recommended best practice for reducing overall a
• No communications are required between the VM guest and VM host. • No security issues arise because application management authority is confined to the VM guest. • Monitoring mechanisms (for example, ps command to monitor process IDs) are readily available from the VM guest operating system. The shortcomings of guest-based application monitoring include: • A custom application monitor must be developed, tested, and maintained.
Figure 14 shows a representation of the Serviceguard VM guest application monitoring architecture. The “VM guest package” in the figure is the Serviceguard package, which contains the cmappmgr services and the VM guest service.
The application server cmappserver responds to requests from application manager process cmappmgr running on the VM host to execute a specified process on the VM guest and to periodically check the status of the monitored process and return an exit code when process terminates. The cmappserver is a light-weight daemon (approximately 36k) running on the monitored VM guest and provides connections for up to 30 simultaneous processes that can be monitored by cmappmgr.
Figure 16: Failover of a VM guest due to an application failure VM Guest Package Serviceguard Cluster cmapp server app1 app2 VM guest package failover hpvmsg_mon service_restart exceeded… Serviceguard fails over VM Guest to Standby Node VM Guest cmappmgr app1 cmappmgr app2 VM Host VM Host Primary Node Standby Node Note there are other methods that can be implemented for monitoring VM guest applications using this framework.
cross-subnet cluster by configuring the guest to use DHCP for dynamically obtaining the IP addresses for its virtual network interfaces, and configuring a bootpd or dhcpd server in each data center to provide the IP addresses for the guest. To ensure that the VM guest always gets the same IP addresses in a data center, you must configure the local bootpd or dhcpd server in that data center to assign IP addresses based on the hardware addresses 7 of the virtual network interfaces configured for the VM guest.
Where: VM1 and VM1-2—the hostnames that will be associated with the IP addresses for the VM guest when it is running in data center B ht=ether—required setting ha—set to the hardware address of the virtual network interfaces used by the guest ip—set to the IP address to be assigned to the virtual network interface ns—set to the subnet mask for the networks in data center B ds—set to the IP address of the DNS server for data center B (optional) 4. Now you must enable the bootpd daemon.
• SAP – SAP software running on HP-UX 11i v2 or 11i v3 – For SAP, the latest information can be found on SAP’s Service Marketplace website (www.service.sap.com/notes Note: access to this site requires a support contract).
Networks For VM as package configurations: • Three LAN connections are recommended: one LAN for a dedicated Serviceguard heartbeat for the VM host and a primary/standby LAN pair for VM guests, which are monitored by Serviceguard on the VM host. • HP Auto-Port Aggregation (APA) is supported and can be used to provide network bandwidth scalability, load balancing between the physical links, automatic fault detection, and HA recovery.
When creating VMs: • Consider how using uniprocessor vs. multiprocessor VMs can affect the overall performance of CPU resources on the VM host. The use of multi-processor VMs may require the tuning of Serviceguard node timeout and heartbeat interval parameters in some instances to avoid false detection of Serviceguard node failures. The only way to do this tuning is through “trial and error”.
Selecting the best model to meet your requirements The following steps can help with the selection of the appropriate configuration model for designing and implementing Serviceguard clusters when using VMs for consolidation: Establish design goals: • With consolidation being the primary goal when using Integrity VM, determine the secondary design goals for the consolidation effort (for example, high availability for specific applications, good application performance under normal operations, acceptable per
Glossary Term Definition Integrity VM HP Integrity Virtual Machines product VM Virtual hardware system, implemented with software, representing a collection of virtual hardware devices VM host A physical system that includes a host Operating System and Integrity VM Software for executing one or more VM guests VM guest A VM and its operating system running on a VM host Guest OS Operating system instance installed on the virtual machine Physical node A single server or nPar (hard partition) Serv
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Java and Oracle are registered trademarks of Oracle and/or its affiliates.
