Designing Disaster Tolerant High Availability Clusters Manufacturing Part Number: B7660-90013 March 2003
Legal Notices The information contained in this document is subject to change without notice. Hewlett-Packard makes no warranty of any kind with regard to this manual, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. Hewlett-Packard shall not be liable for errors contained herein or direct, indirect, special, incidental or consequential damages in connection with the furnishing, performance, or use of this material.
Contents 1. Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Evaluating the Need for Disaster Tolerance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . What is a Disaster Tolerant Architecture? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Types of Disaster Tolerant Clusters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Extended Distance Clusters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Contents Creating the Raid Manager Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 Sample Raid Manager Configuration File. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Configuring Automatic Raid Manager Startup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 Verifying the XP Series Disk Array Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . 102 Creating and Exporting LVM Volume Groups . . . . . . . . . . . . . . . . . .
Contents Grouping the Symmetrix Devices at Each Data Center. . . . . . . . . . . . . . . . . . . . . . Building the Symmetrix CLI Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Determining Symmetrix Device Names on Each Node . . . . . . . . . . . . . . . . . . . . . . Setting up 1 by 1 Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Creating Symmetrix Device Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Contents Performing Cluster Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Notes on Packages in a Continental Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . How MC/ServiceGuard commands work in a ContinentalCluster . . . . . . . . . . . . . Designing a Disaster Tolerant Architecture for use with ContinentalClusters . . . . . Mutual Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Contents Newly Created Cluster Will Run Primary Packages . . . . . . . . . . . . . . . . . . . . . . . . Newly Created Cluster Will Function as Recovery Cluster for All Recovery Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Maintaining a Continental Cluster. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Adding a Node to a Cluster or Removing a Node from a Cluster . . . . . . . . . . . . . .
Contents Scenario 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Failback in Scenario 3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Failback When the Primary Has SMPL Status . . . . . . . . . . . . . . . . . . . . . . . . . . . Maintaining the Continuous Access XP Data Replication Environment . . . . . . . . . Rescynchronizing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Contents Setting Up Symmetrix Device Groups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Setting up Volume Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Testing the Volume Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Primary Cluster Package Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Recovery Cluster Package Setup. . . . . . . . . . . .
Contents 10
Printing History Table 1 Editions and Releases Printing Date Part Number Edition Operating System Releases (see Note below) June 1998 B6264-90002 Edition 1 HP-UX 10.20 and 11.0 August 1999 B7660-90001 Edition 2 HP-UX 10.20 and 11.0 December 1999 B7660-90002 Edition 3 HP-UX 10.20 and 11.0 February 2000 B7660-90003 Edition 4 HP-UX 10.20 and 11.00 September 2000 B7660-90005 Edition 5 HP-UX 10.20, 11.0 and 11i March 2001 B7660-90006 Edition 6 HP-UX 10.20, 11.
NOTE 12 This document describes a group of separate software products that are released independently of one another. Not all products described in this document are necessarily supported on all the same operating system releases. Consult your product’s Release Notes for information about supported platforms.
HP Printing Division: Business Critical Computing Business Unit Hewlett-Packard Co. 19111 Pruneridge Ave.
Preface This guide describes how to configure disaster tolerant clusters using MC/ServiceGuard, MetroCluster/CA, MetroCluster/SRDF, and ContinentalClusters. The contents are as follows: • Chapter 1, “Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster,” is an overview of disaster tolerant cluster configurations. • Chapter 2, “Building an Extended Distance Cluster Using MC/ServiceGuard,” describes campus cluster configurations using MC/ServiceGuard.
Related Publications • Appendix C, “Configuration File Parameters for ContinentalClusters,” lists all the configuration file variables used in defining the continental cluster configuration. • Appendix D, “ContinentalClusters Command and Daemon Reference,” documents the ContinentalClusters command set. The following documents contain additional useful information: • Clusters for High Availability: a Primer of HP Solutions, Second Edition.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster 1 Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster This guide shows how to use a variety of Hewlett-Packard high availability cluster technologies to provide disaster tolerance for your mission-critical applications. It is assumed that you are already familiar with MC/ServiceGuard high availability concepts and configurations.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Evaluating the Need for Disaster Tolerance Evaluating the Need for Disaster Tolerance Disaster tolerance is the ability to restore applications and data within a reasonable period of time after a disaster.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Evaluating the Need for Disaster Tolerance line inoperable as well as the computers. In this case disaster recovery would be moot, and local failover is probably the more appropriate level of protection. On the other hand, you may have an order processing center that is prone to floods in the winter. The business loses thousands of dollars a minute while the order processing servers are down.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster What is a Disaster Tolerant Architecture? What is a Disaster Tolerant Architecture? In an MC/ServiceGuard cluster configuration, high availability is achieved by using redundant hardware to eliminate single points of failure. This protects the cluster against hardware faults, such as the node failure in Figure 1-1. Figure 1-1 High Availability Architecture.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster What is a Disaster Tolerant Architecture? impact. For these types of installations, and many more like them, it is important to guard not only against single points of failure, but against multiple points of failure (MPOF), or against single massive failures that cause many components to fail, such as the failure of a data center, of an entire site, or of a small area.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Types of Disaster Tolerant Clusters Types of Disaster Tolerant Clusters To protect against multiple points of failure, cluster components must be geographically dispersed: nodes can be put in different rooms, on different floors of a building, or even in separate buildings or separate cities. The distance between the nodes is dependent on the types of disaster from which you need protection, and on the technology used to replicate data.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Types of Disaster Tolerant Clusters maximum distance between nodes in an extended distance cluster is set by the limits of the data replication technology and networking limits. An extended distance cluster is shown in Figure 1-3.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Types of Disaster Tolerant Clusters In addition, there is no hard requirement on how far the third location has to be from the two main data centers. The third location can be as close as the room next door with its own power source or can be as far as in a site across town. The distance between all three locations dictates the level of disaster tolerance a metropolitan cluster can provide.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Types of Disaster Tolerant Clusters Metropolitan cluster architecture is shown in Figure 1-4.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Types of Disaster Tolerant Clusters Continental Cluster A continental cluster provides an alternative disaster tolerant solution in which distinct clusters are separated by large distances, with wide area networking used between them. Continental cluster architecture is implemented via the ContinentalClusters product, described fully in Chapter 5.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Types of Disaster Tolerant Clusters • The physical connection is one or more leased lines managed by a common carrier. Common carriers cannot guarantee the same reliability that a dedicated physical cable can. The distance can introduce a time lag for data replication, which creates an issue with data currency. This could increase the cost by requiring higher speed WAN connections to improve data replication performance and reduce latency.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Types of Disaster Tolerant Clusters • Recovery Symmetrix—on a site that holds a remote mirror copy of the data, located in the recovery cluster. Figure 1-6 illustrates data centers, clusters, nodes and Symmetrix frames in a cascading failover configuration. Refer to Chapter 8 for details on setting up data replication for this type of cluster.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Disaster Tolerant Architecture Guidelines Disaster Tolerant Architecture Guidelines Disaster tolerant architectures represent a shift away from the massive central data centers towards more distributed data processing facilities.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Disaster Tolerant Architecture Guidelines • Ensure data consistency by replicating data in a logical order so that it is immediately usable or recoverable. Inconsistent data is unusable and is not recoverable for processing. Consistent data may or may not be current. • Ensure data currency by replicating data quickly so that a replica of the data can be recovered to include all committed disk writes that were applied to the local disks.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Disaster Tolerant Architecture Guidelines On-line Data Replication On-line data replication is a method of copying data from one site to another across a link. It is used when very short recovery time, from minutes to hours, is required. To be able to recover use of a system in a short time, the data at the alternate site must be replicated in real time on all disks. Data can be replicated either synchronously or asynchronously.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Disaster Tolerant Architecture Guidelines As shown in Figure 1-7, physical replication can be done in software or hardware. Figure 1-7 Physical Data Replication node 1 node 1a Replication The distance between nodes is limited by the link type (SCSI, ESCON, FC) and the intermediate devices (FC switch, DWDM) on the link path node 1 node 1a Physical Replication in software (MirrorDisk/UX). Direct access to both copies of data is optional.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Disaster Tolerant Architecture Guidelines • Data can be copied in both directions, so that if the primary fails and the replica takes over, data can be copied back to the primary when it comes back up. Disadvantages of physical replication in hardware are: • The logical order of data writes is not maintained in synchronous replication.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Disaster Tolerant Architecture Guidelines • Data copies are peers, so there is no issue with reconfiguring a replica to function as a primary disk after failover. • Because there are multiple read devices, that is, the node has access to both copies of data, there may be improvements in read performance. • Writes are synchronous unless the link or disk is down.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Disaster Tolerant Architecture Guidelines Logical replication can be configured to use synchronous or asynchronous writes. Transaction processing monitors (TPMs) can also perform logical replication. Figure 1-8 Logical Data Replication node 1 node 1a Network Logical Replication in Software. No direct access to both copies of data.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Disaster Tolerant Architecture Guidelines It also uses network bandwidth, whereas most physical replication methods use a separate data replication link. As a result, there may be a significant lag in replicating transactions at the remote site, which affects data currency. • If the primary database fails and is corrupt, and the replica takes over, the process for restoring the primary database so that it can be used as the replica is complex.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Disaster Tolerant Architecture Guidelines Each data center that houses part of a disaster tolerant cluster should be supplied with power from a different circuit. In addition to a standard UPS (uninterrupted power supply), each node in a disaster tolerant cluster should be on a separate power circuit; see Figure 1-9.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Disaster Tolerant Architecture Guidelines they use physically different routes for each network as indicated in Figure 1-10. How you route cables will depend on the networking technology you use. Specific guidelines for some network technologies are listed here.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Disaster Tolerant Architecture Guidelines Disaster Tolerant Local Area Networking The configurations described in this section are for FDDI and Ethernet based Local Area Networks. Figure 1-10 Reliability of the Network is Paramount node 1 node 2 Wrong: Cables use same route. node 1a node 2a X Data Center A node 1 Right: Cables use different route. Data Center A Accident severs both network cables.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Disaster Tolerant Architecture Guidelines These FDDI options are shown in Figure 1-11.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Disaster Tolerant Architecture Guidelines from A, to C, to B, the failure of data center C would make it impossible for clients to connect to data center B in the case of failover.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Disaster Tolerant Architecture Guidelines • Bandwidth affects the rate of data replication, and therefore the currency of the data should you need to switch control to another site. The greater the number of transactions you process, the more bandwidth you will need.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Disaster Tolerant Architecture Guidelines • Chapter 1 A rolling disaster is a disaster that occurs before the cluster is able to recover from a non-disastrous failure. An example is a data replication link that fails, then, as it is being restored and data is being resynchronized, a disaster causes an entire data center to fail.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Managing a Disaster Tolerant Environment Managing a Disaster Tolerant Environment In addition to the changes in hardware and software to create a disaster tolerant architecture, there are also changes in the way you manage the environment. Configuration of a disaster tolerant architecture needs to be carefully planned, implemented and maintained.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Managing a Disaster Tolerant Environment Even if recovery is automated, you many choose to, or need to recover from some types of disasters with manual recovery. A rolling disaster, which is a disaster that happens before the cluster has recovered from a previous disaster, is an example of when you may want to manually switch over.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Using this Guide with Your Disaster Tolerant Cluster Products Using this Guide with Your Disaster Tolerant Cluster Products The rest of this guide provides detailed, task-oriented documentation covering a variety of HP products and technologies, as follows: 46 • Chapter 2, “Building an Extended Distance Cluster with MC/ServiceGuard,” shows different types of extended distance clusters using standard MC/ServiceGuard (B3935BA).
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Using this Guide with Your Disaster Tolerant Cluster Products • Chapter 8, “Cascading Failover in a Continental Cluster,” describes a specific configuration using the EMC Symmetrix disk array to provide data replication with disaster recovery between a metropolitan cluster in one geographic region and an ordinary ServiceGuard cluster in another region. A set of appendixes and a glossary provide additional reference information.
Disaster Tolerance and Recovery in an MC/ServiceGuard Cluster Using this Guide with Your Disaster Tolerant Cluster Products 48 Chapter 1
Building an Extended Distance Cluster Using MC/ServiceGuard 2 Building an Extended Distance Cluster Using MC/ServiceGuard Simple MC/ServiceGuard clusters are usually configured in a single data center, often in a single room, to provide protection against failures in CPUs, interface cards, and software. Extended MC/ServiceGuard clusters are specialized cluster configurations, which allow a single cluster to extend across two or three separate data centers for increased disaster tolerance.
Building an Extended Distance Cluster Using MC/ServiceGuard Types of Data Link for Storage and Networking Types of Data Link for Storage and Networking FibreChannel technology lets you increase the distance between the components in an MC/ServiceGuard cluster, thus making it possible to design a disaster tolerant architecture. The following table shows some of the distances possible with a few of the available technologies, including some of the FibreChannel alternatives.
Building an Extended Distance Cluster Using MC/ServiceGuard Types of Data Link for Storage and Networking NOTE Chapter 2 Increased distance often means increased cost and reduced speed of connection. Not all combinations of links are supported in all cluster types. For a current list of supported configurations and supported distances, refer to the HP 9000 Servers Configuration Guide, available through your HP representative. As new technologies become supported, they will be described in that guide.
Building an Extended Distance Cluster Using MC/ServiceGuard Two Data Center Architecture Two Data Center Architecture The two-data-center architecture is based on a standard MC/ServiceGuard configuration with half of the nodes in one data center, and the other half in another data center. Nodes can be located in separate data centers in the same building, or even separate buildings within the limits of FibreChannel technology.
Building an Extended Distance Cluster Using MC/ServiceGuard Two Data Center Architecture StorageWorks Disk Array XP or EMC Symmetrix Disk Arrays. If Model FC30 disk arrays are used as cluster lock disks, Auto trespass must be disabled for the entire disk array. Chapter 2 • Application data must be mirrored between the primary data centers. If MirrorDisk/UX is used, Mirror Write Cache (MWC) must be the Consistency Recovery policy defined for all mirrored logical volumes.
Building an Extended Distance Cluster Using MC/ServiceGuard Two Data Center Architecture Two Data Center FibreChannel Implementations FibreChannel Using Hubs Although this architecture should work with the maximum number of nodes allowed in an MC/ServiceGuard cluster, this cluster architecture has been tested with a maximum of 4 nodes. Therefore, the largest configuration currently supported is 2 nodes per data center as shown in Figure 2-1.
Building an Extended Distance Cluster Using MC/ServiceGuard Two Data Center Architecture FibreChannel Using Switches The two data center architecture is also possible over longer distances using FibreChannel switches. The following is one example of a switched two data center configuration using FibreChannel and FDDI networking.
Building an Extended Distance Cluster Using MC/ServiceGuard Two Data Center Architecture DWDM with Two Data Centers The following is an example of a two data center configuration using DWDM for both storage and networking. Figure 2-3 Two Data Centers with DWDM Network and Storage Advantages and Disadvantages of a Two-Data-Center Architecture The advantages of a two-data-center architecture are: 56 • Lower cost.
Building an Extended Distance Cluster Using MC/ServiceGuard Two Data Center Architecture • All systems are connected to both copies of data, so that if a primary disk fails but the primary system stays up, there is greater availabilty because there is no package failover. The disadvantages of a two-data-center architecture are: • There is a slight chance of split brain syndrome.
Building an Extended Distance Cluster Using MC/ServiceGuard Two Data Centers and Third Location Architectures Two Data Centers and Third Location Architectures Configurations with two data centers and third location have the following requirements: • In these solutions, there must be an equal number of nodes (1-7) or (1-8 if a Quorum Server is used) in each primary data center, and the third location (known as the arbitrator data center) or Quorum Server can contain 1 or 2 nodes.
Building an Extended Distance Cluster Using MC/ServiceGuard Two Data Centers and Third Location Architectures Chapter 2 • Fibre Channel Arbitrated loops are limited to a maximum of four nodes and nine Fibre Channel Disk Arrays per loop. This restriction does not exist if Direct Fabric Attach configurations are used.
Building an Extended Distance Cluster Using MC/ServiceGuard Two Data Centers and Third Location Architectures The following table shows the possible configurations using a three data center architecture. Table 2-2 Supported System and Data Center Combinations Data Center A 60 Data Center B Data Center C ServiceGuard Version 1 1 1 Arbitrator Node A.11.13 or later 1 1 Quorum Server System A.11.13 or later 2 1 2 Arbitrator Nodes A.11.13 or later 1 2 2 Arbitrator Nodes A.11.
Building an Extended Distance Cluster Using MC/ServiceGuard Two Data Centers and Third Location Architectures Table 2-2 Supported System and Data Center Combinations (Continued) Data Center A Data Center B Data Center C ServiceGuard Version 5 5 1 Arbitrator Node A.11.13 or later 5 5 2* Arbitrator Nodes A.11.13 or later 5 5 Quorum Server System A.11.13 or later 6 6 1 Arbitrator Node A.11.13 or later 6 6 2* Arbitrator Nodes A.11.13 or later 6 6 Quorum Server System A.11.
Building an Extended Distance Cluster Using MC/ServiceGuard Two Data Centers and Third Location Architectures • Quorum Server with APA • Quorum Server For more information on Quorum Server, refer to the MC/ServiceGuard Quorum Server Version A.01.00 Release Notes for HP-UX. The following is an example of a three data center configuration using DWDM, with arbitrator nodes on the third site.
Building an Extended Distance Cluster Using MC/ServiceGuard Two Data Centers and Third Location Architectures The following is an example of a three data center configuration using DWDM, with a quorum server node on the third site.
Building an Extended Distance Cluster Using MC/ServiceGuard Rules for Separate Network and Data Links Rules for Separate Network and Data Links 64 • The network interfaces used must support DLPI (link level). • There must be less than 200 milliseconds of latency in the network between the data centers. • No routing is allowed for the networks between the data centers.
Building an Extended Distance Cluster Using MC/ServiceGuard Rules for Separate Network and Data Links Chapter 2 • Fibre Channel hubs are only supported for distances up to 10 kilometers. For distances longer than 10 kilometers, Fibre Channel switches are required. Fibre Channel switches are always the preferred solution, since they offer much better performance than Fibre Channel hubs, which only support Fibre Channel Arbitrated loop mode.
Building an Extended Distance Cluster Using MC/ServiceGuard Guidelines on DWDM Links for Network and Data Guidelines on DWDM Links for Network and Data • The network interfaces used must support DLPI (link level). There must be less than 200 milliseconds of latency in the network between the data centers. No routing is allowed for the networks between the data centers. The maximum distance supported between the data centers for DWDM configurations is 100 kilometers.
Building an Extended Distance Cluster Using MC/ServiceGuard Guidelines on DWDM Links for Network and Data Refer to the HP 9000 Servers Configuration Guide, available through your HP representative, for more information about specific supported devices Chapter 2 67
Building an Extended Distance Cluster Using MC/ServiceGuard Guidelines on DWDM Links for Network and Data 68 Chapter 2
Building a Metropolitan Cluster Using MetroCluster/CA 3 Building a Metropolitan Cluster Using MetroCluster/CA This chapter describes the high availability toolkit for MetroCluster with Continuous Access XP (hereafter known as MetroCluster/CA), which allows you to install and configure an MC/ServiceGuard metropolitan cluster using the HP StorageWorks E Disk Array XP Series with Continuous Access XP.
Building a Metropolitan Cluster Using MetroCluster/CA Overview of MetroCluster/CA Overview of MetroCluster/CA MetroCluster is a set of scripts and environmental file that work in an MC/ServiceGuard cluster to automate failover to alternate nodes in the case of disaster in metropolitan cluster. The MetroCluster/CA product contains the following files. Table 3-1 MetroCluster/CA Template Files Name Description /opt/cmcluster/toolkit/SGCA/xpca.env The MetroCluster/CA environmental file.
Building a Metropolitan Cluster Using MetroCluster/CA Overview of MetroCluster/CA MetroCluster/CA has to be installed on all nodes that will run a ServiceGuard package whose data are on an HP StorageWorks E Disk Array XP Series where the data are replicated to a second XP using the Continuous Access XP facility.
Building a Metropolitan Cluster Using MetroCluster/CA Designing a Disaster Tolerant Architecture for use with MetroCluster/CA Designing a Disaster Tolerant Architecture for use with MetroCluster/CA MetroCluster/CA is designed for use in an extended distance cluster or metropolitan cluster environment within the 100 km limit of the FDDI network. All nodes must be members of a single MC/ServiceGuard cluster. Two configurations are supported: • A single data center without arbitrators (not disaster tolerant.
Building a Metropolitan Cluster Using MetroCluster/CA Designing a Disaster Tolerant Architecture for use with MetroCluster/CA Two Data Centers and Third Location with Arbitrator(s) This is the recommended and supported disaster tolerant architecture for use with MetroCluster/CA. This architecture consists of two main data centers with an equal number of nodes and a third location with one or more arbitrator nodes or a quorum server node; see Figure 3-1.
Building a Metropolitan Cluster Using MetroCluster/CA Designing a Disaster Tolerant Architecture for use with MetroCluster/CA Series disk array can be the main disk array for one set of packages and the remote disk array for another. In Figure 3-1, the XP disk array in data center A is the main or primary disk array for packages A and B, and the remote or secondary disk array for packages C and D in data center B.
Building a Metropolitan Cluster Using MetroCluster/CA Designing a Disaster Tolerant Architecture for use with MetroCluster/CA Table 3-2 Supported System and Data Center Combinations (Continued) Data Center A Chapter 3 Data Center B Data Center C ServiceGuard Version 2 2 1 Arbitrator Node A.11.13 or later 2 2 2* Arbitrator Nodes A. 11.13 or later 2 2 Quorum Server System A. 11.13 or later 3 3 1 Arbitrator Node A. 11.13 or later 3 3 2* Arbitrator Nodes A. 11.
Building a Metropolitan Cluster Using MetroCluster/CA Designing a Disaster Tolerant Architecture for use with MetroCluster/CA Table 3-2 Supported System and Data Center Combinations (Continued) Data Center A Data Center B Data Center C ServiceGuard Version 6 6 Quorum Server System A.11.13 or later 7 7 1 Arbitrator Node A.11.13 or later 7 7 2* Arbitrator Nodes A.11.13 or later 7 7 Quorum Server System A.11.13 or later 8 8 Quorum Server System A.11.
Building a Metropolitan Cluster Using MetroCluster/CA Designing a Disaster Tolerant Architecture for use with MetroCluster/CA NOTE In the campus or metropolitan environment, the same number of systems must be present in each of the two data centers (Data Center A and Data Center B) whose systems are connected to the XP disk arrays. There must be either one or two arbitrators or a Quorum Server in a third location.
Building a Metropolitan Cluster Using MetroCluster/CA Designing a Disaster Tolerant Architecture for use with MetroCluster/CA XP Series Disk Array Configuration Rules Each XP Series disk array must be configured with redundant CA links, each of which is connected to a different LCP or RCP card. To prevent a single point of failure (SPOF), there must be at least two physical boards in each XP for the CA links. Each board usually has multiple ports.
Building a Metropolitan Cluster Using MetroCluster/CA Designing a Disaster Tolerant Architecture for use with MetroCluster/CA Example Failover Scenarios with One Arbitrator Taking a node off-line for planned maintenance is treated the same as a node failure in these scenarios. Study these scenarios to make sure you do not put your cluster at risk during planned maintenance.
Building a Metropolitan Cluster Using MetroCluster/CA Designing a Disaster Tolerant Architecture for use with MetroCluster/CA Table 3-3 Node Failure Scenarios with One Arbitrator (Continued) Failure Quorum Result nodes 1, 2, arbitrator 1, then node 3 1 of 2 (50%) cluster halts* arbitrator 1, then node 1 3 of 4 (75%) pkg A switches * Cluster can be manually started with the remaining node. Table 3-4 illustrates possible results if a data center fails in a configuration with a single arbitrator.
Building a Metropolitan Cluster Using MetroCluster/CA Designing a Disaster Tolerant Architecture for use with MetroCluster/CA Example Failover Scenarios with Two Arbitrators Having two arbitrator nodes adds extra protection during node failures and allows you to do planned maintenance on arbitrator nodes without losing the cluster should a disaster occur.
Building a Metropolitan Cluster Using MetroCluster/CA Designing a Disaster Tolerant Architecture for use with MetroCluster/CA Table 3-5 Failure Scenarios with Two Arbitrators (Continued) Failure Quorum Result data center A, then arbitrator 1, then node 3 2 of 3 (67%) pkg A, B, and C switch arbitrator 1, then data center A 3 of 5 (60%) pkg A and B switch to data center B node 3, then data center A 3 of 5 (60%) pkg A and B switch to data center B data center B 4 of 6 (67%) pkg C and D switch to
Building a Metropolitan Cluster Using MetroCluster/CA Designing a Disaster Tolerant Architecture for use with MetroCluster/CA Disaster Tolerant Checklist Use the this checklist to make sure you have adhered to the disaster tolerant architecture guidelines for a two main data centers and a third location configuration. Figure 3-4 Disaster Tolerant Checklist Data centers A and B have the same number of nodes to maintain quorum in case an entire data center fails.
Building a Metropolitan Cluster Using MetroCluster/CA Designing a Disaster Tolerant Architecture for use with MetroCluster/CA Cluster Configuration Worksheet Use this cluster configuration worksheet either in place of, or in addition to the worksheet provided in the Managing MC/ServiceGuard manual. If you have already completed an MC/ServiceGuard cluster configuration worksheet, you only need to complete the first part of this worksheet.
Building a Metropolitan Cluster Using MetroCluster/CA Designing a Disaster Tolerant Architecture for use with MetroCluster/CA Package Configuration Worksheet Use this package configuration worksheet either in place of, or in addition to the worksheet provided in the Managing MC/ServiceGuard manual. If you have already completed an MC/ServiceGuard package configuration worksheet, you only need to complete the first part of this worksheet.
Building a Metropolitan Cluster Using MetroCluster/CA Designing a Disaster Tolerant Architecture for use with MetroCluster/CA Figure 3-7 Package Control Script Worksheet Package Control Script Data VG[0]: LV[0]: FS[0]: FS_MOUNT_OPT[0]: VG[1]: LV[1]: FS[1]: FS_MOUNT_OPT[1]: VG[2]: LV[2]: FS[2]: FS_MOUNT_OPT[2]: VXVM_DG[0]: LV[0]: FS[0]: FS_MOUNT_OPT[0]: VXVM_DG[1]: LV[1]: FS[1]: FS_MOUNT_OPT[1]: VXVM_DG[2]: LV[2]: FS[2]: FS_MOUNT_OPT[2]: IP[0]: SUBNET[0]: IP[1]: SUBNET[1]: X.
Building a Metropolitan Cluster Using MetroCluster/CA Preparing an MC/ServiceGuard Cluster for MetroCluster /CA Preparing an MC/ServiceGuard Cluster for MetroCluster /CA When the following procedures are completed, an adoptive node will be able to access the data belonging to a package after it fails over. Setting up the Hardware Ensure that the XP Series disk arrays are correctly cabled using PV links to each node in the cluster that will run packages accessing data on the array.
Building a Metropolitan Cluster Using MetroCluster/CA Preparing an MC/ServiceGuard Cluster for MetroCluster /CA • Figure 3-8 Primary (PVOL) and secondary (SVOL) volumes must be correctly defined and assigned to the appropriate nodes in the XP hardware configuration.
Building a Metropolitan Cluster Using MetroCluster/CA Preparing an MC/ServiceGuard Cluster for MetroCluster /CA Fence Level of NEVER Fence level = NEVER should only be used when the availability of the application is more important than the data currency on the remote XP disk array. In the case when all CA links fail, the application will continue to modify the data on PVOL side, however the new data is not replicated to the SVOL side.
Building a Metropolitan Cluster Using MetroCluster/CA Preparing an MC/ServiceGuard Cluster for MetroCluster /CA Although the risk of this sequence of events taking place is extremely low, if your business cannot afford even this quite small risk, then you must enable Fence level = DATA to ensure that the data at the SVOL side are always consistent.
Building a Metropolitan Cluster Using MetroCluster/CA Preparing an MC/ServiceGuard Cluster for MetroCluster /CA there is enough room in the side file. Before continuing to write, the primary XP will wait until there is enough room in the side file, and will keep trying until it reaches its side file timeout value, which is configured through the SVP.
Building a Metropolitan Cluster Using MetroCluster/CA Preparing an MC/ServiceGuard Cluster for MetroCluster /CA Consistency Group An important property of asynchronous mode volumes is the consistency group (CT group). A CT group is a grouping of LUNs that need to be treated the same from the perspective of data consistency (I/O ordering). A CT group is equal to a device group in the Raid Manager configuration file. A consistency group ID (CTGID) is assigned automatically during pair creation.
Building a Metropolitan Cluster Using MetroCluster/CA Preparing an MC/ServiceGuard Cluster for MetroCluster /CA • If you change the LDEV number associated with a given target/LUN, you must restart all the Raid Manager instances even though the Raid Manager configuration file is not modified. • Any firmware update, cache expansion, or board change, requires a restart of all Raid Manager instances. • pairsplit for asynchronous mode may take a long time depending on how long the synchronization takes.
Building a Metropolitan Cluster Using MetroCluster/CA Preparing an MC/ServiceGuard Cluster for MetroCluster /CA Creating the Raid Manager Configuration The Raid Manager configuration file must be edited and customized on each node that is attached to one of the XP Series disk arrays. The file is named using the following convention: horcm.conf All MetroCluster packages must use the same Raid Manager instance, and must be configured in the same configuration file.
Building a Metropolitan Cluster Using MetroCluster/CA Preparing an MC/ServiceGuard Cluster for MetroCluster /CA 4. Start the Raid Manager instance by using the command horcmstart.sh as in the following example: # horcmstart.sh 0 5. Export the environment variable that specifies the Raid Manager instance to be used by the Raid Manager commands. For example, with the POSIX shell, type: # export HORCMINST=0 Now, you can use Raid Manager commands to get further information from the disk arrays. 6.
Building a Metropolitan Cluster Using MetroCluster/CA Preparing an MC/ServiceGuard Cluster for MetroCluster /CA 8. Determine which devices will be used by the application package. Define a device group that contains all of these devices. The device group name (dev_group) is user-defined and must be the same on each host in the MetroCluster that accesses the XP Series disk array. It is recommended that you use a name that is easily associated with the package.
Building a Metropolitan Cluster Using MetroCluster/CA Preparing an MC/ServiceGuard Cluster for MetroCluster /CA 11. Repeat these steps on each host that will run this particular application package. If a host may run more than one application package, you must incorporate device group and host information for each of these packages. Note that the Raid Manager configuration file must be different for each host, especially for the HORCM_MON and HORCM_INST fields. 12.
Building a Metropolitan Cluster Using MetroCluster/CA Preparing an MC/ServiceGuard Cluster for MetroCluster /CA # with the "-c" option. # # NOTE: The Raid Manager command device (RORCM_CMD) cannot be used for # data storage (it is reserved for private Raid Manager usage). #/************************ HORCM_MON *************************************/ # # The HORCM_MON parameter is used for monitoring and control of device groups # by the Raid Manager.
Building a Metropolitan Cluster Using MetroCluster/CA Preparing an MC/ServiceGuard Cluster for MetroCluster /CA #Primary #dev_name /dev/rdsk/c4t1d0 Primary Alt-Link dev_name /dev/rdsk/c5t1d0 Secondary dev_name /dev/rdsk/c4t0d1 Secondary Alt-link dev_name /dev/rdsk/c5t0d1 #/************************* HORCM_DEV *************************************/ # # The HORCM_DEV parameter is used to define the addresses of the physical # volumes corresponding to the paired logical volume names.
Building a Metropolitan Cluster Using MetroCluster/CA Preparing an MC/ServiceGuard Cluster for MetroCluster /CA # # # # # # # # # # # # # # # # for each of the device group secondary volumes. The remote Raid Manager instances are required to get status or provide control of the remote devices in the device group. All remote hosts must be defined here, so that the failure of one remote host will prevent obtaining status. This is the same device group names as defined in dev_group of HORC_DEV.
Building a Metropolitan Cluster Using MetroCluster/CA Preparing an MC/ServiceGuard Cluster for MetroCluster /CA # startup parameters for a HP StorageWorks E Disk Array XP Raid Manager # instance. The Raid Manager instance must be running before any # MetroCluster package can start up successfully. # # @(#) $Revision: 1.
Building a Metropolitan Cluster Using MetroCluster/CA Preparing an MC/ServiceGuard Cluster for MetroCluster /CA Verifying the XP Series Disk Array Configuration Use the following checklist to verify the configuration. Figure 3-10 XP Series Disk Array Configuration Checklist Raid Manager XP software installed on all nodes. PVOLs and SVOLs created for use with all packages. /etc/horcm0.conf created for all nodes with appropriate differences. Raid Manager automatic startup configured on all nodes.
Building a Metropolitan Cluster Using MetroCluster/CA Preparing an MC/ServiceGuard Cluster for MetroCluster /CA Make sure that you copy the map files to all of the nodes. The sample script Samples/ftpit shows a semi-automated way (using ftp) to copy the files. You need only enter the password interactively. Importing Volume Groups on Other Nodes Use the following procedure to import volume groups. The sample script mk2imports can be modified to automate these steps. 1.
Building a Metropolitan Cluster Using MetroCluster/CA Preparing an MC/ServiceGuard Cluster for MetroCluster /CA commands if you are using PV links. The -s option to the vgexport command saves the volume group id (VGID) in the map file, but it does not preserve the order of PV links.
Building a Metropolitan Cluster Using MetroCluster/CA Preparing an MC/ServiceGuard Cluster for MetroCluster /CA # newfs -F vxfs /dev/vx/rdsk/logdata/logfile 9. Create a directory to mount the volume group. # mkdir /logs 10. Mount the volume group. # mount /dev/vx/dsk/logdata/logfile /logs 11. Check if file system exits, then unmount the file system # umount /logs Validating VxVM Disk Groups for Use with MetroCluster/CA The following section shows how to validate VERITAS disk groups.
Building a Metropolitan Cluster Using MetroCluster/CA Preparing an MC/ServiceGuard Cluster for MetroCluster /CA 9. Resynchronize the CA pair device.
Building a Metropolitan Cluster Using MetroCluster/CA Configuring Packages for Automatic Disaster Recovery Configuring Packages for Automatic Disaster Recovery When you have completed the following steps, packages will be able to automatically fail over to an alternate node in another data center and still have access to the data that they need in order to operate.
Building a Metropolitan Cluster Using MetroCluster/CA Configuring Packages for Automatic Disaster Recovery If you are using a fence level of ASYNC, then the RUN_SCRIPT_TIMEOUT should be greater than the value of HORCTIMEOUT in the package environment file (see step 7g below). NOTE If you are using the EMS disk monitor as a package resource, you must not use NO_TIMEOUT. Otherwise, package shutdown will hang if there is no access from the host to the package disks.
Building a Metropolitan Cluster Using MetroCluster/CA Configuring Packages for Automatic Disaster Recovery b. Uncomment the behavioral configuration environment variables starting with AUTO_. It is recommended that you retain the default values of these variables unless you have a specific business requirement to change them. See Appendix A for an explanation of these variables. c. Uncomment the PKGDIR variable and set it to the full path name of the directory where the control script has been placed.
Building a Metropolitan Cluster Using MetroCluster/CA Configuring Packages for Automatic Disaster Recovery If any messages are returned, you should correct the syntax errors. 9. Check the configuration using the cmcheckconf -P pkgname.config, then apply the MC/ServiceGuard configuration using the cmapplyconf -P pkgname.config command or SAM. 10.
Building a Metropolitan Cluster Using MetroCluster/CA Maintaining a Cluster that uses MetroCluster/CA Maintaining a Cluster that uses MetroCluster/CA While the cluster is running, manual changes of state for devices on the XP Series disk array can cause the package to halt due to unexpected conditions or can cause the package to not start up after a failover. In general, it is recommended that no manual changes of state be performed while the package and the cluster are running.
Building a Metropolitan Cluster Using MetroCluster/CA Maintaining a Cluster that uses MetroCluster/CA cluster operation, shows the percentage of the side file that is full: # pairdisplay -g pkgB -fc -CLI Group PairVol L/R Port# TID LU Seq# LDEV# P/S Status Fence % P-LDEV# M pkgB pkgD-disk0 L CL1-C 0 3 35422 463 P-VOL PAIR ASYNC 35 3 pkgB pkgD-disk0 R CL1-F 0 3 35663 3 S-VOL PAIR ASYNC 0 463 - This output shows that 35% of the side file is full.
Building a Metropolitan Cluster Using MetroCluster/CA Maintaining a Cluster that uses MetroCluster/CA Resynchronizing After certain failures, data is no longer remotely protected. In order to restore disaster tolerant data protection after repairing or recovering from the failure, you must manually run the command pairresync. This command must successfully complete for disaster-tolerant data protection to be restored.
Building a Metropolitan Cluster Using MetroCluster/CA Maintaining a Cluster that uses MetroCluster/CA • Failure of the secondary XP Series disk array for a given application package followed by application startup on a primary host • Failure of all CA links with Fence Level NEVER and ASYNC with restart of the application on a primary host Using the pairresync Command The pairresync command can be used with special options after a failover in which the recovery site has started the application and has pr
Building a Metropolitan Cluster Using MetroCluster/CA Maintaining a Cluster that uses MetroCluster/CA Failback After resynchronization is complete, you can halt the package on the failover site, and restart it on the primary site. MetroCluster will then swap the personalities between the PVOL and the SVOL, returning PVOL status to the primary site.
Building a Metropolitan Cluster Using MetroCluster/CA XP/CA Device Group Monitor XP/CA Device Group Monitor In the MetroCluster/CA environment where the device group state is not actively monitored and the end user may not be aware when the application data is not remotely protected for an extended period of time. Under these circumstances, the XP/CA device group monitor provides the capability to monitor the status of the XP/CA device group that is used in a package.
Building a Metropolitan Cluster Using MetroCluster/CA XP/CA Device Group Monitor XP/CA Device Group Monitor Operation Overview The XP/CA device group monitor runs as a package service. The user can configure the monitor's setting through the package's environment file. Once the package has started the XP/CA device group monitor, the monitor will periodically check the status of the XP/CA device group.
Building a Metropolitan Cluster Using MetroCluster/CA XP/CA Device Group Monitor • If you want to receive notification messages over email, uncomment the MON_NOTIFICATION_EMAIL variable and set it to a fully qualified email address. Multiple email addresses can be configured using comma as separator between the addresses. • If you want notification messages to be logged in the syslog file, uncomment the MON_NOTIFICATION_SYSLOG variable and set it to 1.
Building a Metropolitan Cluster Using MetroCluster/CA XP/CA Device Group Monitor • send a notification on every third polling, if the state of the device group remains the same. • send the notifications to sysadmin1@hp.com and sysadmin2@hp.com. • log notifications to system log file, syslog. • display notifications to system console. • perform automatic resynchronization with BC management when detecting the device group local state change to PVOL-PSUE or PVOL-PDUB.
Building a Metropolitan Cluster Using MetroCluster/CA XP/CA Device Group Monitor Configure XP/CA Device Group Monitor as a Service of the Package Add the monitor as a service in the package's configuration file and control script file as follows: • In the package's configuration file, add the following lines: SERVICE_NAME pkgXdevgrpmon.srv SERVICE_FAIL_FAST_ENABLED NO SERVICE_HALT_TIMEOUT 5 NOTE The SERVICE_HALT_TIMEOUT value of 5 is a recommended value.
Building a Metropolitan Cluster Using MetroCluster/CA XP/CA Device Group Monitor Troubleshooting the XP/CA Device Group Monitor The following is a guideline to help the user identify the cause of possible problems with the XP/CA device group monitor. Problems with email notifications XP/CA device group monitor uses SMTP to send out email notifications. All email notification problems are logged in the package log file.
Building a Metropolitan Cluster Using MetroCluster/CA XP/CA Device Group Monitor 122 Chapter 3
Building a Metropolitan Cluster Using MetroCluster/SRDF 4 Building a Metropolitan Cluster Using MetroCluster/SRDF This chapter describes the high availability toolkit template for MetroCluster with EMC SRDF (hereafter known as MetroCluster/SRDF), which allows you to install and configure an MC/ServiceGuard metropolitan cluster using the EMC Symmetrix disk array.
Building a Metropolitan Cluster Using MetroCluster/SRDF Overview of MetroCluster/SRDF Overview of MetroCluster/SRDF MetroCluster is a set of scripts and an environment file that work in an MC/ServiceGuard cluster to automate failover to alternate nodes in the case of disaster in a metropolitan cluster. The MetroCluster/SRDF product contains the following files. Table 4-1 MetroCluster/SRDF Template Files Name Description /opt/cmcluster/toolki t/SGSRDF/srdf.env The MetroCluster/SRDF environmental file.
Building a Metropolitan Cluster Using MetroCluster/SRDF Overview of MetroCluster/SRDF MetroCluster with Symmetrix SRDF is specifically for configuring one or more MC/ServiceGuard packages whose data reside on EMC Symmetrix ICDAs (Integrated Cache Disk Arrays) replicated with SRDF (Symmetrix Remote Data Facility). MetroCluster with Symmetrix SRDF can be used in metropolitan cluster configuration with nodes in a loop of up to 100 km.
Building a Metropolitan Cluster Using MetroCluster/SRDF Designing a Disaster Tolerant Architecture for use with MetroCluster/SRDF Designing a Disaster Tolerant Architecture for use with MetroCluster/SRDF MetroCluster with Symmetrix SRDF is designed for use metropolitan cluster environment within the 100 km loop limit of the FDDI network. All nodes must be members of a single MC/ServiceGuard cluster. Two configurations are supported: • A single data center without arbitrators (not disaster tolerant.
Building a Metropolitan Cluster Using MetroCluster/SRDF Designing a Disaster Tolerant Architecture for use with MetroCluster/SRDF Two Data Centers and Third Location with Arbitrator(s) This is the recommended and supported disaster tolerant architecture for use with MetroCluster with Symmetrix SRDF. This architecture consists of two main data centers with an equal number of nodes and a third location with one or more arbitrator nodes; see Figure 4-1.
Building a Metropolitan Cluster Using MetroCluster/SRDF Designing a Disaster Tolerant Architecture for use with MetroCluster/SRDF the EMC Symmetrix disk array in data center A is the R1 disk for packages A and B, and the R2 disk for packages C and D in data center B. Likewise the EMC Symmetrix disk array in data center B is the R1 for packages C and D, and the R2 for packages A and B.
Building a Metropolitan Cluster Using MetroCluster/SRDF Designing a Disaster Tolerant Architecture for use with MetroCluster/SRDF Table 4-2 Possible Number of Nodes in a Three Data Center Configuration Primary Data Center A (with Symmetrix) Primary Data Center B (with Symmetrix) Arbitrator Third Location (No Symmetrix) 6 6 2* 7 7 1 7 7 2* * Configurations with two arbitrators are preferred because they provide a greater degree of availability, especially in cases when a node is down due to a fa
Building a Metropolitan Cluster Using MetroCluster/SRDF Designing a Disaster Tolerant Architecture for use with MetroCluster/SRDF Arbitrator systems can be used to perform important and useful work such as: • Running mission-critical applications not protected by disaster recovery • IT/Operations or NetworkNodeManager • Backup • Application servers Calculating a Cluster Quorum When a cluster initially forms, all systems must be available to form the cluster (100% Quorum requirement).
Building a Metropolitan Cluster Using MetroCluster/SRDF Designing a Disaster Tolerant Architecture for use with MetroCluster/SRDF Example Failover Scenarios with One Arbitrator Taking a node off-line for planned maintenance is treated the same as a node failure in these scenarios. Study these scenarios to make sure you do not put your cluster at risk during planned maintenance.
Building a Metropolitan Cluster Using MetroCluster/SRDF Designing a Disaster Tolerant Architecture for use with MetroCluster/SRDF * Cluster can be manually started with the remaining node. Table 4-4 illustrates possible results if a data center fails in a configuration with a single arbitrator.
Building a Metropolitan Cluster Using MetroCluster/SRDF Designing a Disaster Tolerant Architecture for use with MetroCluster/SRDF Example Failover Scenarios with Two Arbitrators Having two arbitrator nodes adds extra protection during nodes failures and allows you to do planned maintenance on arbitrator nodes without losing the cluster should a disaster occur.
Building a Metropolitan Cluster Using MetroCluster/SRDF Designing a Disaster Tolerant Architecture for use with MetroCluster/SRDF Table 4-5 Data Center Failure Scenarios with Two Arbitrators (Continued) Node Failure Quorum Result arbitrator 1, then data center A 3 of 5 (60%) pkg A and B switch to data center B node 3, then data center A 3 of 5 (60%) pkg A and B switch to data center B data center B 4 of 6 (67%) pkg C and D switch to data center A third location 4 of 6 (67%) nothing * Cluster
Building a Metropolitan Cluster Using MetroCluster/SRDF Designing a Disaster Tolerant Architecture for use with MetroCluster/SRDF Disaster Tolerant Checklist Use this checklist to make sure you have adhered to the disaster tolerant architecture guidelines for a two main data center and third location configuration. Figure 4-4 Disaster Tolerant Checklist Each data center must have the same number of nodes to maintain a proper quorum in case an entire data center fails.
Building a Metropolitan Cluster Using MetroCluster/SRDF Designing a Disaster Tolerant Architecture for use with MetroCluster/SRDF Cluster Configuration Worksheet Use this cluster configuration worksheet either in place of, or in addition to the worksheet provided in the Managing MC/ServiceGuard manual. If you have already completed an MC/ServiceGuard cluster configuration worksheet, you only need to complete the first part of this worksheet.
Building a Metropolitan Cluster Using MetroCluster/SRDF Designing a Disaster Tolerant Architecture for use with MetroCluster/SRDF Package Configuration Worksheet Use this package configuration worksheet either in place of, or in addition to the worksheet provided in the Managing MC/ServiceGuard manual. If you have already completed an MC/ServiceGuard cluster configuration worksheet, you only need to complete the first part of this worksheet.
Building a Metropolitan Cluster Using MetroCluster/SRDF Designing a Disaster Tolerant Architecture for use with MetroCluster/SRDF NOTE It is recommended to use the same Symmetric Device Group Name on all nodes.
Building a Metropolitan Cluster Using MetroCluster/SRDF Preparing a Cluster for MetroCluster/SRDF Preparing a Cluster for MetroCluster/SRDF When the following procedures are completed, an adoptive node will be able to access the data belonging to a package after it fails over. Use the convenience scripts in the /opt/cmcluster/toolkits/SGSRDF/Samples to automate some of the tasks in the following sections: mk3symgrps.nodename — to create EMC Symmetrix device groups. mk4gatekpr.
Building a Metropolitan Cluster Using MetroCluster/SRDF Preparing a Cluster for MetroCluster/SRDF See the SymCLI manual for instructions on creating the appropriate pseudo device files. • Figure 4-8 R1 and R2 devices must have been correctly defined and assigned to the appropriate nodes in the internal configurations that is downloaded by EMC support staff.
Building a Metropolitan Cluster Using MetroCluster/SRDF Preparing a Cluster for MetroCluster/SRDF • Only Synchronous Mode is supported; Adaptive Copy must be disabled. • Domino Mode is recommended to ensure data currency on all Symmetrix frames and that there is no possibility of inconsistent data at the R2 side in case of SRDF links failure.
Building a Metropolitan Cluster Using MetroCluster/SRDF Preparing a Cluster for MetroCluster/SRDF Setting up Hardware for M by N Configurations You can configure up to four Symmetrix disk arrays in the following combinations: WARNING 142 • An array in Data Center A connected to one array in Data Center B. • An array in Data Center A connected to two arrays in Data Center B. • Two arrays in Data Center A connected to an array in Data Center B.
Building a Metropolitan Cluster Using MetroCluster/SRDF Preparing a Cluster for MetroCluster/SRDF Figure 4-9 shows a 2 by 1 configuration with BCVs. Figure 4-9 2 by 1 Node and Data Center Configuration Data Center A Data Center B R2 vols R1 vols node3 node1 pkg A node4 SRDF Links BCVs Third Location (Arbitrators) node5 pkg B node6 node2 R1 vols The figure indicates R1 volumes at Data Center A and R2 volumes and BCVs at Data Center B for pkg A and pkg B.
Building a Metropolitan Cluster Using MetroCluster/SRDF Preparing a Cluster for MetroCluster/SRDF Figure 4-10 shows a 2 by 2 configuration with R1 volumes for pkg A and pkg B on the Symmetrix frames located in Data Center A and R2 volumes and BCVs at Data Center B. Many of the examples given later in this chapter are based on this configuration.
Building a Metropolitan Cluster Using MetroCluster/SRDF Preparing a Cluster for MetroCluster/SRDF at Data Center A, and R2 volumes are at Data Center B. R1 volumes for pkg C and pkg D are at Data Center B, and R2 volumes are at Data Center A.
Building a Metropolitan Cluster Using MetroCluster/SRDF Preparing a Cluster for MetroCluster/SRDF due to a failure of a remote device or an RDF link, the data flow to the other Symmetrix will be halted in less than one second. Once mirroring is resumed, any updates will propagate with normal SRDF operation. Figure 4-12 shows that when there is a break in the links between two of the Symmetrix frames, the use of consistency groups (dashed oval lines) ensures that the other two links are also suspended.
Building a Metropolitan Cluster Using MetroCluster/SRDF Preparing a Cluster for MetroCluster/SRDF Building the Symmetrix CLI Database The Symmetrix CLI (Command Line Interface) should be installed on all nodes running packages that use data on the EMC Symmetrix disk arrays. Create the SymCLI database on each system using the following steps. For complete information, refer to the Symmetrix SymCLI manual.
Building a Metropolitan Cluster Using MetroCluster/SRDF Preparing a Cluster for MetroCluster/SRDF 1. Obtain a list of data for the Symmetrix devices available, using the following command on each node without any options: # syminq Sample output from both the R1 and R2 sides is shown in Figure 4-13 and Figure 4-14.
Building a Metropolitan Cluster Using MetroCluster/SRDF Preparing a Cluster for MetroCluster/SRDF 2. You need the following information from these listings for each Symmetrix logical device: • HP-UX device file name (e.g. /dev/rdsk/c3t3d2). • device type (R1, R2, BCV, GK, or blank) • Symmetrix serial number (e.g. 50006161), useful in matching the HP-UX device names to the actual devices in the Symmetrix configuration downloaded by EMC support staff. This number is further explained in Figure 4-15.
Building a Metropolitan Cluster Using MetroCluster/SRDF Preparing a Cluster for MetroCluster/SRDF 3. Use the symrdf command on each Symmetrix disk array (that is, from both the R1 and the R2 side) to pair the logical device names for the R1 and R2 sides of each SRDF link: # symrdf list Sample output is shown in Figure 4-16 and Figure 4-17.
Building a Metropolitan Cluster Using MetroCluster/SRDF Preparing a Cluster for MetroCluster/SRDF 4. Match the logical device numbers in the symrdf listings with the HP-UX device file names in the output from the syminq command to see which devices are seen from each node to make sure that this node can see all necessary devices. Use the Symmetrix ID to determine which Symmetrix array is connected to the node.
Building a Metropolitan Cluster Using MetroCluster/SRDF Preparing a Cluster for MetroCluster/SRDF Table 4-6 Symmetrix Device and HP-UX Device Correlation (Continued) Symmetrix ID, device #, and type NOTE Type R1 ID 95 Dev# 040 Type GK ID 50 Dev# 041 Type GK ID 95 Dev# 028 Type BCV Node 1 /dev/rdsk device file name Node 2 /dev/rdsk device file name Nodes 4 /dev/rdsk device file name Node 3 /dev/rdsk device file name c0t15d0 c0t15d0 c3t15d1 c5t15d1 c4t3d2 c4t3d2 n/a n/a The Symm
Building a Metropolitan Cluster Using MetroCluster/SRDF Setting up 1 by 1 Configurations Setting up 1 by 1 Configurations The most common Symmetrix configuration used with MetroCluster/SRDF is a 1 by 1 configuration in which there is a single Symmetrix frame at each Data Center. This section describes how to set up this configuration using SymCLI and HP-UX commands.
Building a Metropolitan Cluster Using MetroCluster/SRDF Setting up 1 by 1 Configurations 1. Use the symdg command, or modify the mk3symgrps.nodename script to define an R1 and an R2 device group for each package: # symdg create -type RDF1 devgroupname Issue this command on nodes attached to the R1 side. # symdg create -type RDF2 devgroupname Issue this command on nodes attached to the R2 side. The group name must be the same on each node on the R1 and R2 side.
Building a Metropolitan Cluster Using MetroCluster/SRDF Setting up 1 by 1 Configurations • Symmetrix device group name (arbitrary, but unique name may be chosen for each group that defines all of the volume groups (VGs) belonging to a particular MC/ServiceGuard package). • keyword RDF1 or RDF2 Configuring Gatekeeper Devices NOTE The sample scripts mk4gatekpr.nodename can be modified to automate these steps.
Building a Metropolitan Cluster Using MetroCluster/SRDF Setting up 1 by 1 Configurations Verifying the EMC Symmetrix Configuration When you are finished with all these steps, use the symrdf list command to get a listing of all devices and their states. You may want to back up the SymCLI database on each node so that you do not have to do these configuration steps again if a failure corrupts the database. The SymCLI database is a binary file in the directory /var/symapi/db.
Building a Metropolitan Cluster Using MetroCluster/SRDF Setting up 1 by 1 Configurations 1. Import the VGs on all of the other systems that might run the MC/ServiceGuard package and backup the LVM configuration. Make sure that you split the logical SRDF links before importing the VGs, especially if you are importing the VGs on the R2 side. Use the commands: # symrdf -g devgrpname split -v # vgimport -v -s -m mapfilename vgname 2. Back up the configuration.
Building a Metropolitan Cluster Using MetroCluster/SRDF Setting up M by N Configurations Setting up M by N Configurations Metropolitan clusters using EMC/SRDF can be built in configurations that use more than two EMC Symmetrix disk arrays. In such configurations, M arrays located in Data Center A may be connected to N arrays located in Data Center B. This section describes how to set up this configuration using SymCLI and HP-UX commands.
Building a Metropolitan Cluster Using MetroCluster/SRDF Setting up M by N Configurations Figure 4-19 shows a 2 by 2 configuration. Data in this figure are used in the example commands given in the following sections. The example shows R1 devices at one data center and R2 devices with Business Continuity Volumes (BCVs) at the other, but a bidirectional configuration is also possible, with R1 devices on both sites.
Building a Metropolitan Cluster Using MetroCluster/SRDF Setting up M by N Configurations The following examples are based on the 2 by 2 configuration shown in Figure 4-19. Create device groups using the following commands on each node on the R1 side: # symdg -type RDF1 create dgoraA # symdg -type RDF1 create dgoraB For each node on the R2 side (node3 and node4), create the device groups as follows. Note that you have to create two device groups because device groups do not span frames.
Building a Metropolitan Cluster Using MetroCluster/SRDF Setting up M by N Configurations To manage the BCV devices from the R1 side, you need to associate the BCV devices with the device groups that are configured on the R1 side.
Building a Metropolitan Cluster Using MetroCluster/SRDF Setting up M by N Configurations Defining the Consistency Groups To configure consistency groups for use with MetroCluster/SRDF, you first create device groups and gatekeeper groups as described in previous sections. Then you create the consistency groups and add the devices to them as well. NOTE Each package requires its own unique consistency group name. Use the following steps for each package: 1.
Building a Metropolitan Cluster Using MetroCluster/SRDF Setting up M by N Configurations # symcg -g cgoradb enable 4. Establish the BCV devices in the secondary Symmetrix as a mirror of the standard device.
Building a Metropolitan Cluster Using MetroCluster/SRDF Setting up M by N Configurations 4. Create the logical volumes. (XXXX indicates size in MB) # lvcreate -L XXXX /dev/vgoraA # lvcreate -L XXXX /dev/vgoraB 5. Install a VxFS file system on the logical volumes: # newfs -F vxfs /dev/vgoraA/rlvol1 # newfs -F vxfs /dev/vgoraB/rlvol1 6. Create map files to permit exporting the volume groups to other systems: # vgchange -a n vgoraA # vgchange -a n vgoraB # vgexport -v -s -p -m /tmp/vgoraA.
Building a Metropolitan Cluster Using MetroCluster/SRDF Setting up M by N Configurations Creating VxVM Disk Groups for Use with MetroCluster/SRDF If you are using VERITAS storage, use the following procedure to create disk groups. It is assumed that you have already created a VERITAS root disk (rootdg) on the system where you are configuring the storage. The following section shows how to set up VERITAS disk groups. On one node do the following: 1.
Building a Metropolitan Cluster Using MetroCluster/SRDF Setting up M by N Configurations # umount /logs Validating VxVM Disk Groups for Use with MetroCluster/SRDF The following section shows how to validate VERITAS diskgroups. On one node do the following: 1. Deport the disk group. # vxdg deport logdata 2. Enable other cluster nodes to have access to the disk group. # vxdctl enable 3. Split the SRDF link to enable R2 Read/Write permission. # symrdf -g dgoraA split # symrdf -g dgoraB split 4.
Building a Metropolitan Cluster Using MetroCluster/SRDF Configuring MC/ServiceGuard Packages for Automatic Disaster Recovery Configuring MC/ServiceGuard Packages for Automatic Disaster Recovery Before you can implement these procedures you must: • Configure your cluster hardware according to disaster tolerant architecture guidelines outlined in “Designing a Disaster Tolerant Architecture for use with MetroCluster/SRDF” on page 126.
Building a Metropolitan Cluster Using MetroCluster/SRDF Configuring MC/ServiceGuard Packages for Automatic Disaster Recovery # cd /etc/cmcluster/pkgname # cmmakepkg -p pkgname.ascii Customize the package configuration file as appropriate to your application. Be sure to include the pathname of the control script (/etc/cmcluster/pkgname/pkgname.cntl) for the RUN_SCRIPT and HALT_SCRIPT parameters. 3. In the .ascii file, list the node names in the order in which you want the package to fail over.
Building a Metropolitan Cluster Using MetroCluster/SRDF Configuring MC/ServiceGuard Packages for Automatic Disaster Recovery 5. Add customer-defined run and halt commands in the appropriate places according to the needs of the application. See Managing MC/ServiceGuard for more information on these functions. 6. In the package_name.ascii file, list the node names in the order in which you want the package to fail over.
Building a Metropolitan Cluster Using MetroCluster/SRDF Configuring MC/ServiceGuard Packages for Automatic Disaster Recovery the first package. For other packages RETRYTIME should be altered to avoid contention when more than one package is starting on a node. RETRY * RETRYTIME should be approximately five minutes to keep package startup time under 5 minutes. RETRYTIM E RETRY pkgA 60 seconds 5 attempts pkgB 43 seconds 7 attempts pkgC 33 seconds 9 attempts f.
Building a Metropolitan Cluster Using MetroCluster/SRDF Configuring MC/ServiceGuard Packages for Automatic Disaster Recovery other files Any other scripts you use to manage MC/ServiceGuard packages The MC/ServiceGuard cluster is ready to automatically switch packages to nodes in remote data centers using MetroClusters/SRDF 11. Check the configuration using the cmcheckconf -P package_name.ascii, then apply the MC/ServiceGuard configuration using the cmapplyconf -P package_name.ascii command or SAM. 12.
Building a Metropolitan Cluster Using MetroCluster/SRDF Maintaining a Cluster that Uses MetroCluster/SRDF Maintaining a Cluster that Uses MetroCluster/SRDF While the cluster is running, all EMC Symmetrix disk arrays that belong to the same MC/ServiceGuard package, and are defined in a single SRDF group must be in the same state at the same time. Manual changes of these states can cause the package to halt due to unexpected conditions.
Building a Metropolitan Cluster Using MetroCluster/SRDF Managing Business Continuity Volumes Managing Business Continuity Volumes The use of Business Continuity Volumes is recommended with all implementations of MetroCluster/SRDF, and it is required with M by N configurations, which employ consistency groups. These BCV devices will provide a good copy of the data when it is necessary to recover from a rolling disaster—a second failure that occurs while we are attempting to recover from the first failure.
Building a Metropolitan Cluster Using MetroCluster/SRDF Managing Business Continuity Volumes 3. Use the following command to monitor the status of the RDF pair state between the R1 and R2 devices: # symrdf -cg cgoradb query Once the column with the “RDF Pair STATE” shows the state as “Synchronized” for all the devices, you can proceed to the next step. 4. Re-establish the BCV devices in the secondary Symmetrix as a mirror of the standard device.
Building a Metropolitan Cluster Using MetroCluster/SRDF R1/R2 Swapping R1/R2 Swapping This section shows how the R1/R2 swapping can be done via MetroCluster/SRDF package and manual procedures. Each of these allow you to swap the SRDF personality for each device designations of a specified devicegroup, where each source R1 device(s) becomes a target R2 device(s) and a target R1 device(s) becomes a source R1 device(s).
Building a Metropolitan Cluster Using MetroCluster/SRDF R1/R2 Swapping 1. Swap the personalities of the devices and mark the old R1 devices to be refresh from the old R2 devices. # symrdf -g swap -refresh R1 2. After swapping is completed, the devices will be in Suspended state. Next establish the device group for data replication from the new R1 devices to the new R2 devices.
Building a Continental Cluster 5 Building a Continental Cluster Unlike metropolitan and campus clusters, which have a single-cluster architecture, a continental cluster uses multiple MC/ServiceGuard clusters to provide application recovery over wide areas. Using the ContinentalClusters product, two independently functioning clusters are set up in such a way that in the event of a disaster, one cluster can take over the critical operations formerly carried out by the other cluster.
Building a Continental Cluster Oracle Standby Database is available from your HP representative. Also see the “White Papers” area on web page http://docs.hp.com/hpux/ha.
Building a Continental Cluster Understanding Continental Cluster Concepts Understanding Continental Cluster Concepts The ContinentalClusters product provides the ability to monitor a high availability cluster and fail over mission critical applications to another cluster if the monitored cluster should become unavailable.
Building a Continental Cluster Understanding Continental Cluster Concepts Two packages are running on the cluster in Los Angeles, and their data is replicated to the cluster in New York. Physical data replication is carried out using ESCON (Enterprise Storage Connect) links between the disk array hardware in New York and Los Angeles via an ESCON/WAN converter at each end. The New York cluster is running a monitor that checks the status of the Los Angeles cluster.
Building a Continental Cluster Understanding Continental Cluster Concepts In the above figure, the salespkg is running on the New York cluster and can be recovered by the Los Angeles cluster. Similarly, the custpkg running on the Los Angeles cluster can be recovered by the New York cluster. As stated previously, physical data replication is carried out using ESCON (Enterprise Storage Connect) links between the disk array hardware in New York and Los Angeles via an ESCON/WAN converter at each end.
Building a Continental Cluster Understanding Continental Cluster Concepts Application Recovery in a Continental Cluster If a given cluster of a continental cluster should become unavailable, ContinentalClusters allows an administrator to issue a single command (cmrecovercl, described later) to transfer mission critical applications from the that cluster to another cluster, making sure that the packages do not run on both clusters at the same time.
Building a Continental Cluster Understanding Continental Cluster Concepts • You verify that the monitored cluster has failed. • You issue the cluster recovery command. Monitoring over a Wide Area Network A monitor package running on one cluster tracks the health another cluster and sends notification to system administrators if the state of the monitored cluster changes. (If a cluster contains any recovery packages it must be monitored.
Building a Continental Cluster Understanding Continental Cluster Concepts with regard to both the monitored cluster and the WAN. It is clear that in many cases, the causes of cluster events are indeterminate without additional information that is not available to the software.
Building a Continental Cluster Understanding Continental Cluster Concepts Table 5-1 Monitored States and Possible Causes (Continued) Cluster Event (Old state -> New state) NOTE Cluster-related causes WAN-related causes Unreachable -> Up Cluster nodes were rebooted and the cluster started WAN came up and the cluster was already running Error -> Up Error resolved, cluster is up WAN problem was fixed, cluster is up There is only one condition under which cmclsentryd will determine that the cluster h
Building a Continental Cluster Understanding Continental Cluster Concepts How Notifications Work A central part of the operation of ContinentalClusters is the transmission of notifications following the detection of a cluster event. Notifications occur at specifically coded times, and at two different levels: • Alert—when a cluster event should be considered noteworthy. • Alarm—when an event shows evidence of a cluster failure.
Building a Continental Cluster Understanding Continental Cluster Concepts The expected process in dealing with alerts is to continue watching for additional notifications and to contact individuals at the site of the monitored cluster to see whether problems exist. Alarms Alarms are intended to indicate that a cluster failure might have taken place.
Building a Continental Cluster Understanding Continental Cluster Concepts Performing Cluster Recovery When a CLUSTER_ALARM is issued, there may be a need for recovery, and the recovery command, cmrecovercl, is enabled for use by the root user. Cluster recovery is carried out at the site of the recovery cluster by using the cmrecovercl command, as follows: # cmrecovercl This command will fail if a cluster alarm has not been issued.
Building a Continental Cluster Understanding Continental Cluster Concepts • Startup and Switching Characteristics • Network Attributes Startup and Switching Characteristics Normally, an application (package) can run on only one node at a time in a cluster. However, in a continental cluster, there are two clusters in which an application—the primary package or the recovery package—could operate on the same data.
Building a Continental Cluster Understanding Continental Cluster Concepts Network Attributes Another important difference between the packages in a continental cluster and the packages configured in a standard ServiceGuard cluster is that different subnets are used in recovery packages than the subnets in the primary packages. The client application must be designed to reconnect to the appropriate IP address following a recovery operation.
Building a Continental Cluster Understanding Continental Cluster Concepts Table 5-2 ServiceGuard and ContinentalClusters Commands How the commands work in SG How the commands work in ContinentalClusters cmrunpkg runs a package will not start a recovery package if any of the primary, data receiver, or data sender package in the same recovery group is running or enabled.
Building a Continental Cluster Understanding Continental Cluster Concepts Table 5-2 ServiceGuard and ContinentalClusters Commands (Continued) How the commands work in SG How the commands work in ContinentalClusters cmhaltnode -f halts a node in a highly available cluster will not re-enable switching on a recovery package if any of the primary, data receiver, or data sender package in the same recovery group is running or enabled.
Building a Continental Cluster Designing a Disaster Tolerant Architecture for use with ContinentalClusters Designing a Disaster Tolerant Architecture for use with ContinentalClusters The ContinentalClusters product operates as a configuration of two MC/ServiceGuard clusters, which can run a package on a cluster and a Recovery Cluster.
Building a Continental Cluster Designing a Disaster Tolerant Architecture for use with ContinentalClusters many nodes as are permitted in an ordinary MC/ServiceGuard cluster, and each may be running packages that are not configured to fail over between clusters. NOTE Remember that when cluster A takes over for cluster B, it must run cluster B’s packages as well as any packages that it was already running on its own, unless you choose to stop those packages.
Building a Continental Cluster Designing a Disaster Tolerant Architecture for use with ContinentalClusters Table 5-3 • ContinentalClusters product is only responsible for the following: ContinentalClusters configuration and management commands, the monitoring of remote cluster status, and the notification of remote cluster events. • ContinentalClusters product provides a single recovery command to start all recovery packages that are configured in the ContinentalClusters configuration file.
Building a Continental Cluster Designing a Disaster Tolerant Architecture for use with ContinentalClusters Table 5-3 Data Replication and ContinentalClusters (Continued) Replication Type Physical Replication of Disk Units via Hardware How it Works Replication of the LUNs within a disk array through dedicated hardware links such as EMC SRDF or Continuous Access XP.
Building a Continental Cluster Designing a Disaster Tolerant Architecture for use with ContinentalClusters Both of these templates can be purchased separately with the product MetroCluster/CA or MetroCluster/SRDF. Details on configuring the special ContinentalClusters control scripts are in Chapters 6 and 7. Some additional notes are provided below. Highly Available Wide Area Networking Disaster tolerant networking for ContinentalClusters is directly tied to the data replication method.
Building a Continental Cluster Designing a Disaster Tolerant Architecture for use with ContinentalClusters ContinentalClusters Worksheets Planning is an essential effort in creating a robust continental cluster environment. It is recommended that you record the details of your configuration on planning worksheets. These worksheets can be filled in partially before configuration begins, and then completed as you build the continental cluster.
Building a Continental Cluster Designing a Disaster Tolerant Architecture for use with ContinentalClusters Node Names: ______________________________________________________ Monitor Package Name: __ccmonpkg__________________________________ Monitor Interval: __60 seconds____________________________________ ========================================================== ============= Recovery Data Center Information: Recovery Cluster Name: ___________________________________________ Data Center Name and Location
Building a Continental Cluster Designing a Disaster Tolerant Architecture for use with ContinentalClusters ========================================================== ============= Continental Cluster Name: _____________________________________________ ========================================================== ============= Recovery Group Data: Recovery Group Name: _____________________________________________ Primary Cluster/Package Name:_____________________________________ Data Sender Cluster/Package Nam
Building a Continental Cluster Designing a Disaster Tolerant Architecture for use with ContinentalClusters _____________________________________________ Primary Cluster/Package Name:_____________________________________ Data Sender Cluster/Package Name:_________________________________ Recovery Cluster/Package Name:____________________________________ Data Receiver Cluster/Package Name:_______________________________ Cluster Event Worksheet The following worksheet will help you organize and record the clu
Building a Continental Cluster Designing a Disaster Tolerant Architecture for use with ContinentalClusters Alarm Interval:___________________________________________________ Notification:________________________________________ _____________ Notification:________________________________________ _____________ Notification:________________________________________ _____________ DOWN: Alert Interval:___________________________________________________ Notification:________________________________________ _____
Building a Continental Cluster Designing a Disaster Tolerant Architecture for use with ContinentalClusters Notification:________________________________________ _____________ Chapter 5 203
Building a Continental Cluster Preparing the Clusters Preparing the Clusters The steps for configuring the clusters needed by ContinentalClusters are as follows: • Set up and test data replication between the sites. • Configure each cluster for MC/ServiceGuard operation. Setting up and Testing Data Replication Depending on which data replication method you choose, it can take a week or more to set up and test a data replication method.
Building a Continental Cluster Preparing the Clusters Table 5-4 shows the types of packages that are needed for each type of data replication.
Building a Continental Cluster Preparing the Clusters 5. Configure and test each primary package according to the instructions in chapters 6 and 7 of Managing MC/ServiceGuard manual. Use the cmapplyconf command to apply the package configuration. 6. Be sure that AUTO_RUN (PKG_SWITCHING_ENABLED) is set to NO in the package ASCII configuration file for any package that is in a recovery group, and therefore might at some time be a candidate for recovery.
Building a Continental Cluster Preparing the Clusters The primary cluster is shown in Figure 5-4. Figure 5-4 Sample Local Cluster Configuration Los Angeles Cluster LAnode1 salespkg WAN LAnode2 custpkg Highly Available Network Configuring a Cluster with Recovery Packages Use the following steps and the instructions in chapters 4 through 7 of Managing MC/ServiceGuard manual as guidelines for creating a new Recovery Cluster or preparing an existing cluster to run in a ContinentalClusters environment: 1.
Building a Continental Cluster Preparing the Clusters 2. For new clusters, install minimum required versions of HP-UX and MC/ServiceGuard. For existing clusters, perform a rolling upgrade to the minimum required versions of HP-UX and MC/ServiceGuard if necessary. Coordinate with the other site to make sure the same versions and patches are installed at both sites. This may include coordinating between HP support personnel if the sites have separate support contracts. 3.
Building a Continental Cluster Preparing the Clusters c. NOTE • Package services • Failfast settings Modify the package control script (salespkg_bak.cntl), checking for anything that may be different between clusters: • Volume groups (VGs) may be different. • IP addresses will be different. • Site-specific customer-defined routines (for example routines that send messages to a local administrator) may be different. • Control script files must be executable.
Building a Continental Cluster Preparing the Clusters The New York cluster is shown in Figure 5-5.
Building a Continental Cluster Building the ContinentalClusters Configuration Building the ContinentalClusters Configuration If necessary, use the swinstall command to install the ContinentalClusters product on all nodes in both clusters. Then create the ContinentalClusters configuration using the following steps (each step is described in detail in the sections that follow): • Prepare the security files. • Create the monitor package on each cluster containing a recovery package.
Building a Continental Cluster Building the ContinentalClusters Configuration After the cmapplyconcl command has been run successfully, you can remove this entry from the /.rhosts file if you wish. Remember, however, that the entry must be present in the /.rhosts file when you use cmapplyconcl at a later time. NOTE The cmclnodelist file does not provide the required type of access for the cmapplyconcl command. You must also create the /etc/opt/cmom/cmomhosts file on all nodes.
Building a Continental Cluster Building the ContinentalClusters Configuration classless interdomain routing, a type of routing supported by the Border Gateway protocol (BGP). allow from lists all the nodes that are allowed access. Permissible entries are: all All hosts are allowed access. domain Hosts whose names match, or end in, this string are allowed access, e.g. hp.com. hostname The named host (for example, kitcat.myco.com) is allowed access.
Building a Continental Cluster Building the ContinentalClusters Configuration Creating the Monitor Package The ContinentalClusters monitoring software is configured as an MC/ServiceGuard package so that it remains highly available. The following steps should be carried out on the recovery cluster and repeated on the primary cluster if you want to monitor the recovery site from the primary site: 1.
Building a Continental Cluster Building the ContinentalClusters Configuration 5. Use the cmapplyconf command to add the package to the MC/ServiceGuard configuration: # cmapplyconf -P ccmonpkg.config 6. Copy the package control script ccmonpkg.cntl to the monitor package directory (default name /etc/cmcluster/ccmonpkg) on all the other nodes in the cluster. Make sure this file is executable.
Building a Continental Cluster Building the ContinentalClusters Configuration # cmqueryconcl -C cmconcl.config This file has three editable sections: • Cluster information • Recovery groups • Monitoring definitions Customize each section according to your needs. The following are some guidelines for editing each section. Editing Section 1—Cluster Information Enter cluster-level information as follows in this section of the file: 1.
Building a Continental Cluster Building the ContinentalClusters Configuration comment out the MONITOR_PACKAGE_NAME line and the MONITOR_INTERVAL line. For mutual recovery, create the monitor package on both the first and second clusters. NOTE Monitoring of a cluster not containing recovery packages is optional. For example, you might set up monitoring of such a cluster so you can check the status of the data replication technology being used. 5. Repeat steps 2 through 4 for the alternate cluster.
Building a Continental Cluster Building the ContinentalClusters Configuration #### #### For complete details about how to set the parameters in #### #### this file, consult the cmqueryconcl(1m) manpage or your manual. #### #### #### ############################################################## ################# #### #### #### Section 1. Cluster Information #### #### #### #### This section contains the name of the continental cluster #### #### followed by the names of member clusters and all their nodes.
Building a Continental Cluster Building the ContinentalClusters Configuration use the #### #### same name for the monitoring package on all clusters; #### #### "ccmonpkg" is suggested. Monitoring of the recovery cluster #### #### by the primary cluster is optional. If you do not wish to #### #### monitor the recovery cluster, you must delete or comment out the #### #### MONITOR_PACKAGE_NAME and MONITOR_INTERVAL lines that follow the #### #### name of the primary cluster.
Building a Continental Cluster Building the ContinentalClusters Configuration #### #### #### #### #### #### #### #### #### SECONDS #### #### CLUSTER_DOMAIN #### NODE_NAME eastnet.myco.
Building a Continental Cluster Building the ContinentalClusters Configuration Examples of recovery groups are shown graphically in Figure 5-6 and Figure 5-7. Figure 5-6 Sample ContinentalClusters Recovery Groups New York Cluster Recovery Group for Sales Application: RECOVERY_GROUP_NAME PRIMARY_PACKAGE RECOVERY_PACKAGE NYnode1 Sales LAcluster/salespkg NYcluster/salespkg_bak Los Angeles Cluster LAnode1 LAnode2 custpkg.config custpkg.cntl Chapter 5 salespkg_bak.config custpkg_bak.conf salespkg_bak.
Building a Continental Cluster Building the ContinentalClusters Configuration Figure 5-7 Sample Bi-directional Recovery Groups New York Cluster Recovery Group for Sales Application: RECOVERY_GROUP_NAME PRIMARY_PACKAGE RECOVERY_PACKAGE Sales LAcluster/salespkg NYcluster/salespkg_bak NYnode1 NYnode2 salespkg_bak.config custpkg.config salespkg_bak.cntl WAN Los Angeles Cluster LAnode1 LAnode2 salespkg.config salespkg.cntl custpkg.cntl custpkg_bak.conf custpkg_bak.
Building a Continental Cluster Building the ContinentalClusters Configuration 2. After the PRIMARY_PACKAGE keyword, enter a primary package definition consisting of the cluster name followed by a slash (/) followed by the package name. Example: PRIMARY_PACKAGE LAcluster/custpkg 3. Optionally, enter a data sender package definition consisting of the cluster name, a slash (/), and the data sender package name after the DATA_SENDER_PACKAGE keyword.
Building a Continental Cluster Building the ContinentalClusters Configuration #### for each ServiceGuard package that will be started on the #### #### recovery cluster when the cmrecovercl(1m) command is issued. #### #### #### #### A recovery group consists of a primary package running on #### #### one cluster, and a recovery package that is ready to run on a #### #### different cluster.
Building a Continental Cluster Building the ContinentalClusters Configuration #### Enter the name of each package recovery group together with #### #### the fully qualified names of the primary and recovery #### #### packages. If appropriate, enter the fully qualified name #### #### of a data receiver package. Note that the data receiver #### #### package must be on the same cluster as the recovery package.
Building a Continental Cluster Building the ContinentalClusters Configuration #### DATA_RECEIVER_PACKAGE eastcoast/nfsreceiverpkg #### #### #### #### RECOVERY_GROUP_NAME hpgroup #### #### PRIMARY_PACKAGE westcoast/hppkg #### #### DATA_SENDER_PACKAGE westcoast/hpsenderpkg #### #### RECOVERY_PACKAGE eastcoast/hpbackuppkg #### #### DATA_RECEIVER_PACKAGE eastcoast/hpreceiverpkg #### #### #### RECOVERY_GROUP_NAME PRIMARY_PACKAGE # DATA_SENDER_PACKAGE RECOVERY_PACKAGE # DATA_RECEIVER_PACKAGE Editing Section 3—M
Building a Continental Cluster Building the ContinentalClusters Configuration comments in the file excerpt below for a list of destination types). Note that the message text in the notification must be on a separate line in the file. 3. If the event is for a cluster in an Unreachable condition, define a CLUSTER_ALARM at appropriate times.
Building a Continental Cluster Building the ContinentalClusters Configuration #### #### A cluster event takes place when a monitor that is located on #### #### one cluster detects a significant change in the condition #### #### of another cluster. The monitored cluster conditions are: #### #### #### #### UNREACHABLE - the cluster is unreachable.
Building a Continental Cluster Building the ContinentalClusters Configuration Some events #### #### are noteworthy at the time they occur, and some are noteworthy #### #### when they persist over time. Setting the elapsed time to zero #### #### results in a message being sent as soon as the event takes place. #### #### Setting the elaspsed time to 5 minutes results in a message #### #### being sent when the condition has persisted for 5 minutes.
Building a Continental Cluster Building the ContinentalClusters Configuration string in #### #### a notification is entered in double quotes on a separate line; #### #### it can be no more than 170 characters long. Enter notifications #### #### in one of the following forms: #### #### #### #### NOTIFICATION CONSOLE #### #### #### #### Message written to the console.
Building a Continental Cluster Building the ContinentalClusters Configuration #### #### #### #### #### #### #### #### #### #### #### #### SYSLOG #### The message is sent as an SNMP trap. The value of may be 1 (normal), 2 (warning), 3 (minor), 4 (major), 5 (critical). NOTIFICATION #### #### #### A notice of the event is appended to the #### #### syslog file.
Building a Continental Cluster Building the ContinentalClusters Configuration #### #### Message is sent to a UDP port on the #### #### specified node. #### #### #### #### For the cluster event, enter a cluster name followed by #### #### a slash ("/") and a cluster condition (UP, DOWN, UNREACHABLE, #### #### ERROR) that may be detected by a monitor program. #### #### #### #### Each cluster event must be paired with a monitoring cluster.
Building a Continental Cluster Building the ContinentalClusters Configuration cmrecovercl(1m) #### #### command for immediate execution. #### #### #### #### NOTIFICATION #### #### #### #### A string value which is sent from the #### #### monitoring cluster for a given event #### #### to a specified destination.
Building a Continental Cluster Building the ContinentalClusters Configuration #### #### #### CLUSTER_ALERT 10 MINUTES #### #### NOTIFICATION EMAIL admin@primary.site #### #### "westcoast unreachable for 10 min. Call secondary site." #### #### NOTIFICATION EMAIL admin@secondary.site #### #### "Call primary admin. (555) 555-6666." #### #### NOTIFICATION CONSOLE #### #### "Cluster ALERT: westcoast not responding." #### #### #### #### CLUSTER_ALARM 15 MINUTES #### #### NOTIFICATION EMAIL admin@primary.
Building a Continental Cluster Building the ContinentalClusters Configuration #### CLUSTER_EVENT westcoast/DOWN #### #### MONITORING_CLUSTER eastcoast #### #### CLUSTER_ALERT 0 MINUTES #### #### NOTIFICATION EMAIL admin@secondary.site #### #### "Cluster westcoast is down." #### #### #### #### CLUSTER_EVENT westcoast/ERROR #### #### MONITORING_CLUSTER eastcoast #### #### CLUSTER_ALERT 0 MINUTES #### #### NOTIFICATION EMAIL admin@secondary.site #### #### "Error in monitoring cluster westcoast.
Building a Continental Cluster Building the ContinentalClusters Configuration NOTIFICATION NOTIFICATION CLUSTER_EVENT /UP MONITORING_CLUSTER CLUSTER_ALERT NOTIFICATION CLUSTER_EVENT /ERROR MONITORING_CLUSTER CLUSTER_ALERT NOTIFICATION Selecting Notification Intervals The monitor interval determines the amount of time between distinct attempts by the monitor to obtain the status of a cluster.
Building a Continental Cluster Building the ContinentalClusters Configuration The following sequence could provide meaningful notifications, since a change of state is possible between notification intervals: MONITOR_PACKAGE_NAME ccmonpkg MONITOR_INTERVAL 1 MINUTE ...
Building a Continental Cluster Building the ContinentalClusters Configuration 3. Be sure to make a backup copy of the configuration ascii file and save it on the other cluster after it is applied. NOTE Remember that the /.rhosts file on each node must contain an entry that allows write access. See “Preparing Security Files” on page 211 If any problems occur during the execution of cmapplyconcl, you can repeat the command as often as necessary.
Building a Continental Cluster Building the ContinentalClusters Configuration When configuration is finished, your systems should have sets of files similar to those shown in Figure 5-8. Figure 5-8 ContinentalClusters Configuration Files New York Cluster NYnode1 recovery package files salespkg_bak.config salespkg_bak.cntl custpkg_bak.config custpkg_bak.cntl contclust config file cmconcl.config contclust monitor pkg ccmonpkg.config ccmonpkg.cntl contclust config file cmconcl.
Building a Continental Cluster Building the ContinentalClusters Configuration Starting the ContinentalClusters Monitor Package Starting the monitoring package enables all ContinentalClusters functionality. Before you do this, ensure that the primary packages you wish to protect are running normally and that data sender and receiver packages, if they are being used for logical data replication, are working correctly. If you are using physical data replication, make sure that it is operational.
Building a Continental Cluster Building the ContinentalClusters Configuration Use the following steps to make sure the components are functioning correctly: 1. Use the following command to make sure all daemons are running: # ps -ef | grep cmcl Two important ContinentalClusters daemons are cmclsentryd and cmclrmond. 2. Check the cluster configuration on each cluster using the cmviewcl -v command. a. Ensure that each primary package is running correctly. b.
Building a Continental Cluster Building the ContinentalClusters Configuration CAUTION You should never issue the cmrunpkg command for a recovery package when ContinentalClusters is enabled, because there is no guaranteed way of preventing a package that is running on one cluster from running on the other cluster if the package is started using this command. The potential for data corruption is great.
Building a Continental Cluster Building the ContinentalClusters Configuration Documenting the Recovery Procedure Once everything is configured and the ContinentalClusters monitor is running, you must define your recovery procedure and train the administrators and operators at both sites. The checklist in Figure 5-9 is an example of how you might document the recovery procedure. Figure 5-9 Recovery Checklist Identify the level of alert that the monitoring site received.
Building a Continental Cluster Building the ContinentalClusters Configuration Reviewing the Recovery Procedure Using the checklist described in the previous section, step through the recovery procedure to make sure that all necessary steps are included. If possible, create simulated failures to test the alert and alarm scenarios coded in the ContinentalClusters configuration file.
Building a Continental Cluster Testing the Continental Cluster Testing the Continental Cluster This section presents some test procedures and scenarios. Some scenarios presume certain configurations that may not apply to all environments. Additionally, these tests do not eliminate the need to perform standard MC/ServiceGuard testing for each cluster individually. NOTE Data and system corruption can occur as a result of testing. System and data backups should always be done prior to testing.
Building a Continental Cluster Testing the Continental Cluster Testing ContinentalClusters Operations Use the following procedures to exercise typical ContinentalClusters behaviors. 1. Halt both clusters, then restart both clusters. The monitor packages on both clusters should start automatically. The ContinentalClusters packages (primary, data sender, data receiver, and recovery) should not start automatically. Any other packages may or may not start automatically, subject to their configuration.
Building a Continental Cluster Testing the Continental Cluster If physical data replication is used disconnect the physical replication links between the disk arrays: Powering off the disk array at the primary site Powering off the disk array at the recovery site • Testing cmrecovercl -f as well as cmrecovercl Depending on the condition, the primary packages should be running to test real life failures and recovery procedures. 4.
Building a Continental Cluster Switching to the Recovery Packages in Case of Disaster Switching to the Recovery Packages in Case of Disaster Once the clusters are configured and tested, packages will be able to fail over to an alternate node in another data center and still have access to the data they need to function. The primary steps for failing over a package are: 1. Receive notification that a monitored cluster is unavailable. 2. Verify that it is necessary and safe to start the recovery packages. 3.
Building a Continental Cluster Switching to the Recovery Packages in Case of Disaster alert or alarm, the timer is reset to 0 after each change of state; thus, the time to the alert or alarm will be the configured interval plus the time used by all the earlier state changes. NOTE The cmrecovercl command is fully enabled only after a CLUSTER_ALARM is issued; however, the command may be used with the -f option when a CLUSTER_ALERT has been issued.
Building a Continental Cluster Switching to the Recovery Packages in Case of Disaster • Halt the data sender packages running at the primary site if a database or software data replication scheme is used, and if the data sender package is configured for any ContinentalClusters recovery group. In addition, the recovery command, cmrecovercl,cannot be started until the data replicated from the primary site is properly applied to the database, data files, or devices at the recovery site.
Building a Continental Cluster Switching to the Recovery Packages in Case of Disaster # cmrecovercl -f This should only be used after positive confirmation from the remote site. If the monitored cluster comes back up following an alert or alarm, but you are certain that the primary packages cannot start (say, because of damage to the disks on the primary site), you need to use a special procedure to initiate recovery: 1. Use the cmhaltcl command to halt the primary cluster. 2.
Building a Continental Cluster Switching to the Recovery Packages in Case of Disaster Halting data receiver package nfsreceiverpkg on recovery cluster eastcoast Starting recovery package nfsbackuppkg on recovery cluster eastcoast enabling package nfsbackuppkg in cluster eastcoast ----------------exit status = 0 ----------------- The command cmrecovercl starts up all the recovery packages that are configured in the recovery groups.
Building a Continental Cluster Switching to the Recovery Packages in Case of Disaster change in the remote cluster’s state. The following table shows the status of ContinentalClusters packages after recovery has taken place, and applications are now running on the local cluster.
Building a Continental Cluster Switching to the Recovery Packages in Case of Disaster NOTE 254 If the remote cluster comes back up following a cluster event but the primary packages cannot run, halt the primary cluster with the cmhaltcl command, then issue cmrecovercl with the -f option.
Building a Continental Cluster Forcing a Package to Start Forcing a Package to Start The cmforceconcl command is used to force a ContinentalClusters package to start even if the status of a remote package in the recovery group is unknown. This command is used as a prefix to a cmrunpkg and cmmodpkg command. Under normal circumstances, ContinentalClusters will not allow a package to start in the recovery cluster unless it can determine that the package is not running in the primary cluster.
Building a Continental Cluster Restoring Disaster Tolerance Restoring Disaster Tolerance After a failover to a cluster occurs, restoring disaster tolerance has many challenges, the most significant of which are: • Restoring the failed cluster Depending on the nature of the disaster you may need to create a new cluster, or you may be able to restore the cluster. Steps for each scenario are discussed in the following sections.
Building a Continental Cluster Restoring Disaster Tolerance b. Halt the application on the surviving cluster if necessary, and start it on the new cluster. c. To keep application down time to a minimum, start the primary package on the cluster before resynchronizing the data of the next recovery group. 5.
Building a Continental Cluster Restoring Disaster Tolerance 4. Restart the monitor packages. Issue the following command on each cluster: # cmmodpkg -e ccmonpkg 5. View the status of the ContinentalCluster. # cmmviewconcl Before applying the edited configuration, the data storage associated with each cluster needs to be prepared to match the new role. In addition, the data replication direction needs to be changed to mirror data from the new primary cluster to the new recovery cluster.
Building a Continental Cluster Restoring Disaster Tolerance -c oldPrimaryClusterName \ [-a] [-F NewContinentalClustersConfigFileName] The above command switches the roles of the primary and recovery packages of the ContinentalClusters recovery groups for which “oldPrimaryClusterName” is defined as the primary cluster.
Building a Continental Cluster Restoring Disaster Tolerance CLUSTER_DOMAIN cup.hp.com NODE_NAME node1 NODE_NAME node2 MONITOR_PACKAGE_NAME ccmonpkg CLUSTER_NAME ClusterB CLUSTER_DOMAIN cup.hp.com NODE_NAME node3 NODE_NAME node4 MONITOR_PACKAGE_NAME ccmonpkg MONITOR_INTERVAL ### Section 2.
Building a Continental Cluster Restoring Disaster Tolerance “CC alert: DOWN” NOTIFICATION SYSLOG “CC alert: DOWN” CLUSTER_ALARM NOTIFICATION 90 SECONDS TEXTLOG /home/users/logs/events.log “CC alarm: DOWN” NOTIFICATION SYSLOG “CC alarm: DOWN” sample.output ### Section 1. Cluster Information CONTINENTAL_CLUSTER_NAME Sample_CC_Cluster CLUSTER_NAME ClusterA CLUSTER_DOMAIN cup.hp.
Building a Continental Cluster Restoring Disaster Tolerance PRIMARY_PACKAGE ClusterB/pkgZ RECOVERY_PACKAGE ClustserA/pkgZ' RECOVERY_GROUP_NAME RG4 PRIMARY_PACKAGE ClusterB/pkgW RECOVERY_PACKAGE ClusterA/pkgW' DATA_RECEIVER_PACKAGE ClusterA/pkgR2 ### Section 3.
Building a Continental Cluster Restoring Disaster Tolerance CLUSTER_ALERT NOTIFICATION 0 MINUTES SYSLOG “CC alert: UP” Newly Created Cluster Will Run Primary Packages After you create a new cluster to replace the damaged cluster, you may choose to restore the critical applications to the new cluster and restore the other cluster to its role as a backup for the recovered packages. 1. Configure the new cluster as an MC/ServiceGuard cluster.
Building a Continental Cluster Restoring Disaster Tolerance 6. Restart the monitor package on the surviving cluster: # cmrunpkg ccmonpkg 7. View the status of the ContinentalCluster.
Building a Continental Cluster Maintaining a Continental Cluster Maintaining a Continental Cluster The following common maintenance tasks are described in this section: CAUTION • Adding a Node to a Cluster or Removing a Node from a Cluster • Adding a Package to a Continental Cluster • Removing a Package from the Continental Cluster • Changing Monitoring Definitions • Checking the Status of Clusters, Nodes and Packages • Reviewing Log Files • Renaming a Continental Cluster • Deleting a Cont
Building a Continental Cluster Maintaining a Continental Cluster 3. Edit the ContinentalClusters configuration ASCII file to add or remove the node in the cluster. 4. For added nodes, ensure the the /etc/opt/cmom/cmomhosts file is set up correctly on the new node. Refer to “Preparing Security Files” on page 211. Ensure that the /.rhosts file on all nodes (including the new node) contains an entry allowing write access by the host on which you are running the configuration commands. 5.
Building a Continental Cluster Maintaining a Continental Cluster 7. Use the cmapplyconcl command to apply the new ContinentalClusters configuration. 8. Restart the monitor packages on both clusters. 9. View the status of the continental cluster. # cmviewconcl Removing a Package from the Continental Cluster To remove a package from the ContinentalClusters configuration, you must first remove the recovery group from the ContinentalClusters configuration file.
Building a Continental Cluster Maintaining a Continental Cluster 3. Ensure that the /.rhosts file on all nodes (including the new node) contains an entry allowing write access by the host on which you are running the configuration commands. 4. Use the cmapplyconcl command to apply the new configuration. 5. Restart the monitor packages on both clusters. 6. View the status of the continental cluster.
Building a Continental Cluster Maintaining a Continental Cluster cluster cjc1234 CONTINENTAL CLUSTER cjccc1 RECOVERY CLUSTER cjc1234 PRIMARY CLUSTER cjc838 STATUS down EVENT LEVEL ALARM POLLING INTERVAL 20 CONFIGURED EVENT STATUS DURATION LAST NOTIFICATION SENT alert unreachable 15 sec -alarm unreachable 30 sec -alarm down 0 sec Fri May 12 12:13:06 PDT 2000 alert error 0 sec -alert up 20 sec -alert up 40 sec -PACKAGE RECOVERY GROUP prg1 PACKAGE cjc838/primary cjc1234/recovery ROLE primary recovery
Building a Continental Cluster Maintaining a Continental Cluster alert up RECOVERY CLUSTER 1 min -- PTST_sanfran PRIMARY CLUSTER INTERVAL PTST_dts1 STATUS Unmonitored CONFIGURED EVENT NOTIFICATION SENT alert alert alarm alert alert alarm alert alert PACKAGE RECOVERY GROUP POLLING unmonitored 1 min STATUS DURATION LAST unreachable unreachable unreachable down down down error up 1 2 3 1 2 3 0 1 --------- min min min min min min sec min hpgroup10 PACKAGE PTST_sanfran/PACKAGE1 PTST_dts1/PAC
Building a Continental Cluster Maintaining a Continental Cluster NODE nynode1 STATUS up STATE running Network_Parameters: INTERFACE STATUS PRIMARY up PRIMARY up NODE nynode2 PATH 12.1 56.1 STATUS up STATE running Network_Parameters: INTERFACE STATUS PRIMARY up PRIMARY up PACKAGE ccmonpkg nynode2 NAME lan0 lan2 STATUS up PATH 4.1 56.1 NAME lan0 lan1 STATE running PKG_SWITCH enabled Script_Parameters: ITEM NAME RESTARTS Service ccmonpkg.
Building a Continental Cluster Maintaining a Continental Cluster Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING Primary down Alternate down NAME nynode1 nynode2 You can also use the ps command to check for the status of the ContinentalClusters monitor daemons cmclrmond and cmclsentryd. They should be running on the cluster node where the monitor package is running.
Building a Continental Cluster Maintaining a Continental Cluster You can also review the monitor startup and shutdown log file /etc/cmcluster/ccmonpkg/ccmonpkg.cntl.log on any node where a ContinentalClusters monitor has been running. Information about the primary or recovery packages may be found in their respective startup and shutdown log files. Messages from the ContinentalClusters daemon are reported in log file /var/adm/cmconcl/sentryd.log, and Object Manager messages appear in /var/opt/cmom/cmomd.
Building a Continental Cluster Maintaining a Continental Cluster # cmdeleteconcl This command deletes the ContinentalCluster configuration. NOTE If you are modifying the configuration, you simply re-issue the cmapplyconcl command. There is no need to delete the previous configuration.
Building a Continental Cluster Support for Oracle 9i RAC Instances in a ContinentialClusters Environment Support for Oracle 9i RAC Instances in a ContinentialClusters Environment Starting with ContinentalClusters A.04.00, support is available for Oracle 9i Real Application Cluster (RAC) instances.
Building a Continental Cluster Support for Oracle 9i RAC Instances in a ContinentialClusters Environment As shown in the above example, Oracle 9i RAC instances are configured to run in ServiceGuard packages. The instance packages are running on the primary cluster and will be recovered on the recovery cluster upon a primary cluster failure. Figure 5-11 shows a recovery using an Oracle 9i RAC configuration after failover.
Building a Continental Cluster Support for Oracle 9i RAC Instances in a ContinentialClusters Environment Configuring the Environment for ContinentalClusters to Support Oracle 9i RAC In order to enable ContinentalClusters support for Oracle 9i RAC, there needs to be a set of configurations, which include HP StorageWorks Continuous Access XP, Oracle 9i RAC, and ContinentalClusters.
Building a Continental Cluster Support for Oracle 9i RAC Instances in a ContinentialClusters Environment the number of RAC instances configured to run on that cluster. It is recommended that the same number of Oracle 9i RAC instances are configured on both primary and recovery clusters. The AUTO_RUN parameter in the package configuration file needs to be set to “NO” similar to other applications packages running in the ContinentalClusters environment. 5. Setup XP/CA environment file.
Building a Continental Cluster Support for Oracle 9i RAC Instances in a ContinentialClusters Environment c. Copy the file: # cp /opt/cmconcl/scripts/ccrac.config \ ccrac.config.mycopy d. Edit ccrac.config.mycopy to fit your environment. The following parameters need to be edited: CCRAC_ENV- fully qualified XP/CA environment file name. This file name has to be appended with "_xpca.
Building a Continental Cluster Support for Oracle 9i RAC Instances in a ContinentialClusters Environment # CCRAC_ENV[2]=/etc/cmconcl/ccrac/db3 \ /db3EnvFile_xpca.env CCRAC_SLVM_VGS[2]=ccracvg5 ccracvg6 CCRAC_INSTANCE_PKGS[1]=ccracPkg5 ccracPkg6 e. Copy the edited file to the final directory: # cp ccrac.config.mycopy \ /etc/cmconcl/ccrac/ccrac.config f. Copy file /etc/cmconcl/ccrac/ccrac.config to all the other nodes of the cluster. g.
Building a Continental Cluster Support for Oracle 9i RAC Instances in a ContinentialClusters Environment Initial Startup of Oracle 9i RAC Instance in a ContinentalClusters Environment To ensure that the XP disk array will be ready for the access in shared mode for the Oracle 9i RAC instances, it is recommended the user runs the ContinentalClusters tool /opt/cmconcl/bin/ccrac_mgmt.ksh to initially startup the configured instance packages.
Building a Continental Cluster Support for Oracle 9i RAC Instances in a ContinentialClusters Environment Failover of Oracle 9i RAC Instances to the Recovery Site Upon a disaster that disables the primary cluster, use the following command to start up a ContinentalClusters recovery process: # cmrecovercl NOTE Make sure that the primary site is unavailable and all of the Oracel 9i RAC instance packages are not running on the primary cluster before triggering the recovery process.
Building a Continental Cluster Support for Oracle 9i RAC Instances in a ContinentialClusters Environment The Oracle 9i RAC instance package can be started in sequence by using: # cmrecovercl -g If option -g is used to start up the first instance package, you should wait until the XP disk arrays are synchronized to the “PAIR” state before starting up the second instance package.
Building a Continental Cluster Support for Oracle 9i RAC Instances in a ContinentialClusters Environment Failback of Oracle 9i RAC Instances After a Failover After failover, the configured XP disk array at the old recovery cluster becomes the primary storage of the database. The Oracle 9i RAC instances are running at the recovery cluster after a successful recovery.
Physical Data Replication for ContinentalClusters Using Continuous Access XP 6 Physical Data Replication for ContinentalClusters Using Continuous Access XP This chapter shows how to use the MetroCluster/CA package environment file template to set up data replication between the primary and secondary packages in a continental cluster that uses the HP StorageWorks E Disk Array XP Series with Continuous Access XP. These packages run on different clusters within the continental cluster.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Overview Overview Physical data replication with the HP StorageWorks E Disk Array XP Series uses the Continuous Access XP integration scripts and environment file, which are installed in the /usr/sbin and /opt/cmcluster/toolkit/SGCA directories when you install the separately purchased product MetroCluster with Continous Access XP. These directories include the following files: • /opt/cmcluster/toolkit/SGCA/xpca.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Overview NOTE • Raid Manager XP host-based software for control and status of the XP series disk arrays. • MC/ServiceGuard Ensure that you have the most recent version for HP-UX, ContinentalCluster, Raid Manager, and MC/ServiceGuard by referring to most current ContinentalClusters release notes.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Preparing the Continental Cluster for Data Replication Preparing the Continental Cluster for Data Replication These procedures will prepare your MC/ServiceGuard clusters for use with Continuous Access XP data replication in a continental cluster. 1. Ensure that the XP Series disk arrays are correctly cabled to each host system in the two clusters that will run packages whose data reside on the arrays.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Preparing the Continental Cluster for Data Replication If there are no configured command devices on the disk array, you must create two before proceeding. Each command device must have alternate links (PVLinks). The first command device is the primary command device. The second command device is a redundant command device and is used only upon failure of the primary command device.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Preparing the Continental Cluster for Data Replication 9. Export the environment variable that specifies the Raid Manager instance to be used by the Raid Manager commands. For example, with the POSIX shell type: # export HORCMINST=0 Now, you can use Raid Manager commands to get further information from the disk arrays.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Preparing the Continental Cluster for Data Replication package. For example, a device group name of “db-payroll” is easily associated with the database for the payroll application. A device group name of “group1” would be more difficult to easily relate to an application. Edit the Raid Manager configuration file (horcm0.conf in the above example) to include the devices and device group used by the application package.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Preparing the Continental Cluster for Data Replication # horcmstart.sh 12. Repeat these steps on each host that will run this particular application package. If a host may run more than one application package, you must incorporate device group and host information for each of these packages.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Preparing the Continental Cluster for Data Replication The TargetID and LU# fields for each device name may be different on different hosts in the clusters, to allow for different hardware I/O paths on different hosts. See the sample convenience scripts in the Samples-CC directory included with this toolkit for examples.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Creating and Exporting LVM Volume Groups Creating and Exporting LVM Volume Groups Use the following procedure to create and export volume groups. 1. Define the appropriate Volume Groups on each host system that might run the application package. Use the commands: # mkdir /dev/vgxx # mknod /dev/vgxx/group c 64 0xnn0000 where the name /dev/vgxx and the number nn are unique within the entire cluster. 2.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Creating and Exporting LVM Volume Groups # pairsplit -g -rw # vgimport -s -m # vgchange -a y # vgcfgbackup # vgchange -a n # pairresync -g -c 15 See the sample script /opt/cmcluster/toolkit/SGCA/Samples-CC/mk2imports. You can skip the pairsplit/pairresync, but you will not be able to activate the volume group to perform the vgcfgbackup.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Creating VxVM Disk Groups for Use with ContinentalCluster/CA Data Replication Creating VxVM Disk Groups for Use with ContinentalCluster/CA Data Replication If you are using VERITAS storage, use the following procedure to create disk groups. It is assumed that you have already created a VERITAS root disk (rootdg) on the system where you are configuring the storage. The following section shows how to set up VERITAS disk groups.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Creating VxVM Disk Groups for Use with ContinentalCluster/CA Data Replication 11.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Validating VxVM Disk Groups for Use with ContinentalCluster/CA Data Replication Validating VxVM Disk Groups for Use with ContinentalCluster/CA Data Replication The following section shows how to validate VERITAS disk groups. On one node do the following: 1. Deport the disk group. # vxdg deport logdata 2. Enable other cluster nodes to have access to the disk group. # vxdctl enable 3.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Configuring Packages for Automatic Disaster Recovery Configuring Packages for Automatic Disaster Recovery When you have completed the following steps, packages will be able to automatically fail over to an alternate node in another data center and still have access to the data that they need in order to operate.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Configuring Packages for Automatic Disaster Recovery If you are using a fence level of ASYNC, then the RUN_SCRIPT_TIMEOUT should be greater than the value of HORCTIMEOUT in the package environment file (see step 7g below). NOTE If you are using the EMS disk monitor as a package resource, you must not use NO_TIMEOUT. Otherwise, package shutdown will hang if there is not access from the host to the package disks.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Configuring Packages for Automatic Disaster Recovery b. Uncomment the behavioral configuration environment variables starting with AUTO_. It is recommended that you retain the default values of these variables unless you have a specific business requirement to change them. See Appendix A for an explanation of these variables. c.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Configuring Packages for Automatic Disaster Recovery If any messages are returned, you should correct the syntax errors. 9. Check the configuration using the cmcheckconf -P pkgname.config, then apply the MC/ServiceGuard configuration using the cmapplyconf -P pkgname.config command or SAM. 10.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Setting up a Primary Package on the Primary Cluster Setting up a Primary Package on the Primary Cluster Use the procedures in this section to configure a primary package on the primary cluster. Consult the MC/ServiceGuard documentation for more detailed instructions on setting up MC/ServiceGuard with packages, and for instructions on how to start, halt, and move packages and their services between nodes in a cluster.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Setting up a Primary Package on the Primary Cluster Create an MC/ServiceGuard package configuration file in the primary cluster with the commands: # cd /etc/cmcluster/ # cmmakepkg -p .ascii Customize it as appropriate to your application. Be sure to include the pathname of the control script (/etc/cmcluster// .cntl) for the RUN_SCRIPT and HALT_SCRIPT parameters.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Setting up a Primary Package on the Primary Cluster a. If necessary, add the path where the RaidManager software binaries have been installed to the PATH environment variable. If the software is in the usual location, /usr/bin, you can just uncomment the line in the script. b. Uncomment the behavioral configuration environment variables starting with AUTO_.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Setting up a Primary Package on the Primary Cluster # rcp -p /etc/cmcluster/pkgname/* \ other_node:/etc/cmcluster/pkgname See the example script Samples/ftpit to see how to semi-automate the copy using ftp. This script assumes the package directories already exist on all nodes. Using ftp may be preferable at your organization, since it does not require the use of a.rhosts file for root. Root access via .
Physical Data Replication for ContinentalClusters Using Continuous Access XP Setting up a Recovery Package on the Recovery Cluster Setting up a Recovery Package on the Recovery Cluster Use the procedures in this section to configure a recovery package on the recovery cluster. Consult the MC/ServiceGuard documentation for more detailed instructions on setting up MC/ServiceGuard with packages, and for instructions on how to start, halt, and move packages and their services between nodes in a cluster.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Setting up a Recovery Package on the Recovery Cluster Create an MC/ServiceGuard package configuration file in the recovery cluster with the commands: # cd /etc/cmcluster/ # cmmakepkg -p .ascii Customize it as appropriate to your application. Be sure to include the pathname of the control script (/etc/cmcluster// .cntl) for the RUN_SCRIPT and HALT_SCRIPT parameters.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Setting up a Recovery Package on the Recovery Cluster 6. Copy the environment file template /opt/cmcluster/toolkit/SGCA/xpca.env to the package directory, naming it pkgname_xpca.env: # cp /opt/cmcluster/toolkit/SGCA/xpca.env \ /etc/cmcluster/pkgname/pkgname_xpca.env 7. Edit the environment file _xpca.env as follows: a.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Setting up a Recovery Package on the Recovery Cluster less than the RUN_SCRIPT_TIMEOUT set in the package configuration file. The default setting is the side file timeout value + 60 seconds. i. Uncomment the CLUSTER_TYPE variable and set it to CONTINENTAL. 8.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Setting up a Recovery Package on the Recovery Cluster 12. Make sure the packages on the primary cluster are not running. Using standard MC/ServiceGuard commands (cmruncl, cmhaltcl, cmrunpkg, cmhaltpkg) test the recovery cluster for cluster and package startup and package failover. 13. Any running package on the recovery cluster that has a counterpart on the primary cluster should be halted at this time.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Setting up the Continental Cluster Configuration Setting up the Continental Cluster Configuration The steps below are the basic procedure for setting up the ContinentalClusters configuration file and the monitoring packages on the two clusters. For complete details on creating and editing the configuration file, refer to Chapter 5, “Building a Continental Cluster.” 1.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Setting up the Continental Cluster Configuration 6. Apply the continental cluster configuration file using cmapplyconcl. Files are placed in /etc/cmconcl/instances. There is no change to /etc/cmcluster/cmclconfig nor is there an equivalent file for ContinentalClusters. Example: # cmapplyconcl -C cmconcl.config NOTE You must create or edit /.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Switching to the Recovery Cluster in Case of Disaster Switching to the Recovery Cluster in Case of Disaster It is vital the administrator verify that recovery is needed after receiving a cluster alert or alarm. Network failures may produce false alarms. After validating a failure, start the recovery process using the cmrecovercl [-f] command.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Failback Scenarios Failback Scenarios The goal of HP ContinentalClusters is to maximize system and application availability. However, even systems configured with ContinentalClusters can experience hardware failures at the primary site or the recovery site, as well as the hardware or networking failures connecting the two sites.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Failback Scenarios NOTE In ASYNC mode, the package will halt unless a force flag is present or unless the auto variable AUTO_SVOLPSUE is set to 1. The fence level of the paired volume—NEVER, ASYNC, or DATA—will not impact the starting of the packages at the recovery site.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Failback Scenarios 3. Start the cluster at the primary site. Assuming they have been properly configured, the ContinentalClusters primary packages should not start. The monitor package should start automatically. 4. Manually start the ContinentalClusters primary packages at the primary site using the command # cmrunpkg The CAXP integration script is programmed to handle this case. 5.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Failback Scenarios Subsequently, the paired volume will have a status of SVOL_SWSS. To view the local status of the paired volumes run # pairvolchk -g -s To view the remote status of the paired volumes run # pairvolchk -g -c (While the remote XP disk array and primary cluster systems are down, the command will timeout with an error code of 242).
Physical Data Replication for ContinentalClusters Using Continuous Access XP Failback Scenarios # cmapplyconcl -C 7. Start the monitor packages at both the primary and recovery sites. 8.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Failback Scenarios # cmrunpkg The control script is programmed to handle this case. The control script recognizes that the paired volume is synchronized and will proceed with the programmed package startup. 5. Ensure that monitor packages are running at both sites.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Maintaining the Continuous Access XP Data Replication Environment Maintaining the Continuous Access XP Data Replication Environment There might be situations where a package has to be taken down for maintenance purposes without having the package move to another node. The following procedure is recommended for normal maintenance of ContinentalClusters: 1. Shut down the package with the appropriate MC/ServiceGuard command.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Maintaining the Continuous Access XP Data Replication Environment This command must successfully complete for disaster-tolerant data protection to be restored.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Maintaining the Continuous Access XP Data Replication Environment Pairs must be manually recreated if both the primary and recovery XP disk arrays are in the SMPL (simplex) state. Make sure you periodically review the following files for messages, warnings and recommended actions. You should particularly review these files after system, data center and/or application failures: • /var/adm/syslog/syslog.
Physical Data Replication for ContinentalClusters Using Continuous Access XP Maintaining the Continuous Access XP Data Replication Environment Long delays in package startup time will occur in those situations when recovering from broken pair affinity. NOTE 324 • The value of RUN_SCRIPT_TIMEOUT in the package ASCII file should be set to NO_TIMEOUT or to a large enough value to take into consideration the extra startup time due to getting status from the XP disk array.
Physical Data Replication for ContinentalClusters Using EMC SRDF 7 Physical Data Replication for ContinentalClusters Using EMC SRDF This chapter shows how to use the MetroCluster/SRDF environment file template to set up data replication between the primary and secondary packages in a Continental Cluster that uses the EMC Symmetrix Disk Array with the Symmetrix Remote Data Facility (SRDF). These packages run on different clusters within the Continental cluster configuration.
Physical Data Replication for ContinentalClusters Using EMC SRDF Overview Overview Physical data replication with the EMC Symmetrix Disk Array uses the EMC SRDF environment file template, which is installed in the /opt/cmcluster/toolkit/SGSRDF directory when you install the separately purchased product MetroCluster with EMC SRDF. This directory includes the following files: • /opt/cmcluster/toolkit/SGSRDF/srdf.env—the EMC SRDF environment file template.
Physical Data Replication for ContinentalClusters Using EMC SRDF Preparing a Continental Cluster for Data Replication Preparing a Continental Cluster for Data Replication These procedures will prepare your MC/ServiceGuard clusters for use in a ContinentalClusters configuration. 1. Ensure that the Symmetrix ICDAs are correctly cabled to each host system in the cluster that will run packages whose data reside on the Symmetrix.
Physical Data Replication for ContinentalClusters Using EMC SRDF Preparing a Continental Cluster for Data Replication • Symmetrix serial number Gatekeeper devices are usually of size 2880. The Symmetrix device serial number is useful in matching the devices to the actual devices in the Symmetrix configuration (downloaded by EMC). The following is a key to interpreting the number: • The first two digits equal the last two digits of the serial number of the Symmetrix frame.
Physical Data Replication for ContinentalClusters Using EMC SRDF Preparing a Continental Cluster for Data Replication • hostname • Symmetrix ID (first two digits of the device serial number) • Symmetrix device number (next three hexidecimal digits of the device serial number) • Symmetrix device type (BCV, R1, R2, or gatekeeper) to help you correlate the devices on all of the host systems. The table might look like Table 7-1.
Physical Data Replication for ContinentalClusters Using EMC SRDF Preparing a Continental Cluster for Data Replication Table 7-1 Symmetrix Device and HP-UX Device Correlation (Continued) Symmetrix ID, device #, and type Dev# 041 Type GK ID 95 Dev# 028 Type BCV Node 1 /dev/rdsk device file name Node 2 /dev/rdsk device file name Node 3 /dev/rdsk device file name Nodes 4 /dev/rdsk device file name c5t15d1 c1t1d0 c6t1d0 n/a n/a In this example, the target and LUN numbers do not always match for
Physical Data Replication for ContinentalClusters Using EMC SRDF Preparing a Continental Cluster for Data Replication 7. Use the symdg, symld and symgate commands on all nodes in both clusters. to create the Symmetrix device groups. All devices belonging to volume groups that are owned by an application package must be defined in a single Symmetrix device group. The device group name must be the same on both R1 side and the R2 side of the SRDF link.
Physical Data Replication for ContinentalClusters Using EMC SRDF Preparing a Continental Cluster for Data Replication You can display what is in the SymCLI database with the commands: • symdg list • symld -g list • symgate list You may also use the following command to get a listing of all devices and their states: # symrdf list 332 Chapter 7
Physical Data Replication for ContinentalClusters Using EMC SRDF Creating and Exporting Volume Groups Creating and Exporting Volume Groups Create and export the volume groups using the following procedure: 1. Define the appropriate volume groups on each host system that might run the application package. Use the commands: # mkdir /dev/vgxx # mknod /dev/vgxx/group c 64 0xnn0000 where the name /dev/vgxx and the number nn are unique within the entire cluster. 2.
Physical Data Replication for ContinentalClusters Using EMC SRDF Creating and Exporting Volume Groups # symrdf -g split -v # vgimport -s -m # vgchange -a y # vgcfgbackup # vgchange -a n # symrdf -g establish -v # symrdf list NOTE The vgexport -s option creates the special device files for the vgimport on adoptive nodes. The vgimport -s option will look for the marked disks for that volume group.
Physical Data Replication for ContinentalClusters Using EMC SRDF Creating and Exporting Volume Groups If the mounted volumes show, then: # vgchange -a n vgpkgCCA d. Activate the volume group on all failover nodes in the primary cluster. On each node, activate the volume group, and mount LV’s.
Physical Data Replication for ContinentalClusters Using EMC SRDF Creating VxVM Disk Groups for Use with ContinentalCluster/SRDF Data Replication Creating VxVM Disk Groups for Use with ContinentalCluster/SRDF Data Replication If you are using VERITAS storage, use the following procedure to create disk groups. It is assumed that you have already created a VERITAS root disk (rootdg) on the system where you are configuring the storage. The following section shows how to set up VERITAS disk groups.
Physical Data Replication for ContinentalClusters Using EMC SRDF Validating VxVM Disk Groups for Use with ContinentalCluster/SRDF Data Replication Validating VxVM Disk Groups for Use with ContinentalCluster/SRDF Data Replication The following section shows how to validate VERITAS diskgroups. On one node do the following: 1. Deport the disk group. # vxdg deport logdata 2. Enable other cluster nodes to have access to the disk group. # vxdctl enable 3. Slit the SRDF link to enable R2 Read/Write permission.
Physical Data Replication for ContinentalClusters Using EMC SRDF Setting up a Primary Package on the Primary Cluster Setting up a Primary Package on the Primary Cluster Use the procedures in this section to configure a primary package on the primary cluster. Consult the MC/ServiceGuard documentation for more detailed instructions on setting up MC/ServiceGuard with packages, and for instructions on how to start, halt, and move packages and their services between nodes in a cluster. 1.
Physical Data Replication for ContinentalClusters Using EMC SRDF Setting up a Primary Package on the Primary Cluster 6. Create an MC/ServiceGuard Application package configuration file with the commands: # cd /etc/cmcluster/ # cmmakepkg -p .conf Customize it as appropriate to your application. Be sure to include Node names, the pathname of the control script (/etc/cmcluster//.cntl) for the RUN_SCRIPT and HALT_SCRIPT parameters.
Physical Data Replication for ContinentalClusters Using EMC SRDF Setting up a Primary Package on the Primary Cluster b. Uncomment the DEVICE_GROUP variable and set them to the Symmetrix device group names given in the ’symdg list’ command. The DEVICE_GROUP variable may also contain the consistency group name if using a M by N configuration. c. Uncomment the RETRY and RETRYTIME variables. The defaults should be used for the first package. The values should be slightly different for other packages.
Physical Data Replication for ContinentalClusters Using EMC SRDF Setting up a Primary Package on the Primary Cluster When using ftp, be sure to make the file executable on any destination systems. 13. Verify that each host in both clusters in the continental cluster has the following files in the directory /etc/cmcluster/: • .cntl (EMC SRDF package control script) • .conf (MC/ServiceGuard package ASCII config file) • .
Physical Data Replication for ContinentalClusters Using EMC SRDF Setting up a Recovery Package on the Recovery Cluster Setting up a Recovery Package on the Recovery Cluster The installation of EMC SRDF, MC/ServiceGuard, and ContinentalClusters software is exactly the same as in the previous section. The procedures below will install and configure a recovery package on the recovery cluster. Consult the MC/ServiceGuard documentation for instructions on setting up an MC/ServiceGuard cluster (i.e.
Physical Data Replication for ContinentalClusters Using EMC SRDF Setting up a Recovery Package on the Recovery Cluster 6. Copy the package files from the primary cluster to a bkpkgXXX directory, and rename it to .cntl and _srdf.env. Edit the recovery package control file from the primary cluster for the secondary cluster. Change the subnet, relocatable IP, and nodes. Be sure to set AUTO_RUN to NO in the package ASCII file. 7.
Physical Data Replication for ContinentalClusters Using EMC SRDF Setting up a Recovery Package on the Recovery Cluster .sh applicable) (Package monitor shell script, if _srdf.env (MC/SRDF environment file) 10. Split the SRDF logical links for the disks associated with the application package. See the script Samples-CC/pre.cmquery for an example of how to automate this task. The script must be customized with the Symmetrix device group names. 11.
Physical Data Replication for ContinentalClusters Using EMC SRDF Setting up the Continental Cluster Configuration Setting up the Continental Cluster Configuration The procedures below will configure ContinentalClusters and the monitoring packages on the two clusters. For complete details on creating and editing the configuration file, refer to Chapter 5, “Building a Continental Cluster.” 1. Split the SRDF logical links for the disks associated with the application package. See the script Samples-CC/pre.
Physical Data Replication for ContinentalClusters Using EMC SRDF Setting up the Continental Cluster Configuration # cmapplyconf -P /etc/cmcluster/ccmonpkg/ccmonpkg.config 7. Restore the logical SRDF links for the package. See the script Samples-CC/post.cmapply for an example of how to automate this task. The script must be customized with the appropriate Symmetrix device group names. Example: # Samples-CC/post.cmapply 8. Generate the cluster configuration file using cmapplyconcl.
Physical Data Replication for ContinentalClusters Using EMC SRDF Switching to the Recovery Cluster in Case of Disaster Switching to the Recovery Cluster in Case of Disaster It is vital the administrator verify that recovery is needed after receiving a cluster alert or alarm. Network failures may produce false alarms. After validating a failure, start the recovery process using the cmrecovercl [-f] command.
Physical Data Replication for ContinentalClusters Using EMC SRDF Failback Scenarios Failback Scenarios There is no failback counterpart to the “pushbutton” failover from the primary cluster to the recovery cluster. Failback is very dependent on the original nature of the failover, the state of primary and secondary Symmetrix SRDF volumes (R1 and R2) and the condition of the primary cluster.
Physical Data Replication for ContinentalClusters Using EMC SRDF Failback Scenarios Dev RDev Typ:G RDev Pair --- ---- -------- ------000 000 R2:2 Synchronized 001 001 R2:2 Failed Over 002 002 R2:2 Synchronized 003 003 R2:2 Synchronized 004 004 R1:1 Synchronized 005 005 R1:1 Synchronized 006 006 R1:1 Synchronized 007 007 R1:1 Synchronized SA RA LNK Mode Dom ACp Tracks Tracks Dev --------- ------------ ------ ------ --- RW WD RW SYN DIS OFF 0 0 WD RW RW RW NR SYN DIS OFF 12 0 RW WD RW WD R
Physical Data Replication for ContinentalClusters Using EMC SRDF Failback Scenarios 4. Manually start the ContinentalClusters primary packages at the primary site using # cmrunpkg or # cmmodpkg -e The control script is programmed to handle this case. The control script will issue an SRDF failback command to move the device group back to the R1 side and to resynchronize the R1 from the R2 side.
Physical Data Replication for ContinentalClusters Using EMC SRDF Failback Scenarios target.......Started. Device: 001 ............................................. Merged. Merge device track tables between source and target.......Done. Resume RDF link(s)........................................Done. Read/Write Enable device(s) on SA at source (R1)..........Done. The RDF ’Failback’ operation successfully executed for device group ’pkgCCB_r1’. 8.
Physical Data Replication for ContinentalClusters Using EMC SRDF Failback Scenarios Dev RDev Typ:G SA RA LNK Mode Dom ACp RDev Pair --- ---- ----- --------- --------------------------000 RW 001 RW 000 R2:2 RW WD RW Synchronized 001 R2:2 RW WD RW SyncInProg Tracks Tracks Dev ------ ------ --- SYN DIS OFF 0 0 WD SYN DIS OFF 2 0 WD 9. Halt the recovery cluster and restart it: # cmhaltcl -f (if not already down) # cmruncl 10. Verify the data for data consistency and currency.
Physical Data Replication for ContinentalClusters Using EMC SRDF Failback Scenarios # symgate define pd /dev/rdsk/c7t15d0 # symgate define pd /dev/rdsk/c7t15d1 # symgate -g pkgCCA_r1 associate pd /dev/rdsk/c7t15d0 2. Halt the ContinentalClusters recovery packages at the recovery site using # cmhaltpkg This will halt any applications, remove any floating IP addresses, unmount file systems and deactivate volume groups as programmed into the package control files.
Physical Data Replication for ContinentalClusters Using EMC SRDF Failback Scenarios 8. Ensure that the monitor packages at the primary and recovery sites are running.
Physical Data Replication for ContinentalClusters Using EMC SRDF Maintaining the EMC SRDF Data Replication Environment Maintaining the EMC SRDF Data Replication Environment Normal Startup The following is the normal ContinentalClusters startup procedure. On the primary cluster: 1. Start the primary cluster: # cmruncl -v The primary cluster comes up with ccmonpkg up. The application packages are down, and ccmonpkg is up. 2.
Physical Data Replication for ContinentalClusters Using EMC SRDF Maintaining the EMC SRDF Data Replication Environment # cmviewconcl -v Normal Maintenance There might be situations where a package has to be taken down for maintenance purposes without having the package move to another node. The following procedure is recommended for normal maintenance of the ContinentalCluster with EMC SRDF data replication: 1. Shut down the package with the appropriate command. Example: # cmhaltpkg 2.
Physical Data Replication for ContinentalClusters Using EMC SRDF Maintaining the EMC SRDF Data Replication Environment It is highly recommended that the R2 device is locally protected with RAID 1 or RAID S. If the R2 device is protected with BCV, and if it fails and there is a failover, the package cannot operate on the BCV device. The R2 device has to be fixed, the data has to be restored from the BCV device to the new R2 device, before the package can start.
Physical Data Replication for ContinentalClusters Using EMC SRDF Maintaining the EMC SRDF Data Replication Environment devices) until the SRDF link is restored, or manual intervention is undertaken to disable Domino Mode. Applications may fail or may continuously retry the I/Os (depending on the application) if Domino Mode is enabled and the SRDF link fails.
Physical Data Replication for ContinentalClusters Using EMC SRDF Maintaining the EMC SRDF Data Replication Environment For example, if a package is configured to failover across four nodes in the cluster, there should be eight gatekeeper devices (two for each node) that are assigned to the Symmetrix device group belonging to this package. It is required that there be a pool of four additional gatekeeper devices that are NOT associated with any device group.
Physical Data Replication for ContinentalClusters Using EMC SRDF R1/R2 Swapping R1/R2 Swapping This section shows how the R1/R2 swapping can be done via MetroCluster/SRDF package and manual procedures. Each of these allow you to swap the SRDF personality for each device designations of a specified devicegroup, where each source R1 device(s) becomes a target R2 device(s) and a target R1 device(s) becomes a source R1 device(s).
Physical Data Replication for ContinentalClusters Using EMC SRDF R1/R2 Swapping 1. Swap the personalities of the devices and mark the old R1 devices to be refresh from the old R2 devices. # symrdf -g swap -refresh R1 2. After swapping is completed, the devices will be in Suspended state. Next establish the device group for data replication from the new R1 devices to the new R2 devices.
Physical Data Replication for ContinentalClusters Using EMC SRDF R1/R2 Swapping 362 Chapter 7
Cascading Failover in a Continental Cluster 8 Cascading Failover in a Continental Cluster This chapter shows how to create a configuration that allows cascading failover between two sites that are connected via ContinentalClusters. Cascading failover is the ability of an application to fail from a primary to a secondary location, and then to fail to a recovery location on a different site.
Cascading Failover in a Continental Cluster Overview Overview The basic design of the cascading failover solution is shown in Figure 8-1. The primary cluster, shown on the left, is configured as a metropolitan cluster with three data centers physically located on three different sites—two main sites (Site 1 and Site 2) and an arbitrator site (Site 3). A fourth data center (Site 4) is used for the recovery cluster, which is a standard MC/ServiceGuard configuration.
Cascading Failover in a Continental Cluster Overview played by either Symmetrix in the primary cluster. In Figure 8-1, Symm1 holds the R1 volumes for application X data, and Site 1 is known as the primary site for application X. Symm2 holds R1 volumes for application Y data, and Site 2 is the primary site for application Y. The recovery Symmetrix holds a remote replicated copy of the data in the recovery cluster. The data center that houses the recovery Symmetrix is called the recovery site.
Cascading Failover in a Continental Cluster Overview The Symmetrix configuration is shown in Figure 8-2. Figure 8-2 Symmetrix Configuration for Cascading Failover Using Template Files Cascading failover configurations are built using the EMC SRDF control script template, /opt/cmcluster/toolkit/SGSGRDF/srdfpkg.cntl.
Cascading Failover in a Continental Cluster Data Storage Setup Data Storage Setup These procedures will prepare the MC/ServiceGuard clusters for use in a ContinentalClusters Cascading Failover configuration. Setting Up Symmetrix Device Groups Work with your EMC support engineer to configure the devices in three Symmetrix disk arrays. Figure 8-3 shows an example of the Symmetrix devices setup for an application that runs on clear. Figure 8-3 Example of Symmetrix Device Setup Use the following steps: 1.
Cascading Failover in a Continental Cluster Data Storage Setup connection upon failure of all links. This is required so the user would have time to split the BCV/R1 devices in the secondary Symmetrix from the mirror group before re-establishing the SRDF volume pairs between the primary Symmetrix and the secondary Symmetrix upon link recovery. 2. Ensure that the Symmetrix ICDAs are correctly cabled to each host system in the cluster that will run packages whose data reside on the Symmetrix. 3.
Cascading Failover in a Continental Cluster Data Storage Setup • The next three digits equal the Symmetrix host adapter (SA or FA) and port numbers; this is useful to see multiple host links to the same Symmetrix device (PVLinks). 6. The BCV devices are configured to be visible on a different SA/FA than R1 or R2 devices. These devices will not show on the output of the syminq command unless there is a host connection to this SA/FA and the BCV’s are split from the mirror group.
Cascading Failover in a Continental Cluster Data Storage Setup connect to the primary Symmetrix. Based on the example shown in Figure 8-3, Figure 8-4 shows the devices that need to be included in the primary Symmetrix device group. Figure 8-4 Primary Device Group • A second device group contains the R2 devices in the secondary Symmetrix and its local BCV/R1 devices in the same Symmetrix. This device group has to be defined in all nodes that connect to the secondary Symmetrix.
Cascading Failover in a Continental Cluster Data Storage Setup • A third device group contains the R2 devices in the recovery Symmetrix and its local BCV devices in the same Symmetrix. This device group has to be defined in all nodes that connect to the recovery Symmetrix. Figure 8-6 shows the devices that need to be included in the recovery Symmetrix device group.
Cascading Failover in a Continental Cluster Data Storage Setup • mkprigrp—create device group on nodes that connect to the primary Symmetrix. Run this script only on each node that connects to the primary Symmetrix. • mksecgrp—create device group on nodes that connect to the secondary Symmetrix. Run this script only on each node that connects to the secondary Symmetrix. • mkrecgrp—create device group on nodes that connect to the recovery Symmetrix.
Cascading Failover in a Continental Cluster Data Storage Setup Setting up Volume Groups Use the following procedure to set up volume groups for the volumes on the Symmetrix frames. 1. Before creating the volume groups, make sure that the SRDF link is established between the R1 devices in primary Symmetrix and the R2 devices in the secondary Symmetrix, and the BCV/R1 devices in the secondary Symmetrix are established as mirrors of the local standard devices.
Cascading Failover in a Continental Cluster Data Storage Setup c. Use the following command to check the data synchronization from the BCV/R1 devices to the R2 devices. If the “RDF Pair STATE” column shows the state “Synchronized” for all the devices, the copying completed. # symrdf -g query -rbc d. Once the copy completes, split the SRDF link between the secondary Symmetrix and the recovery Symmetrix: # symrdf -g split -rbcv e.
Cascading Failover in a Continental Cluster Data Storage Setup # vgcfgbackup # vgchange -a n # symrdf -g est -v Testing the Volume Groups Before MC/ServiceGuard and ContinentalClusters can be configured, HP-UX must be satisfied. Use the following procedure to confirm that the disks have been shared, and that the volume groups can be seen on the primary and adoptive nodes: 1. Confirm the volume groups have been imported using command: # strings /etc/lvmtab 2.
Cascading Failover in a Continental Cluster Primary Cluster Package Setup Primary Cluster Package Setup This section only describes the required changes to the package control script to support the cascading failover configuration. To create a primary package on the primary cluster, the reader should follow the procedure in “Setting up a Primary Package on the Primary Cluster” on page 338. Cascading failover uses a ContinentalClusters in which the primary cluster is configured as a metropolitan cluster.
Cascading Failover in a Continental Cluster Recovery Cluster Package Setup Recovery Cluster Package Setup This section only describes the required changes to the package control script to support the cascading failover configuration. To create a recovery package on the recovery cluster, the reader should follow the procedure in “Setting up a Recovery Package on the Recovery Cluster” on page 342.
Cascading Failover in a Continental Cluster Continental Cluster Configuration Continental Cluster Configuration To set up the Continental Cluster, the reader should follow the procedure in “Setting up the Continental Cluster Configuration” on page 345. Step 1 in this section calls for a split of SRDF logical links; this is the link between the secondary Symmetrix and the recovery Symmetrix.
Cascading Failover in a Continental Cluster Data Replication Procedures Data Replication Procedures This section describes the procedures that should be used to manage the data in a Symmetrix cascading failover configuration. Data Initialization Procedures The following procedures are needed if you already have existing data prior to implementing this solution.
Cascading Failover in a Continental Cluster Data Replication Procedures 3. Start the application to load the data to the R1 devices in the primary Symmetrix. Refer to the sample script datainit in the /opt/cmcluster/toolkit/SGSRDF/cascade/Samples directory for examples of how these commands are used. This script is designed to run only on a node that connects to the primary Symmetrix. 2. Mirroring from the Secondary to the Recovery Symmetrix This procedure is illustrated in Figure 8-8.
Cascading Failover in a Continental Cluster Data Replication Procedures # symrdf -g est -rbcv 5. Use the following command to check the data synchronization from the BCV/R1 devices to the R2 devices. If the “RDF Pair STATE” column shows the state “Synchronized” for all the devices, the copying completed. # symrdf -g query -rbcv 6.
Cascading Failover in a Continental Cluster Data Replication Procedures always be behind. The level of data currency on the recovery Symmetrix is dictated by the frequency at which it is refreshed. The refresh process is shown in Figure 8-9. Figure 8-9 Data Refresh in Steady State The following procedure describes the steps necessary to periodically copy the data from the secondary Symmetrix to the recovery Symmetrix while the application is running on the primary site. 1.
Cascading Failover in a Continental Cluster Data Replication Procedures 5. Split the BCV devices in the recovery Symmetrix from the mirror group. This is required to preserve an old copy of the data just in case a failure occurs during data synchronization between the secondary Symmetrix and the recovery Symmetrix that may cause data corruption on the R2 devices in the recovery Symmetrix.
Cascading Failover in a Continental Cluster Data Replication Procedures Scenario 1—Primary Site within the Primary Cluster Fails When a failure occurs at the primary site, the hosts are down or the whole site is down, the application package is automatically failover to the secondary site within the primary cluster. Until the problems at the primary site are fixed, and data replication is reestablished, there is no data protection for the package at the secondary site.
Cascading Failover in a Continental Cluster Data Replication Procedures The is a file that contains a list of the standard device and BCV device pair in the recovery Symmetrix. 2. Freeze the application I/O if the application is writing data to the R1 devices in the primary Symmetrix. The method of freezing the I/O is application dependent. 3.
Cascading Failover in a Continental Cluster Data Replication Procedures Failback from the Secondary Site to the Primary Site Once the problems at the primary site have been fixed, the application can fail back to the primary site. The current RDF pair states of the package device groups will be “Split,” which is not handled automatically by the package control script. The following steps are required to move the package back to the primary site. 1.
Cascading Failover in a Continental Cluster Data Replication Procedures the package device group. The failback will synchronize the R1 devices from the R2 devices. Until the synchronization is complete, the package application may run at a lower performance level. 5. Use the following command to check the RDF pair state between the R1 devices in the primary Symmetrix to the R2 devices in the secondary Symmetrix.
Cascading Failover in a Continental Cluster Data Replication Procedures Scenario 2—Secondary Site within the Primary Cluster Fails When the secondary site fails, or all SRDF links between the primary Symmetrix and the secondary Symmetrix fail, unless domino mode is used, the application running on the primary site is not aware of this failure and continues to run on the primary site. This scenario is illustrated in Figure 8-11.
Cascading Failover in a Continental Cluster Data Replication Procedures From a host that connects to the secondary Symmetrix: # symmir -g split 2. Incrementally establish the SRDF volume pairs between the primary Symmetrix and the secondary Symmetrix. From a host that connects to the primary Symmetrix: # symrdf -g est From a host that connects to the secondary Symmetrix: # symrdf -g est 3.
Cascading Failover in a Continental Cluster Data Replication Procedures Scenario 3—Entire Primary Cluster Fails In this scenario, the assumption is that both primary site and secondary site fail at the same time or very close to each other. This scenario is illustrated in Figure 8-12.
Cascading Failover in a Continental Cluster Data Replication Procedures 3. Once the restore completed, split the BCV devices from the mirror group: # symmir -g split The data in the recovery Symmetrix may not be current but should be consistent. There is no additional procedure needed. The package control script is programmed to handle this case.
Cascading Failover in a Continental Cluster Data Replication Procedures # symmir -g split 4. Re-establish the RDF volume pairs between the recovery Symmetrix and the secondary Symmetrix. Restore the data from the recovery Symmetrix and the secondary Symmetrix.
Cascading Failover in a Continental Cluster Data Replication Procedures # symmir -g -full restore 8. Use the following command to check the data restore from the BCV/R1 devices in the R2 standard devices in the secondary Symmetrix. If the “RDF Pair STATE” column shows the state "Restored" for all the devices, the data restore completed.
Cascading Failover in a Continental Cluster Data Replication Procedures 12. Start the application package at the secondary site. Issue the following command for all hosts in secondary site that may run this package: # cmmodpkg -e -n Then issue the following command from any node on the primary cluster: # cmmodpkg -e 13. Start the ContinentalClusters monitor package.
Cascading Failover in a Continental Cluster Data Replication Procedures Use the following command from a host that connects to the secondary Symmetrix: # symrdf -g -full restore -bcv 5. Use the following command to check the data restore from the R2 devices in the recovery Symmetrix to the BCV/R1 devices in the secondary Symmetrix. If the "RDF Pair STATE" column shows the state "Synchronized" for all the devices, the data restore completed.
Cascading Failover in a Continental Cluster Data Replication Procedures 9. Once the data restore completed, split the BCV/R1 devices from the mirror group. From a host that connects to the recovery Symmetrix: # symmir -f -sid split From a host that connects to the secondary Symmetrix: # symmir -g split 10. Set the RDF pairs between primary Symmetrix and secondary Symmetrix to fail over state.
Cascading Failover in a Continental Cluster Data Replication Procedures If the “RDF Pair STATE” column shows the state “Synchronized” for all the devices, then proceed to the next step. 15. Re-establish the BCV/R1 devices in the secondary Symmetrix as mirrors of the standard devices.
Cascading Failover in a Continental Cluster Data Replication Procedures 398 Chapter 8
Environment File Variables for MetroCluster/CA A Environment File Variables for MetroCluster/CA This appendix lists all Environment File variables that have been modified or added for MetroCluster with Continuous Access XP.
Environment File Variables for MetroCluster/CA AUTO_NONCURDATA (Default=0) This parameter applies when the package is starting up with possible non-current data under certain CA pair states. During failover, this paramter will apply when the SVOL is in the PAIR or PFUL state and the PVOL side is in the PSUE, EX_ENORMT, or EX_CMDIOE state. During failback, this parameter will apply when the PVOL is in the PSUS state and the SVOL is in the EX_ENORMT or EX_CMDIOE state.
Environment File Variables for MetroCluster/CA secondary site, the PVOL will become PSUE and the SVOL will become PSUS(SSWS). When we fail back to the primary site, any differential data that was on the PVOL prior to failover will be lost during resynchronization. NOTE This variable is also used for the combination of PVOL_PFUS and SVOL_PSUS. When the side file has reached side file threshold timeout, the PVOL will become PFUS.
Environment File Variables for MetroCluster/CA latest data. When starting the package in this state on the PVOL side, you run the risk of losing any changed data in the PVOL. Values: 0—(DEFAULT) Do NOT startup the package at the primary site. Require user intervention to choose which side has the good data and resynchronizing the PVOL and SVOL or force the package to start by creating the FORCEFLAG file. 1—Startup the package after resynchronize the data from the SVOL side to the PVOL side.
Environment File Variables for MetroCluster/CA This parameter applies when the PVOL and SVOL both have the state of PSUE. This state combination will occur when there is an CA link, or other hardware failure, or when the SVOL side is in a PSUE state while we can not communicate with the PVOL side. This will only apply while in the Asynchronous mode.
Environment File Variables for MetroCluster/CA 1—Startup the package after making the SVOL writeable. The risk of using this option is the SVOL data may be inconsistent and the application may fail. However, there is also a chance that the data is actually consistent, and it is okay to startup the application. CLUSTER_TYPE This parameter defines the clustering environment in which the script is used.
Environment File Variables for MetroCluster/CA package can be successfully started. (Note: If this variable is not exported, Raid Manager commands used in this script may fail). HORCMPERM This variable supports the security feature, RAID Manager Protection Facility on the Continuous Access devices. (Note: If the RAID Manager Protection Facility is disabled, set this variable to MGRNOINST. This is the default value).
Environment File Variables for MetroCluster/CA Pkg Startup Timeout > HORCTIMEOUT > CA link timeout value MULTIPLE_PVOL_OR_SVOL_FRAME_FOR_PKG (Default=0) This parameter must be set to 1 if a PVOL or an SVOL for this package resides on more than one XP frame. Currently, only a value of 0 is supported for this parameter. Future releases may allow a value of 1. NOTE Values: 0—(Default) Single frame. 1—Multiple frames.
Environment File Variables for MetroCluster/CA Below is a summary table on the setting of the AUTO_* variables and the package’s startup behavior that are supported with MetroCluster with Continuous Access XP version A.04.10 on HP-UX 11.0 and 11i or later systems Table A-1 AUTO_FENCEDATA_SPLIT Remote State Fence Level AUTO_FENCEDATA _SPLIT =0 (Default) PVOL_PSUE SVOL_PAIR DATA PVOL_PSUE EX_ENORMT Do not start with exit 1. PVOL_PSUE EX_CMDIOE Local State Table A-2 Perform a PVOL takeover.
Environment File Variables for MetroCluster/CA Table A-3 AUTO_PSUEPSUS Local State Remote State Fence Level AUTO_PSUEPSUS =0 (Default) PVOL_PSUE SVOL_PSUS ASYNC PVOL_PFUS SVOL_PSUS Do not start with exit 1 AUTO_PSUEPSUS =1 or FORCEFLAG=yes Pairresync-swapp works, package starts up. * If pairresync-swapp works, package starts up. *If pairresync-swapp fails, package does not start with exit 1.
Environment File Variables for MetroCluster/CA Table A-5 Local State AUTO_SVOLPFUS Remote State SVOL_PFUS PVOL_PFUS SVOL_PFUS EX_ENORMT SVOL_PFUS EX_CMDIOE Table A-6 Local State AUTO_SVOLPFUS =0 (Default) ASYNC Do not start on the local node with exit 2. AUTO_SVOLPFUS =1 or FORCEFLAG=yes Perform SVOL takeover, which changes SVOL to PSUS(SSWS).
Environment File Variables for MetroCluster/CA Table A-7 Local State AUTO_SVOLPSUS Remote State SVOL_PSUS PVOL_PSUS SVOL_PSUS EX_ENORMT SVOL_PSUS EX_CMDIOE Fence Level AUTO_SVOLPSUS =0 (Default) NEVER /DATA/ ASYNC Do not start with exit 2. AUTO_SVOLPSUS =1 or FORCEFLAG=yes Perform a SVOL to PSUS(SSWS). After the takeover succeeds, package starts with a warning message about non-current data in the package’s control log file.
Environment File Variables for MetroCluster/CA MON_NOTIFICATION_EMAIL ( (Default=empty string) This parameter defines the email addresses that the monitor will use to send email notifications. The variable must use fully qualified email addresses. If multiple email addresses are defined, the comma must be used as a separator. If the parameter is not defined (commented out) or the default value is an empty string, this will indicate to the monitor that no email notifications will be sent.
Environment File Variables for MetroCluster/CA will try to resynchronize every polling interval. Once the device group has been completely resynchronized, the monitor will resynchronize the remote BC. 2—When the parameter is set to 2 and the data replication link is down, the monitor will only try to perform resynchronization if a file named MON_RESYNC exists in the package directory (PKGDIR). The monitor will not perform any operations to the remote BC (i.e. split and resynchronize the remote BC).
Environment File Variables for MetroCluster/SRDF B Environment File Variables for MetroCluster/SRDF This appendix lists all MC/ServiceGuard control script variables that have been modified or added for MetroCluster with EMC SRDF.
Environment File Variables for MetroCluster/SRDF AUTOR2RWNL Default: 1 AUTOR2WDNL=1 indicates that when the package is being started on an R2 host, the Symmetrix is in a Write-disabled state, and the SRDF links are down, the package will not be started. Since we cannot check the state of the Symmetrix on the R1 side to validate conditions, the data on the R2 side may be non-current and thus a risk that data loss will occur when starting the package up on the R2 side.
Environment File Variables for MetroCluster/SRDF perform a R1/R2 swap, set AUTOSWAPR2=1 and an automatic R1/R2 swap will occur only when the SRDF link and the two Symmetrix are properly functioning. If AUTOSWAPR2 is set to 1, then CONSISTENCYGROUPS can not be set to 1. Ensure that you have the minumum requirements for R1/R2 Swapping by referring to most recent version of the MetroClusters release notes. CLUSTER_TYPE This parameter defines the clustering environment in which the script is used.
Environment File Variables for MetroCluster/SRDF RETRY • symcfg.out – contains the results of symcfg list command, used for model and revision information. • symrdf.out – contains the results of symrdf query commands run from the package control script. • awk.out – contains the output from using awk to parse the symrdf.out file. • FORCEFLAG – forces a package to start automatically under certain circumstances if this file is present.
Configuration File Parameters for ContinentalClusters C Configuration File Parameters for ContinentalClusters This appendix lists all ContinentalClusters configuration file variables. See Chapter 5, “Building a Continental Cluster,” for suggestions on coding these parameters.
Configuration File Parameters for ContinentalClusters • unreachable - the cluster is unreachable • down - the cluster is down, but nodes are responding • error - an error is detected The maximum length is 47 characters. When the MONITORING_CLUSTER detects a change in status, one or more notifications are sent, as defined by the NOTIFICATION parameter, at time intervals defined by the CLUSTER_ALERT and CLUSTER_ALARM parameters.
Configuration File Parameters for ContinentalClusters The parameter consists of a pair of names: the name of the cluster that receives the data to be replicated (usually the Recovery Cluster) as defined in the MC/ServiceGuard cluster configuration ASCII file, followed by a slash (“/”), followed by the name of the data replication receiver package as defined in the MC/ServiceGuard package configuration ASCII file.
Configuration File Parameters for ContinentalClusters MONITORING_CLUSTER Name This is name of the cluster that polls the cluster named in the CLUSTER_EVENT and sends notification. Maximum length is 40 characters. NODE_NAME nodename This is the unqualified node name as defined in the DNS name server configuration. Maximum size is 40 characters. NOTIFICATION Destination “Message” This is a destination and message associated with a specific CLUSTER_ALERT or CLUSTER_ALARM.
Configuration File Parameters for ContinentalClusters • TEXTLOG Pathname - append the specified message to a specified text log file. • UDP Nodename:Portnumber - send the specified message to a UDP port on the specified node. Any number of notifications may be associated with a given alert or alarm.
Configuration File Parameters for ContinentalClusters 422 Appendix C
ContinentalClusters Command and Daemon Reference D ContinentalClusters Command and Daemon Reference This appendix lists all commands and daemons used with ContinentalClusters. Manual pages are also available online. cmapplyconcl [-v] [-C] filename This command verifies the ContinentalClusters configuration as specified in filename, creates or updates the binary, and distributes it to all nodes in the continental cluster.
ContinentalClusters Command and Daemon Reference This command verifies the ContinentalClusters configuration specified in filename. It is not necessary to halt the ServiceGuard cluster in order to run this command; however, the ContinentalClusters monitor package must be halted. This command will parse the ASCII_file to ensure proper syntax, check parameter lengths, and validate object names such as the CLUSTER_NAME and NODE_NAME. Options are: -C filename The name of the ASCII configuration file.
ContinentalClusters Command and Daemon Reference whether to proceed with deleting the configuration on the reachable nodes. If this option is used and some node has configuration files for a continental cluster with a different name, you will be prompted to indicate whether to proceed with deleting the configuration on that node. cmforceconcl ServiceGuardPackageEnableCommand This command is used to force a ContinentalClusters package to start.
ContinentalClusters Command and Daemon Reference Declares an alternate location for the configuration file. The default is/etc/cmcluster/cmoncl.config. cmreadlog -f input_file [output_file] This command formats the content of Object Manager and other log files for easier reading. The command is used when reading the /var/opt/cmom/cmomd.log file and the /var/adm/cmconcl/sentryd.log file. Options are: -f input_file Specifies the name of the managed object file (MOF file) to be read.
ContinentalClusters Command and Daemon Reference packages are started by enabling package switching globally (cmmodpkg -e) for each package. This will cause the package to be started on the first available node within the recovery cluster. The cmrecovercl command can only be run on a recovery cluster. The cmrecovercl command will fail if there has not been sufficient time since the primary cluster became unreachable.
ContinentalClusters Command and Daemon Reference 428 Appendix D
Glossary Business Recovery Service Glossary A application restart Starting an application, usually on another node, after a failure. Application can be restarted manually, which may be necessary if data must be restarted before the application can run (example: Business Recovery Services work like this.) Applications can by restarted by an operator using a script, which can reduce human error.
Glossary campus cluster C campus cluster A single cluster that is geographically dispersed within the confines of an area owned or leased by the organization such that it has the right to run cables above or below ground between buildings in the campus. Campus clusters are usually spread out in different rooms in a single building, or in different adjacent or nearby buildings. See also extended distance cluster.
Glossary disaster database. Consistency groups allow you to configure R1/R2 devices on multiple Symmetrix frames in MetroCluster/SRDF. continental cluster A group of clusters that use routed networks and/or common carrier networks for data replication and cluster communication to support package failover between separate clusters in different data centers. Continental clusters are often located in different cities or different countries and can span 100s or 1000s of kilometers.
Glossary disaster protection disaster protection (Don’t use this term?) Processes, tools, hardware, and software that provide protection in the event of an extreme occurrence that causes application downtime such that the application can be restarted at a different location within a fixed period of time. disaster recovery The process of restoring access to applications and data after a disaster.
Glossary mission critical solution H, I M heartbeat network A network that provides reliable communication among nodes in a cluster, including the transmission of heartbeat messages, signals from each functioning node, which are central to the operation of the cluster, and which determine the health of the nodes in the cluster. M by N A type of Symmetrix grouping in which up to two Symmetrix frames may be configured on either side of a data replication link in a MetroCluster/SRDF configuration.
Glossary multiple points of failure (MPOF) multiple points of failure (MPOF) More than one point of failure that can bring down an MC/ServiceGuard cluster. P package alert Time at which a message is sent indicating a problem with a package. multiple system high availability Cluster technology and architecture that increases the level of availability by grouping systems into a cooperative failover design.
Glossary rolling disaster Primary Cluster A cluster in production that has packages protected by the HP ContinentalClusters product. primary package The package that normally runs on the Primary Cluster in a production environment. pushbutton failover Use of the cmrecovercl command to allow all package recovery groups to start up on the Recovery Cluster following a significant cluster event on the Primary Cluster.
Glossary single point of failure (SPOF) S single point of failure (SPOF) A component of a cluster or node that, if it fails, affects access to applications or services. See also multiple points of failure. SVOL A secondary volume configured in an XP series disk array that uses Continuous Access. SVOLs are the secondary copies in physical data replication with Continous Access on the XP.
Glossary WAN data replication solutions transparent IP failover Moving the IP address from one network interface card (NIC), in the same node or another node, to another NIC that is attached to the same IP subnet so that users or applications may always specify the same IP name/address whenever they connect, even after a failure. U-Z volume group In LVM, a set of physical volumes such that logical volumes can be defined within the volume group for user access.
Glossary transparent IP failover 438 Glossary
Index A adding a node to ContinentalClusters configuration, 265 adding a recovery group in ContinentalClusters, 255, 266, 267 alarms and cluster events, 186 how used, 187 alerts and cluster events, 186 how used, 186 application recovery in a continental cluster, 182 applying the continental clusters configuration, 237 arbitrator nodes, 73 , 77, 78, 127, 129, 130 asynchronous data replication, 31 AUTO_FENCEDATA_SPLIT in MetroCluster/CA, 399, 411 AUTO_NONCURDATA in MetroCluster/CA, 400 AUTO_PSUEPSUS in MetroC
Index cmclrmond ContinentalClusters daemon, 424 cmclsentryd ContinentalClusters daemon, 424 cmdeleteconcl ContinentalClusters daemon, 424 cmdeleteconcl command, 273 cmomd Object Manager daemon, 425 cmqueryconcl ContinentalClusters command, 425 cmreadlog ContinentalClusters command, 426 cmrecovercl, 249 ContinentalClusters command, 426 how the command works in ContinentalClusters, 253 command line cmrecovercl, 249 symdg, 153 symgate, 155 command line interface, EMC Symmetrix, 147 concepts in ContinentalClus
Index ideal, 36 logical, 34 off-line, 30 online, 31 over WAN, 194 physical, 31, 285, 325 restoring after a disaster, 256 synchronous or asynchronous, 31 data replication procedures for cascading failover, 379 DATA_RECEIVER_PACKAGE ContinentalClusters configuration file parameter, 418 DATA_SENDER_PACKAGE ContinentalClusters configuration file parameter, 419 DC1HOST package control script variables, 109, 301, 305, 309 deleting a continental cluster, 273 Design of disaster tolerant cluster MetroCluster/CA, 72
Index building, 49 definition, 22 mapping EMC Symmetrix and HP-UX devices, 151, 154 F failback scenarios in ContinentalClusters with Continuous Access XP, 315 in ContinentalClusters with EMC SRDF, 348 failover cascading, 363 FDDI, disaster tolerant, 39 FENCE in MetroCluster/CA, 404 FibreChannel clusters, 50 G gatekeeper devices, 155 geographic dispersion of nodes, 29 H hardware configuration checklist for MetroCluster, 83, 135 hardware for ContinentalCluster Recovery cluster, 207 HORCMINST in MetroCluste
Index disaster tolerant Ethernet, 40 disaster tolerant FDDI, 39 disaster tolerant WAN , 41 node adding to ContinentalClusters, 265 NODE_NAME ContinentalClusters configuration file parameter, 420 nodes allowed in three-data-center architecture, 74, 128 arbitrator nodes, 77, 129 arbitrators, 127 used as arbitrators, 73 NOTIFICATION ContinentalClusters configuration file parameter, 420 notifications entering in ContinentalClusters configuration file, 226 how they work, 186 receiving, 248 O off-line data repli
Index RECOVERY_PACKAGE ContinentalClusters configuration file parameter, 421 redundant power sources, 36 remote control unit on XP series, 73 Remote Control Unit.
Index worksheet for package configuration MetroCluster/CA, 85 worksheet, MetroCluster, 84, 136 worksheet, package, 85, 137 X XP series supported configurations, 88 verify configuration, 102 XP series disk array with MetroCluster, 69 445
