Designing Disaster Recovery HA Clusters using Metrocluster and Continentalclusters HP Part Number: 5900-1881 Published: October 2011
Legal Notices © Copyright 2011 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor’s standard commercial license. The information contained herein is subject to change without notice.
Contents Printing History...........................................................................................17 Preface......................................................................................................19 Guide to Disaster Recovery Solutions Documentation...................................................................20 1 Designing a Metrocluster...........................................................................
Continentalclusters Worksheets............................................................................................52 Data Center Worksheet .................................................................................................52 Recovery Group Worksheet ...........................................................................................53 Cluster Event Worksheet ................................................................................................
To Start the Failover Process................................................................................................95 How the cmrecovercl Command Works.................................................................................97 Forcing a Package to Start.......................................................................................................97 Restoring Disaster Tolerance.....................................................................................................
Configuring the Storage Device for the Complex Workload at the Recovery Cluster.........136 Configuring the Identical Complex Workload Stack at the Recovery Cluster....................139 Halting the Complex Workload on the Recovery Cluster..............................................139 Configuring the Site Controller Package in the Recovery Cluster.........................................139 Configuring Site Safety Latch Dependencies....................................................................
Journal Cache, Journal Volumes, and Inflow Control........................................................161 Continuous Access Journal Pair State.............................................................................161 Limitations of Continuous Access Journal........................................................................161 One-to-One Volume Copy Operations...........................................................................162 One-to-One Journal Group Operations......................
Resynchronizing..........................................................................................................194 Using the pairresync Command....................................................................................195 Failback.....................................................................................................................195 Timing Considerations.................................................................................................
Setting a Default Management Server............................................................................219 Displaying the List of Management Servers.....................................................................219 Adding or Updating Management Server Information......................................................220 Deleting a Management Server.....................................................................................221 Defining EVA Storage Cells and DR Groups....................
5 Building Disaster Recovery Serviceguard Solutions Using Metrocluster with EMC SRDF.......................................................................................................255 Files for Integrating Serviceguard with EMC SRDF.....................................................................255 Overview of EMC and SRDF Concepts....................................................................................256 Preparing the Cluster for Data Replication.....................................
Metrocluster SRDF Topology using SRDF/Asynchronous....................................................291 Configuring Metrocluster with EMC SRDF using SRDF/Asynchronous...........................................292 Building a Device Group for SRDF/Asynchronous................................................................292 Package Configuration using SRDF/Synchronous or SRDF/Asynchronous................................294 First-time installation of Metrocluster with EMC SRDF using SRDF/Synchronous.......
Defining Storage Units......................................................................................................320 Creating and Exporting LVM Volume Groups .................................................................320 Creating VxVM Disk Groups ........................................................................................321 Configuring Modular Packages.....................................................................................322 Starting the Metrocluster Package......
Configuring the Storage Device for Complex Workload at the Target Disk Site...............353 Configure the Identical Complex Workload Stack at the Recovery SIte...........................355 Halting the Complex Workload on the Recovery Site...................................................355 Configuring the Site Controller Package for the Complex Workload...................................355 Configuring the Site Safety Latch Dependencies for a Complex Workload..........................
Configuring the Site Controller Package..............................................................................390 Configuring the Site Safety Latch Dependencies...................................................................390 Starting the Disaster Tolerant Oracle RAC Database with ASM in the Metrocluster....................391 Understanding Failure Scenarios in a Site Aware Disaster Tolerant Architecture.............................391 Failure Scenarios in a Complex Workload.....................
3DC DR Solution Configuration.........................................................................................424 Tri-Link Configuration...................................................................................................425 Bi-Link Configuration....................................................................................................427 Overview of Device Group Monitor Feature in 3DC DR Solution.............................................428 Infrastructure Requirements.....
Data Maintenance with the Failure of a Metrocluster with Continuous Access for P9000 and XP Failover...............................................................................................................................473 Swap Takeover Failure (for Continuous Access Device Group Pair Between DC1 and DC2).......473 Takeover Timeout (for third data center)...............................................................................
Printing History Table 1 Editions and Releases Printing Date Part Number Edition Operating System Releases (see Associated Release Notes Note below) December 2006 B7660-90019 Edition 1 HP-UX 11i v1 and 11i v2 September 2007 B7660-90021 Edition 2 HP-UX 11i v1, 11i v2 and 11i v3 December 2007 B7660-90023 Edition 3 HP-UX 11i v1, 11i v2 and 11i v3 September 2008 B7660–90025 Edition 4 HP-UX 11i v1, 11i v2 and 11i v3 March 2009 B7660-90026 Edition 5 HP-UX 11i v2 and 11i v3 September 2009 B76
NOTE: This document describes a group of separate software products that are released independently of one another. Not all products described in this document are necessarily supported on all the same operating system releases. Consult your product’s Release Notes for information about supported platforms. HP Printing Division: ESS Software Division Hewlett-Packard Co. 19111 Pruneridge Ave.
Preface The following two guides describe disaster tolerant clusters solutions using Serviceguard, Metrocluster with Continuous Access for P9000 and XP, Metrocluster Continuous Access EVA, Metrocluster EMC SRDF, and Continentalclusters: • Understanding and Designing Serviceguard Disaster Tolerant Architectures • Designing Disaster Recovery HA Clusters Using Metrocluster and Continentalclusters The Understanding and Designing Serviceguard Disaster Tolerant Architectures guide provides an overview of Hewl
Table 2 outlines the types of disaster tolerant solutions and their related documentation.
Table 2 Disaster Tolerant Solutions Document Road Map (continued) To Set up Read Continentalclusters using EMC SRDF data replication Understanding and Designing Serviceguard Disaster Tolerant Architectures • Chapter 1: Disaster Tolerance and Recovery in a Serviceguard Cluster Designing Disaster Recovery HA Clusters Using Metrocluster and Continentalclusters • Chapter 2: Designing Continentalclusters • Chapter 5: Building Disaster Recovery Serviceguard Solutions Using Metrocluster with EMC SRDF Continent
• Using High Availability Monitors • Using the Event Monitoring Service When using VxVM storage with Serviceguard, refer to the following: • VERITAS Volume Manager Administrator’s Guide. This contains a glossary of VERITAS terminology.
1 Designing a Metrocluster This chapter describes the configuration and management of a basic metropolitan cluster through the following topics: • Designing a Disaster Recovery Architecture for use with Metrocluster Products • Single Data Center • Two Data Centers and Third Location with Arbitrator(s) • Package Configuration Worksheet • Disaster Tolerant Checklist • Cluster Configuration Worksheet • Next Steps In addition, this chapter outlines the general characteristics of the metropolitan c
Following are the guidelines that must be followed to configure a Serviceguard cluster across network subnets: • All the nodes in the cluster must belong to the same domain. • The latency period in the heartbeat network that is configured across subnets must be less than 200 milliseconds. • A minimum of two heartbeat subnets must be configured for all cluster nodes. • Each heartbeat subnet on a node must be routed using a different physical route to the other heartbeat subnet on the other node.
Figure 1 Two Data Centers and Third Location with Arbitrators A disk array can be the main disk array for one set of packages and the remote disk array for another. In Figure 1, the P9000 and XP disk array in data center A is the main or primary disk array for packages A and B, and the remote or secondary disk array for packages C and D in data center B. For packages A and B, data is written to PVOLs on the array in Data Center A and replicated to SVOLs on the array in Data Center B.
Table 3 Supported System and Data Center Combinations Data Center A Data Center B Data Center C Serviceguard Version 1 1 1 Arbitrator Node A.11.16 or later 1 1 Quorum Server System A.11.16 or later 2 1 2 Arbitrator Nodes A.11.16 or later 1 2 2 Arbitrator Nodes A.11.16 or later 2 2 1 Arbitrator Node A.11.16 or later 2 2 2* Arbitrator Nodes A. 11.16 or later 2 2 Quorum Server System A. 11.16 or later 3 3 1 Arbitrator Node A. 11.16 or later 3 3 2* Arbitrator Nodes A. 11.
nodes are listed in decreasing order of preference. In a Metrocluster configuration, the node names list in the package configuration is ordered by site. Node names from the same site are listed sequentially. The node names of the site with the primary node must be specified first in the list. The site with the primary node is referred as primary site and the other site is referred as alternate site in this section.
NODE_NAME SJC_2 SITE san_jose ........ Use the cmviewcl command to view the list of sites that are configured in the cluster and their associated nodes. Following is a sample of the command, and the output: cmviewcl -l node SITE_NAME NODE SFO_1 SFO_2 .........
Arbitrator Node Configuration Rules Although you can use one arbitrator, having two arbitrators provides greater flexibility in taking systems down for planned outages as well as providing better protection against multiple points of failure. Using two arbitrators: • Provides local failover capability to applications running on the arbitrator. • Protects against multiple points of failure (MPOF). • Provides for planned downtime on a single system anywhere in the cluster.
Figure 2 Failover Scenario with a Single Arbitrator The scenarios in Table 4, based on Figure 2, illustrate possible results if one or more nodes fail in a configuration with a single arbitrator.
Figure 3 Failover Scenario with Two Arbitrators The scenarios in Table 5 illustrate possible results if a data center or one or more nodes fail in a configuration with two arbitrators. Note that 3 of the 4 scenarios that caused a cluster halt with a single arbitrator, do not cause a cluster halt with two arbitrators.
Figure 4 Disaster Tolerant Checklist Cluster Configuration Worksheet Use this cluster configuration worksheet either in place of, or in addition to the worksheet provided in the Managing Serviceguard user’s guide. If you have already completed a Serviceguard cluster configuration worksheet, you only need to complete the first part of this worksheet.
Figure 6 Modular package configuration worksheet Worksheets 33
Designing a Metrocluster
Figure 7 Package Configuration Worksheet Figure 8 Package Control Script Worksheet Next Steps To implement the metropolitan cluster design, use the procedures in the following chapters: • “Building Disaster Recovery Serviceguard Solutions Using Metrocluster with Continuous Access for P9000 and XP” (page 153) • “Building Disaster Recovery Serviceguard Solutions Using Metrocluster with Continuous Access EVA” (page 211) • “Building Disaster Recovery Serviceguard Solutions Using Metrocluster with EMC SRD
2 Designing Continentalclusters Unlike metropolitan and campus clusters, which have a single-cluster architecture, a Continentalclusters uses multiple Serviceguard clusters to provide application recovery over local or wide area network (LAN and WAN).
Because clusters may be separated over wide geographical distances, and because they have independent function, the operation of clusters in a Continentalclusters configuration is somewhat different from that of typical Serviceguard clusters. A typical Continentalclusters recovery pair environment with versions prior than Continentalclusters A.08.00 is shown in Figure 9. A sample Continentalclusters configuration with Continentalclusters version A.08.00 is shown in Figure 10 (page 37).
York cluster is configured with a recovery version of the packages that are running on the Los Angeles cluster. These packages do not run under normal circumstances, but are set to start up when they are needed. In addition, either cluster may run other packages that are not involved in Continentalclusters operation. Mutual Recovery Configuration Bi-directional failover is supported in what is called a “mutual recovery configuration.
In the above figure, the salespkg is running on the New York cluster and can be recovered by the Los Angeles cluster. Similarly, the custpkg running on the Los Angeles cluster can be recovered by the New York cluster. As stated previously, physical data replication is carried out using ESCON (Enterprise Storage Connect) links between the disk array hardware in New York and Los Angeles via an ESCON/WAN converter at each end. Each cluster is running a monitor that checks the status of the alternate cluster.
Monitoring over a Network A monitor package running on one cluster tracks the health of another cluster in the recovery pair and sends notification to configured destinations if the state of the monitored cluster changes. (If a cluster contains any packages to be recovered it must be monitored.) The monitor software polls the monitored cluster at a specific MONITOR_INTERVAL defined in an ASCII configuration file, which also indicates when and where to send messages if there is a state change.
Table 6 Monitored States and Possible Causes (continued) Cluster Event (Old state -> New state) Cluster-related Causes Network-related Causes Unreachable -> Up Cluster nodes were rebooted and the cluster started Network came up and the cluster was already running Error -> Up Error resolved, cluster is up Network problem was fixed, cluster is up NOTE: There is only one condition under which cmclsentryd will determine that the cluster has Error status: all nodes are unreachable except those which hav
• Notification that a cluster came down for any reason. • Notification that a cluster has been in an unreachable state for a short period of time. An alert is sent in this case as a warning that an alarm might be issued later if the cluster’s state remains unreachable for a longer time. The expected process in dealing with alerts is to continue watching for additional notifications and to contact individuals at the site of the monitored cluster to see whether problems exist.
Maintenance mode for recovery groups is an optional feature. You must explicitly configure Continentalclusters to use this feature. Consider the following guidelines when you move a recovery group into the maintenance mode: • Configure a shared disk with a file system in all primary clusters and in the recovery cluster. This shared disk is local to the cluster and need not be replicated.
Run this command only on recovery cluster nodes. This command succeeds only when Continentalclusters is configured for maintenance mode. The command checks for the following conditions to successfully disable a recovery group: • The recovery package is down and package switching is disabled. • The primary cluster and the primary package are up. If the cluster is down or unreachable, use the force -f option to forcefully disable the recovery group.
NOTE: After a recovery, it is not possible to reverse directions and return a package to its original cluster without first reconfiguring the data replication hardware and/or software and synchronizing data. Therefore, be very cautious when deciding to use the cmrecovercl command. It is for this reason, HP recommends that stringent procedures and processes are in place to aid in making the decision to complete a recovery process.
replication from the primary cluster. Finally, you need to move the recovery group out of the maintenance mode by enabling it using the cmrecovercl -e command. WARNING! Ensure that the storage system of the recovery group is synchronized with the latest data and the replication environment is restored before the recovery group is moved out of the maintenance mode. Failure to do so can result in the recovery package using production data that was invalidated by the rehearsal run during a subsequent recovery.
To prevent packages from being started at the wrong time and in the wrong place, use the following strategies: • Set the AUTO_RUN (PKG_SWITCHING_ENABLED used prior to Serviceguard A.11.12) parameter for all primary and recovery packages to NO. • Ensure that recovery package names are well known, and that personnel understand they should never be started with a cmrunpkg or cmmodpkg command unless the cmrecovercl command has been invoked first.
Table 8 Serviceguard and Continentalclusters Commands (continued) Commands How the commands work in Serviceguard How the commands work in Continentalclusters cmhaltnode -f halts a node in a highly available cluster Will not re-enable switching on a recovery package if any of the primary, data receiver, or data sender package in the same recovery group is running or enabled.
NOTE: Take note when cluster A takes over for cluster B, it must run cluster B’s packages as well as any packages that it was already running on its own, unless those packages are stopped intentionally. Data Replication Data replication between the Serviceguard clusters in a Continentalclusters recovery pair extends the scope of high availability to the level of the Continentalclusters. Select a technology for data replication between the two clusters.
Table 9 Data Replication and Continentalclusters (continued) Replication Type How it Works Continentalclusters Implication replication inappropriate for Continentalclusters. Physical Replication of Disk Units via Hardware Replication of the LUNs across disk arrays through dedicated hardware links such as EMC SRDF, or Continuous Access P9000, or XP, or Continuous Access EVA, or 3PAR Remote Copy.
Details on configuring the special Continentalclusters control scripts are in Chapters 3, 4, and 5. Some additional notes are provided below. Multiple Recovery Pairs in Continentalclusters One or more than one recovery pair can be configured in Continentalclusters. In the Continentalclusters configuration that contains more than one recovery pair, more than one primary cluster is configured to share a common recovery cluster.
Data Center Processes Continentalclusters provides the cmrecovercl command that fails over all applications on the primary cluster in a recovery pair that are protected by Continentalclusters. However, application failover also requires well-defined processes for the two sites of a recovery pair. These processes and procedures should be written down and made available at both sites.
Recovery Group Worksheet The following worksheet will help you organize and record your specific recovery groups. Fill out the worksheet and keep it for future reference.
Understanding Changes in Continentalclusters A.08.00 Continentalclusters version A.08.00 uses Secure Shell (SSH) for secure communication across all nodes. The following additional components are used in Continentalclusters A.08.00: • Continentalclusters User and Group • Continentalclusters Configuration Package Continentalclusters User and Group Continentalclusters version A.08.00 uses Secure Shell (SSH) for all inter-cluster and node communication.
are used as your choice of data replication method, configure and apply them as described in the next sections before applying the Continentalclusters configuration. Table 10 shows the types of packages that are needed for each type of data replication.
7. 8. Test local failover of the packages. In our sample case, this would mean enabling package switching for salespkg (cmmodpkg -e salespkg) and then testing that salespkg fails over from LAnode1 to LAnode2. If using logical data replication, configure and test the data sender package if one is needed.
5. will be consistent. See chapter 5 in the Managing Serviceguard user’s guide for details on cluster configuration. Set up the recovery package(s). Starting with Continentalclusters A.08.00, packages in Continentalclusters can be configured as modular packages. HP recommends configuring this as a modular package. For information about configuring primary and recovery packages as modular, see “Configuring Primary and Recovery Packages as Modular Packages” (page 67).
Figure 16 Sample Cluster Configuration with Recovery Packages Configuring Recovery Groups with Rehearsal Packages The rehearsal package is a regular Serviceguard package configured on the recovery cluster of the recovery group. You must configure the rehearsal package with the same volume group and file system mount points as that of the recovery package. The application setup and configuration used for the recovery package are also used for the rehearsal package.
Setting up Security With Continentalclusters version prior to A.08.00, setting up security involves preparing the security files, and ensuring that the network security requirements are met. Starting with Continentalclusters A.08.00, a secure communication using SSH must be set up for inter-cluster operations. Issue the following command to retrieve the Continentalclusters version from a node: swlist -l product | grep -i Continentalclusters ContClusters A.08.00.00.
supported by the Border Gateway Protocol (BGP). allow from This lists all the nodes that are allowed access. Permissible entries are: all All hosts are allowed access. domain Hosts whose names match, or end in, this string are allowed access, for example, hp.com. hostname The named host (for example, kitcat.myco.com) is allowed access. IP address Either a full IP address, or a partial IP address of 1 to 3 bytes for subnet inclusion is allowed.
Complete the following steps to set up the SSH environment for Continentalclusters A.08.00 on all nodes of all clusters to be configured in a Continentalclusters: 1. Set a password for the Continentalclusters user, conclusr, on all nodes of all clusters to be configured in a Continentalclusters: a. Log in as root user. b. Set the password for conclusr on the node passwd conclusr passwd conclusr 2. Setup SSH equivalence between the nodes in the Continentalclusters. a.
Cannot view the cluster configuration: Permission denied. This user doesn't have access to view the cluster configuration. To resolve this error, edit the cluster configuration file and add the following information: USER_NAME USER_HOST USER_ROLE conclusr ANY_SERVICEGUARD_NODE MONITOR Apply the cluster configuration file. Now, you should be able to view the cluster configuration using the cmviewcl command.
5. disk. For more information configuring this maintenance mode feature, see “Configuring the Maintenance Mode Feature for Recovery Groups in Continentalclusters” (page 63). Use the cmcheckconf command to validate the package. # cmcheckconf -P ccmonpkg.config 6. 7. Copy the package configuration file ccmonpkg.config and control script ccmonpkg.cntl to the monitor package directory (default name /etc/cmcluster/ccmonpkg) on all the other nodes in the cluster. Make sure this file is executable.
4. Create the volume group: vgcreate /dev/ccvg /dev/c0t10d0 5. Activate the volume group: vgchange -a y ccvg 6. Create the logical volume: lvcreate -L 250M ccvg Run the following command to create a file system on the volume: mkfs vxfs /dev/ccvg/lvol1 Complete the following procedure to export the volume group configuration and import the volume group on all the nodes at the recovery cluster: 1.
NOTE: Starting with Continentalclusters A.08.00, the monitor package can be configured as a modular package. In addition, the existing monitor package configured as a legacy package can be migrated to a modular style package. HP recommends configuring this as a modular package.
3. Skip this step if you are not using the DR Rehearsal feature.
Configuring Primary and Recovery Packages as Modular Packages In a Continentalclusters configuration, starting with version A.08.00, packages specified as a primary_package or a recovery_package in a recovery group can be configured as a modular package. The packages can be created using any modules supplied with and approved in HP Serviceguard.
1. Run the following command to create an XPCA modular package configuration file: # cmmakepkg –m dts/ccxpca temp.config 2. Edit the following attributes in the temp.config file: • dts/dts/dts_pkg_dir This is the package directory for this modular package. The Metrocluster environment file is generated for this package in this directory. This value must be unique for all packages.
Complete the following procedure to migrate legacy style primary and recovery packages to modular packages using Continentalclusters A.08.00: 1. Create a modular package configuration file for the legacy package. # cmmigratepkg -p [-s] -o IMPORTANT: This command generates a package configuration file. Do not apply this configuration file until you complete the migration procedure.
with Continuous Access EVA is not installed on all the nodes, then the following error message is displayed: The file does not exist or read/search permission not set for a component of the path: No such file or directory /etc/cmcluster/modules/dts/mc.2:4: Could not find include ....... When configuring modular packages using Continuous Access EVA, only the package configuration file needs to be edited.
files are created. Only the final package configuration file that is created at the end of the procedure must be applied. Complete the following procedure to migrate Continuous Access EVA legacy packages to modular packages using Continentalclusters A.08.00: 1. Create a modular package configuration file for the Continentalclusters legacy package. # cmmigratepkg -p [-s] -o IMPORTANT: This command generates a package configuration file.
be installed on all nodes in the Continentalclusters. If Metrocluster with EMC SRDF is not installed on all the nodes, then the following error message is displayed: The file does not exist or read/search permission not set for a component of the path: No such file or directory /etc/cmcluster/modules/dts/mc.2:4: Could not find include ....... When configuring modular packages with EMC SRDF, only the package configuration file needs to be edited.
the legacy package configuration. While completing the migration procedure, multiple package configuration files are created. Only the final package configuration file that is created at the end of the procedure must be applied. Complete the following procedure to migrate Continentalclusters with EMC SRDF legacy packages to modular packages using Continentalclusters A.08.00: 1. Create a modular package configuration file for the legacy package. # cmmigratepkg -p [-s] -o
Configuring primary and recovery packages as modular packages using 3PAR Remote Copy When using HP 3PAR Remote Copy in Continentalclusters, the primary and recovery packages must be created using the dts/cc3parrc module. To use this module, Metrocluster with 3PAR Remote Copy must be installed on all nodes in the Continentalclusters. To configure primary and recovery packages in Continentalclusters when using 3PAR Remote Copy : 1.
For example, in a Continentalclusters configuration using Continuous Access P9000 or XP, the recovery package must be created with dts/ccxpca module. While creating the rehearsal package for this recovery group, the dts/ccxpca module must not be included. Complete the following procedure to create a modular rehearsal package using Continentalclusters A.08.00: 1. Create a package configuration identical to the recovery package configuration but without any Continentalclusters module. 2.
Table 11 Configuration rules for using Modular style packages in Continentalclusters packages (continued) Continentalclusters Continuous Access Continuous Access Continuous Package Type P9000 or XP EVA Access SRDF 3PAR Remote Copy Logical Replication Continentalclusters specific module required. Data Receiver Package Use any Serviceguard supported module. No Continentalclusters specific module required.
It is suggested to use the name “ccmonpkg” for all Continentalclusters monitors. Create this package on each cluster containing a recovery package. If it is not desired to monitor a cluster, which does not containing a recovery package, it is required to delete or comment out the MONITOR_PACKAGE_NAME line and the MONITOR_INTERVAL line. For mutual recovery, create the monitor package on both the first and second clusters. NOTE: 5. • Starting with Continentalclusters A.08.
#### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### 78 #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ##
Editing Section 2 – Recovery Groups In this section of the file, define recovery groups, which are sets of Serviceguard packages that are ready to recover applications in case of cluster failure. Create a separate recovery group for each package that will be started on a cluster when the cmrecovercl(1m) command is issued on that cluster. Examples of recovery groups are shown graphically in Figure 17 and Figure 18.
Figure 18 Sample Bi-directional Recovery Groups Enter data in Section 2 as follows: 1. Enter a name for the recovery group following the RECOVERY_GROUP_NAME keyword. This can be any name you choose. 2. After the PRIMARY_PACKAGE keyword, enter a primary package definition consisting of the cluster name followed by a slash (/) followed by the package name. Example: PRIMARY_PACKAGE LAcluster/custpkg 3. 4.
A printout of Section 2 of the Continentalclusters ASCII configuration file follows. ############################################################### #### #### Section 2. Recovery Groups #### #### #### #### This section defines recovery groups--sets #### #### #### #### of ServiceGuard packages that are ready to #### #### #### #### recover applications in case of cluster #### #### #### #### failure.
#### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### primary cluster. The recovery package name #### #### includes the recovery cluster name, followed #### #### by a slash ("/")followed by the package name #### #### on the recovery cluster.
#### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #
#### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### 84 #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### ###
#### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #
If you specify any other location for logging, the following error message appears: The target after textlog “ ” is not valid. Please specify a file under /var/opt/resmon/log directory If you upgraded Continentalclusters but are still using the old configuration file, the textlog location is still specified as /var/adm/cmconcl. As a result, the following error message appears: The file path “s” specified for textlog is invalid. The destination file must be under /var/opt/resmon/log directory.
CLUSTER_DOMAIN cup.hp.com NODE_NAME node11 NODE_NAME node12 MONITOR_PACKAGE_NAME ccmonpkg MONITOR_INTERVAL 60 seconds CLUSTER_NAME CLUSTER_DOMAIN NODE_NAME NODE_NAME cluster2 cup.hp.com node21 node22 CLUSTER_NAME cluster3 CLUSTER_DOMAIN cup.hp.
“DRT: (Ora-test) DOWN alert” NOTIFICATION SYSLOG “DRT: (Ora-test) cluster3 DOWN alert” CLUSTER_ALARM 30 SECONDS NOTIFICATION TEXTLOG /var/opt/resmon/log/CCTextlog “DRT: (Ora-test) DOWN alarm” NOTIFICATION SYSLOG “DRT: (Ora-test) cluster3 DOWN alarm” CLUSTER_EVENT cluster1/UP MONITORING_CLUSTER cluster3 CLUSTER_ALERT 30 SECONDS NOTIFICATION TEXTLOG /var/opt/resmon/log/CCTextlog “DRT: (Ora-test) UP alert” NOTIFICATION SYSLOG “DRT: (Ora-test) cluster1 UP alert” CLUSTER_EVENT cluster2/UP MONITORING_CLUSTER clus
NOTE: If any problems occur during the execution of cmapplyconcl, repeat the command as often as necessary. Issuing the command will delete the existing Continentalclusters configuration and apply the new one. When configuration is finished, your systems should have sets of files similar to those shown in Figure 19. Figure 19 Continentalclusters Configuration Files Starting the Continentalclusters Monitor Package Starting the monitoring package enables all Continentalclusters monitoring functionality.
Validating the Configuration The following table shows the status of Continentalclusters packages in a recovery pair when each cluster is running normally and no recovery has taken place.
7. 8. Change each cluster’s state to test that the monitor running on the monitoring cluster will detect the change in status and send notification. View the status of the Continentalclusters primary and recovery clusters, including configured event data.
Migrating to Continentalclusters A.08.00 Continentalclusters version A.08.00 includes enhanced features and capabilities, such as support for modular packages, IPv6 support, and a secure communication protocol for inter-cluster operations. HP recommends that you migrate Continentalclusters to the latest version to obtain the benefits of these features. Upgrading to Continentalclusters A.08.00 requires re-applying the Continentalclusters configuration.
7. 8. If using physical data replication, do not resync from the recovery cluster to the primary cluster. Instead, manually issue a command that will overwrite any changes on the recovery disk array that may inadvertently have been made. Start the package up on the primary cluster and allow connection to the application. Testing Continentalclusters Operations Use the following procedures to exercise typical Continentalclusters behaviors: 1.
NOTE: 7. Continentalclusters monitors cluster status, but not package status. View the status of the Continentalclusters. # cmviewconcl Switching to the Recovery Packages in Case of Disaster Once the clusters are configured and tested, packages will be able to fail over to an alternate node in another data center and still have access to the data they need to function. The primary steps for failing over a package are: 1. Receive notification that a monitored cluster is unavailable. 2.
Using the Recovery Command to Switch All Packages If Metrocluster Continuous Access P9000 or XP, or Metrocluster Continuous Access EVA, or Metrocluster with EMC SRDF, or Metrocluster with 3PAR Remote Copy is not the chosen data replication technology, use the following steps before executing the Continentalclusters recovery command, cmrecovercl.
If the monitored cluster comes back up following an alert or alarm, but it is certain that the primary packages cannot start (say, because of damage to the disks on the primary site), then use a special procedure to initiate recovery: 1. 2. 3. Use the cmhaltcl command to halt the primary cluster. Wait for the monitor to send an alert. Use cmrecovercl -f to perform recovery.
Table 13 Status of Continentalclusters Packages After Recovery Primary Cluster Recovery Cluster Data Replication Method Primary Package Data Sender Package Optional Monitor Package Recovery Package Data Receiver Required Package Monitor Package Physical— Symmetrix Halted Not used Halted or Running Running Not used Halted or Running Physical— P9000 Halted or XP Series Not used Halted or Running Running Not used Halted or Running Physical—EVA Series Halted Not used Halted or Running R
Restoring Disaster Tolerance After a failover to a cluster occurs, restoring disaster tolerance has many challenges, the most significant of which are: • Restoring the failed cluster. Depending on the nature of the disaster it may be necessary to either create a new cluster or to restore the cluster. Before starting up the new or the failed cluster, make sure the AUTO_RUN flag for all of the Continentalclusters application packages is disabled.
3. Check and apply the Continentalclusters configuration. # cmcheckconcl -v -C cmconcl.config # cmapplyconcl -v -C cmconcl.config 4. Restart the monitor packages on each cluster. # cmmodpkg -e ccmonpkg 5. View the status of the Continentalcluster. # cmmviewconcl Before applying the edited configuration, the data storage associated with each cluster needs to be prepared to match the new role.
If editing of the default values are desired, do it with file “NewContinentalclusterConfigFileName” if -F is specified, or with file, “CurrentContinentalclustersConfigFileName” if -F is not specified. If editing of the new configuration file is needed, do not use the -a option. If option -a is specified the new configuration will be applied automatically. 3. If option -a is specified with cmswitchconcl in step 2, skip this step. Otherwise manually apply the new Continentalclusters configuration.
RECOVERY_GROUP_NAME RIMARY_PACKAGE ECOVERY_PACKAGE RECOVERY_GROUP_NAME PRIMARY_PACKAGE RECOVERY_PACKAGE DATA_RECEIVER_PACKAGE RECOVERY_GROUP_NAME PRIMARY_PACKAGE RECOVERY_PACKAGE RECOVERY_GROUP_NAME PRIMARY_PACKAGE RECOVERY_PACKAGE DATA_RECEIVER_PACKAGE ### Section 3.
NOTIFICATION SYSLOG “CC alert: ERROR” CLUSTER_EVENT ClusterB/UP MONITORING_CLUSTER ClusterA CLUSTER_ALERT 0 MINUTES NOTIFICATION SYSLOG "CC alert: UP” Newly Created Cluster Will Run Primary Packages After creating a new cluster to replace the damaged cluster, restore the critical applications to the new cluster and restore the other cluster to its role as a backup for the recovered packages. 1. Configure the new cluster as a Serviceguard cluster.
Performing a Rehearsal Operation in your Environment Use the cmrecovercl -r -g command to start the disaster recovery rehearsal process in your environment. This command checks for the following prerequisites before starting the rehearsal process: • The recovery group is in the maintenance mode. • The data receiver package, if configured in the recovery group, is halted and disabled in the recovery cluster. The rehearsal package runs regardless of the state of the primary cluster.
For additional examples of setting up and running DR rehearsal in different environments, see the Disaster Recovery Rehearsal in Continentalclusters whitepaper available at: http://www.hp.com/ go/hpux-serviceguard-docs. Data Replication Rehearsal in a Sample Environment This section describes how to set up and run data replication (DR) rehearsal with the example of a single instance Oracle application with Continentalcluster with Continuous Access P9000 or XP integration.
NOTE: Starting with Continentalclusters version A.08.00, rehearsal packages can be configured as modular packages. For more information on configuring rehearsal packages as modular packages, see “Configuring Continentalclusters Rehearsal packages as Modular Packages” (page 74). Primary Package Metrocluster Environment File If Continentalclusters is configured with MC/SRDF, then before starting rehearsal, set the AUTOSPLITR1 variable to 1.
5. Start the rehearsal: $) cmrecovercl -r -g billing_recgp NOTE: You can use the cmviewcl command to verify that the rehearsal package billing_rhpkg was started. 6. Stop rehearsal package: $) cmhaltpkg billing_rhpkg 7. Restore replication environment for recovery. You must first re-synchronize the recovery disk from the primary disk and then synchronizing the recovery disk business recovery copy with the recovery disk.
1. Halt any monitor packages that are running both clusters. # cmhaltpkg ccmonpkg 2. Add or remove the node in a cluster by editing the Serviceguard cluster configuration file and applying the configuration. # cmapplyconf -C cluster.config 3. 4. 5. 6. 7. Edit the Continentalclusters configuration ASCII file to add or remove the node in the cluster. For added nodes, ensure that the /etc/cmcluster/cmclnodelist and /etc/opt/ cmom/cmomhosts files are set up correctly on the new node.
Update the Continentalclusters configuration file by specifying the new rehearsal package name for the REHEARSAL_PACKAGE parameter in the recovery group definition. Distribute the Continentalclusters configuration by reapplying the configuration file. Removing a Package from the Continentalclusters To remove a package from the Continentalclusters configuration, you must first remove the recovery group from the Continentalclusters configuration file.
cjc1234/recovery cjc1234/rehearsal recovery rehearsal up down The following is an example of cmviewconcl output from a primary cluster that is down.
alert up 1 min -- PACKAGE RECOVERY GROUP hpgroup10 PACKAGE ROLE PTST_sanfran/PACKAGE1 primary TST_dts1/PACKAGE1 recovery PACKAGE RECOVERY GROUP hpgroup20 PACKAGE PTST_dts1/PACKAGE1x_ld PTST_sanfran/PACKAGE1x_ld STATUS down down ROLE primary recovery STATUS down down For a more comprehensive status of component clusters, nodes, and packages, use the cmviewclcommand on both clusters.
ITEM Subnet Subnet STATUS unknown unknown NODE_NAME nynode1 nynode2 Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING Primary down Alternate down NAME 195.14.171.0 195.14.171.0 NAME nynode1 nynode2 Use the ps command to check for the status of the Continentalclusters monitor daemons cmclrmond and cmclsentryd, which should be running on the cluster node where the monitor package is running.
IMPORTANT: Starting with Continentalclusters A.08.00, multiple log files are used to log various operations. All log messages are stored in the /var/adm/cmconcl/logs directory with appropriate names. • The monitoring daemon, by default, logs messages into the /var/adm/cmconcl/logs/ cmclsentryd.log file. • Continentalclusters commands, such as cmquerycl, cmcheckconcl, cmapplyconcl, and cmrecovercl, log messages into the standard output as prior versions.
Checking the Version Number of the Continentalclusters Executables In Continentalclusters version prior to A.08.00, some components of Continentalclusters are executed from Java .jar files. To obtain version information about these files, use the what.sh script provided in the /opt/cmconcl/jar directory. Example: # /opt/cmconcl/jar/what.sh configcl.jar For Continentalclusters version A.08.
recovery cluster upon a primary cluster failure. Figure 22 shows a recovery using an Oracle RAC configuration after failover. Oracle RAC instances are only supported in the Continentalclusters environment for physical replication set up using HP StorageWorks Continuous Access P9000 or XP, HP StorageWorks Continuous Access EVA or EMC Symmetrix Remote Data Facility (SRDF) using SLVM or Cluster Volume Manager (CVM) or Cluster File System (CFS) or Automatic Storage Management (ASM) for volume management.
To support this feature, Continentalclusters must be configured with an environment that has physical replication set up using HP StorageWorks Continuous Access P9000 or XP, HP StorageWorks Continuous Access EVA or EMC Symmetrix Remote Data Facility (SRDF) using SLVM or Cluster Volume Manager (CVM) or Cluster File System (CFS) for volume management. For more information on specific Oracle RAC configurations that are supported, refer Table 2-8.
a. b. Make sure that the primary and recovery clusters are running. Configure and start the CFS or CVM multi-node package using the command cfscluster config -s. When CVM starts, it automatically selects the master node. This master node is the node from which you must issue the disk group configuration commands. To determine the master node, run the following command from any node in the cluster. # vxdctl -c mode c. Create disk groups and mount points.
information on configuring Oracle RAC, refer to the Oracle RAC installation and configuration user’s guide. If you have Oracle Clusterware and Serviceguard running in your environment, you need to complete certain additional configuration procedures. For more information on these configuration procedures, see “Serviceguard/Serviceguard Extension for RAC and Oracle Clusterware Configuration” (page 121). 4. 5. Configure Continentalclusters.
a. b. Login as root on one node of the primary cluster. Change to your own directory: # cd c. Copy the file: # cp /opt/cmconcl/scripts/ccrac.config \ ccrac.config.mycopy d. Edit the file ccrac.config.mycopy to fit your environment. The following parameters need to be edited: CCRAC_ENV - fully qualified Metrocluster environment file name. This file naming convention as required by the Metrocluster software. It has to be appended with _.
CCRAC_SLVM_VGS[0]=ccracvg1 ccracvg2 CCRAC_INSTANCE_PKGS[0]=ccracPkg1 ccracPkg2 CCRAC_CLUSTER[0]=PriCluster1 CCRAC_ENV_LOG[0]=/tmp/db1_prep.log (Multiple values for CCRAC_SLVM_VGS and CCRAC_INSTANCE_PKGS should be separated by space). If multiple sets of Oracle instances accessing different databases are configured in your environment and need Continentalclusters recovery support, repeat this set of parameters with an incremented index. For example, CCRAC_ENV[0]=/etc/cmconcl/ccrac/db1/db1EnvFile_xpca.
When recovering a recovery group with multi-node packages, Continentalcluster will start an instance in each cluster node configured in the MNP. After editing the Continentalclusters configuration file to add in the recovery group specification for Oracle RAC instance packages, you must manually apply the new configuration by running the cmapplyconcl command. When you finish configuring a recovery pair with RAC support, your systems must have sets of files similar to those shown in Figure 23.
Serviceguard/Serviceguard Extension for RAC and Oracle Clusterware Configuration The following are the required configurations for Continentalclusters RAC instance recovery support for the cluster environment running with Serviceguard/Serviceguard Extension for RAC and CRS (Oracle Cluster Software): 1.
# /opt/cmconcl/bin/ccrac_mgmt.ksh -i start is the index used in the /etc/cmconcl/ccrac/ccrac.config file for the target set of the Oracle RAC instance packages. 4. To stop all the RAC instance packages configured to run as primary packages on the local cluster. # /opt/cmconcl/bin/ccrac_mgmt.ksh stop To stop a specific set of RAC instance packages. # /opt/cmconcl/ccrac_mgmt.ksh -i stop is the index used in the /etc/cmconcl/ccrac/ccrac.
applications are running on the primary cluster may result in data corruption. Are you sure that the primary packages are not running and will not come back, and are you certain that you want to start the recovery packages [y/n]? y cmrecovercl: Attempting to recover Recovery Groups from cluster LACluster. NOTE: The configuration file /etc/cmconcl/ccrac/ccrac.config for cluster shared storage recovery exists. Data storage specified in the file for this cluster will be prepared for this recovery process.
cmrecovercl: Completed recovery process for each recovery group. Recovery packages have been started. Use cmviewcl or check package log file to verify that the recovery packages are successfully started. These message prompts can be disabled by running cmrecovercl with option -y. If you have configured the Oracle RAC instance package such that there is one instance for every package, the instance or recovery group can be recovered individually.
NOTE: Ensure that the SG-CFS-PKG (system multi-node) package is running for the CFS/CVM environment. 6. Startup the Oracle RAC instance packages on the primary cluster. If you have configured CFS or CVM in your environment, issue the following command from the master node: # /opt/cmconcl/bin/ccrac_mgmt.ksh start Alternatively, you can run the command on any node in the primary cluster. This command fails back all of the RAC instance packages configured to adopt to this cluster as the primary cluster.
Figure 24 SADTA Configuration in Continentalclusters NOTE: Complex workload packages are not Continentalclusters primary or recovery packages. They are regular Serviceguard packages and are defined as managed packages in the Site Controller Package configuration. You can set the AUTO_RUN flag for the complex workload packages as YES or NO.
IMPORTANT: • Complex workloads can be configured in Continentalclusters with SADTA using physical replication technology. • Disaster Recovery (DR) rehearsal is not supported in SADTA with Continentalclusters. • Continentalclusters maintenance mode feature for recovery groups is not supported in SADTA with Continentalclusters.
Complete the following procedure to configure the primary cluster with a single site defined in the Serviceguard configuration file: 1. Run the cmquerycl command to create a cluster configuration file from any node. 2. Edit the created cluster configuration file to specify a site configuration. Following is a sample of the site configuration: SITE_NAME NODE_NAME SITE ... ... NODE_NAME SITE ...
• Configuring the Storage Device using SG SMS CVM in a Legacy Style Package • Configuring the Storage Device using VERITAS CVM • Configuring the Storage Device using SLVM Configuring the Storage Device using CFS or SG SMS CVM in a Modular Style Package By using modular style packages, Serviceguard enables you to manage all the CVM diskgroups and the CFS mountpoints required by an application within a single package.
Complete the following procedure to configure a storage device using CFS: 1. Initialize the source disks of the replication pair. # /etc/vx/bin/vxdisksetup -i # /etc/vx/bin/vxdisksetup -i 2. Create a disk group for the complex workload data. # vxdg –s init \ 3. Create Serviceguard Disk Group MNP packages for the disk groups with a unique name in the cluster.
In this command, and are nodes in the primary cluster. 4. Activate the CVM disk group in the primary cluster CFS sub-cluster. # cfsdgadm activate 5. Create a volume from the disk group. # vxassist -g make \ 4500m Configuring the Storage Device using VERITAS CVM Complete the following procedure on the CVM cluster master node in the primary cluster to setup the CVM disk group volumes: 1.
4. Specify the Serviceguard dependency: dependency_name SG-CFS-pkg_dep dependency_condition SG-CFS-pkg=up dependency_location same_node 5. Apply the newly created package configuration. # cmapplyconf -v -P .conf Configuring the Storage Device using SLVM Complete the following procedure to create volume groups on the primary cluster: 1. Define the appropriate volume groups on each host system in the primary cluster.
Following are the topics discussed in this section: • “Configuring Complex Workload Packages to Use CFS” (page 133) • “Configuring Complex Workload Packages to Use SG SMS CVM or Veritas CVM in Legacy Style Package” (page 133) • “Configuring Complex Workload Packages to use SLVM” (page 133) • “Halting the Complex Workload in the Primary Cluster” (page 133) Configuring Complex Workload Packages to Use CFS When the storage for the complex workload is configured on a Cluster File System (CFS), the compl
Halting the Complex Workload in the Primary Cluster You need to first halt the complex workload stack on the node in the primary cluster using the cmhaltpkg command. Configuring the Site Controller Package in the Primary Cluster Complete the following procedure on a node in the primary cluster to configure the Site Controller Package: 1. Create a Site Controller Package configuration file using the dts/sc and array-specific module as shown below.
Configuring the Site Safety Latch Dependencies in the Primary Cluster After the Site Controller Package configuration is applied, the corresponding Site Safety Latch is automatically configured in the cluster. This section describes the procedure to configure Site Safety Latch dependencies. Complete the following procedure to configure the Site Safety Latch dependencies: 1.
NOTE: 4. • Do not add any comments after specifying the critical and managed packages. • Always set auto_run parameter to yes for failover packages configured as critical or managed packages. • The packages configured with mutual dependency must not be configured as critical or managed packages. Re-apply the Site Controller Package configuration. # cmapplyconf -v -P /etc/cmcluster/cw_sc/cw_sc.
1. Import the diskgroup: # vxdg -stfC import 2. Create a package configuration file: # cmmakepkg -m sg/cfs_all /etc/cmcluster/cfspkg1.ascii 3. Edit the following package parameters in the cfspkg1.
In this command, and are nodes in the recovery cluster. 6. Mount the cluster file systems in this CFS sub-cluster. # cfsmount /cfs/ Configuring the Storage Device using SG SMS CVM in a Legacy Style Packaging NOTE: HP recommends you to use the modular style of packaging. Complete the following procedure to import CVM disk groups on the nodes in the recovery cluster and to create CVM disk group MNP package: 1.
To activate LVM or SLVM volume groups in the recovery cluster, the cluster ID of the LVM or SLVM volume groups must be changed as shown in the following sample.
SADTA to configure Oracle RAC. When using SADTA, you must use the Site Controller Package instead of the /etc/cmconcl/ccrac/ccrac.config file to configure Oracle RAC. CAUTION: Until the migration is complete, do not perform any administrative operations in the Continentalclusters for this Oracle RAC database recovery group. Complete the following procedure to convert an existing Oracle RAC configuration to use SADTA in Continentalclusters: 1. Halt the primary cluster using the cmhaltcl command. 2.
SITE_NAME NODE_NAME SITE ... ... NODE_NAME SITE ... NOTE: Only one site needs to be defined in the cluster configuration and all nodes in the cluster must belong to this site. 3. 4. Run the cmapplyconf command to apply the configuration file. Run the cmruncl command to start the cluster. After the cluster is started, you can run the cmviewcl command to view the single site configuration.
• Ensure that the AUTO_RUN flag is set to NO. • Edit the array specific parameters. For configuring these parameters, see the following sections based on the type of the array used in your environment. When using Continuous Access P9000 or XP, see “Configuring primary and recovery packages as modular packages when using Continuous Access P9000 or XP” (page 67). When using Continuous Access EVA, see “Configuring primary and recovery packages as modular packages when using Continuous Access EVA” (page 69).
Figure 26 Sample Oracle RAC Database with ASM in SADTA The CRS daemons at the clusters must be configured as a Serviceguard package using the HP Serviceguard extension for RAC (SGeRAC) toolkit at each Serviceguard cluster. The CRS Home must be installed on a file system that is local to the cluster. The CRS voting and OCR disks must not be configured for replication. The RAC database software must be installed at each cluster in the Continentalclusters.
NOTE: If Three Data Center (3DC) configuration using P9000 or XP Continuous Access 3DC replication technology is being created, then the primary cluster must be configured as a Metrocluster with two sites. For more information on configuring a complex workload in 3DC configuration , see “Deploying a Complex Workload in Three Data Center Solution using SADTA” (page 447).
configuring Continentalclusters with sites for SADTA, see “Configuring the Recovery Cluster with a Single Site” (page 128) Installing and Configuring Oracle Clusterware After setting up replication in your environment, you must install Oracle Clusterware. Use the Oracle Universal Installer to install and configure the Oracle Clusterware. Install and configure Oracle Clusterware in both the primary cluster and recovery cluster.
the latest edition of the Using Serviceguard Extension for RAC manual, available at www.hp.com/ go/hpux-serviceguard-docs. Configuring SGeRAC Toolkit Packages for the ASM disk group in the Primary cluster To configure SADTA in Continentalclusters for Oracle RAC database with ASM, the ASM disk group must be packaged in Serviceguard MNP packages in both clusters. Configure ASM Disk group MNP package dependency on the Clusterware MNP package on both the clusters.
# cmhaltpkg Suspending the replication to the recovery cluster In the earlier procedures, the RAC database and Site Controller packages were created at the primary cluster with the source disk of the replication disk group. A RAC MNP stack was also created at that cluster. Now, an identical RAC database using the target replicated disk must be configured with the RAC MNP stack in the recovery cluster.
7. ln –s /opt/app/oracle/admin/+ASM/pfile/init.ora init+ASM2.ora chown -h oracle:oinstall init+ASM2.ora chown oracle:oinstall orapw+ASM2 Add the ASM instances with the CRS cluster on the recovery cluster. In this example, run the following commands from any node on cluster2: export ORACLE_SID=”+ASM” srvctl add asm -n -i “+ASM1” –o srvctl add asm -n -i “+ASM2” –o /opt/app/oracle/product/11.1.
9. Copy the tnsnames.ora file from the primary cluster CRS and modify it to fit the local environment. In this example, the file contents would appear as follows: rcp :$ORACLE_HOME/network/admin/tnsnames.ora :$ORACLE_HOME/network/admin/tnsnames.ora rcp :$ORACLE_HOME/network/admin/tnsnames.ora :$ORACLE_HOME/network/admin/tnsnames.ora 10. Edit the tnsnames.
After applying the Site Controller Package configuration, run the cmviewcl command to view the packages that are configured. 5. Repeat the above steps in the Recovery cluster as well. Configuring the Site Controller Package at the recovery cluster The site controller package needs to be configured in the recovery cluster. The procedure to configure the Site Controller Package is identical to the procedure in configuring complex workload in Continentalclusters with SADTA.
Troubleshooting Continentalclusters Version A.08.00 This section contains a list of error messages that a user may encounter while using Continentalclusters Version A.08.00. It also provides the probable cause for these errors and recommended solutions. Table 15 Troubleshooting Continentalclusters Version A.08.00 Command/Component Symptoms Cause Resolution ccmonpkg The ccmonpkg package fails to start. The following error message is written to the /var/opt/ resmon/ log/client.
Table 15 Troubleshooting Continentalclusters Version A.08.00 (continued) Command/Component Symptoms Cause Resolution cmcheckconcl The following error message is encountered while using the cmcheckconcl command: Error: “Global package switching flag is set to true for package on cluster ”. The cause could be one of the following: • Set the AUTO_RUN flag in the package configuration file to NO. cmclsentryd The cmclsentryd daemon Volume Group (VG) is not fails to start.
3 Building Disaster Recovery Serviceguard Solutions Using Metrocluster with Continuous Access for P9000 and XP The HP StorageWorks P9000 Disk Array family or HP StorageWorks XP Disk Array series allows you to configure data replication solutions to provide disaster tolerance for Serviceguard clusters over long distances. This chapter describes the Continuous Access P9000 and XP software and the additional files that integrate the P9000 or XP Disk Arrays with Metrocluster.
Metrocluster with Continuous Access for P9000 and XP needs to be installed on all nodes that will run a Serviceguard package whose data are on an HP StorageWorks P9000 Disk Array family or HP StorageWorks XP Disk Array series, and where the data is replicated to a second P9000 or XP using the Continuous Access P9000 or XP facility.
define it. All devices defined in a given device group must be configured with the same fence level. A fence level of DATA or NEVER results in synchronous data replication; a fence level of ASYNC is used to enable asynchronous data replication. Fence Level of NEVER Fence level = NEVER should only be used when the availability of the application is more important than the data currency on the remote P9000 or XP disk arrays.
Fence Level of ASYNC Fence level = ASYNC is recommended to improve performance in data replication between the primary and the remote site. The XP disk array supports asynchronous mode with guaranteed ordering. When the host does a write I/O to the XP disk array, as soon as the data is written to cache, the array sends a reply to the host. A copy of the data with a sequence number is saved in an internal buffer, known as the side file, for later transmission to the remote XP disk array.
Consistency Group An important property of asynchronous mode volumes is the Consistency Group (CT group). A CT group is a grouping of LUNs that need to be treated the same from the perspective of data consistency (I/O ordering). A CT group is equal to a device group in the Raid Manager configuration file. A consistency group ID (CTGID) is assigned automatically during pair creation. NOTE: Different P9000 and XP models support different maximum numbers of Consistency Groups.
Remote Array RAID Manager Instance A remote array RAID Manager instance allows the Metrocluster package to determine the status of the device group on the remote P9000 or XP array even when the remote hosts are down or are inaccessible due to a network link failure. The remote array RAID Manager instance is configured on a node using the command device of the remote P9000 or XP array.
Continuous Access Journal Overview Continuous Access Journal is an asynchronous data replication between two HP P9500, HP XP10000, HP XP12000, HP XP20000 or HP XP24000 storage disk arrays. As depicted in Figure 29, Continuous Access Journal uses two main features, “disk-based journaling” and “pull-style replication”. These two features reduce XP12000 internal cache memory consumption, while maintaining performance and operational resilience.
Pull-Based Replication In addition to disk-based journaling, Continuous Access Journal uses pull-style replication. The primary storage system does not dedicate resources to pushing data across the replication link. Rather, a replication process on the remote system pulls the data from the primary system's journal volume, across the Continuous Access link, and writes it to the journal volume at the receiving site.
Journal Cache, Journal Volumes, and Inflow Control When a primary array performs an update (host-requested write I/O) on PVOL, the primary array creates the journal data (metadata and new write data) to be transferred to secondary array. The journal data is stored in the journal cache or journal volumes depending on an amount of data in cache. If available cache memory for Continuous Access Journal is low, the journal data is stored in the journal volumes.
One-to-One Volume Copy Operations Continuous Access Journal requires a one-to-one relationship between the logical volumes of the volume pairs. A volume can only be assigned to one journal group pair at a time. NOTE: Continuous Access Journal does not support operations in which one primary data volume is copied to more than one secondary data volume, or more than one primary data volume is copied to one secondary data volume.
Data Replication Connections The remote copy connections are the physical paths used by the primary array to communicate with the secondary array. The primary P9000 or XP array and secondary P9000 or XP array are connected using fiber-channel interface (Note: ESCON is not supported with the XP12000). Ensure the connection is established in a bidirectional manner. Metrocluster package vs. Journal Group Metrocluster with Continuous Access for P9000 and XP supports only one journal group pair per package.
HP StorageWorks Smart Tiers Metrocluster with Continuous Access for P9000 and XP Version A.11.00 includes support for Smart Tiers on HP StorageWorks P9000 Disk Array family. Using Smart Tiers one can configure the storage consisting of multiple kinds of hard disk drives including SSD, SAS, and SATA. All restrictions and guidelines that are applicable to HP StorageWorks P9000 Smart Tiers are applicable in the Metrocluster environment.
horcm0 4. 11000/udp #Raid Manager instance 0 Use the ioscan command to determine what devices on the P9000 or XP disk array have been configured as command devices. The device-specific information in the right most column of the ioscan output will have the suffix-CM for these devices; for example, OPEN-3-CM. If there are no configured command devices on the disk array, you must create two before proceeding. Each command device must have alternate links (PVLinks).
Following is the output that is displayed: DEVICE_FILE UID S/F PORT TARG LUN SERIAL LDEV PRODUCT_ID /dev/rdsk/c5t0d0 0 F CL3-E 0 0 10053 321 OPEN-3 This output displays the mapping between the legacy DSFs and the CU:LDEVs. In this output the value for LDEV specifies the CU:LDEV without the : mark.
11. Restart the Raid Manager instance so that the new information in the configuration file is read. # horcmshutdown.sh # horcmstart.sh 12. Repeat these steps on each host that will run this particular application package. If a host may run more than one application package, you must incorporate device group and host information for each of these packages.
# # NOTE: The Raid Manager command device (RORCM_CMD) cannot be used for # data storage (it is reserved for private Raid Manager usage). #/************************ HORCM_MON *************************************/ # # The HORCM_MON parameter is used for monitoring and control of device groups # by the Raid Manager. # It is used to define the IP address, port number, and paired volume error # monitoring interval for the local host. # # Defines a network address used by the local host.
# # This parameter is used to define the SCSI target ID of the physical # volume on the port specified in "port#". # # This parameter is used to define the SCSI logical unit number (LUN) of # the physical volume specified in "targetID".
the same nodes. This is done by editing the rc script /etc/rc.config.d/raidmgr. Set the START_RAIDMGR parameter to 1, and define RAIDMGR_INSTANCE as the number of the Raid Manager instances being used. By default, this is zero (0). Following is an example of the edited configuration file for XP Raid Manager instance. In this example, the instance number zero (0) is the local array RAID Manager instance and the instance number 1 is the remote array RAID Manager instance.
Figure 31 Remote array RAID Manager instance Complete the following steps, to configure the remote array RAID Manager instance in a Metrocluster environment with Continuous Access P9000 or XP: 1. Configure the command device to manage the remote P9000 or XP arrays. 2. Configure the remote array RAID Manager instance. 3. Configure remote array RAID Manager instance in the /etc/rc.config.d/raidmgr file for automatic startup.
Configuring the command device to manage the remote P9000 or XP arrays A command device to manage the remote P9000 or XP arrays can be configured using one of the following methods: • Configure a command device (remote command device) in a Metrocluster site by mapping a command device in the remote P9000 or XP array to a local device in the local P9000 or XP array using the P9000 or XP external storage feature. Present this remote command device to all Metrocluster nodes in the local site.
command devices. The remote array RAID Manager instance at both sites use the remote command devices. • Configure a dedicated command device in the P9000 or XP array at the remote site that is directly presented to the nodes in the local site over an extended SAN without using the P9000 or XP external storage feature. Similarly, configure a command device in the remote site P9000 or XP array and present it to the nodes of the local site over an extended SAN. Figure 33 illustrates this configuration.
1. 2. Select a node in the site to create a remote array RAID Manager instance. Edit the /etc/services file to add an entry for the remote array RAID Manager instance. The format of the entry must be: horcm /udp For example: horcm1 11001/udp #Raid Manager instance 1 NOTE: The UDP ports used for the local array RAID Manager and the remote array RAID Manager must be different. 3.
Sample RAID Manager Configuration File for the Remote Command Device This section includes a sample RAID Manager configuration file for the remote command device.
Configuring the remote array RAID Manager instance for automatic startup After editing the remote array RAID Manager configuration files on the nodes, you must configure the remote array RAID Manager instance to start automatically during node boot and package startup. Complete the following procedure to configure the remote array RAID Manager instance for automatic startup: 1. Edit the following parameters in the configuration file /etc/rc.config.d/raidmgr: • START_RAIDMGR Set this parameter to 1.
# vgcreate /dev/vgname /dev/dsk/cxtydz 3. 4. Create the logical volume(s) for the volume group. Deactivate and export the Volume Groups on the primary system without removing the special device files. # vgchange -a n Make sure that you copy the mapfiles to all of the host systems. # vgexport -s -p -m 5. On the source disk site import the VGs on all of the other systems that might run the Serviceguard package and backup the LVM configuration.
3. Initialize disks to be used with VxVM by running the vxdisksetup command only on the primary system. # /etc/vx/bin/vxdisksetup -i c5t0d0 4. Create the disk group to be used with the vxdg command only on the primary system. # vxdg init logdata c5t0d0 5. Verify the configuration. # vxdg list 6. Use the vxassist command to create the logical volume. # vxassist -g logdata make logfile 2048m 7. Verify the configuration. # vxprint -g logdata 8. Make the filesystem.
9. Deport the disk group. # vxdg deport logdata Repeat steps 4 through 9 on all nodes in the cluster that require access to this disk group. 10. Resynchronize the Continuous Access pair device. # pairresync -g devicegroupname -c 15 Configuring Legacy Packages for Disaster Recovery When you have completed the following steps, packages will be able to fail over to an alternate node in another data center and still have access to the data that they need in order to operate.
5. 6. Add customer-defined run and halt commands in the appropriate places according to the needs of the application. See Managing Serviceguard for more information on these functions. Copy the environment file template /opt/cmcluster/toolkit/SGCA/ xpca.env to the package directory, naming it pkgname_xpca.env. # cp /opt/cmcluster/toolkit/SGCA/xpca.env /etc/cmcluster/pkgname/pkgname_xpca.
9. Check the configuration using the cmcheckconf -P pkgname.config, then apply the Serviceguard configuration using the cmapplyconf -P pkgname.config command or SAM. 10. Distribute Metrocluster/Continuous Access configuration, environment and control script files to other nodes in the cluster by using ftp, rcp or scp: # rcp -p /etc/cmcluster/pkgname/* \ other_node:/etc/cmcluster/pkgname See the example script Samples/ftpit to see how to semi-automate the copy using ftp.
1. Run the following command to create a Metrocluster XPCA modular package configuration file: # cmmakepkg –m dts/mcxpca temp.config In this command, dts/mcxpca is the Metrocluster XPCA module that needs to be included to create a Metrocluster XPCA package configuration file. By default, the Metrocluster XPCA module includes only the Serviceguard volume group module.
e. f. g. The volume group attribute. Specify the required logical volume group name to the volume group attribute. If other Serviceguard modules are included, then specify values for the included module attributes accordingly. If any application module is included, then list values for the included module attributes accordingly. NOTE: If the EMS disk monitor is used as a package resource, then the NO_TIMEOUT value must not be used.
To distribute the Metrocluster environment file to other nodes in the cluster, you can also use the Command Fanout utility that is available from the Distributed Systems Administration Utilities (DSAU) application. The Command Fanout utility uses SSH. DSAU can be opened from the HP System Management Homepage (HP SMH), the single-system management application that is available with the HP-UX Operating Environment.
#cmmakepkg –i modular_pkgname1.ascii -m ecmt/oracle/oracle -t haoracle.conf modular_pkgname2.ascii 2. Include the Metrocluster XPCA module into this modular package configuration file. Run the following command to include Metrocluster module. #cmmakepkg –i modular_pkgname2.ascii -m dts/mcxpca The new modular package configuration file will contain the dts/dts/dts_pkg_dir attribute unassigned. 3.
c. d. HORCMINST - Specify the name of the RAID manager instance that manages the XPCA device group used by this package. FENCE - Specify the fence level configured for the XPCA device group that is managed by this package. There are additional Metrocluster parameters available in the package configuration file. It is recommended that the default values of these variables are retained unless there is a specific business requirement to change them.
IMPORTANT: This command will generate a package configuration file. Do not apply this configuration file until you complete the migration procedure. For more information on the cmmigratepkg command, see the Managing Serviceguard manual available at http://www.hp.com/go/hpux-serviceguard-docs. 2. If the Metrocluster Legacy package uses ECM toolkits, then generate a new Modular package configuration file using the package configuration file generated in the step 1.
a. Create a new package configuration file using the Metrocluster modules and all other modules used by the package that has to be upgraded. # cmmakepkg -m dts/mcxpca [ -m ...]\ For example, for a Metrocluster ECM Oracle toolkit modular package, run the following command: # cmmakepkg -m dts/mcxpca -m ecmt/oracle/oracle example.conf This new package configuration file includes all the Metrocluster environment file parameters. b.
Checking the Metrocluster Configuration Starting from HP Serviceguard version A.11.20, the cmcheckconf -v command validates the cluster and the package configuration. Starting from the April 2011 patch release, Metrolcuster uses this functionality to ensure the sanity of Metrocluster and the Site Controller package configuration. HP recommends you to set up a cron job to run the cmcheckconf command regularly.
Table 17 Validating Metrocluster Package (continued) the disk names in a persistent dsf format. NOTE: The checks and validations mentioned in “Validating Metrocluster Package” (page 189) is not applicable for legacy packages. It is recommended that you add the location of the package environment file available in the Metrocluster package directory to the list of files in /etc/ cmcluster/cmfiles2check. Support for Easy Deployment Feature Easy Cluster Creation Starting with Serviceguard version A.11.
package configuration file automatically based on the type of disks used by the supported toolkit application. If the disks used are HP StorageWorks Continuous Access P9500 or XP disks and are replicated, then the dts/mcxpca module is automatically inserted into the package configuration file. The following dts/mcxpca parameters are auto-discovered.
# rcp / : 6.
Viewing the Continuous Access Journal Status The following two sections describe using the pairdisplay and raidvchkscan commands for viewing the Continuous Access Journal Status. Viewing the Pair and Journal Group Information - Raid Manager using the “pairdisplay” Command The command option “-fe” is added to the Raid Manager pairdisplay command. This option is used to display the Journal Group ID (and other data) of a device group pair.
• In case of the S-JNL, Q-Marker shows the latest sequence number putting on the cache. • Q-CNT: Displays the number of remaining Q-Marker of a journal group. Figure 34 Q-Marker and Q-CNT • U(%): Displays the usage rate of the journal data. • D-SZ: Displays the capacity for the journal data on the journal group. • Seq#: Displays the serial number of the XP12000. • Num: Displays the number of LDEV (journal volumes) configured for the journal group.
• Failure of the entire secondary Data Center for a given application package • Failure of the secondary P9000 or XP Series disk array for a given application package while the application is running on a primary host Following is a partial list of failures that require full resynchronization to restore disaster-tolerant data protection.
Timing Considerations In a journal group, many journal volumes can be configured to hold a significant amount of the journal data (host-write data). The package startup time may increase significantly when a Metrocluster Continuous Access package fails over. Delay in package startup time will occur in these situations: 1. When recovering from broken pair affinity. On failover, the SVOL pull all the journal data from PVOL site.
1. 2. 3. Split the device group pair completely (pairsplit -g -S). Re-create a pair from original PVOL as source (use the paircreate command). Startup package on PVOL site (or SVOL site). PVOL-PAIR with SVOL-PSUS(SSWS) State (for Continuous Access Journal Mode) PVOL-PAIR with SVOL-PSUS(SSWS) is an intermediate state. The following is one scenario that leads to this state: • At T1, device pair is in PVOL-PAIR/SVOL-PAIR and the AP value is 0 in SVOL site.
Configure the Monitor’s Variables in the Package Environment File Edit the following variables of the monitor’s section in the environment file _xpca.env as follows: NOTE: See Appendix A for an explanation of these variables. • Uncomment the MON_POLL_INTERVAL variable and set it to the desired value in minutes. If this variable is not set, it will default to a value of 10 minutes. • Uncomment the MON_NOTIFICATION_FREQUENCY variable and set it to the desired value.
MON_NOTIFICATION_SYSLOG=1 MON_NOTIFICATION_CONSOLE=1 AUTO_RESYNC=1 Configure Metrocluster with Continuous Access for P9000 and XP Device Group Monitor for Modular Packages While using Modular style packages, the parameters used for configuring the Device Group Monitor that were previously available in the Metrocluster environment file, are now available in the package configuration file. Complete the following procedure to configure the Device Group Monitor when using Modular style packages: 1.
in the Site Controller Package directory and the same file path must be passed to the Device Group Monitor service. The Site Controller Package can be halted in the DETACH mode for maintenance in the Site Controller Package configuration. The Site Controller Package can also fail in the cluster. In these conditions, the Device Group Monitoring service is not available for the workloads. The service resumes once the Site Controller Package is restarted.
NOTE: Neither the source disk site nor the target disk site may configure an P9000 or XP series paired volume, PVOL or SVOL, as a cluster lock disk. A cluster lock disk must always be writable. Since it cannot be guaranteed that either half of a paired volume is always writable, neither half may be used as a cluster lock disk. A configuration with a cluster lock disk that is part of a paired volume is not a supported configuration. 1. 2.
a. b. c. d. e. f. g. h. i. 8. If necessary, add the path where the Raid Manager software binaries have been installed to the PATH environment variable. If the software is in the usual location, /usr/bin, you can just uncomment the line in the script. Uncomment the behavioral configuration environment variables starting with AUTO_. It is recommended that you retain the default values of these variables unless you have a specific business requirement to change them.
12. Using standard Serviceguard commands (cmruncl, cmhaltcl, cmrunpkg, cmhaltpkg), test the source disk site for cluster and package startup and package failover. 13. Any running package on the source disk site that will have a counterpart on the target disk site must be halted at this time. Setting up a Recovery Package on the Recovery Cluster Use the procedures in this section to configure a recovery package on the target disk site.
NOTE: Some of the control script variables, such as VG and LV, on the target disk site must be the same as on the source disk site. Some of the control script variables, such as, FS, SERVICE_NAME, SERVICE_CMD and SERVICE_RESTART are probably the same as on the source disk site. Some of the control script variables, such as IP and SUBNET, on the target disk site are probably different from those on the source disk site. Make sure that you review all the variables accordingly. 5. 6.
9. Apply the Serviceguard configuration using the cmapplyconf command or SAM. 10. Verify that each node in the Serviceguard cluster has the following files in the directory /etc/cmcluster/pkgname: bkpbkgname.cntl Metrocluster/Continuous Access package control script bkpkgname_xpca.env Metrocluster/Continuous Access environment file bkpkgname.ascii Serviceguard package ASCII configuration file bkpkgname.
NOTE: The monitor package for a cluster checks the status of the other cluster and issues alerts and alarms, as defined in the Continentalclusters configuration file, based on the other cluster’s status. 8. Check /var/adm/syslog/syslog.log for messages. Also check the ccmonpkg package log file. 9. Start the primary packages on the source disk site using cmrunpkg. Test local failover within the source disk site. 10.
mode) and a remote status of EX_ENORMT, PSUE or PSUS, indicating that there is an error accessing the primary site. The control file script is programmed to handle this condition and will enable the volume groups, mount the logical volumes, assign floating IP addresses and start any processes as coded into the script. NOTE: In ASYNC mode, the package will halt unless a force flag is present or unless the auto variable AUTO_SVOLPSUE is set to 1.
This will halt any applications, remove any floating IP addresses, unmount file systems and deactivate volume groups as programmed into the package control files. The status of the paired volumes will remain SMPL at the recovery site and PSUE at the primary site. 2. 3. Start the cluster at the primary site. Assuming they have been properly configured the Continentalclusters primary packages should not start. The monitor package should start automatically.
NOTE: The preceding steps are automated provided the default value of 1 is being used for the auto variable AUTO_PSUEPSUS. Once the Continuous Access link failure has been fixed, the user only needs to halt the package at the failover site and restart on the primary site. However, if you want to reduce the amount of application downtime, you should manually invoke pairresync before failback.
stopped and restarted. The Raid Manager instance must be running before any Continentalclusters package movement occurs. 210 • A given file system must not reside on more than one P9000 or XP frames for either the PVOL or the SVOL. A given LVM Logical Volume (LV) must not reside on more than one P9000 or XP frames for either the PVOL or the SVOL. • The application is responsible for data integrity, and must use the O_SYNC flag when ordering of I/Os is important.
4 Building Disaster Recovery Serviceguard Solutions Using Metrocluster with Continuous Access EVA The HP StorageWorks Enterprise Virtual Array (EVA) allows you to configure data replication solutions to provide disaster tolerance for Serviceguard clusters over long distances. This chapter describes the Continuous Access EVA software and the additional files that integrate the EVA with Serviceguard clusters. It then shows how to configure metropolitan cluster solutions using Continuous Access EVA.
Table 18 Metrocluster Continuous Access EVA Template Files (continued) Name Description package. The Metrocluster Continuous Access EVA environment file is generated automatically starting with version A.05.00. /opt/cmcluster/toolkit/SGCAEVA/ Samples A directory containing sample convenience shell scripts that must be edited before using. These shell scripts may help to automate some configuration tasks. These scripts are contributed, and not supported.
DR Groups A data replication (DR) group is a software construct comprising one or more Vdisks in an HSV storage system so that they: • Replicate to the same specified destination storage array • Fail over together • Preserve write order within the data replication collection groups • Share a log disk All virtual disks used for replication must belong to a DR group, and a DR group must contain at least one Vdisk. A DR group can be thought of as a collection of copy sets.
IMPORTANT: Metrocluster with Continuous Access EVA does not support asynchronous replication on arrays prior to those that are listed above. For more information on supported arrays, see the Disaster Tolerant Clusters Products Compatibility Feature Matrix available at: http://www.hp.com/go/hpux-serviceguard-docs —> High Availability. ◦ • Synchronous mode: An I/O completion acknowledgement is sent to the host after data is written to the source and destination caches.
When creating disk groups and distributing Vdisks within them, sufficient capacity must remain for log disks to expand to their maximum level. The log is declared full, and reaches its maximum level, whenever the first of the following conditions is reached: • The size of the log data file exceeds twice the capacity of the DR group. • No free space remains in the physical disk group. • The log reaches 2 TB of Vraid1 (4 TB total).
Preparing a Serviceguard Cluster for Metrocluster Continuous Access EVA When the following procedures are completed, an adoptive node will be able to access the data belonging to a package after it fails over. Setting up the Storage Hardware 1. 2. 3. 4. Before configuring Metrocluster Continuous Access EVA, the EVA must be correctly cabled with redundant paths to each node in the cluster that will run packages accessing data on the array.
NOTE: For more detailed information on the sssu commands used in the sample input file, refer to the sssu ReadMe file found at "/opt/cmcluster/toolkit/SGCAEVA/Samples/ Readme.sssu_sample_input Follow the steps below when copying and editing the sample file: 1. Copy the sample file /opt/cmcluster/toolkit/SGCAEVA/Samples/ sssu_sample_input to the /etc/dtsconf/directory. # cp /opt/cmcluster/toolkit/SGCAEVA \/Samples/sssu_sample_input /etc/dtsconf/sssu_input 2. 3. Customize the file sssu_input.
information. These processes take time and are not necessary since the IDs are static in the EVA system. To improve the query performance, the software will cache these IDs in the clustered nodes. To cache the object IDs in the clustered nodes, it is required to run the evadiscovery tool after the EVA and Continuous Access EVA are configured, and the storage is accessible from the hosts. The tool will query the active Management Server for the needed information and save it in a mapping file.
## data; fields on each line should be separated either by ## ## space(s)or tab(s). The order of fields is significant. ## ## The first field must be a hostname or IP address, the ## ## second field must be a user login name on the host. The ## ## third field must be ‘y’ or ‘n’ to use SSL connect. The ## ## last field must be the namespace of the SMI-S service. ## ## For details of each field data, refer to the smispasswd ## ## man page, ‘man smispasswd’.
------------------------------------------------------15.13.172.11 administrator N 15.13.172.
% Deleting a Management Server To delete a Management Server from the group used by the cluster, use the smispasswd command with the -r option. Example: # smispasswd -r 15.13.172.12 The Management Server 15.13.172.
## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## Node World Wide Name (WWN)which both contain the DR groups## defined under the tag. You can define as ## many DR groups as you need, but each DR group must belong ## to only one of the storage pairs. A storage pair can have ## a maximum of 64 DR groups.
Copying the Storage Map File After running the smispasswd and evadiscovery commands to generate the /etc/dtsconf/ caeva.map file, copy this file to all cluster nodes so that they can be used by Metrocluster Continuous Access EVA to communicate with the EVA units. Be sure to use the same full pathname. Displaying Information about Storage Devices Use the evadiscovery command to display information about the storage systems and DR groups in your configuration.
Below is a sample output after running the spmgr command: TGT/LUN 0/ 3 Device c12t0d3 WWLUN_ID H/W_Path 6000-1FE1-0016-6C30-0009-2030-2549-000A 255/0.0.3 Path_Instance HBA Controller Path_Status ZG20302549 c4t0d4 c10t0d4 Controller Path_Instance Path_Status ZG20400420 c6t0d4 c8t0d4 0/ 4 Preferred? td1 td3 HBA no no Active no Available Preferred? td1 td3 no no Standby no Standby c12t0d4 6000-1FE1-0016-6C30-0009-2030-2549-000E 255/0.0.
Array WWN : 5000-1FE1-5000-2EE0 ================================================================= Lun WWN : 6005-08B4-0010-0E01-0001-B000-0287-0000 Load Balancing Policy : No Load Balancing ================================================================= Device Path Status ================================================================= /dev/dsk/c3t0d1 Active /dev/dsk/c9t0d1 Active /dev/dsk/c15t0d1 Active /dev/dsk/c21t0d1 Active /dev/dsk/c4t0d1 Active /dev/dsk/c10t0d1 Active /dev/dsk/c16t0d1 Active /dev
In the following sample listing there are eight device files that correspond to different paths to the same vdisk. Use all the device files identified while creating a volume group which is described in section, “Configuring Volume Groups using PVLinks”. ======================= mc-node1.cup.hp.com ======================= Virtual Disk Name..: \\XL-1\Vdisk001-DRGSynDCN Disk...............: /dev/dsk/c16t0d1 Disk...............: /dev/dsk/c17t0d1 Disk...............: /dev/dsk/c18t0d1 Disk...............
Ctl-B/FP-1/Optimized/dev/rdsk/c17t1d6 6005-08B4-0010-78F1-0000-E000-0034-0000 Ctl-B/FP-3/Optimized/dev/rdsk/c18t1d6 6005-08B4-0010-78F1-0000-E000-0034-0000 Ctl-B/FP-2/Optimized/dev/rdsk/c19t1d6 6005-08B4-0010-78F1-0000-E000-0034-0000 Ctl-B/FP-4/Optimized 5000-1FE1-5007-DBD0 25600MB 5000-1FE1-5007-DBD0 25600MB 5000-1FE1-5007-DBD0 25600MB For more information on using the EVAInfo tool, see HP StorageWorks EVAInfo Release Notes.
8. Use the vgexport command with the -p option to export the Volume Groups on the primary system without removing the HP-UX device files. # vgexport -s -p -m mapfile /dev/vgname Make sure that you copy the map files to all of the nodes. The sample script Samples/ftpit shows a semi-automated way (using ftp) to copy the files. It is necessary to only enter the password interactively. 9. De-activate the volume group.
Make sure to copy the map files to all of the nodes. The sample script Samples/ftpit shows a semi-automated way (using ftp) to copy the files. Only enter the password interactively. 8. De-activate the volume group. # vgchange -a n /dev/vgname NOTE: With the introduction of mass storage stack’s Native Multi-pathing functionality in HP-UX 11i v3, it is no longer required nor recommended to configure PV links.
IMPORTANT: From HP-UX 11i v3 onwards, HP recommends that you use agile DSF naming model for mass storage devices. For more information on the agile view, mass storage on HP-UX, DSF migration and LVM Online Disk Replacement, see the following documents that are available at http://www.hp.
NOTE: Use the vgimport -N command to configure the volume group using agile DSF. In the absence of the -N option, legacy DSF is used. 3. Verify the Volume Group configuration with the following procedures: • 4. From the command view EVA, shown in Figure 38 failover the DR group to make it the source on the REMOTE site instead of the destination by following the steps described below: a. Select the destination site storage system from the command view EVA. b.
Customize the package configuration file as appropriate to your application. Be sure to include the pathname of the control script (/etc/cmcluster/pkgname/pkgname.cntl) for the RUN_SCRIPT and HALT_SCRIPT parameters. 3. In the .config file, list the node names in the order in which you want the package to fail over. It is recommended for performance reasons, that you have the package fail over locally first, then to the remote data center.
a. b. c. d. e. f. g. Set the CLUSTER_TYPE variable to metro if this a Metrocluster. Set the PKGDIR variable to the full path name of the directory where the control script has been placed. This directory, which is used for status data files, must be unique for each package. For example, set PKGDIR to/etc/cmcluster/, removing any quotes around the file names. The operator may create the FORCEFLAG file in this directory. See Appendix B for an explanation of these variables.
10. Verify that each node in the Serviceguard cluster has the following files in the directory /etc/cmcluster/: .cntl Seviceguard package control script _caeva.env Metrocluster Continuous Access EVA environment file .config Serviceguard package ASCII configuration file .sh Package monitor shell script, if applicable other files Any other scripts used to manage Serviceguard packages 11.
then they need to be explicitly specified while creating the Metrocluster Continuous Access modular package configuration file. The following example shows the package IP, the filesystem, and the monitor subnet module included along with the Metrocluster Continuous Access EVA module. # cmmakepkg –m dts/mccaeva –m sg/filesystem –m sg/monitor_subnet -m\ sg/package_ip temp.config NOTE: Metrocluster is usually used with applications such as Oracle.
NOTE: If the EMS disk monitor is used as a package resource, then the NO_TIMEOUT value must not be used. If used, the package shutdown hangs if there is no access from the host to the package disks. 5. Copy the environment file template /opt/cmcluster/toolkit/SGCA/caeva.env to the dts/dts/dts_pkg_dir directory specified earlier. # cp /opt/cmcluster/toolkit/SGCAEVA/caeva.env \ /etc/cmcluster/pkgname 6. Edit the environment file, caeva.env, as follows: a.
FTP may be preferable at your organization, since it does not require the use of an .rhosts file for root. Root access using the .rhosts may create a security issue. To distribute Metrocluster environment file to other nodes in the cluster, you can also use the Command Fanout utility that is available from the Distributed Systems Administration Utilities (DSAU) application. The Command Fanout utility uses SSH.
#cmmakepkg –i modular_pkgname1.ascii -m ecmt/oracle/oracle -t haoracle.conf modular_pkgname2.ascii 2. Include the Metrocluster Continuous Access EVA module into this modular package configuration file. Use the following command to include Metrocluster module. #cmmakepkg –i modular_pkgname2.ascii -m dts/mccaeva The new modular package configuration file will contain the dts/dts/dts_pkg_dir attribute unassigned. 3.
f. g. h. i. DC1_HOST_LIST— This is a parameter used to define a list of clustered nodes that reside in Data Center 1. DC2_STORAGE_WORLD_WIDE_NAME— This is the World Wide Name of the EVA storage system that resides in Data Center 2. DC2_SMIS_LIST— This is a list of management servers that reside in Data Center 2. DC2_HOST_LIST— This is a parameter used to define a list of clustered nodes that reside in Data Center 2. There are additional Metrocluster parameters available in the package configuration file.
1. Create a Modular package configuration file for the Metrocluster Legacy package. # cmmigratepkg -p [-s] -o IMPORTANT: This command will generate a package configuration file. Do not apply this configuration file until you complete the migration procedure. For more information on the cmmigratepkg command, see the Managing Serviceguard manual available at http://www.hp.com/go/hpux-serviceguard-docs. 2.
1. Upgrade the modular package configuration file of the package created with the A.04.00 version of Metrocluster with Continuous Access EVA. When using HP Serviceguard A.11.18, complete the following steps to upgrade the modular package configuration file: a. Create a new package configuration file using the Metrocluster modules and all other modules used by the package that has to be upgraded. # cmmakepkg -m dts/mcxpca [ -m ...]\
6. Run the package on a node in the Serviceguard cluster. # cmrunpkg -n 7. Enable global switching for the package. # cmmodpkg -e Checking the Metrocluster Package Configuration Starting from HP Serviceguard version A.11.20, the cmcheckconf -v command validates the cluster and the package configuration.
Table 20 Validating Metrocluster Package (continued) greater than 512 (This happens when VxVM uses an internal format to recognize devices that do not support legacy dsf format). In such cases, it is recommended that you change the naming scheme using the vxddladm command so that VxVM output shows the disk names in a persistent dsf format. NOTE: The checks and validations mentioned in “Validating Metrocluster Package” (page 242) is not applicable for legacy packages.
information about the Package Easy Deployment feature, see Using Easy Deployment in Serviceguard and Metrocluster Environments available at http://www.hp.com/go/hpux-serviceguard-docs —> HP Serviceguard. The advantage offered by the Package Easy Deployment feature is that none of the Metrocluster module parameters have to be entered by the user in the modular package configuration file.
3. # rcp /etc/dtsconf/caeva.map :/etc/dtsconf/caeva.map If you are using Metrocluster legacy packages, follow the steps mentioned below: a. Create the Metrocluster package directory on the newly added node. b.
Normal Maintenance There might be situations when the package has to be taken down for maintenance purposes without having the package move to another node. The following procedure is recommended for normal maintenance of the Metrocluster Continuous Access EVA: 1. Stop the package with the appropriate Serviceguard command. # cmhaltpkg pkgname 2. Distribute the Metrocluster Continuous Access EVA configuration changes. # cmapplyconf -P pkgname.config 3.
Set the AUTO_RUN flag to NO. This is to ensure the package will not start when the cluster starts. Only after the primary packages start, use cmmodpkg to enable package switching on all primary packages. By enabling package switching in the package configuration, it will automatically start the primary package when the cluster starts.
g. Set the DC1_SMIS_LIST variable to the list of Management Servers which resides in Data Center 1. Multiple names are defined using a comma as a separator between the names. If a connection to the first management server fails, attempts are made to connect to the subsequent management servers in the order that they are specified. h. i. j. Set the DC1_HOST_LIST variable to the list of clustered nodes which resides in Data Center 1.
NOTE: Serviceguard should already be installed on all the cluster nodes. Run swinstall(1m) to install Continentalclusters from an SD depot. 2. When swinstall(1m) has completed, create a directory as follows for the new package in the target disk site. # mkdir /etc/cmcluster/ Create an Serviceguard package configuration file in the target disk site. # cd /etc/cmcluster/ # cmmakepkg -p .ascii Customize it as appropriate to your application.
e. f. g. Set the DR_GROUP_NAME variable to the name of DR Group used by this package. This DR Group name is defined when the DR Group is created. Set the DC1_STORAGE_WORLD_WIDE_NAME variable to the world wide name of the EVA storage system which resides in Data Center 1. This WWN can be found on the front panel of the EVA controller, or from command view EVA UI. Set the DC1_SMIS_LIST variable to the list of Management Servers which resides in Data Center 1.
Setting up the Continentalclusters Configuration The steps below are the basic procedure for setting up the Continentalclusters configuration file and the monitoring packages on the two clusters. For complete details on creating and editing the configuration file, refer to Chapter 2: “Designing Continentalclusters”. 1. Generate the Continentalclusters configuration using the following command: # cmqueryconcl -C cmconcl.config 2. Edit the configuration file cmconcl.
Switching to the Recovery Cluster in Case of Disaster It is vital the administrator verify that recovery is needed after receiving a cluster alert or alarm. Network failures may produce false alarms. After validating a failure, start the recovery process using the cmrecovercl [-f] command. Note the following: • During an alert, the cmrecovercl will not start the recovery packages unless the -f option is used. • During an alarm, the cmrecovercl will start the recovery packages without the -f option.
the data in the primary site's EVA is not consistent and is not usable until the full copy (resynchronization) completes. It is recommended to wait until the resynchronization is complete before failing back the packages to the primary site. The state of the DR group in the primary site’s EVA can be checked either via Command View (CV) EVA or SSSU command.
Use the cmswitchconclcommand (only in a two cluster configuration) to swap the site identities for all or a selected application’s recovery group. This is so that the applications can now be monitored and recovered from their once source disk site.
5 Building Disaster Recovery Serviceguard Solutions Using Metrocluster with EMC SRDF The EMC and Symmetrix Remote Data Facility (EMC SRDF) disk arrays allows configuration of physical data replication solutions to provide disaster tolerance for Serviceguard clusters over long distances. This chapter describes the EMC SRDF software and the additional files that integrate the EMC with Serviceguard clusters. It then shows how to configure both metropolitan and Continentalclusters solutions using EMC SRDF.
using the SRDF facility. In the event of node failure, the integration of Metrocluster with EMC SRDF with the package will allow the application to fail over in the following ways: • Among local host systems that are attached to the same EMC Symmetrix. • Between one system that is attached locally to its EMC Symmetrix and another “remote” host that is attached locally to the other EMC Symmetrix.
Preparing the Cluster for Data Replication When the following procedures are completed, an adoptive node will be able to access the data belonging to a package after it fails over. Use the convenience scripts in the /opt/cmcluster/ toolkits/SGSRDF/Samples to automate some of the tasks in the following sections: • mk3symgrps.nodename —to create EMC Symmetrix device groups • mk4gatekpr.
NOTE: Do not set the SYMCLI_SID and SYMCLI_DG environment variables before running the symcfg command. These environment variables limit the amount of information gathered when the EMC Solutions Enabler database is created, and therefore will not be a complete database. Also, the SYMCLI_OFFLINE variable should not be set since this environment variable disables the command line interface.
Figure 41 Sample syminq Output from a Node on the R2 Side 2. The following information is needed from these listings for each Symmetrix logical device: • HP-UX device file name (for example, /dev/rdsk/c3t3d2). • Device type (R1, R2, BCV, GK, or blank) • Symmetrix serial number (for example, 50006161), useful in matching the HP-UX device names to the actual devices in the Symmetrix configuration downloaded by EMC support staff. This number is further explained in Figure 42.
------------------------------------------------------------------------STATUS MODES RDF S T A T E S Sym RDF --------- ----- R1 Inv R2 Inv ---------------------Dev RDev Typ:G SA RA LNK MDA Tracks Tracks Dev RDev Pair ---- ---- ------ --------- ----- ------- ------- --- ---- ------------0196 0197 0198 0199 019A 019B 019C 019C 0012 0013 0014 0015 0016 0017 0018 0019 R1:5 R1:5 R1:5 R1:5 R1:5 R1:5 R1:5 R1:5 RW RW RW RW RW RW RW RW RW RW RW RW RW RW RW RW RW RW RW RW RW RW RW RW S.. S.. S.. S.. S.. S.. S..
Figure 44 Sample symrdf list Output from R2 Side 4. Match the logical device numbers in the symrdf listings with the HP-UX device file names in the output from the syminq command. This displays which devices are seen from each node to ensure this node can see all necessary devices. Use the Symmetrix ID to determine which Symmetrix array is connected to the node.
Table 22 Mapping for a 4 Node Cluster connected to 2 Symmetrix Arrays (continued) Symmetrix ID, Node 1 /dev/rdsk device #, and type device file name ID 95 Dev# 028 Type BCV Node 2 /dev/rdsk device file name Node 3 /dev/rdsk device file name Nodes 4 /dev/rdsk device file name n/a n/a c4t3d2 c4t3d2 NOTE: The Symmetrix device number may be the same or different in each of the Symmetrix units for the same logical device.
Issue the above command on nodes attached to the R1 side. # symdg create -type RDF2 devgroupname Issue the above command on nodes attached to the R2 side. The group name must be the same on each node on the R1 and R2 side. The devgroup name used will be later placed in variable DEVICE_GROUP defined in pkg.env file. 2. Use the symld command to add all LUNs that comprise the Volume Group for that package on that host.
Verifying the EMC Symmetrix Configuration When finished with all these steps, use the symrdf list command to get a listing of all devices and their states. Back up the EMC Solutions Enabler database on each node, so that these configuration steps do not have to be repeated if a failure corrupts the database. The EMC Solutions Enabler database is a binary file located in the directory /var/symapi/db.
Configuring PV Links The examples in the previous sections describe the use of the vgimport and vgexport commands with the -s option. In addition, the mk1VGs script uses a -s in the vgexport command, and the mk2imports script uses a -s in the vgimport command. Optionally, remove this option from both commands if using PV links. The -s option to the vgexport command saves the volume group id (VGID) in the map file, but it does not preserve the order of PV links.
Symmetrix CLI database on each node has already been setup, as described in the section, “Preparing the Cluster for Data Replication” (page 257). CAUTION: M by N configurations cannot be used with R1/R2 swapping. Figure 47 depicts a 2 by 2 configuration. Data in this figure are used in the example commands given in the following sections. This example shows R1 devices at one data center and R2 devices with Business Continuity Volumes (BCVs) at the other.
5. On each node on the R2 side (node3 and node4), associate the local BCV devices to the R2 device group. # symbcv -g dgoraA add dev 01A # symbcv -g dgoraA add dev 01B # symbcv -d dgoraB add dev 052 # symbcv -d dgoraB add dev 053 6. To manage the BCV devices from the R1 side, it is necessary to associate the BCV devices with the device groups that are configured on the R1 side. Use the following commands on hosts directly connected to the R1 Symmetrix.
To create a consistency group using RDF-ECA on the R1 side, use # symcg create cgoradb -rdf_consistency -type rdf1 Replace rdf1 with rdf2 in the command to create the consistency group on the R2 side. Use the same name on all nodes. To use RDF-ECA, ensure that the RDF process daemon is running on any of the locally attached hosts. For redundancy, it is recommended that you run multiple instances of the RDF daemon on different hosts.
# mkdir /dev/vgoraB # mknod /dev/vgoraA/group c 64 0x01000 # mknod /dev/vgoraB/group c 64 0x02000 3. Create the volume groups. Be careful not to span Symmetrix frames. # vgcreate /dev/vgoraA /dev/rdsk/c6t0d0 # vgextend /dev/vgoraA /dev/rdsk/c6t0d1 # vgcreate /dev/vgoraB /dev/rdsk/c5t0d2 # vgextend /dev/vgoraB /dev/rdsk/c5t0d3 4. Create the logical volumes. (XXXX indicates size in MB) # lvcreate -L XXXX /dev/vgoraA # lvcreate -L XXXX /dev/vgoraB 5. Install a VxFS file system on the logical volumes.
NOTE: While creating a volume group, you can choose either the legacy or agile Device Special File (DSF) naming convention. To determine the mapping between these DSFs, use the # ioscan –m dsf command. Creating VxVM Disk Groups using Metrocluster with EMC SRDF If using VERITAS storage, use the following procedure to create disk groups. It is assumed VERITAS root disk (rootdg) has been created on the system where configuring the storage. The following section shows how to set up VERITAS disk groups.
5. Start the logical volume in the disk group. # vxvol -g logdata startall 6. Create a directory to mount the volume. # mkdir /logs 7. Mount the volume. # mount /dev/vx/dsk/logdata/logfile /logs 8. Check to make sure the file system is present, then unmount the file system and deport the disk group. # umount /logs # vxdg deport logdata Repeat steps 4 through 8 on all nodes that will access the disk group. 9. Establish the SRDF link. # symrdf -g devgrpA establish IMPORTANT: VxVM 4.
Figure 49 Bidirectional 2 by 2 Configuration Configuring Serviceguard Legacy Packages for Automatic Disaster Recovery Before implementing these procedures it is necessary to do the following: • Configure your cluster hardware according to disaster tolerant architecture guidelines. See the Understanding and Designing Serviceguard Disaster Tolerant Architectures user’s guide. • Configure the Serviceguard cluster according to the procedures outlined in Managing Serviceguard user’s guide.
Customize the package configuration file as appropriate to your application. Be sure to include the pathname of the control script (/etc/cmcluster/pkgname/pkgname.cntl) for the RUN_SCRIPT and HALT_SCRIPT parameters. 3. In the .ascii file, list the node names in the order for which the package is to fail over. It is recommended for performance reasons, that the package fail over locally first, then to the remote data center.
NOTE: If not use a package name as a filename for the package control script, it is necessary to follow the convention of the environment file name. This is the combination of the file name of the package control script without the file extension, an underscore and type of the data replication technology (srdf) used. The extension .env of the file must be used. The following examples demonstrate how the environment file name should be chosen: Example 1: If the file name of the control script is pkg.
See the example script Samples/ftpit to see how to semi-automate the copy using ftp. This script assumes the package directories already exist on all nodes. Using ftp may be preferable at your organization, since it does not require the use of a.rhosts file for root. Root access via .rhosts may create a security issue. 10. Verify that each node in the Serviceguard cluster has the following files in the directory /etc/cmcluster/pkgname: pkgname.cntl Serviceguard package control script pkgname_srdf.
By default, the Metrocluster SRDF module includes only the Serviceguard volume group module. If modules other than the Serviceguard modules need to be included, then they need to be explicitly specified while creating the Metrocluster SRDF modular package configuration file. The following example shows the package IP, the filesystem, and the monitor subnet module included along with the Metrocluster SRDF module. # cmmakepkg –m dts/mcsrdf –m sg/filesystem –m sg/monitor_subnet -m\ sg/package_ip temp.
4. Copy the environment file template /opt/cmcluster/toolkit/SGSRDF/srdf.env to the dts/dts/dts_pkg_dir directory specified earlier. # cp /opt/cmcluster/toolkit/SGSRDF/srdf.env \ /etc/cmcluster/ 5. Edit the environment file, srdf.env as follows: a. Add the path where the EMC Solutions Enabler software binaries have been installed to the PATH environment variable.
NOTE: If external_pre_script is specified in a Metrocluster package configuration, the external_pre_script will be executed after the execution of Metrocluster module scripts in package startup. Metrocluster module scripts are always executed first during package startup. For a sample Metrocluster SRDF modular package configuration file, see Appendix C (page 491). Migrating a Metrocluster Legacy Package to a Metrocluster SRDF A.08.
3. Locate the Metrocluster environment file directory and add the following as the value to the dts/dts/dts_pkg_dir attribute in the package configuration file: dts/dts/dts_pkg_dir /etc/cmcluster/pkgA 4. Halt the package. # cmhaltpkg pkgA 5. Apply the package configuration with the modular configuration file. # cmapplyconf –P modular_sg_ecm_mc.ascii 6. Run the package in the Serviceguard cluster.
6. Run the package on a node in the Serviceguard cluster. # cmrunpkg -n 7. Enable global switching for the package. # cmmodpkg -e Once the package is created, if the value of any Metrocluster parameter needs to be changed, then edit this package configuration file and re-apply it. Migrating Legacy Packages to Modular Packages using Metrocluster EMC SRDF A.09.
6. Apply the package configuration with the modular configuration file created in step 3. # cmapplyconf -P 7. Run the package on a node in the Serviceguard cluster. # cmrunpkg -n 8. Enable global switching for the package. # cmmodpkg -e Upgrading Metrocluster SRDF Modular Packages to the Latest Version Modular Metrocluster packages must be upgraded to the latest versions to use the features enabled by newer versions.
3. Halt the package, # cmhaltpkg 4. Validate the package configuration file. # cmcheckconf -P 5. Apply the package configuration file. # cmapplyconf -P 6. Run the package on a node in the Serviceguard cluster. # cmrunpkg -n 7. Enable global switching for the package. # cmmodpkg -e Checking the Metrocluster Package Configuration Starting from HP Serviceguard version A.11.
Table 23 Validating Metrocluster Package (continued) Restrictions Cannot confirm if the disks are being replicated in case they are being used as raw disks or in case they belong to an Oracle ASM diskgroup. Skips this check on VxVM diskgroups if the naming convention used by VxVM is legacy and if the controller ID is greater than 512 (This happens when VxVM uses an internal format to recognize devices that do not support legacy DSF format). Also skips the check if the naming convention is enclosure based.
For further information, see the cmpreparestg(1m) man page. All restrictions imposed by Serviceguard are applicable when using cmpreparestg in a Metrocluster. Furthermore, the restrictions and prerequisite mentioned in this section apply to cmpreparestg as well. Easy Deployment of Metrocluster Modular Packages Starting with Serviceguard version A.11.20, the Package Easy Deployment feature is introduced. This feature is available from the Serviceguard Managed version B.03.10.
a. b. 5. Create the Metrocluster package directory on the newly added node.
At time T0, all the SRDF links go down. The application continues to run on the R1 side. At time T1, the SRDF links are restored, and at T2 a manual resynchronization is started to resync new data from the R1 to the R2 side. At time T3, while resynchronization is in progress, the R1 site fails, and the application starts up on the R2 side. Since the resynchronization did not complete when there was a failure on the R1 side, the data on the R2 side is corrupt.
# symmir -g dgname split -rdf 2. Begin to resynchronize the data from R1 to R2 devices. # symrdf -g dgname est 3. Re-establish the BCV to R2 devices on R2 as a mirror. # symmir -g dgname -full est Alternatively, from node on R1 side. # symmir -g dgname -full est -rdf R1/R2 Swapping This section describes how the R1/R2 swapping can be done via the Metrocluster SRDF package and manual procedures.
1. Swap the personalities of the devices and mark the old R1 devices to be refresh from the old R2 devices. # symrdf -g swap -refresh R1 2. After swapping is completed, the devices will be in Suspended state. Next establish the device group for data replication from the new R1 devices to the new R2 devices. # symrdf -g establish Scenario 2: In this scenario, two failures happen before the package fails over to the secondary data center.
failure.
It is required that there be a pool of four additional gatekeeper devices that are NOT associated with any device group. These gatekeepers would be available for other, non-cluster uses, for example, the Symmetrix Manager GUI and other EMC Solutions Enabler or SymAPI requests. After data configuration, each physical device in the Symmetrix has enough space remaining on it for gatekeeper purposes. • This toolkit does not support the HP OmniBack Integration with Symmetrix.
as write-pending to the secondary devices (R2). The data is considered committed to the R2 side devices at cycle switch time. Figure 50 SRDF/Asynchronous Basic Functionality Requirements for using SRDF/Asynchronous in a Metrocluster Environment The following describes the hardware and software requirements for setting up SRDF/Asynchronous in a Metrocluster environment: Hardware Requirements • EMC supports SRDF/Asynchronous on Symmetrix DMX Series only.
Figure 51 Metrocluster Topology using SDRF/Asynchronous Data replication can utilize any extended SAN devices that support SRDF Links, for example DWDM, Fiber Channel over Internet Protocol, etc. However, since the network for a Serviceguard cluster heartbeat requires a “Dark Fiber” link, it is recommended to utilize the DWDM links for SRDF/Asynchronous data replication. This will increase data replication bandwidth and reliability in the Metrocluster environment.
019C 0018 019D 0019 2. R1:5 R1:5 RW RW RW RW RW RW S.. S.. 0 0 0 RW 0 RW WD WD Synchronized Synchronized Create an RDF1 type device group. For example, the group name AsynDG. On R1 side: # symdg create AsynDG -type RDF1 On R2 side: # symdg create AsynDG -type RDF2 3. All devices from the RDF (RA) group configuration are added to the device group for SRDF/Asynchronous operation.
DEV007 DEV007 019C RW 019D RW 0 0 0 RW 0018 WD 0 RW 0019 WD 0 0 0 A..X 0 A..X Consistent Consistent Package Configuration using SRDF/Synchronous or SRDF/Asynchronous The following describes configuring a package using SRDF/Synchronous or SRDF/Asynchronous MSC for first-time installation or pre-existing installations. First-time installation of Metrocluster with EMC SRDF using SRDF/Synchronous If this is a first-time installation of Metrocluster with EMC SRDF do the following steps: 1.
Metrocluster with SRDF/Asynchronous Multi-Session Consistency Data Replication The following sections present the concepts, functionality and requirements for configuring Metrocluster using SRDF/Asynchronous (SRDF/A) Multi-Session Consistency (MSC) Data Replication.
window. When the host discovers that all systems are ready for a cycle switch, it issues a single command to each Symmetrix system that performs a cycle switch to open the SRDF/A window. When the window is open, any I/Os that start will be disconnected, and as a result no dependent I/Os will even be issued by any host to any devices in the multi-session group.
Configuring Metrocluster with EMC SRDF using SRDF/Asynchronous Multi-Session Consistency (MSC) Data Replication The following sections describe the steps for building a composite group and package configuration in a SRDF/Asynchronous MSC environment.
ST Standard A Logical Sym T Dev E Tracks Tracks ------------------- -DEV001 01B6 RW DEV002 01B7 RW DEV003 01B8 RW DEV004 01B9 RW DEV005 01BA RW DEV006 01BB RW DEV007 01BC RW DEV008 01BD RW 6. LI ST N A R1 Inv R2 Inv K T R1 Inv R2 Inv S Dev E Tracks Tracks MDAC ---------------- ----- -----------0 0 RW 0326 WD 0 0 S... 0 0 RW 0327 WD 0 0 S... 0 0 RW 0328 WD 0 0 S... 0 0 RW 0329 WD 0 0 S... 0 0 RW 032A WD 0 0 S... 0 0 RW 032B WD 0 0 S... 0 0 RW 032C WD 0 0 S... 0 0 RW 032D WD 0 0 S...
1. 2. Copy the template file from /opt/cmcluster/toolkit/SGSRDF/srdf.env to the package directory. Change the value of the RDF_MODE variable to Asynchronous. RDF_MODE=async 3. Change the value of the CONSISTENCYGROUPS variable to 1. CONSISTENCYGROUPS = 1 Metrocluster with EMC SRDF is already installed If Metrocluster SRDF is already installed and the Serviceguard applications use SRDF/Asynchronous data replication, you must make the following changes in the srdf.env file: 1.
1. 2. 3. If this was not done previously, split the EMC SRDF logical links for the disks associated with the application package. See the script, Samples/pre.cmquery (edit to the SRDF groups configured) for an example of how to automate this task. The script must be customized with the Symmetrix device group names. Create and test a standard Serviceguard cluster using the procedures described in the Managing Serviceguard user’s guide.
d. e. Uncomment the DEVICE_GROUP variable and set them to the Symmetrix device group names given in the ’symdg list’ command. The DEVICE_GROUP variable may also contain the consistency group name if using a M by N configuration. Uncomment the RETRY and RETRYTIME variables. The defaults should be used for the first package. The values should be slightly different for other packages. RETRYTIME should increase by two seconds for each package.
The source disk site is now ready for the Continentalclusters operation. Setting up a Recovery Package on the Recovery Cluster The installation of EMC SRDF, Serviceguard, and Continentalclusters software is exactly the same as in the previous section. The procedures below will install and configure a recovery package on the target disk site. Consult the Managing Serviceguard user’s guide for instructions on setting up a Serviceguard cluster (that is, LAN, VG, LV, and so on). 1. 2.
9. according to the needs of the application. See the Managing Serviceguard user’s guide for more information on these variables. Change the Subnet IP from ftp copy. Verify that each host in both clusters in the Continentalclusters has the following files in the directory /etc/cmcluster/: .cntl (Continentalclusters package control script) .conf (Serviceguard package ASCII config file) .
4. 5. 6. Edit the Continentalclusters security file /etc/opt/cmom/cmomhosts to allow or deny hosts read access by the monitor software. On all nodes in both clusters copy the monitor package files from /opt/cmconcl/scripts to /etc/cmcluster/ccmonpkg. Edit the monitor package configuration as needed in the file /etc/cmcluster/ccmonpkg/ccmonpkg.config. Set the AUTO_RUN flag to YES. This is in contrast to the flag setting for the application packages.
Failback Scenarios There is no failback counterpart to the “pushbutton” failover from the source disk site to the target disk site. Failback is dependent on the original nature of the failover, the state of primary and secondary Symmetrix SRDF volumes (R1 and R2) and the condition of the source disk site. In Chapter 2: “Designing Continentalclusters”, there is a discussion of failback mechanisms and methodologies in the section “Restoring Disaster Tolerance” (page 98).
ensure that any reads on the R1 side will be current, by reading data through the SRDF link from the R2 side. NOTE: If the system administrator does not want synchronization performed from the remote (recovery) site, the device groups should be split and recreated manually. 5. 6. Ensure that the monitor packages at the primary and recovery sites are running. Verify device group is synchronized. # symrdf list 7.
# cmruncl 10. Verify the data for data consistency and currency. Scenario 2 The primary site Symmetrix experienced a catastrophic hardware failure and all data was lost on the array. After the reception of the Continentalclusters alerts and alarm, the administrators at the recovery site follow prescribed processes and recovery procedures to start the protected applications on the target disk site.
Maintaining the EMC SRDF Data Replication Environment Normal Startup The following is the normal Continentalclusters startup procedure. On the source disk site: 1. Start the source disk site. # cmruncl -v The source disk site comes up with ccmonpkg up. The application packages are down, and ccmonpkg is up. 2. Manually start application packages on the source disk site. # cmmodpkg -e 3. Confirm source disk site status. # cmviewcl -v and # cmviewconcl -v 4. Verify SRDF Links.
5. To apply the new Continentalclusters configuration. # cmapplyconcl -C 6. Restart the monitor package.
6 Building Disaster Recovery Serviceguard Solutions Using Metrocluster with 3PAR Remote Copy HP 3PAR storage systems allow users to configure data replication solutions to provide disaster tolerance for Serviceguard clusters over long distances. This chapter describes the HP 3PAR Remote Copy software and the additional files that integrate the HP 3PAR storage system with Metrocluster. It also shows how to configure both Metrocluster and Continentalclusters solutions using HP 3PAR Remote Copy.
Overview of Solution for Metrocluster with 3PAR Remote Copy Figure 53 depicts applications deployed in a Metrocluster with 3PAR Remote Copy environment. A Metrocluster is configured with the nodes at Site1 and Site2. When Site1 and Site2 form a Metrocluster, a third location is required where Quorum Server or arbitrator nodes must be configured. There is a 3PAR storage system at each site and they are connected to each other through Remote Copy links.
Copy and Continentalclusters solutions. HP 3PAR Remote Copy is a product that allows you to copy virtual volumes from one 3PAR storage system to another. Remote Copy Pairs A Remote Copy pair is a pair of storage systems on which Remote Copy operations are performed. Within this pair, the 3PAR storage system from which the data is being replicated is the primary storage system. The 3PAR storage system to which the data is being replicated to is the remote or backup storage system.
Remote Copy and Thin Provisioning A Common Provisioning Group (CPG) is a user-created storage pool available to all volumes associated with it. There are two types of virtual volumes, which draw spaces from CPGs that can be used with Remote Copy: • Thinly Provisioned Virtual Volumes (TPVVs) • Fully provisioned Virtual Volumes. For TPVVs, all data and snapshot space is allocated on demand from a CPG, and for fully provisioned virtual volumes, only the snapshot space is allocated on demand from the CPG.
Figure 55 Single Storage System Managing Multiple Independent Applications Each domain provides users with various levels of accessibility domain objects. A domain is made of Common Provisioning Groups (CPGs), hosts, and Remote Copy groups. Domains contain derived domain objects such as Virtual Volumes (VVs), Logical Disks (LDs), and volume exports (VLUNs).
setting up the Remote Copy links, see 3PAR Remote Copy User Guide available at: http:// www.hp.com/go/saw or contact your HP representative. • Each node in the Metrocluster must have TCP/IP connectivity to both local and remote 3PAR storage systems. HP recommends you to provide redundant network paths between the cluster nodes and the 3PAR storage systems. • For each node in the site, there must be at least two alternately routed fibre channel paths to the local 3PAR storage system.
size, though they can have different RAID levels. In addition, they must be TPVVs or fully provisioned virtual volumes. NOTE: When a TPVV is configured as a primary volume in a Remote Copy volume group, no data should be written on the secondary volume before adding it to the Remote Copy volume group, or it must match the primary volume. This enables the primary and secondary volumes to match during initial synchronization.
where: 4. ◦ -usr_aw is the allocation warning alert limit for the user space specified in percentage. This generates an alert when the user space of the volume exceeds a specified percentage of the volume’s size. ◦ -usr_al is the allocation limit of the user space specified in percentage. This prevents the user space from exceeding a specified percentage of the volume’s size. Export the created volume to all the nodes in the primary site of the Metrocluster.
7. On the primary storage system, start the group created in step 5. On primary storage system: cli% startrcopygroup 8. To verify the creation of a Remote Copy volume group pair, use either showrcopy command or 3PAR Management Console. cli% showrcopy groups A Remote Copy volume group pair is now created. These are the high level commands. For more information on creating a Remote Copy volume group, see Remote Copy User Guide available at: http://www.hp.
The utility will prompt for the passwords of all the users of the storage systems that are passed as arguments.
SSH public key successfully set! 4. 5. Repeat the steps from 2 to 3 for the other storage system. Repeat the steps from 1 to 4 on all nodes in the cluster. Defining Storage Units Both LVM and VERITAS VxVM storage can be used in disaster tolerant clusters.
7. Login to the 3PAR storage system of the target disk site. Stop the Remote Copy volume group and reverse the direction of replication to make disks read/write. cli% setrcopygroup reverse —stopgroups 8. On the target disk site, import the VGs on all of the systems that might run the Serviceguard package and backup the LVM configuration. # vgimport -s -m # vgchange -a y # vgcfgbackup # vgchange -a n 9.
The following procedure details how to validate the VERITAS disk groups on nodes in the target disk site: 1. Login to the target disk site's 3PAR storage system. Stop the Remote Copy volume group and reverse the direction of replication. cli% setrcopygroup reverse -stopgroups 2. Import the disk group on a node in the target disk site. # vxdg -tfC import logdata 3. Start the logical volume in the disk group. # vxvol -g logdata startall 4. Create a directory to mount the volume.
NOTE: Metrocluster is usually used with applications such as Oracle. So, the application toolkit module must also be included when Metrocluster is used in conjunction with an application. For example, when Metrocluster is used in conjunction with the Oracle toolkit, the Oracle toolkit module and other required modules must also be included with the Metrocluster with 3PAR Remote Copy module. Use the following command: # cmmakepkg –m dts/mc3parrc –m sg/filesystem -m sg/package_ip -m\ ecmt/oracle/oracle temp.
j. Specify the target name associated with the Remote Copy volume group on DC1 for the HP 3PAR storage system in DC2. dts/3parrc/DC1_RC_TARGET_FOR_DC2 “3PAR002” k. Specify the target name associated with the Remote Copy volume group on DC2 for the HP 3PAR storage system in DC1. dts/3parrc/DC2_RC_TARGET_FOR_DC2 “3PAR001” NOTE: For steps f through k, use HP 3PAR Management Console/HP 3PAR CLI to identify the values for configuring Metrocluster with 3PAR Remote Copy attributes. l.
must be configured with sites and each cluster nodes must be associated to a site. For information on configuring the failover policy to site_preferred or site_preferred_manual, see “Site Aware Failover Configuration” (page 26). 3. Validate the package configuration file. # cmcheckconf -P pkgName.config 4. Apply the package configuration file. # cmapplyconf -P pkgName.
3. Distribute the package configuration changes, if any. # cmapplyconf -P pkgname.config 4. Start the package with the appropriate Serviceguard command. # cmmodpkg -e pkgname Planned node maintenance is treated the same as a failure by the cluster. If you take a node down for maintenance, package failover and quorum calculation is based on the remaining nodes. Make sure that nodes are taken down evenly at each site, and that enough nodes remain on-line to form a quorum if a failure occurs.
Figure 56 Sample Mutual Recovery Configuration of Continentalclusters with 3PAR Remote Copy A Serviceguard cluster is configured with the nodes at Site1. A separate Serviceguard cluster is configured with the nodes at Site2. A Continentalclusters is configured using the clusters at Site1 and Site2. Each cluster is configured with either a cluster lock disk or a quorum server as a tie-breaker. There is a HP 3PAR storage system at each site, and they are connected to each other through Remote Copy links.
Setting up the Continentalclusters Configuration The following steps detail the basic procedure for setting up the Continentalclusters configuration file and the monitoring packages on the two clusters. For details on creating and editing the configuration file, see Chapter 2. 1. A secure communication using SSH must be set up for inter-cluster operations. For more information, see section “Setting up Security with Continentalclusters Version A.08.00” (page 60). 2.
NOTE: • During an alert, the cmrecovercl will not start the recovery packages unless the -f option is used. • During an alarm, the cmrecovercl will start the recovery packages without the -f option. • When there is neither an alert nor an alarm condition, cmrecovercl command cannot start the recovery packages on the target disk site.
move the application packages back to the primary site are different depending on the roles of the Remote Copy volume groups on either site. • Failback to the Primary Site The following procedure applies to the situation where the Remote Copy volume group has a status of “Primary" on the primary site and "Primary-Rev" on the recovery site. 1.
Reconfiguring Recovery Group Site Identities in Continentalclusters after a Recovery In a disaster scenario where the primary site goes out of operation, but there was no loss of data on the storage systems or the servers. The recovered application can continue to run at the recovery site without requiring to fail back on the primary site. It is desirable to have the same level of recovery capabilities for the applications in their new site, as they had in their original primary site.
Table 26 Error Messages and their Resolution Log Messages Cause The Remote Copy is not functioning The error was caused by one of the from the remote storage system to the following reasons: local storage system. The package is • The Remote Copy volume group not allowed to start up. Start the was stopped manually by using Remote Copy volume group manually stoprcopygroup command or before restarting the package. via 3PAR Management Console.
Table 26 Error Messages and their Resolution (continued) Log Messages Cause Resolution • Password less SSH is not configured from the node to the storage system. • The Remote Copy volume group for DC1 or DC2 may not exist. Not able to determine the status of the local storage system. This might be because either the storage system is down or because there are SSH connectivity issues. The package is not allowed to start up.
Table 26 Error Messages and their Resolution (continued) Log Messages Cause Resolution in use and the failure was caused because another was not available to control the array. For more information, see “Managing SSH connections to 3PAR array” (page 335). Not able to determine the status of the local storage system. This may be because of SSH connectivity issues or because the local storage system is down. The role of remote Remote Copy volume group is “Primary”.
Table 26 Error Messages and their Resolution (continued) Log Messages Cause Resyncwait timeout has occurred. The package is not allowed to start up on this node. Either data copy is still in progress between the virtual volumes of the primary and secondary Remote Copy volume groups () or the status of the volume group/virtual volume is not “Started or Synced”.
# ping If you are using storage system network name, verify it is resolving to proper IP address using nslookup command from the cluster nodes. # nslookup Issues with the startrcopygroup command If the local replication role is secondary, the remote replication role is primary, and the remote copy link is up, Metrocluster executes stop, reverse, and start operations for the Remote Copy volume group on the local site.
7 Designing a Disaster Recovery Solution Using Site Aware Disaster Tolerant Architecture This chapter describes Site Aware Disaster Tolerant Architecture (SADTA) for deploying complex workloads with inter-dependent packages that are managed collectively for disaster tolerance in Metrocluster. SAP and Oracle RAC database, which includes 10gR2 RAC, 11gR1 RAC, and 11gR2 RAC are some examples of complex workloads.
Components of SADTA This section describes the components of SADTA.
sfo_hrdb_mp sfo_hrdb up up running running enabled enabled no no Oracle Clusterware Sub-cluster Oracle Clusterware is layered above the Serviceguard cluster membership. In SADTA, the Oracle Clusterware sub-cluster is formed with membership exclusively from a set of nodes in an underlying site definition in the Serviceguard cluster. The Oracle Clusterware sub-clusters are installed and configured separately at each site using the Oracle Universal Installer.
sub-cluster automatically. The CFS sub-clusters are formed automatically when the cluster starts. The SG SMS commands, CFS commands, and other utilities operate on the CFS sub-cluster where they are executed. The CFS sub-clusters manage the cluster file systems on disk arrays that are local to a site. NOTE: The CFS sub-cluster functionality is supported only in a Metrocluster environment with sites defined in the underlying cluster.
Figure 58 Web Server Configured as a Complex Workload Site Controller Package The Site Controller Package manages the package configuration of a complex workload in SADTA. The Site Controller Package is a special Metrocluster failover package that is created using the dts/sc module. Each complex workload that is configured using SADTA must have a corresponding Site Controller Package.
3. dts/dts/dts_pkg_dir This variable specifies the absolute path to the Site Controller Package directory. This directory must be present in all the nodes in the Metrocluster. The Site Controller Package looks for the Metrocluster environment file in this directory. All Metrocluster flag files must be created within this directory. 4. monitor_interval This attribute specifies the time interval, in seconds, at which the Site Controller Package monitors the complex-workload packages.
failure and there are no other nodes on the local site that it can run on with package switching enabled. The workload packages can be halted and restarted using the cmhaltpkg and cmrunpkg commands when the Site Controller Package is running. The Site Controller Package is not affected when the workload packages are administratively halted using the cmhaltpkg command. Site Failover The Site Controller Package initiates a site failover when the site is lost or when the complex workload has failed.
Site Safety Latch The Site Safety Latch prevents inadvertent simultaneous startup of the workload configuration on both sites. The Site Safety Latch is an internal mechanism that is created for each Site Controller Package. It is created automatically on all the cluster nodes when the corresponding Site Controller Package configuration is applied in the cluster. The Site Safety Latch for a Site Controller Package is identified by the following convention: /dts/mcsc/
Overview of SADTA Configuration A complex workload is configured redundantly by configuring it at each site sub-cluster. A Site Controller Package is created to manage the workload. The Site Safety Latch is created automatically on all nodes when the Site Controller Package configuration is applied in the cluster. The workload packages at each site are configured with the Site Controller Package and the Site Safety Latch is configured with the appropriate package in the workload.
Configuring Metrocluster with Sites To configure SADTA, a Serviceguard cluster must be created that comprises nodes from both sites. In this example, a Serviceguard cluster is created using nodes from two sites. Complete the following steps to configure a Metrocluster: 1. Create a Serviceguard cluster with the sites configured. 2. Configure the Cluster File System Multi-node Package (SMNP). The following sections describe each of these steps in detail.
# cmdeploycl -s -n -n -s -n -n -c -q -cfs For example, # cmdeploycl -s siteA -n node1 -n node2 -s siteB -n node3 -n node4 -c site_cluster -q quorum.server.com -cfs This creates a cluster with two sites with the CVM infrastructure configured via the SG-CFS-pkg System Mutlti-node (SMNP) package. For additional information on cmdeploycl, see the cmdeploycl man page, cmdeploycl (1m).
Complete the following procedure on the CVM cluster master node in the primary sub-cluster to set up the CVM disk group volumes: 1. Initalize the source disks of the replication pair: # /etc/vx/bin/vxdisksetup -i # /etc/vx/bin/vxdisksetup -i 2. Create a disk group for the complex workload data: # vxdg –s init 3. Activate the CVM disk group in the primary sub-cluster: # vxdg -g set activation=sw 4.
3. Create Serviceguard Disk Group MNP packages for the disk groups with a unique name in the cluster. # cfsdgadm add all=sw\ where node1 and node2 are the nodes in the Source Disk Site. 4. Activate the CVM disk group in the Source Disk Site CFS sub-cluster. # cfsdgadm activate 5. Create a volume from the disk group. # vxassist -g make 4500m 6.
4500m Configuring the Storage Device using VERITAS CVM Complete the following procedure on the CVM cluster master node in the Source Disk Site to set up the CVM disk group volumes: 1. Initialize the source disks of the replication pair. # /etc/vx/bin/vxdisksetup -i # /etc/vx/bin/vxdisksetup -i 2. Create a disk group for the complex workload data. # vxdg –s init 3.
1. Define the appropriate volume groups on each host system in the Source Disk Site. # mkdir /dev/ # mknod /dev/ >/group c 64 0xnn0000 where the name /dev/ and the number nn are unique within the entire cluster. 2. Create the volume group on the source volumes. # pvcreate -f /dev/rdsk/cxtydz # vgcreate /dev/ /dev/dsk/cxtydz 3. Create the logical volume for the volume group. # lvcreate -L XXXX /dev/ where XXXX indicates the size in MB. 4.
complex workload will not run until its dependent CFS mount point MNP package is up, and will halt before the CFS mount point MNP package is halted.
environment, see the respective chapters of this manual to configure replication. After splitting the replication, configure the replica complex workload. Configuring the Storage Device for Complex Workload at the Target Disk Site The storage device for complex workload must be configured for the data of the complex workload from the replicated disks at the target disk site. The procedure to configure the storage device differs if CFS, CVM, and SLVM is used.
1. From the CVM master node at the target disk site, import the disk groups used by the complex workload. # vxdg -stfC import 2. Create Serviceguard disk group MNP packages for the disk groups with a unique name in the cluster. # cfsdgadm add all=sw Where node1 and node2 are the nodes at the target disk site. 3. Activate the complex workload disk groups in the CFS sub-cluster. # cfsdgadm activate 4.
IMPORTANT: VERITAS CVM disk groups must be configured in a dedicated modular MNP package using the cvm_dg attribute. This modular MNP package must be configured to have a package dependency on the SG-CFS-pkg SMNP package. Metrocluster SADTA does not support configuring Legacy style packages for managing VERITAS CVM disk groups. Configuring the Storage Device using SLVM Complete the following procedure to import volume groups on the nodes in the target disk site: 1.
Following are the guidelines that you must follow while configuring the Site Controller Package: • The default value of the priority parameter is set to no_priority. The Site Controller Package should not be subjected to any movement because of package prioritization. So do not change this default value. • The default value of the failover_policy parameter for the Site Controller Package is set to site_preferred. Based on your requirement, the value can be set to site_preferred_manual.
site site2 For example: site san_francisco site san_jose 4. Copy the Metrocluster environment template file to the Site Controller Package directory. For example: cp /opt/cmcluster/toolkit/SGCA/xpca.env\ /etc/cmcluster/cw_sc/cw_sc_xpca.env The command differs based on the arrays that are configured in your environment. For more information on copy the Metrocluster environment template file based on the arrays in your environment, see the respective chapters of this manual.
resource_name /dts/mcsc/cw_sc resource_polling_interval 120 resource_up_value != DOWN resource_start automatic If you have VERITAS CVM configured in your environment, add the EMS resource details in the CVM disk group packages on both the sites.
NOTE: 5. • Do not add any comments after specifying the critical and managed packages. • Always set auto_run parameter to yes for failover packages configured as critical or managed packages. • The packages configured with mutual dependency must not be configured as critical or managed packages. • Site controller package can be created using the Package Easy Deployment feature available in Serviceguard Manager version B.03.10.
Table 27 Additional Validation of Site Controller Packages (continued) Check the managed and critical package types cmapplyconf Check if the Site values are valid cmapplyconf cmcheckconf [-P/-p] cmcheckconf [-P/-p] Verify if the Site Safety Latch cmapplyconf dependencies are configured properly cmcheckconf [-P/-p] for a complex workload Checks if the managed or critical package type is either multi node or failover.
For SADTA, a Site Controller Package must be configured to provide robust site failover semantics for a site aware disaster tolerant RAC database. The Site Controller Package starts the configured local RAC MNP stack packages on the site where it is started. The Site Controller Package monitors the started RAC MNP stack packages. When these packages fail, the Site Controller Package fails over to the remote site.
This section addresses the following topics: • “Summary of Required Procedures” (page 362) • “Sample Configuration” (page 364) • “Configuring SADTA” (page 366) • “Setting up Replication” (page 367) • “Configuring Metrocluster with Sites” (page 367) • “Installing and Configuring Oracle Clusterware” (page 369) • “Installing and Configuring Oracle Real Application Clusters (RAC)” (page 372) • “Creating the RAC Database ” (page 373) • “Creating Identical RAC Database at the Remote Site ” (page
http://www.hp.com/go/hpux-serviceguard-docs. 2. Install Serviceguard extension for Oracle RAC. Install SGeRAC with CFS to use CFS/CVM. 3. Install the required Serviceguard and SGeRAC patches. For more information on the required Serviceguard and SGeRAC patches, see the Disaster Tolerant Clusters Products Compatibility Feature Matrix available at http://www.hp.com/go/ hpux-serviceguard-docs. 4. 5. Install Metrocluster software. Set up replication between sites.
13. 14. 15. 16. Configure the Site Safety Latch dependencies. Start the site aware disaster tolerant RAC database in the Metrocluster. Configure client access for the RAC database. Configure SGeRAC cluster interconnect subnet monitoring. The subsequent sections explain each of these steps in detail.
Figure 61 Sample Configuration To configure SADTA, two CRS sub-clusters; one at the San Francisco site and the other at the San Jose site, must be created. The Oracle Clusterware software must be installed at each site in the Metrocluster. The CRS daemons at the sub-clusters must be configured as a Serviceguard package using the SGeRAC toolkit. The CRS Home is installed on a file system that is local to a site. The CRS voting and OCR disks must not be configured for replication.
the disk groups and mount points are packaged using different packages at each site CFS sub-cluster. Table 29 lists the packages and other resources at each site.
7. 8. Configure the Site Controller Package and the Site Safety Latch. Configure client access for Oracle Database 10gR2 RAC. The subsequent sections discuss each step in detail. Setting up Replication The RAC database data files and the flash recovery area should be replicated between the site disk arrays. The underlying disks must be configured for replication. The replication mechanisms differ depending on the type of arrays in your environment.
NETWORK_INTERFACE STATIONARY_IP NETWORK_INTERFACE NETWORK_INTERFACE STATIONARY_IP NETWORK_INTERFACE lan4 # SFO_CRS CSS HB 192.1.7.2 lan5 # SFO_CRS CSS HB standby lan1 # SFO client access 16.89.140.203 lan6 # SFO client access standby NODE_NAME sjc_1 SITE san_jose NETWORK_INTERFACE lan2 #SG HB 3 HEARTBEAT_IP 192.1.6.1 NETWORK_INTERFACE lan3 #SG HB 4 HEARTBEAT_IP 192.1.4.1 NETWORK_INTERFACE lan4 #SJC_CRS CSS STATIONARY_IP 192.1.8.
Installing and Configuring Oracle Clusterware After setting up replication in your environment and configuring the Metrocluster, you must install Oracle Clusterware. Use the Oracle Universal Installer to install and configure the Oracle Clusterware. As SADTA requires two Oracle Clusterware sub-clusters, one at each site, you must install and configure Oracle Clusterware twice in the Serviceguard cluster.
export CLASSPATH export ORACLE_SID= Configuring the Storage Device for Installing Oracle Clusterware When Oracle Clusterware is installed in a site, it is installed only on a local file system on the Clusterware sub-cluster nodes of that site. Complete the following steps on all nodes at the site: 1. Create a directory path for Oracle Clusterware Home, set an owner, and specify appropriate permissions. 2.
10. Create the Clusterware VOTE directory in the clustered file system. mkdir /cfs/sfo_crs/VOTE chmod 755 /cfs/sfo_crs/VOTE 11. Set oracle as the owner for the Clusterware directories. chown –R oracle:oinstall /cfs/sfo_crs After setting owners for the OCR and Voting directories, Oracle Clusterware can be installed. Installing and Configuring Oracle Clusterware This section describes the procedure to install and configure Oracle Clusterware. Use the Oracle Universal Installer to install Oracle Clusterware.
/cfs/sfo_crs/VOTE/vote 9. Complete the remaining on-screen instructions to complete the installation. Once the installation is complete, you must ensure that Oracle Clusterware is installed appropriately, and that the Clusterware sub-cluster is formed. To ensure that Oracle Clusterware is installed appropriately, check if the /opt/crs/oracle/product/10.2.0/crs/bin/crsd.bin and /opt/crs/oracle/product/10.2.0/crs/bin/ocssd.bin processes are running on all nodes in the current site.
Creating the RAC Database After installing Oracle RAC, create the RAC database from the site which has the source disks of the replication. In this manual, this site is referred to as the local site. The RAC database creation is replicated to the remote site through physical replication and the identical RAC database can be configured on the remote site from the replication target disks. In our example configuration, a database, hrdb, is created from the San Francisco site.
mkdir /cfs/rac 8. Create the Mount Point MNP packages. cfsmntadm add hrdbdg rac_vol /cfs/rac sfo_hrdb_mp all=rw SFO_1\ SFO_2 9. Mount the cluster file system on the CFS sub-cluster. cfsmount /cfs/rac 10. Create a directory structure for the RAC database data files in the cluster file system. Set proper permission and owners for the directory.
9. Mount the RAC database flash recovery file system in the site CFS sub-cluster. cfsmount /cfs/flash 10. Create directory structure in the cluster file system for the RAC database flash recovery area. chmod 775 /cfs/flash cd /cfs/flash mkdir flash chmod 775 flash chown oracle:oinstall flash Creating the RAC Database using the Oracle Database Configuration Assistant After setting up the file systems for the RAC database data files, you must create the RAC database.
to configure replication. After configuring replication in your environment, configure the replica RAC database. For example, complete the following procedure to prepare the replication environment: 1. Ensure the replication is in PAIR state at the replication Target Disk Site node. The STATUS field should have PAIR value.
The -p option retains the permissions of the file. 2. Setup the first RAC database instance on the target site. In this example, run the following commands from the SJC_1 node. cd /opt/app/oracle/product/10.2.0/db_1/dbs ln -s /cfs/rac/oradata/hrdb/orapwhrdb orapwhrdb1 chown -h oracle:oinstall orapwhrdb1 chown oracle:oinstall inithrdb1.ora 3. Copy the second RAC database instance pfile from the source site to the target site second RAC database instance node.
srvctl add instance -d hrdb -i hrdb1 -n SJC_1 srvctl add instance -d hrdb -i hrdb2 -n SJC_2 After registering the database with the CRS sub-cluster on the remote site, you can run the srvctl status command to view the health of the database. Configuring the RAC MNP Stack at the Recovery Cluster The RAC database must be packaged as Serviceguard MNP packages. You must configure the RAC MNP package to have a dependency on the site Clusterware sub-cluster MNP package.
8. Copy the Metrocluster template file to the Site Controller Package directory. cp /opt/cmcluster/toolkit/SGCA/xpca.env\ /etc/cmcluster/hrdb_sc/hrdb_sc_xpca.env This command illustrates copying the template file for Metrocluster with Continuous Access for P9000 and XP. Edit the Metrocluster environment file to match your environment. 9. Distribute the environment file to all nodes under the Site Controller Package directory. From the SFO_1 node: rcp /etc/cmcluster/hrdb_sc/hrdb_sc_xpca.
NOTE: Specify the condition as != DOWN with a space before the word Down. Ignoring the space will cause the cfsdgadm command to fail. If you have SLVM configured in your environment, add the EMS resource details in RAC database package configuration file. RESOURCE_NAME /dts/mcsc/hrdb_sc RESOURCE_POLLING_INTERVAL 120 RESOURCE_UP_VALUE != DOWN RESOURCE_START automatic You must apply the modified RAC database package configuration using the cmapplyconf command. 2.
1. Run the cmviewcl command to view the disaster tolerant RAC database configuration in a Metrocluster.
FAN capabilities can be accessed by the client application using the FAN API directly or by using FAN-integrated clients provided by Oracle. The Metrocluster RAC configuration uses two Oracle sub-clusters in a single SGeRAC cluster. At any time, a given database is accessed through only one of the Oracle sub-clusters. It is referred to as the active sub-cluster for the database.
1. Create a package directory on all nodes in the site. mkdir -p /etc/cmcluster/pkg/sfo_ic 2. Create a package configuration file and control script file. Use site-specific names for the files. You must follow the legacy package creation steps. cmmakepkg -p sfo_ic.conf cmmakepkg -s sfo_ic.cntl 3. 4. 5. 6. 7. Specify a site-specific package name in the package configuration file. Specify only the nodes in the site for the node_name parameter. Specify the package type as MULTI_NODE.
Figure 62 Sample Oracle RAC Database with ASM in SADTA The Oracle Clusterware software must be installed at each site in the Metrocluster. The CRS daemons at the sub-clusters must be configured as a Serviceguard package using the HP Serviceguard extension for RAC (SGeRAC) toolkit at each site. The CRS Home must be installed on a file system that is local to a site. The CRS voting and OCR disks must not be configured for replication.
d. e. 6. Configure and test the RAC MNP stack at the source disk site. Halt the RAC database at the source disk site. Configure the identical ASM disk group at the remote site. This step is required only for Oracle 11g R1. 7. 8. 9. 10. Setup the identical RAC database at the remote site. Configure the Site Controller Package. Configure the Site Safety Latch dependencies. Start the Disaster Tolerant RAC Database in the Metrocluster. The subsequent sections elaborate on each of these steps.
Installing Oracle Real Application Clusters (RAC) Software The Oracle RAC software must be installed twice in the Metrocluster, once at each site. Also, the RAC software must be installed in the local file system in all the nodes in a site. To install Oracle RAC, use the Oracle Universal Installer (OUI). After installation, the installer prompts you to create the database. Do not create the database until you install Oracle RAC at both sites.
Halting the RAC Database on the Source Disk Site After creating the RAC database on the source disk site, you must halt it to replicate it on the target disk site. If you are using 11g R2 RAC, you must change the remote_listener for the database before halting the RAC database MNP stack as explained in step 1. 1. When using Oracle 11g R2 with ASM, the remote_listener for the database is set to the : by default.
rcp -r admin :$PWD 2. Run the following command at the target disk site: chown -R oracle:oinstall /opt/app/oracle/admin 3. Copy the first ASM instance pfile and password file from the source disk site to the first ASM instance node in the target disk site. cd /opt/app/oracle/admin/+ASM/pfile rcp -p init.ora :$PWD cd /opt/app/oracle/product/11.1.0/db_1/dbs rcp -p orapw+ASM1 :$PWD The -p option retains the permissions of the file. 4.
1. Copy the first RAC database instance pfile and password file from the source site to the first RAC database instance node in the target disk site. In this example, run the following commands from the first node in site1: cd /opt/app/oracle/product/11.1.0/db_1/dbs rcp -p inithrdb1.ora :$PWD rcp -p orapwhrdb1 :$PWD The -p option retains the permissions of the file. 2. Setup the first RAC database instance on the target disk site.
10. Edit the tnsnames.ora file on the nodes at the target disk site and modify the HOST = keywords to suit the target disk site environment. In this example, you must edit the tnsnames.ora file on each node in this site. 11. Register the database with the CRS sub-cluster on remote site. srvctl add database -d hrdb -o /opt/app/oracle/product/11.1.
Starting the Disaster Tolerant Oracle RAC Database with ASM in the Metrocluster The procedure to start the disaster tolerant Oracle RAC database with ASM is identical to the procedure for starting a complex workload in a Metrocluster. For more information on starting the complex workload in the Metrocluster, see “Starting the Disaster Tolerant Complex Workload in the Metrocluster” (page 359).
failed but not halted, the Site Controller Package fails over to a remote site node to perform a site failover. Before starting the complex-workload packages configured at the remote site, the Site Controller Package ensures that it is safe to do so. The failed complex-workload packages might not have halted cleanly, leaving stray processes and resources. In such scenarios, it is not safe to start the identical complex workload configuration on the remote site.
Network Partitions Across Sites A network partition across sites is similar to a site failure. The Serviceguard cluster nodes on both sites detect this failure and try to reform the cluster using the Quorum Server. The nodes from only one of the sites will receive the quorum and form the cluster. The nodes on the other site restart and deliberately fail the active complex-workload packages running on them.
to be available in the cluster. However, as the Site Controller Package has failed in the cluster, the complex workload configuration can no longer automatically failover to the remote site. Site Failure A site failure is a scenario where a disaster or an equivalent failure results in all nodes in a site failing or going down. The Serviceguard cluster detects this failure, and reforms the cluster without the nodes from the failed site.
Oracle RAC Database Oracle Clusterware Daemon Failure The Oracle Clusterware is an essential resource for all RAC databases in a site. When the crsd or evmd daemons are aborted on account of a failure, they are automatically restarted on the node. When the cssd daemon is aborted on account of a failure on a node, the node is restarted. The RAC MNP stack continues to run with one less instance on the site.
to yes, will automatically start. If the auto_run flag set to no, then these instances must be manually started on the restarted node. Prior to halting a node in the cluster, the Site Controller Package must be moved to a different node in the site. However, if the node that needs to be halted in the cluster is the last surviving node in the site, then the Site Controller Packages running on this node fail over to the other site.
3. Run the HP-UX touch command with the DETACH flag, in the Site Controller Package directory touch DETACH 4. Halt the Site Controller Package. cmhaltpkg The Site Controller Package halts without halting the complex workload packages. The Site Controller Package leaves the Site Safety Latch open on this site. The DETACH mode file is automatically removed by the Site Controller Package when it halts.
1. 2. Halt the Site Controller Package. Remove all the Site Safety Latch dependencies configured for the packages managed by the Site Controller Package. Also remove the Site Controller EMS resource dependency from packages managed by the Site Controller Packages on both sites. For example, if you have CVM or CFS configured in your environment, run the following commands from a node on both sites: cfsdgadm delete_ems pkg1dg /dts/mcsc/cw_sc 3. Delete Site Controller Package.
The Site Controller Package starts up on a node in the remote site and starts the complex-workload packages that are configured. Restarting a Failed Site Controller Package If the running Site Controller Package fails because of transient error conditions, restart the Site Controller package on a node in the site where it was previously running. Complete the following steps to restart the failed Site Controller Package: 1. Determine the error message logged in the package log, and fix the problem.
e. 3. 4. 5. 6. 7. 8. cmapplyconf -P Get the current configuration of the Site Controller. Modify the Site Controller configuration with the new set of packages that need to be managed on the recovery site. Leave the set of packages that are being managed on the primary site as it is. Start the Site Controller again on the primary site. Halt the Site Controller package on the primary site. This will halt all the complex workload packages that are running on the primary site.
3. 4. Ensure that the new node can access the Clusterware OCR and VOTE disks, and Oracle database disks and add the node to the Serviceguard cluster. Extend the Oracle Clusterware software to the new node. For more information on extending the Oracle Clusterware software, see the Oracle Database 10gR2 RAC documentation available at the Oracle site. 5. 6.
3. Delete an instance from the RAC database. For more information on deleting an instance, see the documentation available at the Oracle documentation site. 4. Delete the RAC database software and Oracle Clusterware. For more information on deleting the RAC database and Oracle Clusterware, see the documentation available at the Oracle documentation site. 5. 6. 7. Remove the node from the node list of the Site Controller Package. Run the cmhaltnode command to halt the cluster on this node.
3. Ensure that the Site Controller Package is enabled on all nodes in the site where the database must be started. cmmodpkg –e –n -n \ 4. Start the Site Controller Package by enabling it. cmmodpkg –e The Site Controller Package starts on the preferred node at the site. At startup, the Site Controller Package starts the corresponding RAC MNP stack packages in that site that are configured as managed packages.
cmhaltpkg Because the RAC MNP is down, as it is halted in the cluster, the Site Controller Package does not interpret it as a failure. The Site Controller Package continues to run on the same site and the Site Safety Latch will remain open. After the maintenance procedures are complete, restart the RAC MNP package at the same site.
Limitations of a Site Aware Disaster Tolerant Architecture Following is a limitation of SADTA: • No Support for Arbitrator Nodes When configuring Metrocluster for applications using CFS sub-clustering storage, no arbitrator nodes must be configured. The Metrocluster, in such configurations, must only use the Quorum Server at a third location to handle network partitions.
But, it does not clean any of the managed packages for the Site Controller Package. Any issues in the packages managed by Site Controller Package must be fixed by the operator and the package’s node switching must be enabled before restarting the Site Controller Package.
cmviewcl –v –f line Check for the field last_halt_failed under each instance of the MNP package. When set to Yes, that instance of the MNP package did not successfully execute the halt script when it was halted. Check for all instances. The unclean nodes might have stray resources. See the MNP package log file on the corresponding node to identify the reason for the halt script run failure.
Table 30 Error Messages and their Resolution (continued) Log Messages Cause Resolution 4. Clean the site using the cmresetsc tool. 5. Restart the Site Controller Package. Refer to Metrocluster documentation for cleanup procedures needed before restarting the Site Controller. Site Controller startup failed. Starting Site Controller (hrdb_sc) on site siteA. The Site Controller 1.
Table 30 Error Messages and their Resolution (continued) Log Messages Cause Resolution Failed to prepare the storage for site. The preparation of the replicated disk and making it read-write on the site nodes failed. 1. Check the host connectivity to disk arrays. 2. Ensure that the replication management software is configured properly and is functioning correctly. 3. Restart the Site Controller Package.
Table 30 Error Messages and their Resolution (continued) Log Messages Cause Unable to execute command. This message is logged because the Site Controller Package has failed to start one or more cmrunpkg: Unable to start some package or packages that it manages. package instances. This situation occurs Check the log files of the packages managed because the package dependency configured by Site Controller for more details.
Table 30 Error Messages and their Resolution (continued) Log Messages Cause Resolution This message is logged because the Serviceguard command cmviewcl failed due to cluster reformation or transient error conditions. 1. Wait for the cluster to reform (until there is no node in reforming state). 2. Restart the Site Controller Package. Check for any error messages in the package log file on all nodes in the site siteA for the packages managed by Site Controller (hrdb_sc).
Table 30 Error Messages and their Resolution (continued) Log Messages Cause Resolution consistent with the environment file naming convention.
8 Designing a Three Data Center Disaster Recovery Solution This chapter describes Three Data Center architecture through the following topics: • “Overview of 3DC DR Solution” (page 413) • “Infrastructure Requirements” (page 429) • “Configuring an P9000 or XP 3DC DR Solution” (page 429) • “Deploying a Complex Workload in Three Data Center Solution using SADTA” (page 447) • “Verifying the Three Data Center Environment” (page 461) • “Automatic Failover between DC1 and DC2” (page 464) • “Recovery a
Figure 63 3DC DR Solution Configuration The 3DC DR Solution consists of three sites: Site1, Site2 and Site3 as shown in Figure 63 (page 414). There is a P9000 or XP array at each site and they are connected to each other using Continuous Access links. The nodes and the two XP arrays at Site1 and Site2 are configured as a Metrocluster. At Site3, an independent Serviceguard cluster is configured and connected to a third P9000 or XP disk array.
Table 31 Difference Between 3DC Sync/CAJ and 3DC CAJ/CAJ Replication (continued) 3DC Sync/CAJ replication 3DC CAJ/CAJ replication Suitable for applications where data currency is more important and the performance can be sustained. Suitable for the write-intensive applications where the performance with synchronous replication can not be sustained. Higher cost for inter site links in Metrocluster as it requires Comparatively lower cost for inter site links as it can work low latency and high bandwidth.
Figure 64 Bi-Link Configuration with the replication link between Site1 and Site3 Figure 65 (page 416) shows the Bi-Link configuration with the replication link configured between Site2 and Site3. Figure 65 Bi-Link Configuration with the replication link between Site2 and Site3 In Bi-Link configuration: 416 • Continuous Access Synchronous data replication is established to replicate data between the arrays at Site1 and Site2 when using 3DC Sync/CAJ replication.
Pros and Cons of Using Bi-Link and Tri-Link Configuration In a Bi-Link configuration, the disaster tolerance to the application is lost for certain failures. For example, in Bi-Link configuration, if the replication link exists between Site1 and Stite3 as shown in “Bi-Link Configuration with the replication link between Site1 and Site3” (page 416), upon the failure of Site1, the data cannot be protected at DC3.
An example for data being replicated in Multi-Hop topology when the application is running at Site2 is shown in Figure 67 (page 418). Figure 67 Multi-Hop Topology when the application is running at Site2 In Multi-Target configuration, the data enters the configuration on a specific node and then splits into two directions. One direction is the replication to one site and the other direction is the replication to the another site.
Figure 69 Multi-Target Topology when the application is running at Site2 Delta Resync Pair The Delta Resync pair must be configured when using 3DC Tri-Link configuration. The Delta Resync pair must be created with the –nocsus option to the paircreate command. The –nocsus option creates a suspended journal volume, without copying the data. Figure 70 (page 420) shows an example 3DC Sync/CAJ replication using Delta Resync pair configured between Site2 and Site3.
Figure 70 Delta Resync Configuration In a 3DC replication configured with Delta Resync functionality, data is replicated to the third site from one of the sites in the Metrocluster; a Delta Resync pair is configured from the other site. In the event of failure of the site from which data is being copied to the third site, the Delta Resync pair can be used. To replicate data to the third site over Delta Resync pair, run the pairresync command on the Delta Resync pair.
http://h20000.www2.hp.com/bizsupport/TechSupport/DocumentIndex.jsp?lang=en&cc=us& taskId=101&prodClassId=-1&contentType=SupportManual&docIndexId=64255& prodTypeId=18964&prodSeriesId=4309847 Figure 71 Configuration of the Delta Resync Operation that utilizes Remote Command Devices HP StorageWorks Mirror Unit Descriptors To setup Raid Manager Configuration for 3DC architecture, “mirror unit descriptors” (MU#) are used to distinguish and manage an application device group among 3 data centers.
Figure 72 Mirror Unit Descriptors In a 3DC Sync/CAJ replication, the Continuous Access Synchronous replication must use MU# 0 and the Continuous Access Journal replication must use any of the remaining three MU# for remote replication. Figure 73 (page 422) shows an example of configuring MU numbers in 3DC Sync/CAJ environment when Delta Resync pair is configured between Site1 and Site3.
Figure 74 3DC CAJ/CAJ solution and mirror unit descriptor usage example Benefits of 3DC DR Solution The 3DC DR Solution provides the following benefits: • Maintains data currency ◦ • When using 3DC Sync/CAJ solution, synchronous replication over a short distance in a Metrocluster environment provides the highest level of data currency and application availability without significant impact to application performance.
functionality the operations can be shifted to the third site and continue unaffected by the disaster. • Allows for additional staff at the remote data center outside the disaster area. A wide-area disaster affects people located within the disaster area, both professionally and personally. By moving operations out of the main data centers to a remotely located recovery data center, operational responsibilities shift to people not directly affected by the disaster.
Figure 75 Three Data Center Architecture An application is deployed in 3DC DR Solution, by configuring it at all three sites. The sites are referred either as DC1 or DC2 or DC3 for an application based on their role. The primary site for the application is where it is expected to run under normal circumstances and is also referred as DC1 site for the application. The hot-standby site for the application is where it automatically fails over to in case of a disaster in its DC1 site.
Figure 76 Applications Deployed in a 3DC DR Solution using Tri-Link Configuration A Continuous Access Synchronous device group pair must be established when using 3DC Sync/CAJ replication to replicate data between the arrays at the application’s DC1 and DC2 sites. A Continuous Access Journal device group pair must be established when using 3DC CAJ/CAJ replication to replicate data between the arrays at the application’s DC1 and DC2 sites.
• Site 3 is the second-standby site or DC3 for both application A and B. • The application A is configured using 3DC Sync/CAJ replication. The application B is configured using 3DC CAJ/CAJ replication. • For the application A, the data is replicated in a Multi-Target topology from Site 1 to Site 2 and then Site 1 to Site 3. A Delta Resync pair is configured between Site 2 and Site 3.
requirements. A Continuous Access Journal device group pair must be established either between DC1 and DC3 or between DC2 and DC3 to replicate data to application’s DC3 site. When Continuous Access Journal device group pair is configured between DC1 and DC3, the data is replicated always in Multi-Target topology from DC1. This configuration is called Multi-Target Bi-Link configuration.
The DGM provides the following functionalities: • Sends a notification message upon the change in a device group status. • Performs automatic resynchronization of the Continuous Access device group upon link recovery. • In case of a Tri-Link configuration, it performs automatic resynchronization of the Delta Resync pair upon some of the failures that result in data not being replicated over the active Continuous Access Journal pair to the third data center (DC3).
3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. Configure the recovery cluster at the third datacenter. Configure RAID Manager in the primary and recovery clusters. Create Device Group pairs. Assign Remote Command Device to Journal Volumes. Configure LVM Volume Groups. Configure VxVM disk groups. Install and configure the application in the primary cluster. Configure a primary package in the primary cluster using the 3DC modules. Start the primary packages in the primary cluster.
horcm0 11000/udp #Raid Manager instance 0. 2. 3. 4. Use the ioscan command to determine what devices on the P9000 or XP disk array have been configured as command devices. There must be two command devices; a primary command device and a secondary command device. Copy the default Raid Manager configuration file to an instance-specific name: # cp /etc/horcm.conf /etc/horcm0.
(secondary volume). To identify HP-UX device files corresponding to each device represented by CU:LDEV execute the following command: #ls /dev/rdsk/* | raidscan -find –fx NOTE: Only OPEN-V LUNs are supported in a Three Data Center configuration. The ioscan output must be checked to verify which LUNs are OPEN-V LUNs. 9. Determine which devices will be used by the application package. 10. Define a device group that contains all of these devices.
Figure 78 Sample Multi-Target Bi-Link Configuration Sample Raid Manager Configuration on a DC1 NodeA (multi-target Bi-Link) HORCM _MON #ip_address service NodeA horcm0 poll(10ms) 1000 timeout(10ms) 3000 HORCM_CMD #dev_name dev_name dev_name /dev/rdsk/c6t12d0 /dev/rdsk/c9t12d0 HORCM_DEV #dev_group dg12 dg13 dev_name port# TargetID LU# MU# dg12_d0 CL3-E 6 dg13_d0 CL3-E 6 HORCM _INST #dev_group ip_address service dg12 NodeB.dc2.net horcm0 dg13 NodeC.dc3.
#dev_group dg12 dg23 ip_address service NodeA.dc1.net horcm0 NodeC.dc3.
/dev/rdsk/c6t12d0 /dev/rdsk/c9t12d0 HORCM_DEV #dev_group dev_name port# TargetID LU# MU# dg12 dg12_d0 CL3-E 6 #Phantom device group dg13 dg13 dg13_d0 CL3-E 6 HORCM _INST #dev_group ip_address service dg12 NodeB.dc2.net horcm0 dg13 NodeC.dc3.
NOTE: • In a 3DC CAJ/CAJ Tri-Link configuration using Delta Resync, only mirror unit descriptors “1”, “2” and “3” must be used for creating device groups. • In a 3DC Sync/CAJ configuration, mirror unit descriptor "0" must be used for the device group between DC1 and DC2.
#dev_group dg23 dg13 ip_address service NodeB.dc2.net horcm0 NodeA.dc1.net horcm0 # communicate with DC2 nodes # communicate with DC1 nodes 11. Restart the Raid Manager instance so that the new information in the configuration file is read: # horcmshutdown.sh # horcmstart.sh 12. Repeat steps 1 through 12 on each host that runs this particular application package.
# pairevtwait -g dg13 -t 300 -s pair 3. Create the Delta Resync pair dg23 between DC2 and DC3 from any node in DC2: # paircreate –g dg23 –f async –nocsus –vl –c 15 –jp -js For configuring Delta Resync pair between DC1 and DC3, the steps differ based on 3DC Sync/CAJ or 3DC CAJ/CAJ replication. For 3DC CAJ/CAJ replication, first create Continuous Access Journal pair between DC1 and DC2 and then between DC2 and DC3.
5. # paircreate –g dg13 –f async –nocsus –vl –c 15 –jp -js Once the Delta Resync pair is created, issue the horctakeover command at DC1 so that the data source is at DC1. This suspends dg23 device group pair.
Assigning Remote Command Devices to Journal Volumes This step is required only when you are using Tri-Link with Delta Resync configuration. Once all the three device groups are created, you must assign the Remote Command Devices (RCMD) to journal volumes in all sites. These Remote Command Devices must have already been configured as part of “Infrastructure Requirements” (page 429). In this step, the configured RCMD must be assigned to the journal volumes in all sites.
# vgexport -s -p -m Make sure to copy the mapfiles to all of the three data centers nodes. 6. Import the LVM volume groups on all of the other nodes in DC1, DC2, and DC3 and backup the LVM configuration. NOTE: If you are using the March 2008 version or later of HP-UX 11i v3, you can skip the mkdir and mknod commands; vgcreate (1m) will create the device file for you.
Configuring Primary Packages in the Primary Cluster using the 3DC Modules A primary package must be created using Metrocluster 3DC primary module dts/ primary_xpca3dc. All 3DC parameters must be configured in the primary package configuration file. There is no need to edit the Metrocluster environment file. The Metrocluster environment file is automatically generated on all nodes when this package configuration is applied in the cluster.
DC1_DC3_DEVICE_GROUP dg13 This is a phantom device group when using Multi-Hop Bi-Link configuration. i. j. k. l.
MON_NOTIFICATION_SYSLOG <0 or 1> If you want notification messages to be logged on the system's console, uncomment the MON_NOTIFICATION_CONSOLE variable and set it to 1. If the variable is not set, the default value is 0. MON_NOTIFICATION_CONSOLE <0 or 1> If you want an automatic resynchronization upon link recovery, uncomment the AUTO_RESYNC variable and set it to either 1 or 2. If the variable is not defined or commented, the default value of 0 is used.
If you do not want the package to fail upon device group monitoring failures, you can set the service_restart to unlimited. 4. Validate the package configuration file. # cmcheckconf -P pkgName.config 5. Apply the package configuration file. # cmapplyconf -P pkgName.config After the modular primary package is created in the primary cluster, if the value of any 3DC parameter needs to be changed, then edit this package configuration file and re-apply it.
3DC_TOPOLOGY multi-target-bi-link For Multi-Hop Bi-Link configuration, specify "multi-hop-bi-link": 3DC_TOPOLOGY multi-hop-bi-link c. d. e. f. g.
NOTE: The Device Group Monitor is not supported for the recovery package. Configure Continentalclusters and Continentalclusters Recovery Group Ensure that the Continentalclusters software is installed on all nodes in both primary and recovery cluster. To configure the Continentalclusters, complete the procedure described in “Designing Continentalclusters” (page 36). For each application in the 3DC solution, configure a recovery group using their corresponding packages at the primary and recovery cluster.
Overview of Deploying a Complex Workload in 3DC using SADTA Site Aware Disaster Tolerant Architecture (SADTA) enables deploying complex workloads in a cluster. Complex workloads are applications configured using multi-node and failover packages with dependencies. This complex workload is a multi-instance application that uses active resources across multiple nodes in a cluster. These workloads are configured using multiple, interdependent multi-node packages or failover packages.
Starting the Site Controller Package at a site implies starting the complex workload packages that are associated with it. In normal circumstances, when the primary cluster is available, the Site Controller Package configured in the primary cluster is running and the Site Controller Package configured in the recovery cluster is halted.
A RAC database is configured redundantly by configuring it at all three sites. The RAC database processes, the disk groups, and file systems at each site are configured as a stack of inter-dependent MNP packages. This stack of inter-dependent MNP packages is referred to as a RAC MNP stack. The RAC database processes are packaged using the HP Serviceguard Extension for Oracle RAC Toolkit (delivered as part of the HP Serviceguard Extension for Oracle RAC).
1. Configure the primary cluster. A primary cluster in a 3DC environment is a Metrocluster with two sites configured. 2. Configure the recovery cluster. A recovery cluster in a 3DC environment is a Serviceguard cluster configured with one site. 3. 4. 5. Configure RAID Manager in the primary and recovery cluster. Create Device Group Pairs. Configure Oracle RAC and the Site Controller Package in the primary cluster. a. Set up the Oracle RAC database redundantly in the primary cluster. b.
Configure RAID Manager in the primary and recovery cluster For 3DC operations, a package requires three different device groups. For Tri-Link configuration, a package requires three different device groups: Continuous Access Synchronous or Continuous Access journal device group between DC1 and DC2, Continuous Access journal device group between DC1 or DC2 and DC3, and a Delta Resync device group from either DC1 or DC2 to DC3 depending on where Continuous Access journal pair to DC3 is configured.
All 3DC parameters must be configured in the Site Controller Package configuration file. There is no need to edit the Metrocluster environment file. The Metrocluster environment file is automatically generated on all nodes when this package configuration is applied in the cluster. All parameters that were previously available in the Metrocluster environment file are now configured from the package configuration file. IMPORTANT: Do not delete the Metrocluster environment file that is generated.
◦ HORCMPERM ◦ HORCMINST The Device Group Monitor can also be configured in this Site Controller package. For more information on the 3DC parameters and Device Group Monitor configuration, see “Configuring Primary Packages in the Primary Cluster using the 3DC Modules ” (page 442). 3. Apply the empty Site Controller Package configuration file in the recovery cluster. # cmapplyconf -P /etc/cmcluster/hrdb_sc/hrdb_sc.
Oracle RAC, see section “Installing and Configuring Oracle Real Application Clusters (RAC)” (page 372). Setup a RAC database in the recovery cluster using the replicated disks You must setup a RAC database in the recovery cluster using the replicated disks and it must be configured with the RAC MNP stack in the recovery cluster. Prior to setting up a RAC database in the recovery cluster, you must first prepare the replication environment.
# rcp -p inithrdb1.ora :$PWD 2. Setup the first RAC database instance on the recovery cluster. In this example, run the following commands from the first RAC database instance node in Site 3. # cd /dbs # ln -s /oradata/hrdb/orapwhrdb orapwhrdb1 # chown -h oracle:oinstall orapwhrdb1 # chown oracle:oinstall inithrdb1.ora 3.
# srvctl add database -d hrdb -o # srvctl add instance -d hrdb -i hrdb1 -n # srvctl add instance -d hrdb -i hrdb2 -n After registering the database with the CRS cluster in the recovery cluster, you can run the srvctl status command to view the health of the database. Configure the RAC MNP Stack at the recovery cluster The RAC database must be packaged as a Serviceguard MNP package.
If Delta Resync pair was deleted in the step Suspending the replication to the recovery cluster of this section, re-create the Delta Resync pair and assign the remote command device to journal volumes of Delta Resync pair using XP Remote Web Console. For details on assigning remote command devices to journal volumes, see “Assigning Remote Command Devices to Journal Volumes” (page 440).
◦ DC1_DC2_DEVICE_GROUP_MU ◦ DC2_DC3_DEVICE_GROUP_MU ◦ DC1_DC3_DEVICE_GROUP_MU ◦ FENCE ◦ HORCMPERM ◦ HORCMINST ◦ HORCTIMEOUT ◦ WAITTIME ◦ AUTO_NONCURDATA For more information on the 3DC parameters, see “Configuring Recovery Packages in the Recovery Cluster” (page 445). 3. Apply the empty Site Controller Package configuration file in the recovery cluster. # cmapplyconf -P /etc/cmcluster/hrdb_sc/hrdb_sc.
If you have SLVM configured in your environment, run the following command to view the EMS resource details: # cmviewcl -v –p If you have VERITAS CVM configured in your environment, run the following command to view the EMS resource details: # cmviewcl -v –p 3. Configure the Site Controller Package with complex-workload packages in the recovery cluster.
At this point, you have completed configuring SADTA in the 3DC solution with the RAC database. It is recommended that you verify the setup during the planned downtime for maintenance using the procedure mentioned in “Verifying the Three Data Center Environment” (page 461). Verifying the Three Data Center Environment To verify the setup: • Preview the data replication storage failover by using the cmdrprev command. • Test the application startup at the recovery cluster.
Table 32 Command Exit Value and its Description Value Description —1 The data replication storage failover preview command failed, due to invalid command usage or due to invalid input parameters. 0 The data replication storage failover preview is successful from a specific node. This indicates if data replication storage failover will be successful in the event of the package failing over to this node. 1 The data replication storage failover preview failed.
# vgchange -c n -S n # vgchange -c y -S y 5. 6. 7. 8. Start the recovery package on a DC3 node using the cmrunpkg command. Verify that the application starts up successfully in the recovery cluster. Halt the recovery package using the cmhaltpkg command. Resume replication from the primary cluster to the recovery cluster.
# cmrecovercl This command starts the recovery package. 7. Verify that the recovery package starts up successfully in the recovery cluster. 8. Halt the recovery package using the cmhaltpkg command. 9. Resume replication from the primary cluster to the recovery cluster.
Similarly, consider another example of the application package is running on DC1 and the data is being replicated to DC3 from DC1 (i.e. Multi-target topology). Upon the failure of application on all nodes in DC1, the application package fails over to a node in DC2. As part of package startup, the data replication between DC1 and DC2 is reversed and the data starts replicating from DC2 to DC1. The replication to DC3 will be continued from DC1 (i.e Multi-hop topology).
Figure 85 Package Failover upon entire DC1 failure NOTE: In Figure 85 (page 466), the writes to the disk at DC2 are not accepted till the Delta Resync pair is re-synchronized when using 3DC CAJ/CAJ replication. So, in the event of 3DC DR software failing to resync the Delta Resync pair, the application package fails to come up. In this case, though the DC1-DC2 device group pair is in SSWS state at DC2, the writes to the disk are rejected.
1. 2. 3. 4. 5. 6. 7. Verify that all the nodes in DC1, DC2 and DC3 are up and running. Start DC1-DC2 cluster if it is not running. Start the RAID Manager instance on each node in DC1, DC2 and DC3. Verify that all the Continuous Access links are up. Halt the 3DC package if it is running on DC3. Recover the latest data from DC3 Change the Cluster ID of all LVM and SLVM volume groups managed by the package.
# pairsplit -g dg12 d. If the above command fails, then split the pair to SMPL state. # pairsplit -g dg12 –S 2. Resync data from DC3 to DC2. Log onto any node at DC2. Resync Journal device group to get the latest data from DC3 to DC2. a. Check the pair status of dg23 (DC3 side). # pairvolchk -g dg23 -s -c b. If dg23 (DC3 side) is in PVOL. # pairresync -g dg23 Or If dg23 (DC3 side) is in SVOL-SSWS. # pairresync -g dg23 -c 15 -swapp c. Wait for PAIR state to come up for the Journal device group.
a. Resync the device group to make local volume PVOL. # pairresync -g dg12 -c 15 -swaps b. Wait for the dg12 device group to attain the PAIR state. # pairevtwait -g dg12 -t 300 -s pair 4. Log on and perform the following from any DC1 node. a. SWAP the roles of the dg12 device group between DC1 and DC2. # horctakeover -g dg12 -t 360 5. Log on and perform the following from any DC2 node: a. Wait for PSUS to come up for the Journal Device Group, dg23. # pairevtwait -g dg23 -t 300 -s psus b.
d. Swap the Journal device group role between DC1 and DC3. # horctakeover -g dg13 -t 360 e. Wait for PAIR state to come up. # pairevtwait -g dg13 -t 300 -s pair 3. Resync the device group to get latest data from DC1 to DC2. • a. If in step 1 the dg12 has been brought to the SMPL state Create the DC1 and DC2 sync device group when using 3DC Sync/CAJ replication. # paircreate -g dg12 -f -c 15 –vl Or Create the DC1 and DC2 Journal device group when using 3DC CAJ/CAJ replication.
On DC2 node, # pairvolchk -g dg12 –s # pairvolchk -g dg12 –s -c On DC3 node, # pairvolchk -g dg12 –s #pairvolchk –g dg13 –s –c 2. Get the recent data from DC3 to DC1 by creating a new journal pair between DC3 and DC1 a. Create a DC1-DC3 device group pair with DC3 as PVOL side. Log in to any DC3 node and perform the following: # paircreate –g dg13 –f async –vl –c 15 –jp js b. Wait for the PAIR state to come up for the Journal device group.
3. the data transfer to the SVOL depends on the bandwidth of the Continuous Access links and amount of outstanding data in the PVOL journal volume. Failback - When systems recover from previous failures, a package can be failed back, within Metrocluster data centers, by manually issuing a Serviceguard command. When a package failback is triggered, the software needs to ensure the application data integrity.
NOTE: If the HORCTIMEOUT parameter configured is too short, the time allowed for the take over operation to complete will expire before the primary P9000 or XP array has flushed all of the outstanding data in it’s cache to the secondary P9000 or XP array. This will cause the takeover action to fail, and the package will fail to start. When this happens, an error message will be logged in the control script log file with instructions on what to do next.
Takeover Timeout (for third data center) When a package is being failed over to the third data center (SVOL of the Continuous Access-Journal device group), the Metrocluster toolkit script issues takeover command on the SVOL. If the journal group pair is flushing the journal data from its PVOL to SVOL and takeover timeout occurs, the following situations would happen: 1. 2.
A Environment File Variables for Serviceguard Integration with Continuous Access P9000 or XP This appendix lists all Environment File variables that have been modified or added for disaster tolerant Serviceguard solutions that employ Continuous Access P9000 or XP.
0 – (DEFAULT) Do NOT startup the application on non-current data. If Metrocluster/Continuous Access cannot determine the data is current, it will not allow the package to start up. (Note: for fence level DATA and NEVER, the data is current when both PVOL and SVOL are in PAIR state.) 1 – Startup the application even when the data may not be current.
PSUS(SSWS). During this transition, horctakeover will attempt to flush any data in the side file on the MCU to the RCU. Data that does not make it to the RCU will be stored on the bit map of the MCU. When failing back to the primary site any data that was in the MCU side file that is now stored on the bit map will be lost during resynchronization. In synchronous mode with fence level NEVER, when the Continuous Access link fails, the application continues running and writing data to the PVOL.
the package to start by creating the FORCEFLAG file. 1—Startup the package after resynchronize the data from the SVOL side to the PVOL side. The risk of using this option is that the SVOL data may not be a preferable one. If the package has been configured for a three data center (3DC) environment, this parameter is applicable only when the package is attempting to start up in either the primary (DC1) or secondary (DC2) data center.
and the data written to the PVOL side after the hardware failure may be loss. This parameter is not required to be set if a package is configured for three data centers environment because three data center does not support Asynchronous mode of data replication. Leave this parameter with its default value in all data centers. AUTO_SVOLPSUS (Default = 0) This parameter applies when the PVOL and SVOL both have the state of suspended (PSUS).
2. topology with two Continuous Access links. The sync Continuous Access link, is between DC1 and DC2 and the other link, the journaling Continuous Access link, is between DC2 and DC3. There is no physical Continuous Access link present between DC1 and DC3 but a phantom device group must be present between DC1 and DC3. multi-target-Bi-Link: When the package is configured for 1:2 or multi target topology with two Continuous Access links.
for two data centers (2DC). If a monitor variable is not defined (commented out) the default value is used. *************************************************** FENCE Fence level. Possible values are NEVER, DATA, and ASYNC. Use ASYNC for improved performance over long distances.
value. The default Continuous Access link timeout value is 5 minutes (300 seconds). A suggested value for HORCTIMEOUT is 360 seconds. During package startup, the default startup timeout value of the package is set to NO_TIMEOUT in the package ASCII file.
Table 34 AUTO_NONCURDATA Local State Remote State Fence Level PVOL_PSUS EX_ENORMT PVOL_PSUS EX_CMDIOE NEVER/DATA/ASYNC Do not start with exit 1 PVOL_PFUS EX_ENORMT ASYNC PVOL_PFUS EX_CMDIOE SVOL_PAIRSVOL_PAIR PVOL_PSUE EX_ENORMT EX_CMDIOE SVOL_ PAIR AUTO_NONCURDATA AUTO_NONCURDATA =1 or =0 (Default) FORCEFLAG=yes NEVER/DATA/ASYNC Do not start with exit 1 SVOL_PFULSVOL_PFUL PVOL_PSUE EX_ENORMT SVOL_PFUL EX_CMDIOE ASYNC SVOL-PAIR;MINAP=0 (Continuous Access-Journal) PVOL-PAIR; MINAP>0, MI
Table 38 AUTO_SVOLPSUE Local State Remote State Fence Level AUTO_SVOLPSUE =0 (Default) AUTO_SVOLPSUE =1 or FORCEFLAG=yes SVOL_PSUE PVOL_PSUS NEVER/DATA/ASYNC SVOL_PSUE EX_ENORMT Do not start with exit 2. SVOL_PSUE EX_CMDIOE Perform a SVOL to PSUS(SSWS). After the takeover succeeds, package starts with a warning message about non-current data in the package’s control log file.
When the parameter is set to 1, the monitor will send notifications to the syslog file. If the parameter is not defined (commented out), the default value is 0. MON_NOTIFICATION_CONSOLE (Default = 0) This parameter defines whether the monitor will send console notifications. When the parameter is set to 0, the monitor will NOT send console notifications. When the parameter is set to 1, the monitor will send console notifications. If the parameter is not defined (commented out), the default value is 0.
module_name sg/failover module_version 1 module_name dts/dts module_version 1 module_name dts/xpca module_version 1 package_type failover node_name * auto_run yes node_fail_fast_enabled no run_script_timeout no_timeout halt_script_timeout no_timeout successor_halt_timeout no_timeout script_log_file /etc/cmcluster/logg operation_sequence /etc/cmcluster/scripts/dts/mc.sh operation_sequence /etc/cmcluster/scripts/dts/xpca.sh operation_sequence $SGCONF/scripts/sg/volume_group.
B Environment File Variables for Metrocluster Continuous Access EVA This appendix lists all Environment File variables that have been modified or added for Metrocluster with Continuous Access EVA. It is recommended that you use the default settings for most of these variables, so exercise caution when modifying them: CLUSTER_TYPE This parameter defines the clustering environment in which the script is used.
The name of the DR group used by this package. The DR group name is defined when the DR group is created. DC1_STORAGE_WORLD_WIDE_NAME The world wide name of the EVA storage system that resides in Data Center 1. This storage system name is defined when the storage is initialized. DC1_SMIS_LIST A list of the management servers that reside in Data Center 1. Multiple names can be defined by using commas as separators.
module_version 1 module_name sg/failover module_version 1 module_name dts/dts module_version 1 module_name dts/caeva module_version 1 package_type failover node_name * auto_run yes node_fail_fast_enabled no run_script_timeout no_timeout halt_script_timeout no_timeout successor_halt_timeout no_timeout script_log_file /etc/cmcluster/logg operation_sequence /etc/cmcluster/scripts/dts/mc.sh operation_sequence /etc/cmcluster/scripts/dts/caeva.
Combined with the DT_APPLICATION_STARTUP_POLICY and the presence or absence of the FORCEFLAG, Metrocluster with Continuous Access EVA handles failover scenarios in different ways when the replication mode is set to Asynchronous or Synchronous. Table 40 describes how failover scenarios are addressed in different replication modes and when the FORCEFLAG is present.
C Environment File Variables for Metrocluster with EMC SRDF This appendix lists all Serviceguard control script variables that have been modified or added for Metrocluster with EMC SRDF.
data on the R2 side may be non-current and thus a risk that data loss will occur when starting the package up on the R2 side. To have automatic package startup under these conditions, set AUTOR2RWNL=0 AUTOR2XXNL Default: 0 A value of 0 for this variable indicates that when the package is being started on an R2 host and at least one (but not all) SRDF links are down, the package will be automatically started. This will normally be the case when the ‘Partitioned+Suspended’ RDF Pairstate exists.
PATH PKGDIR RDF_MODE Has been modified to include the path name of the Symmetrix SymCLI commands. This should be set to the default location of /usr/symcli/ bin unless you have changed the location. If the package is a legacy package, then this variable contains the full path name of the package directory. If the package is a modular package, then this variable contains the full path name of the directory where the Metrocluster SRDF environment file is located.
module_name dts/mcsrdf module_version 1 module_name dts/mc module_version 1 module_name sg/volume_group module_version 1 module_name sg/priority module_version 1 module_name sg/failover module_version 1 module_name dts/dts module_version 1 module_name dts/srdf module_version 1 package_type failover node_name * auto_run yes node_fail_fast_enabled no run_script_timeout no_timeout halt_script_timeout no_timeout successor_halt_timeout no_timeout script_log_file /etc/cm
kill_processes_accessing_raw_devices no priority no_priority Modular Package Sample Configuration file 495
D Configuration File Parameters for Continentalclusters This appendix lists all Continentalclusters configuration file variables. See Chapter 2: “Designing Continentalclusters”, for suggestions on coding these parameters. CLUSTER_ALARM [Minutes] This is a time interval, in minutes and/or seconds, after which the notifications defined in the associated MINUTES [Seconds] SECONDS NOTIFICATION parameters are sent and failover to the Recovery Cluster using the cmrecovercl command is enabled.
The parameter consists of a pair of names: the name of the cluster that receives the data to be replicated (usually the Recovery Cluster) as defined in the Serviceguard cluster configuration ASCII file, followed by a slash (“/”), followed by the name of the data replication receiver package as defined in the Serviceguard package configuration ASCII file. Some replication software may only have a receiver package as separate package because the sender package is built into the application.
that the text of the message is not placed in the syslog file, only a notice from the monitor. • TCP Nodename:Portnumber - send the specified message to a TCP port on the specified node. • TEXTLOG Pathname - append the specified message to a specified text log file. • UDP Nodename:Portnumber - send the specified message to a UDP port on the specified node. Any number of notifications may be associated with a given alert or alarm.
E Continentalclusters Command and Daemon Reference This appendix lists all commands and daemons used with Continentalclusters. Manual pages are also available online. cmapplyconcl [-v] [-C] This command verifies the Continentalclusters configuration as specified in filename, creates or updates the binary, filename and distributes it to all nodes in the Continentalclusters.
cmdeleteconcl [-f] This command is used to delete the continentalclusters configuration from the entire Continentalclusters. This command will not remove the file system configured for recovery group maintenance mode feature. Options are: -f Delete the configuration files on all reachable nodes without further prompting. If this option is not used and if some nodes are unreachable, you will be prompted to indicate whether to proceed with deleting the configuration on the reachable nodes.
is specified, output is written to stdout. cmrecovercl [-f] This command performs the recovery actions necessary to start the recovery groups on current cluster. Care should be taken before issuing this command. It is important to contact the primary cluster site to determine if recovery is necessary prior to running this command. This command will perform recovery actions only for recovery groups that are out of the maintenance mode (i.e. enabled).
cmrecovercl [-r]-g cmviewconcl [-v] This command starts the rehearsal for the specified recovery group. This command should be run only on the recovery cluster. This command will fail if the specified recovery group is not in the maintenance mode. This command allows you to view the status and much of the configuration of Continentalclusters.
F Metrocluster Command Reference for Preview Utility This appendix describes the Data Replication Storage Failover Preview utility. This appendix discusses the following topics: • Overview of Data Replication Storage Failover Preview • Command Reference • Sample Output of the cmdrprev Command Overview of Data Replication Storage Failover Preview In an actual failure, packages are failed over to the standby site.
Table 41 Command Exit Value and its Description (continued) Value Description This indicates that in the event of an actual recovery process, the data replication storage failover will not succeed on that node. Failover may be successful if attempted at a later time or attempted on a different node in the cluster. Command Reference This sections lists the Metrocluster commands that can be used for the preview utility.
CLUSTER_TYPE="metrocluster" RDF_MODE="sync" 3. Run the following command on dtsia14: $> /usr/sbin/cmdrprev -p pkga Following is the output that is displayed: May 10 04:36:40 - Node dtsia14: Entering Data Replication Enabler May 10 04:36:41 - Node dtsia14: Data Replication State: pkga Role: R2 LocalRdfState: WD RemoteRdfState: RW RDF Mode: SYNC RepState: SYNCHRONIZED May 10 04:36:43 - Node dtsia14: Failover of the device group successfully. The package is allowed to start up.
G Site Aware Disaster Tolerant Architecture Configuration Work Sheet This appendix includes the worksheets that you need to use while configuring Site Aware Disaster Tolerant Architecture in your environment.
Table 43 Replication Configuration (continued) Item Data Node Names Name of Nodes at each site Command Device on Nodes Raw device file path at each node Device group device name Site 1 LUN Site 2 LUN Specify luns in CU:LDEV format Specify luns in CU:LDEV format Dev_name parameter 1) 2) 3) 4) 5) 6) 7) 8) 9) 10) CRS Sub-cluster Configuration – using CFS Table 44 Configuring a CRS Sub-cluster using CFS Item Site Site CRS Sub Cluster Name Name of the CRS cluster CRS Home Local FS Path for CRS HOME CR
Table 44 Configuring a CRS Sub-cluster using CFS (continued) Item Site Site Path to the OCR disk or file CRS MP MNP package Path to the OCR disk or file CRS MNP package Path to the OCR disk or file CRS Member Nodes Node Names Private IP IP addresses for RAC Interconnect Private IP names IP address names for RAC Interconnect Virtual IP IP addresses for RAC VIP Virtual IP names IP addresses names for RAC VIP RAC Database Configuration Table 45 RAC Database Configuration Property Database Name Name of the
Table 45 RAC Database Configuration (continued) Property Value Entry Site Site RAC Home Local file system directory to install Oracle RAC RAC MNP Package name for RAC database RAC Data file DG MNP CFS DG MNP package name for RAC data files file system RAC Data file MP MNP CFS MP MNP package name for RAC data files file system RAC Flash Area DG MNP CFS DG MNP package name for RAC flash file system RAC Flash Area MP MNP CFS MP MNP package name for RAC flash file system Node Names Database Instance Names
Table 46 Site Controller Package Configuration (continued) 510 3) 3) 4) 4) Site Aware Disaster Tolerant Architecture Configuration Work Sheet
H Rolling Upgrade for Metrocluster This appendix elaborates the procedures for completing a rolling upgrade of Metrocluster versions.
The subsequent sections describe the procedures for completing a rolling upgrade for Metrocluster configurations with SADTA. These sections describe upgrading HP Serviceguard, HP-UX, and Metrocluster Replication software in Metrocluster configurations. Rolling Upgrade in Metrocluster Configurations This section describes the procedures for completing a rolling upgrade in Metrocluster with and without SADTA configured.
7. Halt the node that is selected for upgrade. # cmhaltnode -f Instances of all MNP packages running on this node are halted and the failover packages move to the adoptive node. 8. Edit the /etc/rc.conf.d/cmcluster file to include the following line: AUTOSTART_CMCLD = 0 9. Upgrade the node to the new HP-UX release, Serviceguard, SGeRAC or Metrocluster version. 10. Edit the /etc/rc.conf.d/cmcluster file to include the following line: AUTOSTART_CMCLD = 1 11.
Following are the topics discussed in this section: • “Upgrading the P9000 or XP Raid Manager Software for Metrocluster with Continuous Access for P9000 and XP” (page 514) • “Upgrading the Solutions Enabler Software for Metrocluster with EMC SRDF” (page 514) • “Upgrading the HP-UX WBEM Services for Metrocluster with Continuous Access EVA” (page 514) Upgrading the P9000 or XP Raid Manager Software for Metrocluster with Continuous Access for P9000 and XP To upgrade the P9000 or XP RAID Manager software
Glossary A application restart Starting an application, usually on another node, after a failure. Application can be restarted manually, which may be necessary if data must be restarted before the application can run (example: Business Recovery Services work like this.) Applications can by restarted by an operator using a script, which can reduce human error. Or applications can be started on the local or remote site automatically after detecting the failure of the primary site.
of nodes, networks, application services, EMS resources, and makes failover decisions based on where the application is able to run successfully. cluster alarm Time at which a message is sent indicating that the cluster is probably in need of recovery. The cmrecoverclcommand is enabled at this time. cluster alert Time at which a message is sent indicating a problem with the cluster.
is transferred to the Recovery Cluster and made available for use on the Recovery Cluster in the event of a recovery. database replication A software-based logical data replication scheme that is offered by most database vendors. disaster An event causing the failure of multiple components or entire data centers that render unavailable all services at a single location; these include natural disasters such as earthquake, fire, or flood, acts of terrorism or sabotage, large-scale power outages.
high availability A combination of technology, processes, and support partnerships that provide greater application or system availability. J, K, L local cluster A cluster located in a single data center. This type of cluster is not disaster tolerant. local failover Failover on the same node; this most often applied to hardware failover, for example local LAN failover is switching to the secondary LAN card on the same node after the primary LAN card has failed.
O off-line data replication. Data replication by storing data off-line, usually a backup tape or disk stored in a safe location; this method is best for applications that can accept a 24-hour recovery time. on-line data replication Data replication by copying to another location that is immediately accessible.
regional disaster A disaster, such as an earthquake or hurricane, that affects a large region. Local, campus, and proximate metropolitan clusters are less likely to protect from regional disasters. rehearsal package The recovery cluster package used to validate the recovery environment and procedure as part of a rehearsal operation. remote failover Failover to a node at another data center or remote location.
transparent IP failover Moving the IP address from one network interface card (NIC), in the same node or another node, to another NIC that is attached to the same IP subnet so that users or applications may always specify the same IP name/address whenever they connect, even after a failure. U-Z volume group In LVM, a set of physical volumes such that logical volumes can be defined within the volume group for user access.
Index A adding a node to Continentalclusters configuration, 106 arbitrator nodes, 29 C cluster calculating quorum, 29 continental, 48 recovery, 56 clusterware, 121 cmdeleteconcl command, 112 cmrecovercl, 95 command line cmrecovercl, 95 symdg, 262 command line interface, EMC Symmetrix, 257 configuring a three-data-center architecture, 25 additional nodes in Continentalclusters, 106 arbitrator nodes, 29 configuring, 54 Continentalcluster Recovery cluster hardware, 56 Continentalclusters recovery cluster, 56
I importing volume groups, 228, 264 installing Continentalclusters product, 58 L log files reviewing in Continentalclusters, 111 logical device names, EMC Symmetrix, 263 M Maintenance Mode cmrecovercl -d -g, 42 mapping EMC Symmetrix and HP-UX devices, 261, 263 mapping Symmetrix and HP-UX devices, 259 MC/ServiceGuard, package configuration, 180, 202, 204, 250 MetroCluster, 257 configuring, 272 Metrocluster, 255 supported configurations, 23 supported distance, 23 with HP 3PAR Remote Copy, 310 Metrocluster/C
Site Controller Package, 341 site coordination for Continentalclusters notification, 94 special device files, mapping to EMC Symmetrix, 261 splitting SRDF links during volume group configuration, 264 SRDF links restoring, 275 splitting, 264 starting for Continentalclusters, 89 starting the monitor packaging in Continentalclusters, 89 status checking status of Continentalclusters objects, 108 switching to a recovery cluster using Continentalclusters, 94 SymCLI database, 257 symdg, 262 symld command, 263 T t
