Building Disaster Recovery Serviceguard Solutions Using Continentalclusters for Linux B.01.00.
Legal Notices © Copyright 2012 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor’s standard commercial license. The information contained herein is subject to change without notice.
Contents 1 Introduction...............................................................................................6 2 Building the Continentalclusters configuration.................................................8 Creating the Serviceguard clusters at both the sites ......................................................................8 Setting up security....................................................................................................................
Overview of maintenance mode feature...........................................................................31 Setting up the file system for Continentalclusters state directory............................................31 Configuring the monitor package to mount the file system from the shared disk......................32 Configuring Continentalclusters rehearsal packages................................................................33 Modifying Continentalclusters configuration.......................
B Configuration file parameters for Continentalclusters......................................56 C Continentalclusters Command and Daemon Reference..................................59 D Package attributes....................................................................................62 Package Attributes for Continentalcluster with Continuous Access P9000 for Linux...........................62 Package Attributes for Continentalcluster with Continuous Access EVA............................................
1 Introduction Continentalclusters for Linux provides disaster recovery between multiple Serviceguard clusters. A single cluster can act as the recovery for a set of primary clusters. It is also possible to have two clusters act as recovery for each other. This allows increased utilization of hardware resources. Continentalclusters for Linux eliminates the cluster itself as a single point of failure.
Figure 1 Sample Continentalclusters Configuration ccmonpkg PRI_SCM_DB_PKG Recovery Group Packages REC_SCM_DB_PKG REC_CRM_DB_PKG Recovery Group Packages PRI_CRM_DB_PKG cconfpkg Continentalclusters Configuration Package cconfpkg Site A Node 1 Monitor Package Site A Node 2 ccmonpkg Site B Node 1 Site B Node 2 FC Switch FC Switch Monitor Package WAN WAN Converters Site A Disk Array Site A Cluster (Primary) Data Replication Links WAN Converters Site B Disk Array Site B Cluster (Recovery
2 Building the Continentalclusters configuration To build a Continentalclusters configuration: The firewall rules must be configured as mentioned in Configuring firewall rules for HP Serviceguard on SLES and Red Hat whitepaper available at http://www.hp.com/go/linux-serviceguard-docs. In addition to the ports mentioned in the whitepaper, SSH port must be open in all the nodes of Continentalclusters. 1. Create a Serviceguard cluster at both the data center sites. 2.
a. b. Log in as root user. Set the password for conclusr on the node. # passwd conclusr 2. Set up SSH equivalence between the nodes in the Continentalclusters. a. Log in to any node in the Continentalclusters as conclusr. b. Create a text file and add the Fully Qualified Domain Names (FQDN) of all the nodes in all the clusters to be configured in the Continentalclusters. For example, consider a Continentalclusters with two clusters, Cluster A and Cluster B, each having two nodes, Node 1 and Node 2.
Creating data replication between the clusters Data replication between the Serviceguard clusters in a Continentalclusters recovery pair extends the scope of high availability to the level of the Continentalclusters. Select a technology for data replication between the two clusters.
Using any other array based physical replication technology If you select a data replication technology is chosen that is not mentioned in the previous section, and if the integration is performed independently, then note the following: • Continentalclusters for Linux product is only responsible for Continentalclusters configuration and management commands, the monitoring of remote cluster status, and the notification of remote cluster events.
Installing and configuring a redundant copy of the application in the recovery site Install the application at the secondary site and configure it to use the same replicated disks as in the previous step. Then, configure the application and its resources as a Serviceguard package. Configuring the Continentalclusters primary and recovery packages The packages can be created using any modules supported by HP Serviceguard for Linux .
dts/xpca/dts_pkg_dir /$SGCONF/ • DEVICE_GROUP Specify the XPCA device group name managed by this package, as defined in the RAID Manager configuration file. • HORCMINST Specify the name of the RAID manager instance that manages the XPCA device group used by this package. • FENCE Specify the fence level configured for the XPCA device group that is managed by this package. • AUTO_RUN Set the value of this parameter to no.
# cmmakepkg –m dts/cccaeva –m sg/filesystem -m sg/package_ip -m tkit/apache/apache temp.config 2. Edit the following attributes in the temp.config file: • dts/caeva/dts_pkg_dir This is the package directory for the modular package. This value must be unique for all the packages. • AUTO_RUN Set the value of this parameter to no. • DT_APPLICATION_STARTUP_POLICY This parameter defines the preferred policy by allowing the application to start with respect to the state of the data in the local volumes.
Configuring the primary and recovery packages as modular packages when using 3PAR Remote Copy When using HP 3PAR Remote Copy in Continentalclusters, the primary and recovery packages must be created using the dts/cc3parrc module. To use this module, Metrocluster with 3PAR Remote Copy must be installed on all the nodes in the Continentalclusters. To configure the primary and recovery packages as modular packages using 3PAR Remote Copy with Continentalclusters for Linux: 1.
• DC2_RC_VOLUME_GROUP The Remote Copy volume group name configured on the HP 3PAR storage system, which is located in Data Center 2, containing the disks used by the application. • DC1_RC_TARGET_FOR_DC2 The target name associated with the Remote Copy volume group on data center 1 for the HP 3PAR storage system in Data Center 2. • DC2_RC_TARGET_FOR_DC1 The target name associated with the Remote Copy volume group on data center 2 for the HP 3PAR storage system in Data Center 1.
If the rehearsal feature is configured, then provide the following information about the filesystem and volume group used as a state directory: • Volume group name • mount point • logical volume name • Filesystem type • mount and unmount options • fsck options For Example: vg ccvg fs_name /dev/ccvg/lvol1 fs_directory /opt/cmconcl/statedir fs_mount_opt "-o rw" fs_umount_opt "" fs_fsck_opt "" fs_type "ext2" 4. Specify a name for the ccmonpkg log file using the script_log_file parameter.
Table 1 Cluster Information Parameters (continued) Parameter Value Mandatory or Optional CONTINENTAL_CLUSTER_STATE_DIR Full path to the directory on the shared Optional: Used only when if the volume. maintenance mode feature is required. For Example: CONTINENTAL_CLUSTER_STATE_DIR /opt/cmconcl/statedir CLUSTER_NAME The name of the Serviceguard cluster Mandatory. that is a part of the Continentalclusters.
Table 2 Recovery Groups Parameters (continued) Parameter Value Mandatory or Optional REHEARSAL_PACKAGE The name of the package that acts as Optional: This is required only when the rehearsal package along with the the DR Rehearsal feature is used. name of the recovery cluster.
Table 3 Monitoring definitions Parameters (continued) Parameter Value Mandatory or Optional NOTIFICATION OPC** The OPC level followed by the Optional. notification message. The value of might be 8 (normal), 16 (warning), 32 (minor), 64 (major), 128 (critical). The notification message is provided in the next line. NOTIFICATION SNMP** The SNMP level followed by the notification message. The value of might be 1 (normal), 2 (warning), 3 (minor), 4 (major), 5 (critical).
1. Halt all the monitor packages if running. # cmhaltpkg ccmonpkg 2. Verify the Continentalclusters configuration. # cmcheckconcl -v -C cmconcl.config This command verifies whether all the parameters are within range, all the fields are filled, and the entries (such as NODE_NAME) are valid. 3. Distribute the Continentalclusters configuration information to all the nodes in the Continentalclusters. # cmapplyconcl -v -C cmconcl.
6. 7. 8. Shut down the application on the recovery cluster using the cmhaltpkg command. If physical data replication is being used, do not resync from the recovery cluster to the primary cluster. Instead, run a command that will overwrite changes on the recovery disk array that might have been made unintentionally. Start the package up on the primary cluster and allow connection to the application. Testing Continentalclusters operations To exercise typical Continentalclusters behaviors: 1.
6. 7. Halt the packages on one cluster, but do not halt the cluster. Notifications that the packages on that cluster have failed are not generated. Test the mechanisms available to detect the manual shutdown or failure of primary packages.
3 Performing a recovery operation in Continentalclusters environment Performing recovery in case of disaster You can also initiate recovery forcefully even if the alarm event has not triggered but the alert event has happened. An administrator can initiate the recovery using cmrecoverclcommand. However, an administrator must confirm from the primary cluster administrator for the need of the recovery.
After the notification is received, and it is determined that the by using the recovery checklist that recovery is required, ensure the following: • The data used by the application is in usable state. Usable state means the data is consistent and recoverable, even though it might not be current. • The secondary devices are in read-write mode. If you are using database or software data replication ensure the data copy at the recovery site is in read-write mode as well.
Viewing the Continentalclusters status The cmviewconcl command is used to view the continentalcluster status.
4 Restoring disaster recovery cluster after a disaster After a failover to a cluster occurs, restoring disaster recovery is a manual processs, the most significant of which are: • Restoring the failed cluster. Depending on the nature of the disaster it might be necessary to either create a new cluster or to repair the failed cluster. Before starting up the new or the failed cluster, ensure the auto_run flag for all of the Continentalclusters application packages is disabled.
3. of recovery groups. It might also be necessary to re-create data sender and data receiver packages. Check and apply the Continentalclusters configuration. # cmcheckconcl -v -C cmconcl.config # cmapplyconcl -v -C cmconcl.config 4. Restart the monitor packages on every cluster. # cmmodpkg -e ccmonpkg 5. View the status of the Continentalclusters. # cmviewconcl Before applying the edited configuration, the data storage associated with every cluster must be prepared to match the new role.
“CurrentContinentalclustersConfigFileName” if -F is not specified. If you want to edit the new configuration file, do not use the -a option. If the option -a is specified, the new configuration is applied automatically. 3. If option -a is specified with cmswitchconcl in step 2, skip this step. Otherwise manually apply the new Continentalclusters configuration.
# cmapplyconf -p ccmonpkg.config 6. Restart the monitor package on the recovery cluster. # cmrunpkg ccmonpkg 7. View the status of the Continentalclusters. # cmviewconcl Creating a new recovery cluster After creating a new cluster to replace the failed primary cluster, if the downtime involved in moving back the applications is a concern, then make the newly created cluster as the recovery cluster.
5 Disaster recovery rehearsal in Continentalclusters Overview of Disaster Recovery rehearsal The disaster recovery setup must be validated to ensure that a recovery can be performed smoothly when disaster strikes. Since disasters are once in a lifetime events, it is likely that a disaster recovery is never performed for long time. In this time, a lot of configuration drift and other changes will appear in either at the production data center or at the recovery data center.
# vgcreate /dev/vgcc -f /dev/sda1 2. Create a logical volume in the volume group and install ’vxfs’ file system in the logical volume: # lvcreate –L mke2fs -j For Example: # lvcreate –L 1000 /dev/vgcc; mke2fs -j /dev/vgcc/rlvol1 3. On every node of the recovery cluster, create the Continentalclusters shared directory /opt/cmconcl/statedir as follows: # mkdir For Example: # mkdir /opt/cmconcl/statedir 4.
7. Halt the monitor package ccmonpkg and apply the edited configuration file. For Example: # cmhaltpkg ccmonpkg # cmapplyconf –P cc_new.config 8. Start the monitor package ccmonpkg after applying the configuration. For Example: # cmrunpkg ccmonpkg Configuring Continentalclusters rehearsal packages The rehearsal packages use all the modules that are used to create the recovery package.
Recovery group inv_rac10g_recgp Primary package Atlanta/inv_rac10g_primpkg Recovery package Houston/inv_rac10g_recpkg Rehearsal package Houston/inv_rac10g_rhpkg 3. Halt the monitor package. # cmhaltpkg ccmonpkg 4. Verify the Continentalclusters configuration ASCII file. # cmcheckconcl -v -C cmconcl.config 5. Apply the Continentalclusters configuration file. # cmapplyconcl -v -C cmconcl.config 6. Start the monitor package.
Verify data replication environment You can use the cmdrprev command to preview the preparation of data replication environment for an actual recovery can be previewed. The command identifies errors in data replication environment which will potentially fail an actual recovery. Run the following command on every node of the recovery cluster and verify that the command returns a value 0.
Restore replication environment for recovery First, synchronize the secondary mirror copy with the primary mirror copy and then synchronize the BC with the secondary mirror copy. # pairresync –g # export HORCC_MRCF=1 # pairresync –g # unset HORCC_MRCF nl Move the recovery group out of maintenance mode After the rehearsal operations are completed, the recovery groups must be taken out of maintenance mode.
1. 2. 3. The replication, preparation, and restoration for rehearsal and restoration for recovery is manual. The operator must prepare and restore the replication environment for every recovery group. The cmdrprev preview command currently supports only verbose output. Since the replication between the primary and recovery cluster is suspended during rehearsal, the production changes to the primary mirror copy must not be replicated to the recovery cluster.
6 Administering Continentalclusters Checking the status of clusters, nodes, and packages To verify the status of the Continentalclusters and associated packages, use the cmviewconcl command, which lists the status of the clusters, associated package status, and status of the configured events. This command also displays, if configured, the mode of the recovery group.
PRIMARY CLUSTER PTST_sanfran STATUS Unmonitored EVENT LEVEL POLLING INTERVAL unmonitored 1 min CONFIGURED EVENT STATUS DURATION alert unreachable 1 min alert unreachable 2 min alarm unreachable 3 min alert down 1 min alert down 2 min alarm down 3 min alert error 0 sec alert up 1 min RECOVERY CLUSTER PRIMARY CLUSTER PTST_dts1 PTST_sanfran STATUS Unmonitored CONFIGURED EVENT alert alert alarm alert alert alarm alert alert STATUS DURATION unreachable 1 min unreachable 2 min unreachable 3 min down 1 min d
INTERFACE PRIMARY PRIMARY PACKAGE ccmonpkg STATUS up up STATUS up PATH STATE NAME 4.1 56.1 lan0 lan1 PKG_SWITCH NODE running enabled Script_Parameters: ITEM NAME STATUS Service ccmonpkg.
Continentalclusters notifications are typically sent as: • Email messages The sendmail command is used to send an email to the address specified in the Continentalclusters configuration file. The sendmail should be configured appropriately to send messages to the specified destination email address. • Syslog For syslog notification, the messages are written to /var/log/messages.
As the processing of each recovery group occurs (the message about the data receiver package appears only using logical data replication with data sender and receiver packages):Processing the recovery group nfsgroup on recovery cluster eastcoast.Disabling switching for data receiver package nfsreceiverpkg on recovery cluster eastcost.Halting data receiver package nfsreceiverpkg on recovery cluster east coast.Starting recovery package nfsbackuppkg on recovery cluster eastcoast.
4. If a new node is added, then setup SSH equivalence as described in the “Sample Continentalclusters Configuration” (page 8). If a node is removed, delete Continentalclusters user along with its HOME directory to remove all SSH credentials. 5. 6. 7. Verify and apply the configuration using the cmcheckconcl and cmapplyconcl commands. Restart the monitor packages. View the status of Continentalclusters.
# cmrunpkg CAUTION: package. Never enable package switching on both the primary package and the recovery Modifying Continentalclusters configuration 1. Halt the monitor package. # cmhaltpkg ccmonpkg 2. Apply the new Continentalclusters configuration. # cmapplyconcl -C 3. Restart the monitor package.
Changing monitoring definitions You can change the monitoring definitions in the configuration without bringing down either cluster. This includes adding, removing, or changing the cluster events, changing the timings, and adding, removing, or changing the notification messages. To change the monitoring definitions: 1. Edit the Continentalclusters configuration file to incorporate the new or changed monitoring definitions. 2. Halt the monitor packages on both clusters. 3.
Table 4 Serviceguard and Continentalclusters Commands (continued) Command How the command How the command works in Continentalclusters works in Serviceguard sender package if the recovery package in the same recovery group is running or enabled. cmhaltcl -f This command halts daemons on all currently running systems. Will not re-enable switching on a recovery package if any of the primary, data receiver, or data sender package in the same recovery group is running or enabled.
Maintaining the data replication environment Continentalclusters supports the pre-integrated physical replication solutions using Continuous Access P9000 and XP, Continuous Access EVA, and 3PAR Remote Copy. • See, “Maintaining Continuous Access P9000 and XP Data Replication Environment” (page 47) for administering Continentalclusters when the Continentalclusters solution is built on Continuous Access P9000 and XP for the physical data replication.
Ensure you periodically review the following files for messages, warnings, and recommended actions. HP recommends to review these files after system, data center, and application failures. • /var/adm/syslog/syslog.log • /etc/cmcluster//.log • /etc/cmcluster/.
Maintaining Metrocluster with Continuous Access EVA P6000 for Linux data replication environment While the package is running, the package might halt because of unexpected conditions in the Continuous Access EVA volumes caused by a manual storage failover on Continuous Access EVA outside of Metrocluster Continuous Access EVA software. HP recommends that manual storage failover must not be performed while the package is running.
Remote Copy Link Failure and Resume Modes When the link is failed, snapshots are created for all the primary volumes, but not for the secondary volumes while replication is stopped. When replication is restarted for the volume, all differences between the base volume and the snapshot taken when the replication was stopped are sent over in order to resynchronize the secondary volume with the primary volume.
7 Troubleshooting Continentalclusters Reviewing Messages and Log Files Continentalclusters logs messages into the standard output. For example, Continentalclusters commands, such as cmqueryconcl, cmcheckconcl, cmapplyconcl, and cmrecovercl output. Multiple log files are also used to log various operations. All log messages are stored in the /var/adm/cmconcl/logs directory with appropriate names. The cmviewconcl command logs messages in the /var/adm/cmconcl/logs/cmviewconcl.log file.
Table 5 Troubleshooting Continentalclusters Error Messages (continued) Command/Component Symptoms Cause (FQDN) cannot be resolved amongst the nodes in the Continentalclusters. • The SSH trust is not established. Continentalclusters configuration file. • Ensure that the FQDNs are resolvable amongs the nodes in the Continentalclusters. • Ensure that the SSH trust is set up properly. The cause must be one • Set the AUTO_RUN flag in the of the following: package configuration file to NO.
A Continentalclusters Worksheets Planning is an essential effort in creating a robust Continentalclusters environment. HP recommends that you to record the details of your configuration on planning worksheets. These worksheets can be filled in partially before configuration begins, and then completed as you build the Continentalclusters. All the participating Serviceguard clusters in one Continentalclusters must have a copy of these worksheets to help coordinate initial configuration and subsequent changes.
Rehearsal Cluster/Package Name:______________________________ Data Receiver Cluster/Package Name:__________________________ Recovery Group Data: Recovery Group Name: ________________________________________ Primary Cluster/Package Name:________________________________ Data Sender Cluster/Package Name:____________________________ Recovery Cluster/Package Name:_______________________________ Rehearsal Cluster/Package Name:______________________________ Data Receiver Cluster/Package Name:______________________
Notify the monitored site of successful recovery using one of the following: Authorized person contacted: Director 1 Admin 1 Confirmation received Human-to-human voice authorization Voice mail Recovery Checklist 55
B Configuration file parameters for Continentalclusters This appendix lists all the variables in the Continentalclusters configuration file. CLUSTER_ALARM [Minutes] This is a time interval, in minutes and seconds or both, after which the notifications defined in the associated MINUTES [Seconds] SECONDS NOTIFICATION parameters are sent and failover to the Recovery Cluster using the cmrecovercl command is enabled. This number must be a positive integer.
The parameter consists of a pair of names — the name of the cluster that receives the data to be replicated (usually the Recovery Cluster) as defined in the Serviceguard cluster configuration ASCII file, followed by a slash (“/”), followed by the name of the data replication receiver package as defined in the Serviceguard package configuration ASCII file. Some replication software might only have a receiver package as separate package because the sender package is built into the application.
• SYSLOG - Append the specified message to the /var/adm/syslog/syslog.log file. Note that the text of the message is not placed in the syslog file, only a notice from the monitor. • TCP Nodename:Portnumber - send the specified message to a TCP port on the specified node. • TEXTLOG Pathname - append the specified message to a specified text log file. • UDP Nodename:Portnumber - send the specified message to a UDP port on the specified node.
C Continentalclusters Command and Daemon Reference This appendix lists all the commands and daemons used with Continentalclusters. Manual pages are also available. cmapplyconcl [-v] [-C] This command verifies the Continentalclusters configuration as specified in filename, creates or updates the binary, filename and distributes it to all nodes in the Continentalclusters.
is used and some node has configuration files for Continentalclusters with a different name, you are prompted to indicate whether to proceed with deleting the configuration on that node. cmforceconcl The command is used to force a Continentalclusters package ServiceguardPackageEnableCommand to start. It allows a package to run even if the status of a remote package in the recovery group is unknown, which indicates that the software will not determine the status of the remote package.
this command enabled and can be run. The -f option can be used to enable the command after the time as configured via CLUSTER_ALERT parameters is reached. Options are: -f The force option enables cmrecovercl to function even though a CLUSTER_ALARM is not received. cmrecovercl {-e | -d [f] }-g This command moves a recovery group in and out of the maintenance mode by disabling or enabling it. This command must be run only on the recovery cluster.
D Package attributes Package Attributes for Continentalcluster with Continuous Access P9000 for Linux This appendix lists all Package Attributes for Metrocluster with Continuous Access XP P9000 for Linux.
0 – (DEFAULT) Do NOT startup the application on non-current data. If Metrocluster/Continuous Access cannot determine that the data is current, it does not allow the package to start up. (Note: For fence level DATA and NEVER, the data is current when both PVOL and SVOL are in PAIR state.) 1 – Startup the application even when the data cannot be current.
the PVOL becomes PSUE and the SVOL becomes PSUS(SSWS). During this transition, horctakeover attempts to flush any data in the side file on the MCU to the RCU. Data that does not make it to the RCU is stored on the bit map of the MCU. When failing back to the primary site, any data that was in the MCU side file that is now stored on the bit map is lost during resynchronization.
1—Startup the package after resynchronizing the data from the SVOL side to the PVOL side. The risk of using this option is that the SVOL data might not be a preferable one. If the package has been configured for a three data center (3DC) environment, this parameter is applicable only when the package is attempting to start up in either the primary (DC1) or secondary (DC2) data center. This parameter is not relevant in (the third data center) in the recovery cluster.
This parameter need not be set if a package is configured for a three data centers environment because this environment does not support Asynchronous mode of data replication. Leave this parameter with its default value in all the data centers. 66 AUTO_SVOLPSUS (Default = 0) This parameter applies when PVOL and SVOL both are in the suspended (PSUS) state. The problem with this situation cannot determine the earlier state: COPY or PAIR.
down, then write(1) calls in the package application will fail, causing the package to fail. NOTE: The Continuous Access Journal is used for asynchronous data replication. Fence level ascyn is used for a journal group pair. If the package is configured for three data centers (3DC), this parameter holds the fence level of device group between DC1 and DC2. As the device group between DC1 and DC2 is always synchronous, the fence level either “data” or “never”.
journal data to be transferred to SVOL. Also, the package startup time might increase significantly when the package fails over and waits for all of the journal data to be flushed. The HORCTIMEOUT must be set long enough to accommodate the outstanding data transfer from PVOL to SVOL. MULTIPLE_PVOL_OR_SVOL_FRAME_FOR_PKG (Default = 0) This parameter must be set to 1 if a PVOL or an SVOL for this package resides on more than P9000 and XP frames. Currently, only a value of 0 is supported for this parameter.
0, the monitor does NOT send notifications to the syslog file. When the parameter is set to 1, the monitor will send notifications to the syslog file. If the parameter is not defined (commented out), the default value is 0. MON_NOTIFICATION_CONSOLE (Default = 0) This parameter defines whether the monitor sends console notifications. When the parameter is set to 0, the monitor does NOT send console notifications. When the parameter is set to 1, the monitor sends console notifications.
Data_Currency_Preferred: The user chooses this policy when the application to start on consistent and current data is preferred. Metrocluster software allows the application to operate only on current data. This policy only focuses on the state of the local data (with respect to the application) being consistent and current. A package can be forced to start on a node by creating the FORCEFLAG in the package directory.
might time out before getting the response from SMI-S, and the package fails to start with an error.
E Configuration rules for using modular style packages in Continentalclusters Table 6 (page 72) summarizes the rules to use modular style packages for various Continentalclusters entities. Table 6 Configuration rules for using Modular style packages in Continentalclusters packages Continentalclusters Package Type Continuous Access P9000 or XP Continuous Access EVA 3PAR Remote Copy Logical Replication Monitor Package Use supplied modular package template.
F Sample Continentalclusters ASCII configuration file Sample Continentalclusters ASCII configuration file: Section 1 of the Continentalclusters ASCII configuration file ############################################################################### #### #### #### CONTINENTAL CLUSTER CONFIGURATION FILE #### #### #### #### #### #### This file contains ContinentalClusters configuration data. #### #### The file is divided into three sections, as follows: #### #### #### #### 1. Cluster Information #### #### 2.
#### #### #### #### #### #### #### #### #### #### #### NODE_NAME MONITOR_PACKAGE_NAME MONITOR_INTERVAL system2 ccmonpkg 1 MINUTE 30 SECONDS CLUSTER_NAME CLUSTER_DOMAIN NODE_NAME NODE_NAME MONITOR_PACKAGE_NAME MONITOR_INTERVAL eastcoast eastnet.myco.
#### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### cmrecovercl -r command. Enter the name of each package recovery group together with the fully qualified names of the primary and recovery packages. If appropriate, enter the fully qualified name of a data receiver package. Note that the data receiver package must be on the same cluster as the recovery package.
#### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### 76 ERROR - there is a mismatch of cluster versions or a security error.
#### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### specified node.
#### #### "Error in monitoring cluster westcoast.
PRIMARY_PACKAGE RECOVERY_PACKAGE RECOVERY_GROUP_NAME RECOVERY_PACKAGE DATA_RECEIVER_PACKAGE cluster2/pkg2 cluster3/pkg2’ ccRG3 cluster3/pkg3 cluster1/pkg3’ # Section 3.
CLUSTER_EVENT cluster3/UP MONITORING_CLUSTER cluster1 CLUSTER_ALERT 30 SECONDS NOTIFICATION TEXTLOG /var/opt/resmon/log/CCTextlog “DRT: (Ora-test) UP alert” NOTIFICATION SYSLOG “DRT: (Ora-test) cluster3 UP alert” 80 Sample Continentalclusters ASCII configuration file
G Sample input and output files for cmswitchconcl command The following is a sample of input and output files for running cmswitchconcl -C sample.input -c clusterA -F Sample.out sample.input ============ ### Section 1. Cluster Information CONTINENTAL_CLUSTER_NAME Sample_CC_Cluster CLUSTER_NAME ClusterA CLUSTER_DOMAIN cup.hp.com NODE_NAME node1 NODE_NAME node2 MONITOR_PACKAGE_NAME ccmonpkg CLUSTER_NAME ClusterB CLUSTER_DOMAIN cup.hp.
PRIMARY_PACKAGE ClusterB/pkgX' RECOVERY_PACKAGE ClusterA/pkgX RECOVERY_GROUP_NAME RG2 PRIMARY_PACKAGE ClusterB/pkgY' RECOVERY_PACKAGE ClusterA/pkgY DATA_RECEIVER_PACKAGE ClusterA/pkgR1 RECOVERY_GROUP_NAME RG3 PRIMARY_PACKAGE clusterB/pkgZ RECOVERY_PACKAGE ClusterA/pkgZ' RECOVERY_GROUP_NAME RG4 PRIMARY_PACKAGE ClusterB/pkgW RECOVERY_PACKAGE ClusterA/pkgW' DATA_RECEIVER_PACKAGE ClusterA/pkgR2 ### Section 3.
Glossary A application restart Starting an application, usually on another node, after a failure. Application can be restarted manually, which might be necessary if the data must be restarted before the application can run (example: Business Recovery Services work like this). Applications can by restarted by an operator using a script, which can reduce human error. Or applications can be started on the local or remote site automatically after detecting the failure of the primary site.
D data center A physically proximate collection of nodes and disks, usually all in one room. data consistency Whether data are logically correct and immediately usable; the validity of the data after the last write. Inconsistent data, if not recoverable to a consistent state, is corrupt.
logical data replication A type of on-line data replication that replicates logical transactions that change either the filesystem or the database. Complex transactions might result in the modification of many diverse physical blocks on the disk. M Maintenance mode A recovery group is in the maintenance mode when it is disabled. The cmrecovercl -d command moves a recovery group into maintenance mode. The cmrecovercl -e command moved the recovery group out of the maintenance mode.
Q quorum server A cluster node that acts as a tie-breaker in a disaster tolerant architecture in case all of the nodes in a data center go down at the same time. See also arbitrator. R Recovery Cluster A cluster on which recovery of a package takes place following a failure on the cluster. recovery group failover A failover of a package recovery group from one cluster to another. recovery package The package that takes over on the Recovery Cluster in the event of a failure on the cluster.
Index Symbols N 3PAR Remote Copy, 47 adding a node to Continentalclusters configuration, 42 node adding to Continentalclusters, 42 notifications receiving, 24 C P cluster continental, 69 recovery, 24 cmdeleteconcl command, 21 cmrecovercl, 24 command line cmrecovercl, 24 configuring additional nodes in Continentalclusters, 42 Continentalcluster Recovery cluster hardware, 24 Continentalclusters recovery cluster, 24 monitoring in Continentalclusters monitor packages in Continentalclusters, 28 Primary cl
