Building Disaster Recovery Serviceguard Solutions Using Continentalclusters A.08.
Legal Notices © Copyright 2013 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor’s standard commercial license. The information contained herein is subject to change without notice.
Contents 1 Introduction...............................................................................................8 2 Building the Continentalclusters configuration...............................................10 Creating the Serviceguard clusters at both the sites ....................................................................10 Easy deployment method....................................................................................................10 Traditional deployment method................
5 Disaster recovery rehearsal in Continentalclusters..........................................36 Overview of Disaster Recovery rehearsal...................................................................................36 Configuring Continentalclusters Disaster Recovery rehearsal.........................................................36 Configuring maintenance mode in Continentalclusters.............................................................36 Overview of maintenance mode feature....................
Adding a Recovery Group to Continentalclusters.........................................................................59 Modifying a package in a recovery group.................................................................................60 Modifying Continentalclusters configuration...............................................................................60 Removing a recovery group from the Continentalclusters..............................................................
C Configuration file parameters for Continentalclusters.....................................82 D Continentalclusters Command and Daemon Reference..................................85 E Package attributes.....................................................................................88 Package Attributes for Continentalcluster with Continuous Access for P9000 and XP........................88 Package Attributes for Continentalcluster with Continuous Access EVA............................................
Installing and configuring Oracle Clusterware..........................................................................138 Installing Oracle Real Application Clusters (RAC) software.........................................................138 Creating the RAC database with ASM in the primary cluster......................................................138 Configuring the ASM disk group in the primary cluster..........................................................
1 Introduction Continentalclusters provides disaster recovery between multiple Serviceguard clusters. A single cluster can act as the recovery for a set of primary clusters. It is also possible to have two clusters act as recovery for each other. This allows increased utilization of hardware resources. Continentalclusters eliminates the cluster itself as a single point of failure.
Figure 1 Sample Continentalclusters Configuration WAN ccmonpkg PRI_SCM_DB_PKG Recovery Group Packages REC_SCM_DB_PKG REC_CRM_DB_PKG Recovery Group Packages PRI_CRM_DB_PKG cconfpkg Continentalclusters Configuration Package cconfpkg Site A Node 1 Monitor Package Site A Node 2 ccmonpkg Site B Node 1 Site B Node 2 FC Switch FC Switch Monitor Package WAN WAN Converters Site A Disk Array Site A Cluster (Primary) Data Replication Links WAN Converters Site B Disk Array Site B Cluster (Rec
2 Building the Continentalclusters configuration To build a Continentalclusters configuration, complete the following list of steps: 1. Create a Serviceguard cluster at both the data center sites. 2. Establish the security credentials for Continentalclusters operation. 3. Create data replication between the two clusters. 4. If required, then create the volume groups or disk groups on the replicated disks. 5. Install and configure an application in the primary site using the replicated disks. 6.
# cmapplyconf -v -C /etc/cmcluster/cluster.config For more information, see Managing Serviceguard, latest edition at http://www.hp.com/go/ hpux-serviceguard-docs —>HP Serviceguard. Setting up security From Continentalclusters, all the nodes in all the clusters must be able to communicate with one another using SSH. When Continentalclusters is installed, a special Continentalclusters user group, conclgrp, and a special user, conclusr are created using groupadd and useradd commands.
Permission denied to 127.0.0.1 cmviewcl: Cannot view the cluster configuration: Permission denied. This user doesn't have access to view the cluster configuration. nl To resolve this error, edit the cluster configuration file, by including the following information: USER_NAME conclusr USER_HOST ANY_SERVICEGUARD_NODE USER_ROLE MONITOR Apply the cluster configuration file. Now, you must be able to view the cluster configuration using the cmviewcl command.
After configuring data replication using any one of the above arrays, the applications in the cluster that needs disaster recovery must be packaged with the appropriate Continentalclusters package module. This must be done at both the primary and the recovery clusters.
Creating and Exporting LVM Volume Groups Run the following procedure to create and export volume groups: NOTE: If you are using the March 2008 version or later of HP-UX 11i v3, skip step1; vgcreate (1m) will create the device file. Define the appropriate Volume Groups on each host system that might run the application package. # mkdir /dev/vgxx # mknod /dev/vgxx/group c 64 0xnn0000 where the name /dev/vgxx and the number nn are unique within the entire cluster. 1. 2.
Creating VxVM Disk Groups Run the following procedure to create VxVM Disk Groups 1. Initialize disks to be used with VxVM by running the vxdisksetup command only on the primary system. # /etc/vx/bin/vxdisksetup -i c5t0d0 2. Create the disk group to be used with the vxdg command only on the primary system. # vxdg init logdata c5t0d0 3. Verify the configuration. # vxdg list 4. Use the vxassist command to create the volume. # vxassist -g logdata make logfile 2048m 5. Verify the configuration.
When any of these pre-integrated solutions are used, the corresponding Continentalclusters specific module must be included in the primary and recovery packages. For example, while using Continuous Access P9000 or XP replication, the dts/ccxpca module must be used to create the primary and recovery packages. NOTE: If none of the above pre-integrated physical replications are used, then it is not required to include any Continentalclusters specific module.
• FENCE Specify the fence level configured for the XPCA device group that is managed by this package. • AUTO_RUN Set the value of this parameter to no. There are additional parameters available in the package configuration file. HP recommends that you retain the default values of these variables unless there is a specific business requirement to change them.
# cmmakepkg –m dts/cccaeva –m sg/filesystem -m sg/package_ip -m tkit/apache/apache temp.config 2. Edit the following attributes in the temp.config file: • dts/caeva/dts_pkg_dir This is the package directory for the modular package. This value must be unique for all the packages. • AUTO_RUN Set the value of this parameter to no. • DT_APPLICATION_STARTUP_POLICY This parameter defines the preferred policy by allowing the application to start with respect to the state of the data in the local volumes.
Configuring the primary and recovery packages as modular packages when using EMC SRDF When using EMC SRDF replication in Continentalclusters, the primary and recovery packages must be created using the dts/ccsrdf module. To use this module, Metrocluster with EMC SRDF must be installed on all the nodes in Continentalclusters.
6. Run the package on a node in the Serviceguard cluster. # cmrunpkg -n 7. Enable global switching for the package. # cmmodpkg -e Configuring the primary and recovery packages as modular packages when using 3PAR Remote Copy When using HP 3PAR Remote Copy in Continentalclusters, the primary and recovery packages must be created using the dts/cc3parrc module.
• DC1_RC_VOLUME_GROUP The Remote Copy volume group name configured on the HP 3PAR storage system, which is located in Data Center 1, containing the disks used by the application. • DC2_RC_VOLUME_GROUP The Remote Copy volume group name configured on the HP 3PAR storage system, which is located in Data Center 2, containing the disks used by the application.
vg ccvg fs_name /dev/ccvg/lvol1 fs_directory /opt/cmconcl/statedir fs_mount_opt "-o rw" fs_umount_opt "" fs_fsck_opt "" fs_type "vxfs" For more information about DR Rehearsal feature, see “Performing Disaster Recovery rehearsal in Continentalclusters” (page 39). 4. Specify a name for the ccmonpkg log file using script_log_file parameter. script_log_file /etc/cmcluster/ccmonpkg/ccmonpkg.log 5. Validate the package configuration file. # cmcheckconf –P ccmonpkg.conf 6. Apply the package configuration.
Parameter Value Mandatory or Optional CLUSTER_DOMAIN The DNS domain of the nodes defined Mandatory. above. MONITOR_PACKAGE_NAME The name of the monitoring package, This parameter is required only when usually ccmonpkg. the cluster specified in CLUSTER_NAME acts as a the recovery cluster. MONITOR_INTERVAL The amount of time between two consecutive monitoring operations. This parameter is required only when the cluster specified in CLUSTER_NAME acts as a the recovery cluster.
** Most software based replication will need either a data sender package or data receiver package while some might need both. Multiple recovery groups in Continentalclusters can be configured by repeating parameters. Monitoring definitions The monitoring definitions has the following parameters: Parameter Value Mandatory or Optional CLUSTER_EVENT The name of the primary cluster followed by Mandatory. cluster status. The following cluster status’ are supported: 1. UNREACHABLE 2. UP 3. DOWN 4.
Parameter Value Mandatory or Optional CLUSTER_EVENT primary_cluster/UNREACHABLE MONITORING_CLUSTER recovery_cluster CLUSTER_ALERT 5 MINUTES NOTIFICATION EMAIL admin@primary.site "primary_cluster status unknown for 5 min. Call recovery site." NOTIFICATION EMAIL admin@recovery.site "Call primary admin. (555) 555-6666." NOTIFICATION CONSOLE "Cluster ALERT: primary_cluster not responding.
Starting the Continentalclusters monitor package Starting the monitoring package enables the recovery clusters to monitor the primary clusters. Before doing this, ensure that the primary packages configured are running normally. If logical data replication is configured, ensure that the data receiver and data sender packages are running properly. If using physical data replication, ensure that it is operational. On every monitoring cluster start the monitor package.
3. command. Any data receiver packages on the monitoring cluster must halt and the recovery packages must start with package switching enabled. Halt the recovery packages. Test 2 should be rerun under a variety of conditions (and multiple conditions) such as the following: • Rebooting and powering off systems one at a time. • Rebooting and powering off all systems at the same time. ◦ Running the monitor package on each node in each cluster. ◦ Disconnecting the WAN connection between the clusters.
PACKAGE ROLE STATUS primary_cluster/test-pri primary down recovery_cluster/test-rec recovery down 28 Building the Continentalclusters configuration
3 Performing a recovery operation in Continentalclusters environment Performing recovery in case of disaster You can also initiate recovery forcefully even if the alarm event has not triggered but the alert event has happened. An administrator can initiate the recovery using cmrecoverclcommand. However, an administrator must confirm from the primary cluster administrator for the need of the recovery.
Once the notification is received, and it is determined that recovery is required by using the recovery checklist (For a sample checklist , see the section “Recovery Checklist” (page 76)) do the following: • Ensure the data used by the application is in usable state. Usable state means the data is consistent and recoverable, even though it might not be current. • Ensure the secondary devices are in read-write mode.
Recovering the entire cluster after a cluster alert If a notification defined in a CLUSTER_ALARM statement in the configuration file is not received, but a CLUSTER_ALERT and the remote site has confirmed the must fail over has been received, then override the disabled cmrecovercl command by using the -f forcing option. Use this command only after a confirmation from the primary cluster site.
4 Restoring disaster recovery cluster after a disaster After a failover to a cluster occurs, restoring disaster recovery is a manual processs, the most significant of which are: • Restoring the failed cluster. Depending on the nature of the disaster it might be necessary to either create a new cluster or to repair the failed cluster. Before starting up the new or the failed cluster, ensure the auto_run flag for all of the Continentalclusters application packages is disabled.
3. Check and apply the Continentalclusters configuration. # cmcheckconcl -v -C cmconcl.config # cmapplyconcl -v -C cmconcl.config 4. Restart the monitor packages on every cluster. # cmmodpkg -e ccmonpkg 5. View the status of the Continentalclusters. # cmviewconcl Before applying the edited configuration, the data storage associated with every cluster needs to be prepared to match the new role.
the new configuration file is required, do not use the -a option. If option -a is specified the new configuration applied automatically. 3. If option -a is specified with cmswitchconcl in step 2, skip this step. Otherwise manually apply the new Continentalclusters configuration. # cmapplyconcl -v -c newContinentalclustersConfigFileName (if -F is specified in step 2) # cmapplyconcl -v -c CurrentContinentalcusterConfigFileName (if -F is not specified in step 2) 4.
5. If the new cluster acts as a recovery cluster for any recovery group, create a monitor package for the new cluster. Apply the configuration of the new monitor package. # cmapplyconf -p ccmonpkg.config 6. Restart the monitor package on the recovery cluster. # cmrunpkg ccmonpkg 7. View the status of the Continentalclusters.
5 Disaster recovery rehearsal in Continentalclusters Overview of Disaster Recovery rehearsal The disaster recovery setup must be validated to ensure that a recovery can be performed smoothly when disaster strikes. Since disasters are once in a lifetime events, it is likely that a disaster recovery is never performed for long time. In this time, a lot of configuration drift and other changes will appear in either at the production data center or at the recovery data center.
# vgcreate /dev/vgcc -f /dev/sda1 2. Create a logical volume in the volume group and install ’vxfs’ file system in the logical volume: # lvcreate –L mke2fs -j For Example: # lvcreate –L 1000 /dev/vgcc; mke2fs -j /dev/vgcc/rlvol1 3. On every node of the recovery cluster, create the Continentalclusters shared directory /opt/cmconcl/statedir as follows: # mkdir For Example: # mkdir /opt/cmconcl/statedir 4.
7. Halt the monitor package ccmonpkg and apply the edited configuration file. For Example: # cmhaltpkg ccmonpkg # cmapplyconf –P cc_new.config 8. Start the monitor package ccmonpkg after applying the configuration. For Example: # cmrunpkg ccmonpkg Configuring Continentalclusters rehearsal packages The rehearsal packages use all the modules that are used to create the recovery package.
CONTINENTAL_CLUSTER_STATE_DIR 2. /opt/cmconcl/statedir Under the recovery group section for which the rehearsal package was configured, enter the rehearsal package name against REHEARSAL_PACKAGE field. For Example: Recovery group inv_rac10g_recgp Primary package Atlanta/inv_rac10g_primpkg Recovery package Houston/inv_rac10g_recpkg Rehearsal package Houston/inv_rac10g_rhpkg 3. Halt the monitor package. # cmhaltpkg ccmonpkg 4. Verify the Continentalclusters configuration ASCII file.
6. 7. 8. Restore the replication environment for recovery. Move the recovery group out of maintenance mode. Clean up the mirror copy. Verify data replication environment You can use the cmdrprev command to preview the preparation of data replication environment for an actual recovery can be previewed. The command identifies errors in data replication environment which will potentially fail an actual recovery.
Stop rehearsal package After performing the rehearsal operations, the rehearsal package must be halted using the cmhatlpkg command. # cmhaltpkg nl Restore replication environment for recovery First, synchronize the secondary mirror copy with the primary mirror copy and then synchronize the BC with the secondary mirror copy.
• Move the recovery group out of maintenance mode # cmrecovercl –e -g • Run cmrrcovercl command. # cmrecovercl Limitations of DR rehearsal feature Following are the limitations of the DR rehearsal feature: 1. The replication, preparation, and restoration for rehearsal and restoration for recovery is manual. The operator must prepare and restore the replication environment for every recovery group. 2. The cmdrprev preview command currently supports only verbose output. 3.
6 Configuring complex workloads in a Continentalclusters environment using SADTA Site Aware Disaster Tolerant Architecture (SADTA) enables automatic recovery of an entire application stack that is protected using physical data replication. The application stack can be packaged using mulit-node packages and failover packages with dependencies among them. SADTA also provides a single interface for manual failover of all the packages configured for an application stack.
5. 6. 7. 8. 9. 10. 11. 12. 13. Configure the Site Controller Package in the primary cluster. Configure the Site Safety Latch dependencies in the primary cluster. Suspend the replication to the recovery cluster. Set up the redundant complex workload in the recovery cluster. Configure the Site Controller Package in the recovery cluster. Configure the Site Safety Latch dependencies in the recovery cluster. Resume the replication to the recovery cluster. Configure Continentalclusters.
SITE site name> ... NOTE: Only one site must be specified in the cluster configuration file, and all the nodes in the cluster must belong to this site. 3. 4. Run the cmapplyconf command to apply the configuration file. Run the cmruncl command to start the cluster. After the cluster is started, you can run the cmviewcl command to view the single site configuration.
5. Create a filesystem. # newfs -F vxfs /dev/vx/rdsk// 6. Create a package configuration file. # cmmakepkg -m sg/cfs_all /etc/cmcluster/cfspkg1.ascii 7. Edit the following package parameters in the cfspkg1.ascii package configuration file.
1. Create a package configuration file using the following modules: # cmmakepkg -m sg/multi_node -m sg/dependency -m\ sg/resource -m sg/volume_group .conf 2. Edit the configuration file and specify values for the following attributes: package_name package_type multi_node cvm_dg cvm_activation_cmd "vxdg -g \${DiskGroup}set activation=sharedwrite" 3. Specify the nodes in the primary cluster using the node_name attribute.
Run this command on the configuration node only. The cluster must be running on all the nodes for the command to succeed. NOTE: Both the -S and the -c options are specified. The -S y option makes the volume group shareable, and the -c y option causes the cluster ID to be written out to all the disks in the volume group. In effect, this command specifies the cluster to which a node must belong in order to obtain shared access to the volume group.
Configuring the Site Controller Package in the primary cluster The procedure on a node in the primary cluster to configure the Site Controller Package as follows: 1. Create a Site Controller Package configuration file using the dts/sc and array-specific module. For example, when using Continuous Access P9000 and XP, the command is: # cmmakepkg -m dts/sc –m dts/ccxpca cw_sc.config When using Continuous Access EVA, the command is: # cmmakepkg -m dts/sc –m dts/cccaeva cw_sc.
1. If you have SG SMS CVM or CFS configured in your environment, add the EMS resource dependency to all DG MNP packages in the complex workload stack in the primary cluster. If you have SLVM configured in your environment, add the EMS resource details in the packages that are the foremost predecessors in the dependency order among the workload packages in the primary cluster.
For information about splitting the replication on HP 3PAR Remote Copy, see Building Disaster Recovery Serviceguard Solutions Using Metrocluster with 3PAR Remote Copy available at http:// www.hp.com/go/hpux-serviceguard-docs. For information about splitting the replication on EMC SRDF, see Building Disaster Recovery Serviceguard Solutions Using Metrocluster with EMC SRDF available at http://www.hp.com/go/ hpux-serviceguard-docs.
1. From the CVM master node at the recovery cluster, import the disk groups used by the complex workload. # vxdg -stfC import 2. Create Serviceguard disk group modular MNP packages for the CVM disk group. IMPORTANT: Veritas CVM disk groups must be configured in a dedicated modular MNP package using the cvm_dg attribute. This modular MNP package must be configured to have a package dependency on the SG-CFS-pkg SMNP package.
Resuming the replication to the recovery cluster Ensure that the Site Controller package and complex workload are halted on the recovery cluster. Re-synchronize the replicated disk in the recovery cluster from the source disk in the primary cluster for the replication. The procedure to resume the replication depends on the type of arrays that are configured in the environment. Based on the arrays in your environment, see the respective manuals to resume the replication.
7 Administering Continentalclusters Checking the status of clusters, nodes, and packages To verify the status of the Continentalclusters and associated packages, use the cmviewconcl command, which lists the status of the clusters, associated package status, and status of the configured events. This command also displays, if configured, the mode of the recovery group.
PRIMARY CLUSTER PTST_sanfran STATUS Unmonitored EVENT LEVEL POLLING INTERVAL unmonitored 1 min CONFIGURED EVENT STATUS DURATION alert unreachable 1 min alert unreachable 2 min alarm unreachable 3 min alert down 1 min alert down 2 min alarm down 3 min alert error 0 sec alert up 1 min RECOVERY CLUSTER PRIMARY CLUSTER PTST_dts1 PTST_sanfran STATUS Unmonitored CONFIGURED EVENT alert alert alarm alert alert alarm alert alert STATUS DURATION unreachable 1 min unreachable 2 min unreachable 3 min down 1 min d
INTERFACE PRIMARY PRIMARY PACKAGE ccmonpkg STATUS up up STATUS up PATH STATE NAME 4.1 56.1 lan0 lan1 PKG_SWITCH NODE running enabled Script_Parameters: ITEM NAME STATUS Service ccmonpkg.
NOTE: • System multi-node packages cannot be configured in Continentalclusters recovery groups. Multi-node packages are supported only for Oracle with CFS or CVM environments. • Starting with Continentalclusters version A.08.00, packages in Continentalclusters can be configured as modular packages. Startup and Switching Characteristics Normally, an application (package) can run on only one node at a time in a cluster.
# cmrecovercl —e recovery_group1 Recovering a cluster when the storage array or disks fail If the monitored cluster returns to UP status following an alert or alarm, but it is certain that the primary packages cannot start (say, because of damage to the disks on the primary site), then use a special procedure to initiate recovery: 1. 2. 3. Use the cmhaltcl command to halt the primary cluster. Wait for the monitor to send an alert. Use the cmrecovercl -f command to perform recovery.
Under normal circumstances, Continentalclusters does not allow a package to start in the recovery cluster unless it can determine that the package is not running in the primary cluster. In some cases, communication between the two clusters might be lost, and it might be necessary to start the package on the recovery cluster anyway.
Modifying a package in a recovery group There might be situations where a package must be halted for modifications purposes without having the package moved to another node. The following procedure is recommended for package maintenance and normal maintenance of Continentalclusters: 1. Shut down the package with the appropriate command. For example, # cmhaltpkg 2. 3. Perform the changes to the packages in primary and recovery cluster. Distribute the package configuration changes, if any.
5. 6. Use the Serviceguard cmdeleteconf command to remove every package in the recovery group. View the status of the Continentalclusters. # cmviewconcl Removing a rehearsal package from a recovery group To remove a rehearsal package from a recovery group: 1. Move the recovery group out of maintenance mode using the cmrecovercl -e command. 2. Delete the rehearsal package from the recovery cluster using the cmdeleteconf command. 3.
The verification is done based on the stable clusters' environment and the proper functioning of the network communication. In case the network communication between clusters can not be established or the cluster or package status cannot be determined, manual verification must be done to ensure that the operation to be performed on the target package will not have a conflict with other packages configured in the same recovery group.
Deleting the Continentalclusters configuration The cmdeleteconcl command is used to delete the configuration on all the nodes in the Continentalclusters configuration. To delete Continentalclusters and the Continentalclusters configuration run the following command. # cmdeleteconcl While deleting a Continentalclusters configuration with the recovery group maintenance feature, the shared disk is not removed.
Following is a partial list of failures that require full resynchronization to restore disaster-tolerant data protection. Resynchronization is automatically initiated by moving the application package back to its primary host after repairing the failure. • Failure of the entire primary Data Center for a given application package. • Failure of all of the primary hosts for a given application package. • Failure of the primary P9000 and XP disk array for a given application package.
NOTE: Long delays in package startup time will occur in situations when recovering from broken pair affinity. • The value of RUN_SCRIPT_TIMEOUT in the package ASCII file must be set to NO_TIMEOUT or to a large enough value to take into consideration the extra startup time required for getting status information from the P9000 and XP disk array. (See the earlier paragraph for more information on the extra startup time).
Maintaining EMC SRDF data replication environment Normal Startup The following is the normal Continentalclusters startup procedure. On the source disk site: 1. Start the source disk site. # cmruncl -v The source disk site comes up with ccmonpkg up. The application packages are down, and ccmonpkg is up. 2. Manually start application packages on the source disk site. # cmmodpkg -e 3. Confirm source disk site status. # cmviewcl -v and # cmviewconcl -v 4. Verify SRDF Links.
Remote Copy Link Failure and Resume Modes When the link is failed, snapshots are created for all the primary volumes, but not for the secondary volumes while replication is stopped. When replication is restarted for the volume, all differences between the base volume and the snapshot taken when the replication was stopped are sent over in order to resynchronize the secondary volume with the primary volume.
cluster to provide continuous service. For more information on moving a site aware disaster tolerant complex workload to a remote cluster, see “Moving a Complex Workload to the Recovery Cluster” (page 70). Maintaining Site Controller Package The Site Controller Package is a Serviceguard failover package. The package attributes that can be modified online can be modified without halting the Site Controller package. Certain package attributes require that the Site Controller package is halted.
3. Halt the Site Controller package. # cmhaltpkg 4. Log in to the other node in the local cluster, and start the Site Controller package. # cmrunpkg Deleting the Site Controller Package To remove a site controller package from the Continentalclusters configuration, you must first remove the associated recovery group from the Continentalclusters configuration file.
Shutting Down a Complex Workload The complex workload in SADTA can be shut down by halting the corresponding Site Controller package. To shutdown the complex workload, run the following command on any node in the cluster: # cmhaltpkg This command halts the Site Controller package and the current active complex-workload packages.
8 Troubleshooting Continentalclusters Reviewing Messages and Log Files Starting with Continentalclusters A.08.00, logs messages into the standard output. Continentalclusters commands, such as cmquerycl, cmcheckconcl, cmapplyconcl, and cmrecovercl output. Multiple log files are also used to log various operations. All log messages are stored in the /var/adm/cmconcl/logs directory with appropriate names. The cmviewconcl command logs messages in the /var/adm/cmconcl/logs/cmviewconcl.log file.
Table 2 Troubleshooting Continentalclusters Error Messages Command/Component Symptoms Cause Resolution ccmonpkg The ccmonpkg package fails to start. The following error message is written to the /var/opt/ resmon/ log/client.log file: Process ID: 26962 (/usr/lbin/cmclsentryd) Log Level: “Error rm_client_connect: Cannot get IP address for localhost.” The system is unable to resolve the IP address of the localhost. As a result, the EMS initialization fails.
Table 2 Troubleshooting Continentalclusters Error Messages (continued) Command/Component Symptoms Cause Resolution to true for package configuration file is set to on cluster YES. ”. • The global switching for the package is enabled using the cmmodpkg -e command. The value is set to YES. recovery or rehearsal packages. Disable the flag using the cmmodpkg –d command. cmclsentryd The cmclsentryd daemon Volume Group (VG) is not fails to start.
A Migrating to Continentalclusters A.08.00 Continentalclusters version A.08.00 includes enhanced features and capabilities, such as support for modular packages, IPv6 support, and a secure communication protocol for inter-cluster operations. HP recommends that you migrate Continentalclusters to the latest version to obtain the benefits of these features. NOTE: Upgrading to Continentalclusters A.08.00 requires re-applying the Continentalclusters configuration. IMPORTANT: Continentalclusters A.06.
B Continentalclusters Worksheets Planning is an essential effort in creating a robust Continentalclusters environment. HP recommends to record the details of your configuration on planning worksheets. These worksheets can be filled in partially before configuration begins, and then completed as you build the Continentalclusters. All the participating Serviceguard clusters in one Continentalclusters must have a copy of these worksheets to help coordinate initial configuration and subsequent changes.
Rehearsal Cluster/Package Name:______________________________ Data Receiver Cluster/Package Name:__________________________ Recovery Group Data: Recovery Group Name: ________________________________________ Primary Cluster/Package Name:________________________________ Data Sender Cluster/Package Name:____________________________ Recovery Cluster/Package Name:_______________________________ Rehearsal Cluster/Package Name:______________________________ Data Receiver Cluster/Package Name:______________________
Notify the monitored site of successful recovery using one of the following: Authorized person contacted: Director 1 Admin 1 Confirmation received Human-to-human voice authorization Voice mail Site Aware Disaster Tolerant architecture configuration worksheet This appendix includes the worksheets that you must use while configuring Site Aware Disaster Tolerant Architecture in your environment.
Table 4 Replication configuration (continued) Item Data Name of the sites Disk Array Serial # Serial Number of Disk Arrays at every site Node Names Name of Nodes at every site Command Device on Nodes Raw device file path at every node Device group device name Cluster 1 LUN CLuster 2 LUN Specify luns in CU:LDEV format Specify luns in CU:LDEV format Dev_name parameter 1) 2) 3) 4) 5) 6) 7) 8) 9) 10) CRS Sub-cluster configuration – using CFS Table 5 Configuring a CRS sub-cluster using CFS Item CRS Sub C
Table 5 Configuring a CRS sub-cluster using CFS (continued) Item Cluster Cluster Path to the vote disk or file CRS OCR Disk Path to the OCR disk or file CRS DG MNP package Path to the OCR disk or file CRS MP MNP package Path to the OCR disk or file CRS MNP package Path to the OCR disk or file CRS Member Nodes Node Names Private IP IP addresses for RAC Interconnect Private IP names IP address names for RAC Interconnect Virtual IP IP addresses for RAC VIP Virtual IP names IP addresses names for RAC VIP RA
Table 6 RAC database configuration (continued) Property Value RAC flash files CVM Disk group name CVM Disk Group name for oracle RAC flash file system Entry Cluster Cluster RAC Home Local file system directory to install Oracle RAC RAC MNP Package name for RAC database RAC Data file DG MNP CFS DG MNP package name for RAC data files file system RAC Data file MP MNP CFS MP MNP package name for RAC data files file system RAC Flash Area DG MNP CFS DG MNP package name for RAC flash file system RAC Flash Are
Table 7 Site Controller package configuration (continued) 1) 1) 2) 2) 3) 3) 4) 4) Site Aware Disaster Tolerant architecture configuration worksheet 81
C Configuration file parameters for Continentalclusters This appendix lists all Continentalclusters configuration file variables. CLUSTER_ALARM [Minutes] This is a time interval, in minutes and/or seconds, after which the notifications defined in the associated MINUTES [Seconds] SECONDS NOTIFICATION parameters are sent and failover to the Recovery Cluster using the cmrecovercl command is enabled. This number must be a positive integer. Minimum is 30 seconds, maximum is 3600 seconds or 60 minutes (one hour).
The parameter consists of a pair of names: the name of the cluster that receives the data to be replicated (usually the Recovery Cluster) as defined in the Serviceguard cluster configuration ASCII file, followed by a slash (“/”), followed by the name of the data replication receiver package as defined in the Serviceguard package configuration ASCII file. Some replication software might only have a receiver package as separate package because the sender package is built into the application.
that the text of the message is not placed in the syslog file, only a notice from the monitor. • TCP Nodename:Portnumber - send the specified message to a TCP port on the specified node. • TEXTLOG Pathname - append the specified message to a specified text log file. • UDP Nodename:Portnumber - send the specified message to a UDP port on the specified node. Any number of notifications can be associated with a given alert or alarm.
D Continentalclusters Command and Daemon Reference This appendix lists all commands and daemons used with Continentalclusters. Manual pages are also available. cmapplyconcl [-v] [-C] This command verifies the Continentalclusters configuration as specified in filename, creates or updates the binary, filename and distributes it to all the nodes in the Continentalclusters.
cmdeleteconcl [-f] This command is used to delete the Continentalclusters configuration from the entire Continentalclusters. This command will not remove the file system configured for recovery group maintenance mode feature. Options are: -f Delete the configuration files on all reachable nodes without further prompting. If this option is not used and if some nodes are unreachable, you prompted to indicate whether to proceed with deleting the configuration on the reachable nodes.
This command can be issued from any node on the recovery cluster. This command first connects to the Continentalclusters monitoring package running on the recovery cluster. This might be a different cluster node than where the cmrecovercl command is being run. cmrecovercl connects to the monitoring package to verify that the primary cluster is in an Unreachable or Down state. If the primary cluster is reachable and the cluster is Up, this command will fail.
E Package attributes Package Attributes for Continentalcluster with Continuous Access for P9000 and XP This appendix lists all Package Attributes for Metrocluster with Continuous Access for P9000 and XP.
0 – (DEFAULT) Do NOT startup the application on non-current data. If Metrocluster/Continuous Access cannot determine the data is current, it will not allow the package to start up. (Note: for fence level DATA and NEVER, the data is current when both PVOL and SVOL are in PAIR state.) 1 – Startup the application even when the data cannot be current.
PSUS(SSWS). During this transition, horctakeover will attempt to flush any data in the side file on the MCU to the RCU. Data that does not make it to the RCU stored on the bit map of the MCU. When failing back to the primary site any data that was in the MCU side file that is now stored on the bit map lost during resynchronization. In synchronous mode with fence level NEVER, when the Continuous Access link fails, the application continues running and writing data to the PVOL.
1—Startup the package after resynchronize the data from the SVOL side to the PVOL side. The risk of using this option is that the SVOL data might not be a preferable one. If the package has been configured for a three data center (3DC) environment, this parameter is applicable only when the package is attempting to start up in either the primary (DC1) or secondary (DC2) data center. This parameter is not relevant in (the third data center) in the recovery cluster.
non-current and the data written to the PVOL side after the hardware failure might be loss. This parameter is not required to be set if a package is configured for three data centers environment because three data center does not support Asynchronous mode of data replication. Leave this parameter with its default value in all data centers. 92 AUTO_SVOLPSUS (Default = 0) This parameter applies when the PVOL and SVOL both have the state of suspended (PSUS).
fence level is that if the package is running and a link goes down, an array goes down, or the remote data center goes down, then write(1) calls in the package application will fail, causing the package to fail. NOTE: The Continuous Access Journal is used for asynchronous data replication. Fence level ascyn is used for a journal group pair. If the package is configured for three data centers (3DC), this parameter holds the fence level of device group between DC1 and DC2.
For Continuous Access Journal mode package, journal volumes in PVOL might contain a significant amount of journal data to be transferred to SVOL. Also, the package startup time might increase significantly when the package fails over and waits for all of the journal data to be flushed. The HORCTIMEOUT must be set long enough to accommodate the outstanding data transfer from PVOL to SVOL.
value is an empty string, this will indicate to the monitor that no email notifications sent. MON_NOTIFICATION_SYSLOG (Default = 0) This parameter defines whether the monitor will send notifications to the syslog file. When the parameter is set to 0, the monitor will NOT send notifications to the syslog file. When the parameter is set to 1, the monitor will send notifications to the syslog file. If the parameter is not defined (commented out), the default value is 0.
name of the directory where the Metrocluster caeva environment file is located. DT_APPLICATION_STARTUP_POLICY This parameter defines the preferred policy to start the application with respect to the state of the data in the local volumes. It must be set to one of the following two policies: Availability_Preferred: The user chooses this policy if he prefers application availability. Metrocluster software allows the application to start if the data is consistent even if the data is not current.
If a connection to the first management server fails, attempts are made to connect to the subsequent management servers in their order of specification.. DC2_HOST_LIST QUERY_TIME_OUT(Default 120 seconds) A list of the clustered nodes that reside in Data Center 2. Multiple names can be defined by using commas as separators. Sets the time in seconds to wait for a response from the SMI-S CIMOM in storage management appliance. The minimum recommended value is 20 seconds.
the data on the R2 side might be non-current and thus a risk that data loss will occur when starting the package up on the R2 side. To have automatic package startup under these conditions, set AUTOR2WDNL=0 AUTOR2RWNL Default: 1 AUTOR2RWNL=1 indicates that when the package is being started on an R2 host, the Symmetrix is in a read/write state, and the SRDF links are down, the package will not be started.
consistency groups. (Consistency groups are required for M by N configurations.) If CONSISTENCYGROUPS is set to 1, AUTOSWAPR2 cannot be set to 1. Ensure that you have the minimum requirements for Consistency Groups by referring to Metrocluster release notes. DEVICE_GROUP DTS_PKG_DIR RDF_MODE This variable contains the name of the Symmetrix device group for the package on that node, which contains the name of the consistency group in an M by N configuration.
F Legacy packages Migrating complex workloads using Legacy SG SMS CVM/CFS Packages to Modular SG SMS CVM/CFS Packages with minimal downtime The procedure to migrate all the legacy SG SMS CVM/CFS packages managed by a Site Controller package to modular SG SMS CVM/CFS packages as follows: 1. Complete the following steps on the recovery cluster where the complex workload packages are not running: a.
1. Halt the monitoring daemon package. # cmhaltpkg ccmonpkg 2. Generate the modular configuration file. # cmmigratepkg -p -o 3. Validate the package configuration file. # cmcheckconf –P modular_ccmonpkg.conf 4. Apply the package configuration. # cmapplyconf –P modular_ccmonpkg.conf 5. Start the monitoring daemon package.
4. Halt the package. # cmhaltpkg 5. Validate the new modular package configuration file. # cmcheckconf -P 6. Apply the package configuration with the modular configuration file created in step 3. # cmapplyconf -P 7. Run the package on a node in the Serviceguard cluster. # cmrunpkg -n 8. Enable global switching for the package.
4. Halt the package. # cmhaltpkg 5. Validate the package configuration file. # cmcheckconf -P 6. Apply the package configuration with the modular configuration file created in step 3. # cmapplyconf -P 7. Run the package on a node in the Serviceguard cluster. # cmrunpkg -n 8. Enable global switching for the package.
5. Validate the new package configuration file. # cmcheckconf -P 6. Apply the package configuration with the modular configuration file created in step 3. # cmapplyconf -P 7. Run the package on a node in the Serviceguard cluster. # cmrunpkg -n 8. Enable global switching for the package.
FS_FSCK_OPT[0]=""; FS_TYPE[0]="vxfs" 5. Use the cmcheckconf command to validate the package. # cmcheckconf -P ccmonpkg.config 6. Copy the package configuration file ccmonpkg.config and control script ccmonpkg.cntl to the monitor package directory (default name /etc/cmcluster/ccmonpkg) on all the other nodes in the cluster. Ensure this file is executable. Use the cmapplyconf command to add the package to the Serviceguard configuration. # cmapplyconf -P ccmonpkg.
3. Create a package control script. # cmmakepkg -s pkgname.cntl Customize the control script as appropriate to your application using the guidelines in the Managing Serviceguard user’s guide. Standard Serviceguard package customizations include modifying the VG, LV, FS, IP, SUBNET, SERVICE_NAME, SERVICE_CMD, and SERVICE_RESTART parameters. Set LV_UMOUNT_COUNT to 1 or greater. NOTE: Some of the control script variables, such as VG and LV, on the target disk site must be the same as on the source disk site.
This script assumes the package directories already exist on all the nodes. Using ftp might be preferable at your organization, because it does not require the use of a .rhosts file for root. Root access via .rhosts might create a security issue. 8. Verify that every node in the Serviceguard cluster has the following files in the directory /etc/cmcluster/pkgname: pkgname.cntl Metrocluster/Continuous Access package control script pkgname_xpca.env Metrocluster/Continuous Access environment file pkgname.
5. available at http://www.hp.com/go/hpux-serviceguard-docs —> HP Serviceguard for more detailed information on these functions. Copy the environment file template /opt/cmcluster/toolkit/SGCA/caeva.env to the package directory, naming it pkgname_caeva.env. # cp /opt/cmcluster/toolkit/SGCA/caeva.env \ /etc/cmcluster/pkgname/pkgname_caeva.env NOTE: If not using a package name as a filename for the package control script, it is necessary to follow the convention of the environment file name.
k. 7. Set the DC2_HOST _LISTvariable to the list of clustered nodes which resides in Data Center 2. Multiple names are defined using a comma as a separator between the names. l. Set the QUERY_TIME_OUT variable to the number of seconds to wait for a response from the SMI-S CIMOM in Management Server. The default timeout is 300 seconds. The recommended minimum value is 20 seconds.
modifying the VG, LV, FS, IP, SUBNET, SERVICE_NAME, SERVICE_CMD, and SERVICE_RESTART parameters. Set LV_UMOUNT_COUNT to 1 or greater. NOTE: Some of the control script variables, such as VG and LV, on the target disk site must be the same as on the source disk site. Some of the control script variables, such as, FS, SERVICE_NAME, SERVICE_CMD and SERVICE_RESTART are probably the same as on the source disk site.
If there are three packages with data on a particular Symmetrix pair (connected by SRDF), then the values for RETRY and RETRYTIME might be as follows: Table 8 RETRY and RETRYTIME Values RETRY RETRYTIME pkgA 60 attempts 5 seconds pkgB 43 attempts 7 seconds pkgC 33 attempts 9 seconds f. g. 7. Uncomment the CLUSTER_TYPE variable and set it to “continental”. Uncomment the RDF_MODE and set it to “async” or “sync” as appropriate to your application.
where node1 and node2 are the nodes in the Source Disk Site. 4. Activate the CVM disk group in the Source Disk Site CFS sub-cluster. # cfsdgadm activate 5. Create a volume from the disk group # vxassist -g make 4500m 6. NOTE: Skip the following steps if you want to use the storage devices as raw CVM volumes.
Where node1 and node2 are the nodes at the target disk site. 6. Mount the cluster file systems in this CFS sub-cluster.
G Configuration rules for using modular style packages in Continentalclusters Table 9 (page 114) summarizes the rules to use modular style packages for various Continentalclusters entities. Table 9 Configuration rules for using Modular style packages in Continentalclusters packages Continentalclusters Continuous Access Continuous Access Continuous Package Type P9000 or XP EVA Access SRDF 3PAR Remote Copy Logical Replication Monitor Package Use supplied modular package template.
H Sample Continentalclusters ASCII configuration file Sample Continentalclusters ASCII configuration file: Section 1 of the Continentalclusters ASCII configuration file ################################################################ #### #### Continentalclusters CONFIGURATION FILE #### #### #### #### This file contains Continentalclusters #### #### #### #### configuration data. #### #### #### #### The file is divided into three sections, #### #### #### #### as follows: #### #### #### #### 1.
#### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### wish to monitor the recovery cluster, you must delete or comment out the MONITOR_PACKAGE_NAME and MONITOR_INTE
#### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #
#### #### #### #### #### #### #### #### DATA_SENDER_PACKAGE westcoast/hpsenderpkg #### RECOVERY_PACKAGE eastcoast/hpbackuppkg #### DATA_RECEIVER_PACKAGE eastcoast/nfsreplicapkg#### REHEARSAL_PACKAGE eastcoast/hprehearsalpkg #### #### #### #### #### Section 3 of the Continentalclusters ASCII configuration file ################################################################ #### #### Section 3.
#### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #
#### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #
#### #### #### #### #### #### #### #### #### #### #### #### #### #### MONITORING_CLUSTER eastcoast CLUSTER_ALERT 0 MINUTES NOTIFICATION EMAIL admin@secondary.site "Error in monitoring cluster westcoast." CLUSTER_EVENT /UNREACHABLE MONITORING_CLUSTER CLUSTER_ALERT #### #### #### #### #### #### #### The following is a sample Continentalclusters configuration file with two recovery pairs.
CLUSTER_EVENT cluster2/DOWN MONITORING_CLUSTER cluster3 CLUSTER_ALERT 30 SECONDS NOTIFICATION TEXTLOG /var/opt/resmon/log/CCTextlog “DRT: (Ora-test) DOWN alert” NOTIFICATION SYSLOG “DRT: (Ora-test) cluster2 DOWN alert” CLUSTER_ALARM 30 SECONDS NOTIFICATION TEXTLOG /var/opt/resmon/log/CCTextlog “DRT: (Ora-test) DOWN alarm” NOTIFICATION SYSLOG “DRT: (Ora-test) cluster2 DOWN alarm” CLUSTER_EVENT cluster3/DOWN MONITORING_CLUSTER cluster1 CLUSTER_ALERT 30 SECONDS NOTIFICATION TEXTLOG /var/opt/resmon/log/logging
I Sample input and output files for cmswitchconcl command The following is a sample of input and output files for running cmswitchconcl -C sample.input -c clusterA -F Sample.out sample.input ============ ### Section 1. Cluster Information CONTINENTAL_CLUSTER_NAME Sample_CC_Cluster CLUSTER_NAME ClusterA CLUSTER_DOMAIN cup.hp.com NODE_NAME node1 NODE_NAME node2 MONITOR_PACKAGE_NAME ccmonpkg CLUSTER_NAME ClusterB CLUSTER_DOMAIN cup.hp.
PRIMARY_PACKAGE ClusterB/pkgX' RECOVERY_PACKAGE ClusterA/pkgX RECOVERY_GROUP_NAME RG2 PRIMARY_PACKAGE ClusterB/pkgY' RECOVERY_PACKAGE ClusterA/pkgY DATA_RECEIVER_PACKAGE ClusterA/pkgR1 RECOVERY_GROUP_NAME RG3 PRIMARY_PACKAGE clusterB/pkgZ RECOVERY_PACKAGE ClusterA/pkgZ' RECOVERY_GROUP_NAME RG4 PRIMARY_PACKAGE ClusterB/pkgW RECOVERY_PACKAGE ClusterA/pkgW' DATA_RECEIVER_PACKAGE ClusterA/pkgR2 ### Section 3.
J Configuring Oracle RAC in Continentalclusters in Legacy style Support for Oracle RAC instances in a Continentalclusters environment When the primary cluster fails, the clients database requests are served by the support of Oracle RAC instances that are restarted by the Continantalclusters on the recovery cluster. Figure 3 (page 125) is a sample of Oracle RAC instances running in the Continentalclusters environment.
Figure 4 Sample Oracle RAC instances in a Continentalclusters environment after failover The Oracle RAC workloads can also be deployed in Continentalclusters using Site Aware Disaster Tolerant Architecture (SADTA). For more information on using SADTA for deploying Oracle RAC Workloads in Continentalclusters, see “Configuring Oracle RAC database with ASM in Continentalclusters using SADTA” (page 136).
1. 2. Configure either Continuous Access P9000 and XP, or Continuous Access EVA for data replication between disk arrays associated with primary and recovery clusters. Configure the database storage using one of the following software: • Shared Logical Volume Manager (SLVM) • Cluster Volume Manager (CVM) • Cluster File Systems (CFS) You must configure the SLVM volume groups or CVM disk groups on the disk arrays to store the Oracle database.
NOTE: After you configure the disk group and mount point multi-node packages, you must deactivate the packages on the recovery cluster. During a recovery process, the cmrecovercl command automatically activates these multi-node packages. h.
a. b. Log in as root on one node of the primary cluster. Change to your own directory: # cd c. Copy the file: # cp /opt/cmconcl/scripts/ccrac.config ccrac.config.mycopy d. Edit the file ccrac.config.mycopy to suit your environment. The following parameters must be edited: CCRAC_ENV - fully qualified Metrocluster environment file name. This file naming convention is required by the Metrocluster software. It must be appended with _.
(Multiple values for CCRAC_SLVM_VGS and CCRAC_INSTANCE_PKGS must be separated by space). If multiple sets of Oracle instances accessing different databases are configured in your environment, and require Continentalclusters recovery support, repeat this set of parameters with an incremented index. For Example, CCRAC_ENV[0]=/etc/cmconcl/ccrac/db1/db1EnvFile_xpca.env CCRAC_SLVM_VGS[0]=ccracvg1 ccracvg2CCRAC_INSTANCE_PKGS[0]=ccracPkg1 ccracPkg2CCRAC_CLUSTER[0]=PriCluster1 CCRAC_ENV_LOG[0]=/tmp/db1_prep.
When you finish configuring a recovery pair with RAC support, your systems must have sets of files similar to those shown in Figure 5. NOTE: If you are configuring Oracle RAC instances in Serviceguard packages in a CFS or CVM environment, do not specify the CVM_DISK_GROUPS, and CVM_ACTIVATION_CMD fields in the package control scripts as CVM disk group manipulation is addressed by the disk group multi node package.
a. Generate the resource profile. # crs_stat -p instance_name > $CRS_HOME/crs/public/instance_name.cap b. c. Edit the resource profile and set AUTO_START value to 2. Register the value. # crs_register -u instance_name d. Verify the value.
started on all the nodes, which the database instance are configured to run before initiating the recovery process. If you have configured CFS or CVM in your environment, ensure the following: • The SG-CFS-PKG (system multi-node package) is up and running. The SG-CFS-PKG package is not part of the continentalclusters configuration. • The cmrecovercl command is run from the CVM master node. Use the following command to find out the CVM master node: # vxdctl -c mode Starting with Continentalclusters A.07.
and that recovery on secondary_cluster is necessary. Continuing with this command while the applications are running on the primary cluster might result in data corruption. Are you sure that the primary packages are not running and will not come back, and are you certain that you want to start the recovery packages [y/n]? y cmrecovercl: Attempting to recover RecoveryGroup subsrecovery1 on cluster secondary_cluster NOTE: The configuration file /etc/cmconcl/ccrac/ccrac.
If you have configured CVM or CFS in your environment: a. Unmount the CFS mount points: # cfsumount 3. 4. 5. b. Deactivate the disk groups: # cfsdgadm deactivate c. Deport the disk groups using the following command: # vxdg deport The recovery cluster is now ready to failback packages and applications to the primary cluster. Synchronize the data between the two participating clusters.
K Configuring Oracle RAC database with ASM in Continentalclusters using SADTA Automatic Storage Management (ASM) is a feature in Oracle Database 10g and 11g that provides the database administrator with a simple storage management interface that is consistent across all server and storage platforms. In Continentalcluster, Oracle RAC with ASM must be configured using the SADTA.
This section describes the procedures that must be followed to configure SADTA with Oracle RAC database with ASM. To explain these procedures, it is assumed that the Oracle RAC home directory is /opt/app/oracle/product/11.1.0/db_1/dbs and the database name is hrdb. Tto configure Oracle RAC database with ASM in Continentalclusters using SADTA: 1. Set up replication between the primary cluster and the recovery cluster. 2.
Configure a recovery cluster with a single site The procedure for configuring Continentalclustrers with sites for Oracle RAC database with ASM is identical to the procedure for configuring Oracle RAC with SADTA. For more information on configuring Continentalclusters with sites for SADTA, see “Configuring the recovery cluster with a single site” (page 45). Installing and configuring Oracle Clusterware After setting up replication in your environment, you must install Oracle Clusterware.
available at http://www.hp.com/go/hpux-serviceguard-docs -> HP Serviceguard Extension for RAC -> Using Serviceguard Extension for RAC. Configuring SGeRAC toolkit packages for the ASM disk group in the primary cluster To configure Oracle RAC database with ASM in Continentalclusters using SADTA, the ASM disk group must be packaged in Serviceguard MNP packages in both the clusters. Configure ASM Disk group MNP package dependency on the Clusterware MNP package on both the clusters.
Suspending the replication to the recovery cluster In the earlier procedures, the RAC database and Site Controller packages were created at the primary cluster with the source disk of the replication disk group. A RAC MNP stack was also created in that cluster. Now, an identical RAC database using the target replicated disk must be configured with the RAC MNP stack in the recovery cluster.
# ln –s /opt/app/oracle/admin/+ASM/pfile/init.ora init+ASM2.ora # chown -h oracle:oinstall init+ASM2.ora # chown oracle:oinstall orapw+ASM2 7. Add the ASM instances with the CRS cluster on the recovery cluster. In this example, run the following commands from any node on cluster2: # export ORACLE_SID=”+ASM” # srvctl add asm -n -i “+ASM1” –o /opt/app/oracle/product/11.1.0/db_1/ # srvctl add asm -n -i “+ASM2” –o /opt/app/oracle/product/11.1.
6. Run the following command at the remote site. # chown -R oracle:oinstall /opt/app/oracle/admin/hrdb 7. Log in to any of the nodes in the remote site using the oracle user credentials. # su – oracle 8. Configure a listener for the database on this site using the Oracle Network Configuration Assistant (NETCA). Copy the tnsnames.ora file from the primary cluster CRS and modify it to fit the local environment.
3. Configure the Site Controller Package in the primary cluster with the RAC MNP stack in primary cluster: # site cc1_site1 # critical_package # managed_package NOTE: Do not add any comments after specifying the critical and managed packages. 4. Re-apply the Site Controller Package configuration.
Database with ASM in the Continentalclusters in the primary cluster The procedure to start the disaster recovery Oracle RAC database with ASM is identical to the procedure for starting a complex workload in a Continentalclusters. Run the cmrunpkg command with the site controller package name managing the Oracle RAC/ASM workload in the primary cluster as the argument.
Glossary A application restart Starting an application, usually on another node, after a failure. Application can be restarted manually, which might be necessary if data must be restarted before the application can run (example: Business Recovery Services work like this.) Applications can by restarted by an operator using a script, which can reduce human error. Or applications can be started on the local or remote site automatically after detecting the failure of the primary site.
data centers. Continentalclusters are often located in different cities or different countries and can span 100s or 1000s of kilometers. Continuous Access A facility provided by the Continuos Access software option available with the HP StorageWorks P9000 Disk Array family, HP StorageWorks E Disk Array XP series. This facility enables physical data replication between P9000 and XP series disk arrays. D data center A physically proximate collection of nodes and disks, usually all in one room.
G gatekeeper A small EMC Symmetrix device configured to function as a lock during certain state change operations. H, I high availability A combination of technology, processes, and support partnerships that provide greater application or system availability. J, K, L local cluster A cluster located in a single data center. This type of cluster is not disaster recovery.
package recovery group A set of one or more packages with a mapping between their instances on the cluster and their instances on the Recovery Cluster. physical data replication An on-line data replication method that duplicates I/O writes to another disk on a physical block basis.
sub-clusters Sub-clusters are clusterwares that run above the Serviceguard cluster and comprise only the nodes in a Metrocluster site. Sub-clusters have access only to the storage arrays within a site. SVOL A secondary volume configured in an P9000 and XP series disk array that uses Continuous Access. SVOLs are the secondary copies in physical data replication with Continuos Access on the P9000 and XP.
Index Symbols H 3PAR Remote Copy, 63 hardware for Continentalclusters Recovery cluster, 29 A L adding a node to Continentalclusters configuration, 59 log files, 69 C M cluster continental, 95 recovery, 29 cmdeleteconcl command, 25 cmrecovercl, 29 command line cmrecovercl, 29 configuring additional nodes in Continentalclusters, 59 Continentalcluster Recovery cluster hardware, 29 Continentalclusters recovery cluster, 29 data replication for Continentalclusters, 19 monitoring in Continentalclusters m
V Veritas, 125 Veritas Cluster Volume Manager/Cluster File System, 125 Veritas CVM/CFS, 126 volume groups creating, 30 W worksheet power supply configuration, 75, 76 worksheet, Metrocluster, 67 worksheet, package, 67 151
