Building Disaster Recovery Serviceguard Solutions Using Continentalclusters for Linux B.01.00.00

6. Shut down the application on the recovery cluster using the cmhaltpkg command.
7. If physical data replication is being used, do not resync from the recovery cluster to the primary
cluster. Instead, run a command that will overwrite changes on the recovery disk array that
might have been made unintentionally.
8. Start the package up on the primary cluster and allow connection to the application.
Testing Continentalclusters operations
To exercise typical Continentalclusters behaviors:
1. Halt both clusters in a recovery pair, then restart both clusters. The monitor packages on both
clusters should start automatically. The Continentalclusters packages (primary, data sender,
data receiver, and recovery) should not start automatically. Any other packages might
or might not start automatically, depending on the configuration.
NOTE: If an UP status is configured for a cluster, then an appropriate alert notification (email,
SNMP, and so on.) should be received at the configured time interval from the node running
the monitor package on the other cluster. Due to delays in email or SNMP, the notifications
may arrive later than expected.
2. While the monitor package is running on a monitoring cluster, halt the monitored cluster
(cmhaltcl -f). An appropriate alert notification (email, SNMP, and so on.) should be
received at the configured time interval from the node running the monitor package. Run the
cmrecovercl. The command should fail. Additional notifications should be received at the
configured time intervals. After the alarm notification is received, run the cmrecovercl
command. Any data receiver packages on the monitoring cluster should halt and the recovery
packages should start with package switching enabled. Halt the recovery packages.
3. Test 2 should be rerun under a variety of conditions (and multiple conditions) such as the
following:
• Rebooting and powering off systems one at a time.
• Rebooting and powering off all systems at the same time.
Running the monitor package on each node in each cluster.◦
◦ Disconnecting the WAN connection between the clusters.
If physical data replication is used, disconnect the physical replication links between the
disk arrays:
◦ Powering off the disk array at the primary site.
◦ Powering off the disk array at the recovery site.
• Testing the cmrecovercl -f as well as the cmrecovercl command.
Depending on the condition, the primary packages should be running to test real life failures
and recovery procedures.
4. After each scenario in tests 2-3, restore both clusters to their production state, restart the
primary package (as well as any data sender and data receiver packages) and note any
issues, including time delays, and so on.
5. Halt the monitor package on one cluster. Halt the other cluster. Notifications that the other
cluster has failed are not generated. Test the mechanisms available to detect manual shutdown
of Continentalclusters monitor daemon.
22 Building the Continentalclusters configuration