Building Disaster Recovery Serviceguard Solutions Using Metrocluster with Continuous Access EVA A.05.01

A special flag, package_halted is set to no when the complex-workload package is down, having
failed in the cluster. This special flag is set to yes when the complex-workload package is down
and manually halted. Serviceguard sets this flag to no only when the last surviving instance of the
complex workload package is halted as a result of a failure. The flag is set to yes if the last surviving
instance is manually halted, even if other instances are halted earlier due to failures.
The Site Controller package determines a failure by checking if the package_halted flag is set
to no for all monitored packages that are in the down state. When the monitored packages have
failed but not halted, the Site Controller Package fails over to a remote site node to perform a site
failover.
Before starting the complex-workload packages configured at the remote site, the Site Controller
package ensures that it is safe to do so. The failed complex-workload packages might not have
halted cleanly, leaving stray processes and resources. In such scenarios, it is not safe to start the
identical complex workload configuration on the remote site. As a result, when it starts on the
remote site node, the Site Controller package checks whether all instances of the failed active
packages have halted cleanly. The Site Controller Package checks the last_halt_failed flag
for each instance of the workload packages. The flag is set to yes for an instance whose halt script
execution resulted in an error. Even if one instance of any of the failed workload's packages did
not halt successfully, the Site Controller package aborts site failover. In these circumstances, the
Site Controller package halts and its state is displayed as failed on the remote site node. To restart
the Site Controller package and the complex workload configuration, the nodes on the site need
to be manually cleaned.
After ensuring a clean halt for all instances of the failed complex-workload packages, the Site
Controller package performs the following steps to activate the corresponding passive complex
workload configured in its current site:
1. Closes the Site Safety Latch for the failed complex-workload package nodes.
2. Waits for all configured packages as part of the failed complex-workload package to halt
successfully.
3. Deports the CVM disk groups used by the database on the failed site.
4. Prepares the replicated data storage on the current site using the Metrocluster environment
file on the node it is starting.
5. Imports the CVM disk groups used by the database in the current site.
6. Opens the Site Safety Latch in the current site.
7. Starts the complex-workload packages configured for the database in the current site.
For the Site Controller package to successfully start the remote complex-workload package
configuration, the packages in the remote configuration must have node switching enabled on
their configured nodes. When the Site Controller package fails to start after successfully preparing
the storage on a site, it sets the Site Safety Latch to a transient state, which is displayed as
INTERMEDIATE. When the Site Safety Latch is in the INTERMEDIATE state, the corresponding Site
Controller package can be restarted only after cleaning the site where it previously failed to start.
For more information on cleaning the Site Controller package, see “Cleaning the site to restart the
Site Controller package” (page 69).
Node failure and rejoining the cluster
When a node in a cluster fails, all Multi-node package (MNP) instances running on the failed node
also fails. The failover type packages failover to the next available adoptive node. If no other
adoptive node is configured or available in the cluster, the failover package fails and is halted.
When a node in the Metrocluster environment is restarted, the active complex-workload packages
on the node are halted before the node restarts. After the node is restarted and joins a cluster, the
active complex-workload package instances on the site with the auto_run flag set to yes, starts
automatically. If the complex workload's packages have the auto_run flag set to no, these
instances must be manually started on the restarted node.
Complex workload failover/failback scenarios 57