Managing HP Serviceguard for Linux Ninth Edition, April 2009

During startup, the cluster manager software checks to see if all nodes specified in the
startup command are valid members of the cluster, are up and running, are attempting
to form a cluster, and can communicate with each other. If they can, then the cluster
manager forms the cluster.
Automatic Cluster Startup
An automatic cluster startup occurs any time a node reboots and joins the cluster. This
can follow the reboot of an individual node, or it may be when all nodes in a cluster
have failed, as when there has been an extended power failure and all SPUs went down.
Automatic cluster startup will take place if the flag AUTOSTART_CMCLD is set to 1 in
the $SGCONF/cmcluster.rc file. When any node reboots with this parameter set to
1, it will rejoin an existing cluster, or if none exists it will attempt to form a new cluster.
Dynamic Cluster Re-formation
A dynamic re-formation is a temporary change in cluster membership that takes place
as nodes join or leave a running cluster. Re-formation differs from reconfiguration,
which is a permanent modification of the configuration files. Re-formation of the cluster
occurs under the following conditions (not a complete list):
• An SPU or network failure was detected on an active node.
• An inactive node wants to join the cluster. The cluster manager daemon has been
started on that node.
• A node has been added to or deleted from the cluster configuration.
• The system administrator halted a node.
• A node halts because of a package failure.
• A node halts because of a service failure.
• Heavy network traffic prohibited the heartbeat signal from being received by the
cluster.
• The heartbeat network failed, and another network is not configured to carry
heartbeat.
Typically, re-formation results in a cluster with a different composition. The new cluster
may contain fewer or more nodes than in the previous incarnation of the cluster.
Cluster Quorum to Prevent Split-Brain Syndrome
In general, the algorithm for cluster re-formation requires a cluster quorum of a strict
majority (that is, more than 50%) of the nodes previously running. If both halves (exactly
50%) of a previously running cluster were allowed to re-form, there would be a
split-brain situation in which two instances of the same cluster were running. In a
split-brain scenario, different incarnations of an application could end up simultaneously
accessing the same disks. One incarnation might well be initiating recovery activity
while the other is modifying the state of the disks. Serviceguard’s quorum requirement
is designed to prevent a split-brain situation.
44 Understanding Serviceguard Software Components