Clustering Linux Servers with the Concurrent Deployment of HP Serviceguard Linux and Red Hat Global File System for RHEL5, October 2008

6
The qdisk feature allows users to configure arbitrary heuristics so that each cluster member can
determine its fitness for participating in a cluster. The fitness information is communicated to other
cluster members through the qdisk residing on shared storage. A qdisk is small 10MB disk
partition shared across the cluster, where each node periodically updates its assigned portion of
the disk with its health information. When there is loss of heartbeat, the vote count for each node
plus the value assigned for the qdisk in cluster configuration file is used for evaluating quorum.
In a large cluster the chance of majority node failure is very small, but not in case of a small
cluster. In Red Hat Cluster, to loose quorum exactly half or more nodes need to fail simultaneously.
For e.g. in a 20 node cluster, the likelihood of losing 10 nodes (to break quorum) is a remote
possibility. However in a 4 node cluster, you’d need to lose 2 nodes to break quorum; which a
possibility especially considering these failures do not need to be simultaneous. The qdisk can be
configured to sustain majority node failure allowing the remaining members to gain quorum and
allow cluster operations to continue. The qdisk is assigned a “vote” of one less than the total
number of nodes in the cluster which means that any one node plus the quorum partition is
sufficient to hold quorum allowing cluster operations to continue.
The qdisk can be used, which, act as a tie-breaker in equal sized partition where each partition
has equal votes. User defined heuristics can be used for deciding which partition is the quorate
partition in a partitioned cluster. For example the heuristics (which are user defined) can be to
check whether a service running on a node or whether there is access to an upstream router. This
prevents the cluster from being suspended.
The qdisk with heuristics can be used in 2 node cluster configuration to prevent the complete
cluster from going when nodes simultaneously fencing each other. For example a heuristic would
be to evaluate a node’s network connectivity (a ping to a network router) and remove itself from
the cluster. However, if both nodes still have a majority score according to their heuristics, then
both nodes will try to fence each other, and the fastest node kills the other. Hence even with qdisk
with heuristics it is possible for nodes to fence each and bring down the cluster.
Asymmetric cluster configuration – heavily weighted voting
In symmetric cluster architecture, all nodes contribute equally to the cluster quorum i.e., a vote of
1. But, it is the opposite in asymmetric client-server architecture where server nodes are the
“heavily-weighted voting” nodes contribute more than the client nodes to the cluster quorum i.e.,
vote of more than 1.
In Red Hat Cluster, if no “votes” value is explicitly assigned to a node in cluster configuration file
(/etc/cluster/cluster.conf), each node will get 1 vote by default. However, user can configure a
“votes” of greater that 1 to a node to provide the asymmetric nature to the cluster.
The asymmetric tuning is largely aimed at reducing the disruption certain nodes create for the
others which is beneficial in certain environments. It is also used for sustaining majority node
failures and preventing freezing of cluster operations and allows applications to be available.
Red Hat Global file system
Red Hat GFS is a cluster file system that provides simultaneous access shared block device to
nodes in a Red Hat Cluster. It uses the DLM to coordinate I/O and maintain data integrity.
GFS suspended
When Red Hat Cluster detects a failure, it communicates to the other cluster-infrastructure
components about the membership change. Then cluster-infrastructure components DLM and GFS
when notified of a node failure, suspends all GFS related operations until failed node is fenced