Optimizing Serviceguard Failover Time, Version A.11.19 and later, April 2009

ManualsBrandsHP ManualsSoftwareHP Serviceguard for Linux ProLiant Cluster

Introduction ...................................................................................................................................2

The HP Serviceguard failover process.................................................................................................2

What happens when failover is triggered by a node failure................................................................2

Node failure detection..............................................................................................................4

Cluster reformation time............................................................................................................4

Election of cluster membership ...................................................................................................5

Lock acquisition.......................................................................................................................5

Quiescence ............................................................................................................................5

Cluster component recovery.......................................................................................................6

Serviceguard implementation: resource recovery...........................................................................6

Serviceguard implementation: applications recovery......................................................................6

Serviceguard with Serviceguard Extension for RAC: group membership reconfiguration.......................6

Serviceguard with Serviceguard Extension for RAC: RAC reconfiguration..........................................6

What happens when failover is triggered by a package failure...........................................................7

Serviceguard implementation: resource failure detection.................................................................8

Serviceguard implementation: package determination....................................................................8

Serviceguard implementation: resource recovery...........................................................................8

Serviceguard implementation: application startup..........................................................................9

Servicegaurd with Serviceguard Extension for RAC: group membership reconfiguration.......................9

Serviceguard with Serviceguard Extension for RAC: RAC reconfiguration and database recovery..........9

How you can optimize failover time...................................................................................................9

Some help in estimating time for failover.......................................................................................10

MEMBER_TIMEOUT value...........................................................................................................10

Testing.................................................................................................................................11

Lock acquisition (cluster lock, also called tie-breaker or arbitrator)......................................................11

Quorum server considerations..................................................................................................12

Heartbeat subnet.......................................................................................................................12

Network failure detection ...........................................................................................................12

Number of nodes and number of packages...................................................................................12

EMS resources..........................................................................................................................12

Package configuration...............................................................................................................12

System restart options................................................................................................................13

Applications.............................................................................................................................13

Conclusion..................................................................................................................................13

For more information.....................................................................................................................14

Optimizing Serviceguard Failover Time

Version A.11.19 & later

Summary of content (14 pages)

PAGE 1
Optimizing Serviceguard Failover Time Version A.11.19 & later Introduction ................................................................................................................................... 2 The HP Serviceguard failover process................................................................................................. 2 What happens when failover is triggered by a node failure................................................................ 2 Node failure detection....................
PAGE 2
Introduction One of the most important measurements for an effective high availability or mission-critical environment is how much delay the end user notices in the event of a failure. In these environments, certain things need to take place, such as detecting a failure, finding a way to restart the service, ensuring data integrity, and restarting applications and making them available to users. Different business needs require different environments.
PAGE 3
Figure 1. Steps in a failover caused by a failed node— Serviceguard implementation Application recovery Serviceguard failover time Resource recovery (VG, FS, IP) Cluster reformation Cluster component recovery Quiescence Lock acquisition Election Node failure detection Application failover time Note: Diagram is not to scale. Application failover time varies by application.
PAGE 4
Figure 2. Steps in a failover caused by a failed node— Serviceguard with Serviceguard Extension for RAC implementation RAC reconfiguration and database recovery Serviceguard failover time Group membership reconfiguration Cluster reformation Cluster component recovery Quiescence Lock acquisition Election Node failure detection Application failover time Note: Diagram is not to scale.
PAGE 5
It should be noted that before Serviceguard A.11.19, a failed node was allowed to rejoin the cluster during cluster reformation time. But with Serviceguard A.11.19, a failed node is not allowed to rejoin the cluster during reformation time. The failed node can rejoin the cluster only after rebooting. Election of cluster membership After a node believes another node has failed, it begins the cluster re-formation process. The cluster nodes elect the nodes that will be members of a newly re-formed cluster.
PAGE 6
Cluster component recovery In this step, Serviceguard does miscellaneous tasks, such as cluster information synchronization and package determination. If Serviceguard extension for RAC is configured then Serviceguard provides new membership to it. If packages are down due to a failure, Serviceguard determines on which node(s), if any, they should be restarted.
PAGE 7
What happens when failover is triggered by a package failure These are the steps in a failover that is caused by a package failure (rather than a node failure). Figure 3. Steps in a failover caused by package failure— Serviceguard implementation Serviceguard failover time Application recovery Package determination Resource recovery (VG, FS, IP) Resource failure detection Application failover time Note: Diagram is not to scale. Application failover time varies by application.
PAGE 8
Figure 4. Steps in a failover caused by package failure—Serviceguard with Serviceguard Extension for RAC implementation RAC reconfiguration and database recovery Group membership reconfiguration Application failover time Note: Diagram is not to scale. With RAC, the two application-dependent steps of failover are different from the steps with Serviceguard implementations. They are: • Group membership reconfiguration—If there is a change in membership, RAC will start reconfiguration.
PAGE 9
The time required for resource recovery depends on the number of resources and types of resources. Serviceguard implementation: application startup Serviceguard completes the application-dependent part of failover by following commands for the recovery and restart of package applications as defined in package configuration. Some packages and some applications may also do their own recovery before startup. The time required for application startup depends on each application and how it is configured.
PAGE 10
Some help in estimating time for failover The following table can help you estimate the total failover time for your Serviceguard cluster. Table 1.
PAGE 11
If your application takes a long time to recover and restart, set the MEMBER_TIMEOUT conservatively. If your database takes several minutes to recover, it isn’t worth risking an unnecessary failover to shave a few seconds off of the Serviceguard failover time. If your application restarts quickly, you can afford to set the MEMBER_TIMEOUT more aggressively. When your application only takes a few seconds to restart, there is little benefit in waiting a few seconds for an interruption to recover.
PAGE 12
3. Dual LVM volume group locks on FC disks: 13 seconds + time calculated in option 1 above Quorum server considerations If you choose a quorum server, be sure the network between it and the nodes is highly available and reliable. If you encounter delays reaching the quorum server, you can configure a QS_TIMEOUT_EXTENSION, but the extension time adds directly to the lock acquisition time and thus to failover time. Serviceguard calculates the time for the actual lock acquisition.
PAGE 13
Linux, a journaled file system (such as Reiser FS and ext3 FS) is faster than a non-journaled file system (ext2). Adding or removing IP addresses takes some time and affects failover time. On HP-UX 11i, it takes at least one second longer to add an IPv6 address if you enable duplicate address detection (DAD).
PAGE 14
For more information To learn more about HP Serviceguard Solutions for high availability and disaster recovery, please visit: www.hp.com/go/serviceguardsolutions For technical documentation: www.docs.hp.com/hpux/ha © Copyright 2009 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services.