Technical Considerations for a Serviceguard Cluster that Spans Multiple IP Subnets, July 2009

ManualsBrandsHP ManualsSoftwareHP Serviceguard for Linux ProLiant Cluster

Introduction......................................................................................................................................... 3

Executive summary........................................................................................................................... 3

Related documents ........................................................................................................................... 3

Definition of terms ............................................................................................................................ 4

Defining cross-subnet support ................................................................................................................ 4

Benefits and challenges of cross-subnet support ....................................................................................... 6

Benefits of a cross-subnet network architecture in a Serviceguard cluster................................................. 6

Challenges for applications in a cross-subnet Serviceguard cluster ......................................................... 6

Network configuration requirements and recommendations ...................................................................... 6

Network domain requirements........................................................................................................... 7

Heartbeat subnet requirements........................................................................................................... 7

Package subnet requirements ............................................................................................................ 8

IP Monitor recommendations ............................................................................................................. 8

Examples of network configuration requirements.................................................................................. 9

Example 1: Dual heartbeat subnet requirement ................................................................................ 9

Example 2: Heartbeat subnets must be routed statically and separately ............................................ 10

Example 3: Package subnet requirements...................................................................................... 12

Example 4: Package subnet IP Monitor recommendations ............................................................... 13

Advantages of modular packages in cross-subnet clusters....................................................................... 14

General application integration considerations ..................................................................................... 14

Handling of application failures from client’s perspective.................................................................... 14

The client is aware of two different destinations.............................................................................. 15

The client is not aware of two potential destinations........................................................................ 15

Name resolution update upon cross-subnet failure.............................................................................. 15

Dynamic DNS............................................................................................................................ 16

Global Server Load Balancer (GSLB)............................................................................................. 17

Oracle in a cross-subnet cluster ........................................................................................................... 18

Client access to Oracle databases ................................................................................................... 18

Client access to a single instance Oracle database ........................................................................ 19

Client access to Oracle RAC instances.......................................................................................... 20

Fast Application Notification (FAN) .............................................................................................. 21

Oracle RAC specific network configuration ....................................................................................... 22

NFS in a cross-subnet cluster ............................................................................................................... 22

Known limitations with the Serviceguard NFS Toolkit in a cross-subnet cluster........................................ 23

Potential workarounds for using NFS in a cross-subnet cluster.............................................................. 23

Workaround 1: Forcible Unmount ................................................................................................ 23

Workaround 2: Automatically unmount idle file systems via AutoFS.................................................. 24

Technical Considerations for a Serviceguard

Cluster that Spans Multiple IP Subnets

July 2009

Summary of content (27 pages)

PAGE 1
Technical Considerations for a Serviceguard Cluster that Spans Multiple IP Subnets July 2009 Introduction......................................................................................................................................... 3 Executive summary ........................................................................................................................... 3 Related documents ........................................................................................................
PAGE 2
Workaround 3: NFS Client-side Failover ....................................................................................... 24 SAP with SGeSAP in a cross-subnet cluster............................................................................................ 25 Known limitations with the SGeSAP in a cross-subnet cluster................................................................ 25 Potential workarounds for using SAP in a cross-subnet cluster ........................................................
PAGE 3
Introduction Executive summary In earlier releases, Serviceguard has required that all nodes in the cluster be on the same IP subnet and that all the interfaces on the same subnet must be connected at the link level. This requirement could be restrictive, particularly for clusters that extend across sites separated by a long distance, or for clusters on a single site that have to follow strict security policies.
PAGE 4
Definition of terms The following terms and abbreviations are used in this whitepaper. Table 1 – Terms and definitions Term Definition AutoFS A client site NFS service that automatically handles NFS mount and umount CTF Connect Time Failover – Oracle client failover mechanism utilizing multiple protocol entries in tnsnames.
PAGE 5
Note: Cross-subnet support does not include automatic notification of a change to the application servers’ IP address to the application clients. Figure 1 shows a non-cross-subnet cluster with two dedicated heartbeat networks. Each heartbeat network (5.1.1.0 and 5.1.2.0) is a single subnet that exists on all cluster nodes. Figure 1 – A non cross-subnet cluster where the subnets are the same on all the nodes Figure 2 shows a cross-subnet cluster with two dedicated heartbeat networks.
PAGE 6
Benefits and challenges of cross-subnet support Benefits of a cross-subnet network architecture in a Serviceguard cluster Cross-subnet support is particularly beneficial for clusters spanning multiple locations such as Metroclusters and some Extended Distance Cluster (EDC) configurations. EDC with CVM/CFS are currently not supported in a cross-subnet configuration since CVM requires level two connectivity between all cluster nodes.
PAGE 7
Network domain requirements Serviceguard recognizes only the hostname element of a fully-qualified domain name. Serviceguard now supports different subnets on different nodes but does not support different network domains.  This means that all the nodes in the cluster must belong to the same network domain. Heartbeat subnet requirements Heartbeat subnets in a cross-subnet cluster must meet the following requirements:  There must be less than 200 milliseconds of latency in the heartbeat network.
PAGE 8
Package subnet requirements To enable Serviceguard’s network monitor to detect failures despite the lack of link-level remote polling (which is not possible across routers), package subnets must meet the following requirement:  Each package subnet must be configured with a standby interface on the local bridged net unless you use the Serviceguard IP Monitor. Link-level polling between the primary interface and the standby interface on the local node ensures that network failures are detected appropriately.
PAGE 9
When the IP Monitor is configured for a package subnet, you are not required, for network monitoring purposes, to have a standby interface for the package subnet. However, a standby interface for the package subnet provides high availability for the package subnet. For more information about the IP Monitor, see “Monitoring LAN Interfaces and Detecting Failure: IP Level” in Chapter 3 of the “Managing Serviceguard” manual, which you can find on http://docs.hp.com under High Availability  Serviceguard.
PAGE 10
configured with subnet 5.1.1.0, and vice versa. Similarly, any IP packet targeted to 5.2.2.0 must be routed out of the interface configured with subnet 5.1.2.0.  The heartbeat subnets 5.1.2.0 and 5.2.2.0 have a physical standby path with standby interfaces and routers. This physical standby path does not qualify as a separate heartbeat path on its own, hence the other heartbeat path with subnets 5.1.1.0 and 5.2.1.0 is needed.  There are no backup routers for the dedicated heartbeat subnets 5.1.1.0 and 5.
PAGE 11
Incorrect configuration of heartbeat routing The following excerpts from the HP-UX file /etc/rc.config.d/netconf on Node A and Node C show a routing configuration that does not meet the requirement for separately routed heartbeat subnets. In these excerpts, green text indicates correct routing entries whereas red text indicates an incorrect routing gateway. Table 2 – Configuration file showing incorrect heartbeat routing Configuration file: /etc/rc.config.d/netconf Node A Node C IP_ADDRESS[0]=5.1.1.
PAGE 12
/etc/rc.config.d/netconf as shown in the excerpt. Note that the static routing configuration in this example is applicable to HP-UX only. Static routing setup may be different in Linux. In this correct configuration, heartbeat packets from subnet 5.1.2.0 to subnet 5.2.2.0 are routed through lan2 and its own router gateway (R2) 5.1.2.254. On node C, heartbeat packets from subnet 5.2.2.0 to subnet 5.1.2.0 are routed through lan2 and its own router gateway (R2) 5.2.2.254.
PAGE 13
Figure 7 – Recommended configuration on a multi-site cross-subnet cluster with dual dedicated heartbeat subnets and a separate package subnet 3rd Location Quorum Service Gateway IPs 5.1.1.254 5.1.2.254 Highly available network for user, client and quorum server access Datacenter 1 5.1.1.0 5.1.2.2 Node A 5.2.1.0 5.2.2.0 R4 R5 R2 R3 5.1.2.1 Datacenter 2 R1 5.1.2.0 5.1.1.1 Gateway IPs 5.2.1.254 5.2.2.254 5.1.1.2 Node B 5.1.3.1 5.2.2.2 5.2.1.1 5.2.2.1 Node C 5.1.3.2 5.2.3.1 5.1.3.0 5.2.
PAGE 14
Advantages of modular packages in cross-subnet clusters As mentioned in the section Challenges for applications in a cross-subnet Serviceguard cluster, upon failover of an application package from one subnet to another, the relocatable IP address which is bound to the package changes. Both legacy and modular packages in Serviceguard enable automatic failover of packages across subnets by means of the new parameter monitored_subnet_access.
PAGE 15
o details on individual timeouts and methods of how to set them on HP-UX are documented in “HP-UX 11i TCP/IP Performance White Paper”. When an error condition is encountered, clients should keep trying to reconnect to the clustered application through its relocatable IP address until the application is eventually up and running on the adoptive node.
PAGE 16
 Updating local system configuration files (e.g.: /etc/hosts) on all involved systems seems to be cumbersome in a larger environment, but takes immediate effect on the updated computer – client system. E.g. as soon as /etc/hosts is updated on a system, new name resolution queries will return the updated information with the new relocatable IP address. Updating local /etc/hosts files seems practical only in an environment with a limited number of client systems.
PAGE 17
resolution requests at the cost of using old records for a long time. Setting it to a low value (e.g. a few seconds) reduces the time old records might be used at the cost of increased number of name resolution requests at the client and a higher load on the DNS server. In a cross-subnet configuration this parameter might be best set at value that is less than the time it takes for an application to fail over to a cluster node in a different subnet.
PAGE 18
“15.84.120.5 ” will return “db-srv1.hp.com” and not “db-srv.hp.com”. Clients and applications need to be analyzed if they require a consistent reverse lookup. Oracle in a cross-subnet cluster Oracle single-instance and RAC databases can be protected in a cross-subnet Serviceguard cluster. The Enterprise Cluster Master Toolkit (ECMT) simplifies the integration of a single-instance Oracle database in a Serviceguard cluster.
PAGE 19
Figure 8 – Example tnsnames.ora configuration file for a single instance and a RAC database in non cross-subnet configuration. DB_SRV = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = (CONNECT_DATA = (SERVICE_NAME = DB_SRV) ) ) DB_RAC = (DESCRIPTION = (ADDRESS_LIST= (FAILOVER = on) (LOAD_BALANCE = on) (ADDRESS = (PROTOCOL = TCP)(HOST (ADDRESS = (PROTOCOL = TCP)(HOST (ADDRESS = (PROTOCOL = TCP)(HOST (ADDRESS = (PROTOCOL = TCP)(HOST ) (CONNECT_DATA= (SERVICE_NAME = DB_RAC) ) ) db-srv.hp.
PAGE 20
database and a domain name update for db-srv.hp.com is not needed on a cross-subnet package failure. Figure 9 shows the sample tnsnames.ora file for this configuration. Figure 9 – Example tnsnames.ora configuration file for a single instance Oracle database with dedicated protocol address entries DB_SRV = (DESCRIPTION = (ADDRESS_LIST= (FAILOVER = on) (ADDRESS = (PROTOCOL = TCP)(HOST = db-srv1.hp.com)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = db-srv2.hp.
PAGE 21
To automatically reconnect clients to the database after a cross-subnet failover, the Oracle Net service names (e.g.: rac1vip1 – rac2vip2) must include VIPs configured at both sub-clusters. Figure 10 shows a sample tnsnames.ora file. Figure 10 – Example tnsnames.
PAGE 22
remoteport=4200 nodes=rac1vip1.hp.com:4200, rac1vip2.hp.com:4200, \ rac2vip1.hp.com:4200, rac2vip2.hp.com:4200 The database nodes publish FAN events which are received by subscribers who then act on it. The easiest way to use (subscribe to) FAN events is to use an integrated Oracle client with your application. These are some of the integrated clients: OCI Session Pools, Connection Manager (CMAN) session pools, JDBC connection pools and Oracle NET Listener.
PAGE 23
This section of the white paper discusses the issues involved with the failover of an NFS server package in a cross-subnet Serviceguard cluster. This paper does not address the concept of a Serviceguard failover involving NFS client systems.
PAGE 24
Most applications do not tolerate having an active file system forcibly removed from the system, even a file system that is not responding to requests. HP therefore strongly recommends that any applications using the hung NFS file system be completely shut down prior to issuing the “ umount –f” command. Once the applications have been stopped and the NFS file systems forcibly un-mounted, the file systems may be re-mounted using the new virtual IP address of the package running in the adoptive subnet.
PAGE 25
the local subnet. Should a failover occur, the NFS client will eventually detect that the primary server is not responding and automatically begin sending requests using the FQDN of the NFS package in the adoptive subnet. If your applications only require read access to NFS file systems, using client-side failover may be a viable solution for automated recovery in a cross-subnet cluster environment.
PAGE 26
documentation and Serviceguard support matrix for updates on the support status or contact your HP support representative. An obvious workaround to the cached name resolution issue is to restart the entire SAP application stack if one of the instances fails over to a different subnet. Stopping the SAP application instance would also be necessary to implement the NFS forcibly un-mount workaround as described in the NFS section.
PAGE 27
 Same domain name for all nodes in the cluster  Maximum 200 millisecond latency in the heartbeat network  Nodes fully connected at the IP level  Minimum of two separate, statically routed heartbeat subnets  Standby interfaces on each node for each package subnet or IP Monitor configured for each package subnet  Packages configured to fail over across subnets should be modular rather than legacy Cross-subnet configurations provide considerable benefits in the area of network architecture and administra