Managing Serviceguard NFS for Linux HP Part Number: T1442-90018 Published: March 2009
Legal Notices © Copyright 2009 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor’s standard commercial license. The information contained herein is subject to change without notice.
Table of Contents 1 Serviceguard NFS for LINUX Introduction...................................................................7 Overview of Serviceguard NFS..............................................................................................................7 Limitations of Serviceguard NFS............................................................................................................7 Overview of the NFS File Lock Migration Feature........................................................
Package Configuration File for pkg03..................................................................................37 Package Control Script for pkg03.........................................................................................38 NFS Toolkit Configuration File for pkg03............................................................................38 Sample Configuration for Modular Packages............................................................................
List of Figures 1-1 1-2 1-3 1-4 3-1 3-2 3-3 3-4 3-5 3-6 3-7 3-8 Simple Failover to an Idle NFS Server..........................................................................................10 Failover from One Active NFS Server to Another........................................................................11 A Host Configured as Adoptive Node for Multiple Packages.....................................................12 Cascading Failover, with Three Adoptive Nodes.......................................
1 Serviceguard NFS for LINUX Introduction This manual describes how to install and configure a Serviceguard NFS toolkit on a Linux system. Serviceguard NFS® is a toolkit that allows you to use Serviceguard to set up highly available NFS servers. The information presented in this manual assumes you are familiar with Serviceguard and NFS operations. Refer to your Serviceguard and/or NFS documentation for additional information.
NOTE: In versions A.02.00 and A.03.01 of the NFS toolkit, you can eliminate the above limitations by enabling the File Lock Migration feature. (For more information, see the “Overview of the NFS File Lock Migration Feature” section). IPv6 addresses are not supported by NFS toolkit for NFS version 3 file lock migration feature. Overview of the NFS File Lock Migration Feature The following describes the File Lock Migration feature, which is part of the NFS toolkit with version A.02.00 and A.03.
statd/sm-notify on Linux does not deal with multi-homed hosts. It does not support multiple IP addresses in sending server reboot notification. • Multiple Package IPs are not supported with NFS A Serviceguard package can support multiple relocatable IP addresses associated with it. But, if lock migration feature is enabled then only one IP should be specified for NFS in the package control script as the NFS server reboot/failover notification can be sent using only one relocatable IP.
Figure 1-1 Simple Failover to an Idle NFS Server Node_A is the primary node for NFS server package Pkg_1. When Node_A fails, Node_B adopts Pkg_1. This means that Node_B locally mounts the file systems associated with Pkg_1 and exports them. Both Node_A and Node_B must have access to the disks that hold the file systems for Pkg_1. Failover between Active Nodes Figure 1-2 shows a failover from one active NFS server node to another active NFS server node.
Figure 1-2 Failover from One Active NFS Server to Another NOTE: During a package failover, the NFS server may receive a status information messages such as: Input/output error, Stale NFS file handle, or Write error: Stale NFS file handle. The result is a retry. If the package fails over during a user file access, the client may experience a momentary hang. Access continues as soon as the package has completed failover to the other node.
Figure 1-3 A Host Configured as Adoptive Node for Multiple Packages Alternatively, you can prevent Node_B from adopting more than one package at a time by setting a package control function in the package control script. With the package control function set, Node_B may adopt the package of the first node that fails, but if the second node fails, Node_B will not adopt its package. The package control function prevents a node from becoming overloaded by adopting too many packages.
Figure 1-4 Cascading Failover, with Three Adoptive Nodes Understanding the Serviceguard NFS Files Serviceguard NFS uses files similar and typical to Serviceguard. These include configuration files, control scripts, monitoring scripts, and templates. As is true for all Serviceguard packages, you configure and view a small number of files. The following is a brief description of the files: • Files that apply to the whole cluster: — Cluster configuration file, cluster.
— — NFS file lock migration script, nfs.flm Toolkit interface script, toolkit.sh This script is invoked by .cntl which invokes the hanfs.sh. — Lock migration script lock_migration.sh This script starts lock migration process in Modular packages — Toolkit module script tkit_module.sh Script used in Modular packages to invoke toolkit.sh — Toolkit configuration generation script. tkit_gen.
Starting File Lock Migration If you call the NFS control script with the lock_migration: parameter after enabling the File Lock Migration feature, the control script does the following: • • • • Populates the /var/lib/nfs/sm directory on SLES and /var/lib/nfs/statd/sm directory on Red Hat with the Status Monitor entries from the configured holding directory of the package, and subsequently removes the entries from the holding directory.
• • Execute any external script specified. In NFS lock_migration.sh is executed if lock migration is enabled in the package configuration file. The NFS script initiates the NFS monitor script to check periodically on the health of NFS services, if you have configured your NFS package to use the monitor script. NOTE: After completing the above sequence, the NFS server is active, and clients can NFS-mount the exported file systems associated with the package.
The monitor script monitors NFS services including: • • • • • • • portmap rpc.statd nfsd rpc.mountd rpc.rquotad, if QUOTA_MON is set to “YES” in hanfs.conf lockd nfs.flm, if LOCK_MIGRATION and NFS_FLM_MONITOR are set to “YES” in hanfs.conf If any of the services are dead or hung, the nfs.mon. will cause the package to fail. NOTE: To configure NFS for maximal availability, you must do the following: • Specify AUTO_RUN=YES in the package configuration file.
2 Installing and Configuring Serviceguard NFS for Linux This chapter explains how to configure Serviceguard NFS. NOTE: You must set up your Serviceguard cluster and make sure NFS server is installed before configuring Serviceguard NFS. For instructions on setting up an Serviceguard cluster, see Chapters 5 and 6 of the Managing HP Serviceguard for Linux user’s guide.
NOTE: The following procedures assume your environment is Red Hat. If your environment is SLES, replace all occurrences of "/usr/local" with /opt The files will be installed in the following directories: /usr/bin and /usr/local/ cmcluster/conf/modules/tkit/nfs, /usr/local/cmcluster/conf/scripts/ tkit/nfs and /usr/local/cmcluster/nfstoolkit for Red Hat. The following files are part of the toolkit: • /usr/local/cmcluster/nfstoolkit/README. Description of the toolkit contents.
# cmmakepkg -m tkit/nfs/nfs pkg.conf NOTE: For more information about creating Modular packaging, see the Modular package support in Serviceguard for Linux and ECM Toolkits white paper available at: http://docs.hp.com -> High Availability. 4. Create a directory for your package files, for example: # mkdir /usr/local/cmcluster/ 5.
4. Extended Distance Cluster which uses the linux software RAID as the basic building block. For information on setting up the Extended Distance Cluster, see the HP Serviceguard Extended Distance Cluster for Linux Deployment Guide. Create a directory for each NFS package. For example: /usr/local/cmcluster/nfs1 5. The names of the volume groups must be unique within the cluster, and the major and minor numbers associated with the volume groups must be the same on all nodes.
1. Except for the variables listed below, use the default values for the variables in the package configuration file, or change them as needed. For instructions on modifying the default values, see the Managing HP Serviceguard for Linux manual, or read the comments in the package configuration file. 2. Set the PACKAGE_NAME variable. For example: PACKAGE_NAME pkg01 Each package must have a unique name. 3. Create a NODE_NAME variable for each node that will run the package.
3. Specify the IP address for the package and the address of the subnet to which the IP address belongs. For example: IP[0]=192.100.112.10 SUBNET[0]=192.100.112.0 The IP address you specify is the relocatable IP address for the package. NFS clients that mount the file systems in the package will use this IP address to identify the server. You should configure a name for this address in the DNS or NIS database, or in the /etc/hosts file. 4.
Do not configure these exported directories in the /etc/exports file. When an NFS server boots up, it attempts to export all file systems in its/etc/exports file. If those file systems are not currently present on the NFS server node, the node cannot boot properly. This happens if the server is an adoptive node for a file system, and the file system is available on the server only after failover of the primary node. 2. If you want to start and monitor rpc.quotad daemon, set QUOTA_MON to YES.
NOTE: The argument to be passed to the HA_APP_SCRIPT for lock migration should contain the same name or IP address used by the clients while mounting the exported file system. Example: If client mounts the NFS file system using the package name, specify the same in customer_defined_run_cmds as shown below: function customer_defined_run_cmds { $HA_APP_SCRIPT lock_migration: test_return 51 } NOTE: A Serviceguard package can support multiple re-locatable IP addresses associated with it.
NOTE: The NFS client may not receive a crash notification if it sends an initial lock request to the NFS server and during the interim, the NFS package failsover to an adoptive node before the FLM script copies the /var/lib/nfs/statd/sm entry on Red Hat and /var/lib/nfs/sm entry on SLES for this client to the package holding directory. Hence the client may not reclaim the lock once the NFS package fails over to the adoptive node.
NOTE: Repeat the configuration process for each NFS package Editing the Package Configuration File (pkg.conf) The following steps describe the required modifications to the Package Configuration file. Make one Package Configuration file for each package. 1. Except for the variables listed below, use the default values for the variables in the package configuration file, or change them as needed.
NOTE: To use the lock migration feature, all the configurations specified above must be done. Also, consider the lock migration feature limitations described in section“Limitations of the NFS File Lock Migration Feature” (page 8). 9. NFS_FLM_HOLDING_DIR: NFS File Lock Migration (FLM) directory is a unique directory created in one of the shared volumes associated with this package. This directory holds copies of the /var/lib/nfs/ sm files on SLES and /var/lib/nfs/statd/sm files on Red Hat for this package.
NOTE: In Red Hat, there are times when sending SIGKILL to the kernel ‘lockd’ thread might not release all the file locks and cause the failure of the unmounting the filesystem. To force unmount the filesystem, the machine has to be restarted. In such cases, it is recommended to set SERVICE_FAIL_FAST_ENABLED to “YES” which reboots the machine upon service failure. In SLES, the SM directory does not get consistently updated with the client entries.
vg vg02 17. Create a separate fs_name, fs_directory, fs_type, and fs_mount_opt variable for each volume group and file system that will be mounted on the server.
3 Sample Configurations This chapter gives sample cluster configuration files, package configuration files, package control script, and NFS control script for configurations supporting the following failover options: • • • • Failover between multiple active nodes. The sample configuration has three servers and three Serviceguard NFS packages and supports a three-server mutual takeover. Each server is the primary node for one package and an adoptive node for the other two packages.
Figure 3-1 Three-Server Mutual Takeover Figure 3-2 shows the three-server mutual takeover configuration after host basil has failed and host sage has adopted pkg02. Dotted lines indicate which servers are adoptive nodes for the packages.
Figure 3-2 Three-Server Mutual Takeover After One Server Fails Cluster Configuration File for Three-Server Mutual Takeover This section shows the cluster configuration file (cluster.conf) for this configuration example. The comments are not shown. CLUSTER_NAME MutTakOvr QS_HOST QS_POLLING_INTERVAL qs 300000000 NODE_NAME NETWORK_INTERFACE HEARTBEAT_IP NETWORK_INTERFACE thyme eth0 192.100.112.146 eth1 NODE_NAME NETWORK_INTERFACE HEARTBEAT_IP basil eth0 192.100.112.
Sample Configuration for Legacy Packages Configuration files of 3 legacy packages are described in this section. These samples are applicable to Red Hat environment. For SLES replace all occurrences of ” /usr/local” with “/opt”. Package Configuration File for pkg01 This section shows the package configuration file (pkg01.conf) for the package pkg01 in this sample configuration. The comments are not shown.
PACKAGE_NAME PACKAGE_TYPE FAILOVER_POLICY FAILBACK_POLICY NODE_NAME NODE_NAME NODE_NAME AUTO_RUN NODE_FAIL_FAST_ENABLED RUN_SCRIPT RUN_SCRIPT_TIMEOUT HALT_SCRIPT HALT_SCRIPT_TIMEOUT SERVICE_NAME SERVICE_FAIL_FAST_ENABLED SERVICE_HALT_TIMEOUT MONITORED_SUBNET MONITORED_SUBNET_ACCESS pkg02 FAILOVER CONFIGURED_NODE MANUAL basil sage thyme YES NO /usr/local/cmcluster/pkg02/pkg02.cntl NO_TIMEOUT /usr/local/cmcluster/pkg02/pkg02.cntl NO_TIMEOUT nfs2.monitor NO 300 192.100.112.
RUN_SCRIPT_TIMEOUT HALT_SCRIPT HALT_SCRIPT_TIMEOUT SERVICE_NAME SERVICE_FAIL_FAST_ENABLED SERVICE_HALT_TIMEOUT MONITORED SUBNET MONITORED_SUBNET_ACCESS NO_TIMEOUT /usr/local/cmcluster/pkg03/pkg03.cntl NO_TIMEOUT nfs3.monitor NO 300 192.100.112.0 FULL Package Control Script for pkg03 This section shows the NFS control script (pkg03.cntl) for the package pkg03 in this sample configuration.
monitored_subnet_access full service_name nfs1.monitor service_cmd "$SGCONF/scripts/tkit/nfs/tkit_module.sh nfs_monitor" service_restart none service_fail_fast_enabled no service_halt_timeout 300 external_script $SGCONF/scripts/tkit/nfs/lock_migration.sh ip_subnet 192.100.112.0 #ip_subnet_node ip_address 192.100.112.
node_name thyme node_name basil auto_run yes node_fail_fast_enabled no script_log_file /usr/local/cmcluster/pkg03/log TKIT_DIR /usr/local/cmcluster/pkg03 XFS[0] "-o rw *:/hanfs/nfsu031" QUOTA_MON YES LOCK_MIGRATION NO monitored_subnet 192.100.112.0 monitored_subnet_access full service_name nfs3.monitor service_cmd "$SGCONF/scripts/tkit/nfs/tkit_module.sh nfs_monitor" service_restart none service_fail_fast_enabled no service_halt_timeout 300 external_script $SGCONF/scripts/tkit/nfs/lock_migration.
NOTE: Setting the package control option (-d) of the cmmodpkg command, prevents host sage from adopting another package, so host sage is no longer an adoptive node for pkg01. This prevent the adoptive node (sage) from becoming overloaded when multiple packages failover. Figure 3-4 One Adoptive Node for Two Packages After One Server Fails Cluster Configuration File for Adoptive Node for Two Packages This section shows the cluster configuration file (cluster.conf) for this configuration example.
AUTO_START_TIMEOUT NETWORK_POLLING_INTERVAL 600000000 2000000 MAX_CONFIGURED_PACKAGES 4 Sample Configuration for Legacy Packages Configuration files of two legacy packages are described in this section. These samples are applicable to Red Hat environment. For SLES user can replace all occurrences of “/usr/local” with “/opt”. Package Configuration File for pkg01 This section shows the package configuration file (pkg01.conf) for the package pkg01 in this sample configuration. The comments are not shown.
Also, add cmmodpkg command with package control option (-e ) in the customer_defined_halt_cmds for enabling the package pkg02 when the package pkg01 is halted. For example: function customer_defined_halt_cmds { cmmodpkg -e -n `hostname` pkg02 & } This package control function can prevent an adoptive node from becoming overloaded when multiple packages fail over. If an adoptive node becomes overloaded, it can fail.
SERVICE_FAIL_FAST_ENABLED SERVICE_HALT_TIMEOUT SUBNET NO 300 192.100.112.0 Package Control Script for pkg02 This section shows shows the package control script (pkg02.cntl) for the package pkg02 in this sample configuration. Only the user-configured part of the script is shown; the executable part of the script and most of the comments have been omitted.
NOTE: The above changes in the customer_defined_halt_cmds and customer_defined_run_cmds must be done only in the package control scripts of the adoptive node and not on the package control script in the primary node. If there are more than 2 packages the user may repeat the cmmodpkg command as many times required with other package names. User may consider making FAILBACK_POLICY as AUTOMATIC to free the adoptive node as early as possible so that adoptive node is ready to accept pkg01 in case it fails.
external_script.template in Red Hat and at /opt/cmcluster/conf/examples/ external_script.template in SLES. Copy and rename the external_script.template to the package directory with the following command: # cp /usr/local/cmcluster/conf/examples/external_script.template \ /usr/local/cmcluster/pkg01/external_script.sh Add the additional external script into the package configuration file. For Example: external_script /usr/local/cmcluster/pkg01/external_script.
auto_run yes node_fail_fast_enabled no script_log_file /usr/local/cmcluster/pkg02/log TKIT_DIR /usr/local/cmcluster/pkg02 XFS[0] "-o rw *:/hanfs/nfsu021" QUOTA_MON YES LOCK_MIGRATION NO monitored_subnet 192.100.112.0 monitored_subnet_access full service_name nfs2.monitor service_cmd "$SGCONF/scripts/tkit/nfs/tkit_module.sh nfs_monitor" service_restart none service_fail_fast_enabled no service_halt_timeout 300 external_script $SGCONF/scripts/tkit/nfs/lock_migration.sh ip_subnet 192.100.112.
The cmmodpkg command in the background allows the control script to complete and finish bringing up the package. There is a small window of time, during which if one package has begun to fail over but the cmmodpkg command has not executed, the other package can fail over and the host will adopt it. In other words, if two packages fail over at approximately the same time, a host may adopt both packages, even though the package control option is specified.
Figure 3-6 Cascading Failover with Three Servers After One Server Fails Cluster Configuration File for Three-Server Cascading Failover This section shows the cluster configuration file (cluster.conf) for this configuration example. The comments are not shown. CLUSTER_NAME Cascading QS_HOST QS_POLLING_INTERVAL qs 300000000 NODE_NAME NETWORK_INTERFACE HEARTBEAT_IP NETWORK_INTERFACE thyme eth0 192.100.112.146 eth1 NODE_NAME NETWORK_INTERFACE HEARTBEAT_IP basil eth0 192.100.112.
Sample Configuration for Legacy Packages Configuration files of two legacy packages are described in the following section. These samples are applicable to Red Hat environment. For SLES user can replace all occurrences of “/usr/ local” with “/opt”. Package Configuration File for pkg01 This section shows the package configuration file (pkg01.conf) for the package pkg01 in this sample configuration. The comments are not shown.
PACKAGE_NAME pkg02 PACKAGE_TYPE FAILOVER FAILOVER_POLICY CONFIGURED_NODE FAILBACK_POLICY MANUAL NODE_NAME thyme NODE_NAME sage NODE_NAME basil AUTO_RUN YES NODE_FAIL_FAST_ENABLED NO RUN_SCRIPT /usr/local/cmcluster/pkg02/pkg02.cntl RUN_SCRIPT_TIMEOUT NO_TIMEOUT HALT_SCRIPT /usr/local/cmcluster/pkg02/pkg02.cntl HALT_SCRIPT_TIMEOUT NO_TIMEOUT SERVICE_NAME nfs2.monitor SERVICE_FAIL_FAST_ENABLED NO SERVICE_HALT_TIMEOUT 300 MONITORED SUBNET 192.100.112.
failback_policy manual node_name thyme node_name sage auto_run yes node_fail_fast_enabled no script_log_file /usr/local/cmcluster/pkg01/log TKIT_DIR /usr/local/cmcluster/pkg01 XFS[0] "-o rw *:/hanfs/nfsu011" QUOTA_MON YES LOCK_MIGRATION NO monitored_subnet 192.100.112.0 monitored_subnet_access full service_name nfs1.monitor service_cmd "$SGCONF/scripts/tkit/nfs/tkit_module.
if a host is an adoptive node for both pkg01 and pkg02, disabling of pkg02, in the external script for pkg01, would prevent the host that is running pkg01 from adopting pkg02, and once the package pkg01 is halted in the node the pkg02 is enabled by using the command cmmodpkg –e in the function stop_command. The ampersand (&) causes the cmmodpkg command to run in the background. The cmmodpkg command in the background allows the control script to complete and finish bringing up the package.
SLES. Copy and rename the external_script.template to the package directory with the following command. # cp /usr/local/cmcluster/conf/examples/external_script.template \ /usr/local/cmcluster/pkg02/external_script.sh Add the additional external script into the package configuration file. For example: external_script /usr/local/cmcluster/pkg02/external_script.sh Modify the external script in the adoptive node. Specify the cmmodpkg command with the package control option (-d) in the function start_command.
Figure 3-7 One Adoptive Node for an NFS Package Figure 3-8 shows the same configuration after one primary server has failed. Figure 3-6 shows this sample configuration after host basil has failed. Host sage has adopted pkg02. Figure 3-8 NFS Package on Adoptive Node After One Server Fails Cluster Configuration File for Adoptive Node for NFS package with File Lock Migration This section shows an example of the cluster configuration file (cluster.conf) for NFS package with File Lock Migration.
CLUSTER_NAME PkgCtrl QS_HOST qs QS_POLLING_INTERVAL 300000000 NODE_NAME thyme NETWORK_INTERFACE eth0 HEARTBEAT_IP 192.100.112.146 NETWORK_INTERFACE eth1 NODE_NAME sage NETWORK_INTERFACE eth0 HEARTBEAT_IP 192.100.112.
The function customer_defined_run_cmds calls the toolkit.sh script with lock_migration: parameter for file lock migration. The argument to be passed to the HA_APP_SCRIPT for lock migration should contain the same name or IP address used by the clients while mounting the exported file system. NFS Toolkit Configuration File for pkg01 (hanfs.conf) This section shows the NFS toolkit configuration file (hanfs.conf) for the package pkg01 on this sample configuration.
Index Symbols -d option, cmmodpkg, 24 A adoptive nodes, 9 configuring, 23 example of cascading failover, 48 example of package control option, 40 for multiple packages, 9, 11, 24, 40 illustration of cascading failover, 12 AUTO_RUN, 23 automounter timeout, 17 B binary configuration files, creating, 31 C cascading failover, 9 example configuration, 48 illustration of, 12 client behavior, 7, 17 cluster configuration file (cluster.
logging, 17 specified in nfs.conf, 23 starting, 14 stopping, 15 unconfiguring, 23 mount points, 22 mount retry, 17 mounting file systems, 14 Multiple node cascading failover, 33 mutual takeover sample configuration, 33 R N S NFS client behavior, 7, 17 NFS control script (nfs.cntl), 14, 21, 23 default values, 24 example, 36, 37, 38, 42, 44, 50, 51 specified in nfs.conf, 23 NFS monitor script (nfs.mon), 16 logging, 17 specified in nfs.