HP Serviceguard Toolkit for Oracle on Linux User Guide Version A.12.00.00 Abstract This guide describes instructions for configuring an Oracle single instance in a Serviceguard cluster on Linux and to integrate an Oracle database server with HP Serviceguard for Linux.
© Copyright 2014 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license. The information contained herein is subject to change without notice.
Contents 1 Overview..................................................................................................5 Advantages.............................................................................................................................5 Prerequisites.............................................................................................................................5 Hardware support....................................................................................................
A Sample package configuration file for Oracle database modular package without ASM..........................................................................................................42 B Sample package configuration file for Oracle ASM instance package and database modular package..........................................................................43 Glossary....................................................................................................45 Index...............................
1 Overview HP Serviceguard toolkit for Oracle on Linux enables you to configure an Oracle single instance in a Serviceguard cluster on Linux and integrate an Oracle database server with HP Serviceguard for Linux (SG/LX). This toolkit consists of a set of scripts that are used to start, stop, and monitor an Oracle database instance, ASM instance, and the configured listeners.
2 Installing and uninstalling HP Serviceguard toolkit for Oracle This chapter covers the details related to installing, uninstalling, and upgrading the Oracle toolkit. Installing Oracle toolkit Oracle toolkit RPM must be installed on all the nodes of the cluster that is configured to run the Oracle toolkit package. For a fresh installation of the Oracle toolkit, run the following command: rpm -ivh serviceguard-oracle-toolkit-..noarch.
Uninstalling Oracle toolkit To uninstall the Oracle toolkit: • To search the version of Oracle toolkit, run the following command: # rpm –qa | grep oracle-toolkit • To remove the earlier version of Oracle toolkit, run the following command: # rpm –e Files and directories created for HP Serviceguard toolkit for Oracle after installation The following table describes the scripts and files that are installed.
Table 1 Scripts for HP Serviceguard toolkit for Oracle File name Description haoracle.sh This script contains a list of internally $SGROOT/oracletoolkit used variables and functions that support the starting and stopping of an Oracle database or ASM instance. The toolkit.sh script invokes the haoracle.sh script to perform the following: • On package startup, it starts the database or ASM instance. It also starts the monitor processes.
Table 1 Scripts for HP Serviceguard toolkit for Oracle (continued) File name Description Available in directory Oracle instance for possible “hung” state. hadbhang.mon script in turn uses hagetdbstatus.sh and hatimeoutdbhang.sh hagetdbstatus.sh It is used to check the database status. hatimeoutdbhang.sh This script checks whether a process with the specified PID exists. If the process exists, then it kills the process and exits. toolkit.
3 Supported configuration This section explains the following supported configurations for Oracle database toolkit packages: • Without Automatic Storage Management (ASM) • With Automatic Storage Management Without Automatic Storage Management This section explains how to configure an Oracle database instance package without ASM in a Serviceguard environment.
With Automatic Storage Management ASM instance can be managed with the HP Serviceguard toolkit for Oracle. This toolkit makes use of the multi-node package and package dependency feature to integrate ASM with Serviceguard. MNP (Multi-node Package) is used to encapsulate the per node ASM instances, with one or more Oracle database instance failover packages dependent on this MNP.
4 Using HP Serviceguard toolkit for Oracle After installing the HP Serviceguard toolkit for Oracle, you can use this toolkit to: • Deploy Oracle ASM, Oracle database instances in Serviceguard cluster environment. • Manage the packages. Configuring modular Oracle ASM instance and Database instance toolkit packages This section describes the steps to configure Oracle ASM instance and Database instance toolkit packages.
Table 2 Lists the attributes that you can edit (continued) Attribute Description Example START_MODE Oracle database startup mode. You open can set this parameter to mount, nomount, or open. The default value is open. LISTENER If this attribute is set to no, you must yes comment the LISTENER_NAME and oracle_listener_service attributes. For example: #LISTENER_NAME The default value is yes. LISTENER_NAME This parameter holds the names of the Oracle listener processes.
Table 2 Lists the attributes that you can edit (continued) Attribute Description Example MONITOR_PROCESSES This is the list of all the critical Oracle processes for an instance that must be executing to assume that the instance is up and running. You can add more instance processes. The processes mentioned will be monitored by default.
Table 2 Lists the attributes that you can edit (continued) Attribute Description Example If abort is specified, the instance is shutdown using the abort option. If immediate is specified, the instance is shutdown using the immediate option. The default value is abort. KILL_ASM_FOREGROUNDS If ASM is being used, this parameter no specifies whether any ASM foreground processes having file descriptors open on the dismounted disk group volumes need to be killed or not.
Table 2 Lists the attributes that you can edit (continued) Attribute Description Example tkit/oracle/tkit_module.sh oracle_hang_monitor 30 alert. By default, email_id is commented. 5. Edit the following service parameters, if required: • • • 6. Monitor service for Oracle processes: Attribute Name Example service_name oracle_service service_cmd "$SGCONF/scripts/tkit/oracle/ tkit_module.
Configuring modular ASM instance package NOTE: The following conditions are applicable for the Oracle Toolkit to support the MetroCluster and ContinentalCluster: • The Oracle ASM MNP package must not be configured as a Metrocluster or Continentalclusters package. • The appropriate Metrocluster or Continentalclusters modules must be included when creating database package. To configure a modular ASM instance Multi-node package (MNP) and Database instance failover package: 1.
Attribute Description Example ASM_DISKGROUP If ASM (Automatic Storage diskgroup2 Management) is being used to manage the storage for the database instance, configure this parameter to indicate the ASM Diskgroups used by the database instance. You can add more than one disk group by adding to the array variable, ASM_DISKGROUP. ASM_VOLUME_GROUP Defines volume groups used in the vgora_asm ASM disk groups for this database instance. You can add more than one volume group by adding to this array variable.
Attribute Description Example instance processes. The processes • asm_lgwr_${ORACLE_ASM_SID} mentioned will be monitored by • asm_reco_${ORACLE_ASM_SID} default. MONITOR_INTERVAL The time interval, in seconds, this script waits between checks to ensure that ASM instance is running. The default value is 30 seconds. TIME_OUT The time period, in seconds, this 30 toolkit waits for a completion of a normal shutdown before initiating forceful halt of the application.
Attribute Description Example immediate option. The default value is abort. KILL_ASM_FOREGROUNDS If ASM is being used, this yes parameter specifies whether any ASM foreground processes having file descriptors open on the dismounted disk group volumes must be killed or not. You can set this parameter to yes or no. The default value is yes. Listener You must set this parameter to no. no email_id This parameter is used to provide abc@domain-name.
NOTE: Remove the Oracle listener and Oracle DB hang monitor services for the ASM instance package, shown in the tables below. Table 3 Attribute Name Example service_name oracle_listener_service service_cmd "$SGCONF/scripts/tkit/oracle/tkit_module.sh oracle_monitor_listener" service_restart none service_fail_fast_enabled no service_halt_timeout 300 Table 4 f. Attribute Name Example service_name oracle_hang_service service_cmd "$SGCONF/scripts/tkit/oracle/tkit_module.
Attribute Description Example SID_NAME The Oracle session name. It is also ORACLE_TEST0 called the session ID (SID). It uniquely identifies an Oracle database instance. NOTE: If you modify this attribute after the # cmmakepkg -i -m -t command is run, you must reconfigure the MONITOR_PROCESSES attribute defined in this file to reflect the same. Also, reconfigure PFILE attribute defined in this file to reflect this change, if PFILE is using this attribute.
Attribute Description Example NOTE: If you modify this attribute after the # cmmakepkg -i -m -t command is run, you must reconfigure the MONITOR_PROCESSES attribute to reflect the same. Also, reconfigure PFILE attribute to reflect the change if PFILE is using this attribute. If this attribute is set to no, you must comment the LISTENER_NAME and oracle_listener_service attributes. For example: #LISTENER_NAME LISTENER yes The default value is yes.
Attribute Description Example MONITOR_PROCESSES This is the list of all the critical Oracle processes for an instance that must be executing to assume that the instance is up and running. You can add more instance processes.
Attribute Description Example If abort is specified, the instance is shutdown using the abort option. If immediate is specified, the instance is shutdown using the immediate option. The default value is abort. KILL_ASM_FOREGROUNDS If ASM is being used, this yes parameter specifies whether any ASM foreground processes having file descriptors open on the dismounted disk group volumes need to be killed or not. You can set this parameter to yes or no. The default value is yes.
Attribute Description Example email_id parameter can have one of the following values: • A complete email address • An alias • A distribution list You can list multiple recipients by repeating the email_id field and address. Toolkit sends an alert email on listener and package failure. If configured, the alert mail is sent for Oracle database hang detection. For example: $SGCONF/scripts/tkit/ oracle/tkit_module.sh oracle_hang_monitor 30 alert. By default, email_id is commented. e.
f. g. Attribute Name Example service_restart none service_fail_fast_enabled No service_halt_timeout 300 To configure the dependency for the modular Oracle ASM database package, you must set the dependency for Oracle DB package on Oracle ASM instance package.
NOTE: You can ignore the errors or warnings in the package log file when the package is set to maintenance mode. 3. Disable the maintenance mode for the package using the cmmodpkg command. cmmodpkg -m off NOTE: Ensure that an application instance is up and running before disabling the maintenance mode.
Table 5 List of Attributes Supported for Online Toolkit Package Modification (continued) Attribute Description MONITOR_INTERVAL Oracle Toolkit supports online modification of this attribute. TIME_OUT Oracle Toolkit supports online modification of this attribute. PARENT_ENVIRONMENT Oracle Toolkit supports online modification of this attribute. CLEANUP_BEFORE_STARTUP Oracle Toolkit supports online modification of this attribute.
1. While package is up and running, retrieve the current package configuration values using the command cmgetconf. For example: #cmgetconf -p <> <> 2. Backup this latest configuration file using a copy utility. This allows you to refer to the previous configuration values. For example: # cp -p <> <> 3. 4. Edit the <> file to modify the required attribute values.
Table 6 Modules affected during online addition (continued) Add hosttags for node test1.ind.hp.com: #vgchange --addtag test1.ind.hp.com vg_dd1 Activate the disk: #vgchange -a y vg_dd1 Run the following commands: #e2fsck -y /dev/vg_dd1/lvol1 #mount -t ext3 /dev/vg_dd1/lvol2 /mnt1 Adding an IP sg/package_ip to the package (package_ip.sh) If an IP address which is added to the package configuration failed to add or not attempted, use the cmmodnet command to add the IP address.
# cmhaltpkg Deleting a package To delete a package from the cluster, run the following command: # cmdeleteconf -p This command prompts for a confirmation, before removing the package configuration from the cluster binary file, unless you use the -f option.
5 Oracle toolkit features Alert mail notification This feature facilitates you to configure email address to get the alert notification from toolkit package. Alert mails are sent to configured email addresses in case of failure scenarios. It can be a resource failure or altogether database failure that results in package failure. Toolkit sends an alert email with failure messages in the following scenarios: • Listener failure.
Run the cmcheckconf –P command to perform the package verification irrespective of the state of the package in the cluster. The following are the steps to initiate the package verification by using the cmcheckconf –P command: 1. While the package is running, retrieve the current package configuration values using the cmgetconf command. For example: # cmgetconf -p 2. Run the cmcheckconf command to verify the package configuration file.
ERROR: Oracle ASM user \"$ORACLE_ASM_USER\" doesn't exist. Solution: Create the Oracle and/or ASM instance users with user ID and Group ID same as on other package nodes. • Verify whether the environment variables ORACLE_HOME and ORACLE_SID are set for the Oracle user. If both these variables are not set, the console displays the following info messages: INFO: ORACLE_HOME environment variable is not set for "${ORACLE_ORACLE_ADMIN}" user.
6 Supporting Oracle features Support for multiple listeners This section explains support for multiple listeners with the HP Serviceguard toolkit for Oracle. It enables you to configure: • Single service to monitor all the listeners together • A separate service to monitor each listener NOTE: In a single package, it is not possible to configure the services using both the approaches. You must either configure all the listeners using a single service, or use a separate service for each listener.
A separate service to monitor each listener This service is recommended if listeners are critical. This approach of packaging listeners provides high availability to listeners. If any one of the configured listeners fail, it leads to a failure of the listener service, and the package fails over to an alternate node. In this case, the default service_cmd in the package configuration file must be modified to include the listener name.
Example: • tkit/oracle/oracle/DB_SERVICE • tkit/oracle/oracle/DB_SERVICE • tkit/oracle/oracle/DB_SERVICE Support for database hang detection You can configure a separate service that will detect the database hang and take appropriate action. Database hang is detected by connecting to the database and checking its status.
7 Troubleshooting This section explains some of the problem scenarios that you might encounter while using the HP Serviceguard toolkit for Oracle in a Serviceguard cluster. Problem scenario Possible cause Recommended action Oracle database toolkit packages Oracle database failed to start on this Look at the sub-directories for database startup fails, and package log contains node.
8 Support and other resources Information to collect before contacting HP Ensure that the following information is available before you contact HP: • Software product name • Hardware product model number • Operating system type and version • Applicable error message • Third-party hardware or software • Technical support registration number (if applicable) How to contact HP Use the following methods to contact HP technical support: • In the United States, see the Customer Service / Contact HP Un
Ctrl+x or Ctrl-x A key sequence that indicates you must hold down the keyboard key labeled Ctrl while you press the letter x. ENVIRONMENT VARIABLE The name of an environment variable, for example, PATH. Key The name of a keyboard key. Return and Enter both refer to the same key. Term A term or phrase that is defined in the body text of the document, not in a glossary. User input Indicates commands and text that you type exactly as shown.
A Sample package configuration file for Oracle database modular package without ASM # ********************************************************************** # ****** HIGH AVAILABILITY PACKAGE CONFIGURATION FILE (template) ******* # ********************************************************************** # ******* Note: This file MUST be edited before it can be used. ******** # * For complete details about package parameters and how to set them, * # * consult the Serviceguard manual.
B Sample package configuration file for Oracle ASM instance package and database modular package ASM multi-node package configuration file # ********************************************************************** # ****** HIGH AVAILABILITY PACKAGE CONFIGURATION FILE (template) ******* # ********************************************************************** # ******* Note: This file MUST be edited before it can be used.
tkit/oracle/oracle/PARENT_ENVIRONMENT no tkit/oracle/oracle/CLEANUP_BEFORE_STARTUP no tkit/oracle/oracle/USER_SHUTDOWN_MODE abort tkit/oracle/oracle/KILL_ASM_FOREGROUNDS yes tkit/oracle/oracle/DB_SERVICE all tkit/oracle/oracle/DISABLE_BACKUP_MODE no email_id abc@hp.com service_name oracle_service service_cmd "$SGCONF/scripts/tkit/oracle/tkit_module.
Glossary $SGCONF /usr/local/cmcluster/conf for RHEL /opt/cmcluster/conf for SLES $SGROOT /usr/local/cmcluster in RedHat /opt/cmcluster in SLES ADF Attribute definition file ASM Automatic Storage Management Configuration directory mode of operation It allows you to maintain your own directory and copy all the toolkits scripts in this directory.
Index A W Alert mail notification, 33 With Automatic Storage Management, 11 Without Automatic Storage Management, 10 Multiple instance configuration, 10 Shared configuration, 10 C Cluster verification of Oracle toolkit, 33 Configuring modular ASM database package, 17 Configuring modular Oracle database package without ASM, 12 H Hardware Support, 5 I Installation uninstallation, 6 M Managing a package, 27 adding the package to the cluster, 27 Managing packages deleting a package, 32 halting packages,
