HP StorageWorks MxDB-Oracle-HiAv installation and administration guide HP MxDB-Oracle-HiAv 3.7.
Legal and notice information © Copyright 2004, 2009 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license. The information contained herein is subject to change without notice.
Contents About this guide ................................................................... 6 Intended audience ............................................................................................... HP technical support ............................................................................................ Subscription service ............................................................................................. HP websites .............................................................
Service loss ................................................................................................ SQL*Net security with Virtual Oracle Services ................................................ Place databases under MxDB-Oracle-HiAv control ................................................. Create the demonstration database ............................................................... Execute the PILOT_create.sh script ...........................................................
Create the OEM repository for this database .................................................. Take the Virtual Oracle Service out of Maintenance Mode ............................... Activate dbconsole on the backup servers ..................................................... Access dbconsole via a browser .................................................................. Troubleshooting .................................................................................................
About this guide This guide provides information about installing MxDB-Oracle-HiAv and managing Oracle databases under MxDB-Oracle-HiAv control. Intended audience This guide is intended for Oracle database administrators and HP Scalable NAS system administrators. HP technical support For worldwide technical support information, see the HP support website: http://www.hp.
HP websites For additional information, see the following HP websites: • http://www.hp.com • http://h18000.www1.hp.com/storage/software/clusteredfs/index.html • http://www.hp.com/go/storage • http://www.hp.com/service_locator • http://www.hp.com/support/manuals Documentation feedback HP welcomes your feedback. To make comments and suggestions about product documentation, please send a message to storagedocsFeedback@hp.com. All submissions become the property of HP.
About this guide
1 Introduction Purpose of this guide This guide is intended as a supplement to the Oracle installation documentation. Oracle DBAs have many choices when installing Oracle products. When using MxDB-Oracle-HiAv to manage Oracle databases running in the cluster, certain of these choices are required or recommended. This document describes those choices; however, it is not intended to be a complete step-by-step guide for installing Oracle. Many steps in this document require root permissions.
NOTE: A Virtual Oracle Service is a type of HP Scalable NAS custom service monitor. For more information about this type of monitor, see Chapter 18, “Configure Service Monitors,” in the HP Scalable NAS File Serving Software administration guide. MxDB-Oracle-HiAv takes the following actions in the event that service for a database is lost: • Server Failure. HP Scalable NAS monitors the health of the servers in the cluster and provides for application failover if a server becomes unavailable.
files that need to be identical, there is only one copy, not several, to maintain. However, with shared Oracle homes, it is not possible to do a rolling upgrade but the down time is typically very brief. With this shared environment, care must be taken so that oracle users and group identifiers are consistent across servers. Otherwise, user “oracle” with user id500 on server 2 will not have ownership to the shared filesystem if on the installing server, server 1, user “oracle” is user id 501.
Introduction
2 Install MxDB-Oracle-HiAv The MxDB-Oracle-HiAv software enables Oracle database administrators to create and manage one or more Virtual Oracle Services. These services allow database administrators to place a single-instance Oracle database running in the cluster under the control of MxDB-Oracle-HiAv. The database can then be configured to run on any number of servers in the cluster.
• MxDB-Oracle-HiAv requires that Oracle databases under its control share Oracle home and datafiles in the cluster. As a prerequisite to creating a shared oracle home, as described under “Installation steps,” below, download a supported version of the Oracle installation software onto at least one cluster node. Installation steps After HP Scalable NAS has been installed, complete the following steps to install Oracle and MxDB-Oracle-HiAv. Each step is described in detail later in this section. 1.
filesystem. This filesystem is mounted with the normal filesystem options but not the dboptimize option. (Note that PSFS filesystems also support asynchronous IO.) Because oradata is, by OFA standards, the root directory for Oracle datafiles, it must be physically located on the dboptimized filesystem. Also, OFA naming convention requires oradata to be located at either $ORACLE_HOME/oradata or $ORACLE_BASE/oradata.
2. Install Oracle Before performing an Oracle install, the DBA should note the following: • For MxDB-Oracle-HiAv to function properly, Oracle Home must be installed into a standard-mounted PSFS filesystem. This is a MxDB-Oracle-HiAv requirement. • MxDB-Oracle-HiAv depends on the standard OFA layout of the Oracle software. Databases created with the Database Configuration Assistant (DBCA) will generally possess the layout required for MxDB-Oracle-HiAv.
3. Set up .matrixrc files When the MxDB-Oracle-HiAv mxdb interface needs to run a script, it checks the file /root/.matrixrc on the server for authentication information. You will need to create this file and supply the authentication information for each server. The root user and password are required for creating the file. The easiest way to do this is to create a bookmark via the HP Scalable NAS Connect dialog, as described below. A bookmark is simply an entry in the .
4. In the Connect to field, specify the hostname of the server. 5. Click the As User button to the right of the Connect to field to display the Authentication Parameters dialog. Specify the root user and valid password, check both Add to bookmarks and Remember this password, and then click OK. 6. The Add Bookmarks dialog appears next. Enter a name for your cluster and click OK. You can call the cluster whatever you want. 7. Add the remaining servers to the bookmarked cluster.
Oracle Service before you place your database under the control of MxDB-Oracle-HiAv. NOTE: For testing purposes, it is sufficient to simply define the Virtual Oracle Service in the /etc/hosts file on each node in the cluster. To verify that your DNS entry is correct before you create the Virtual Oracle Service, run the ping(8) command. Provided that the DNS entry is valid, the ping command will return the IP address assigned to it in DNS. For example: [root@tmr6s18 ~]# ping PROD.customer.com PING PROD.
IMPORTANT: MxDB-Oracle-HiAv operations should never be performed from the HP Scalable NAS Management Console. Read-only access is all that is required. 7. Configure Oracle Virtual Services Now you are ready to begin running databases in a shared clustered environment. Chapter 3, “Operations” explains how this works and describes how to create a small test database to help you become comfortable with the process.
3 Configuration and Operations Management applications MxDB-Oracle-HiAv provides the following management applications: • MxDB-Oracle-HiAv Graphical User Interface. The GUI supports both configuration and operations. The GUI also supports the creation of a demonstration database for self-training. • MxDB-Oracle Command Line Interface. The CLI command, mxdb, supports operations but cannot be used to configure Virtual Oracle Services. • You need to be connected as the oracle user (e.g.
• Configures Oracle Net Services file entries such as listener.ora and tnsnames.ora if desired. This feature is provided as a convenience. The use of Oracle Net Services tools such as netca and netmgr is also supported. • Creates a virtual host that SQL*Net clients use to access the database. • Creates a service monitor that checks the health of servers, the database instance, and listeners. When a virtual host has an associated service monitor and database, it is known as a Virtual Oracle Service.
MxDB-Oracle-HiAv configuration file MxDB-Oracle-HiAv supports an optional configuration file that uses the naming convention mxdbha_.conf where is the name of the service monitor associated with the Virtual Oracle Service (see Naming conventions, page 11). If the file is used, it must be placed in $ORACLE_HOME/dbs. For example, for Virtual Oracle Service PROD: $ cd $ORACLE_HOME/dbs $ ls -l *PROD*conf -rw-r--r-1 root root 75 Feb 8 17:12 mxdbha_PROD.
probe logging information to a local directory. However, to aid in administration, it also sends log messages to a central log in the PSFS cluster filesystem. The system-level logs are placed in /var/log/hpcfs and are named SPACK_ORACLE.-.log, where DBNAME is the name specified when the Virtual Oracle Service is created. type is either start, stop, recovery, or probe. Generally, these logs are used only by HP Support.
$ cd $ORACLE_HOME/dbs $ ls -l *PROD*conf -rw-r--r-- 1 root root 75 Feb 8 17:12 mxdbha_PROD.conf $ cat mxdbha_PROD.conf log /u01/app/oracle/product/10.2.0/rdbms/log lsnr_password 8B1323CCD6F199D5 In this case, the listener.ora file will need to have a matching assignment, such as: $ cd $ORACLE_HOME/dbs $ ls -l *PROD*conf -rw-r--r-1 root root 75 Feb 8 17:12 mxdbha_PROD.conf $ cat mxdbha_PROD.conf log /u01/app/oracle/product/10.2.
that the cluster and your Oracle databases are configured properly for use with MxDB-Oracle-HiAv. MxDB-Oracle-HiAv includes a demonstration database called PILOT that you can use to become familiar with the process of placing a database under MxDB-Oracle-HiAv control and with managing the database via MxDB-Oracle-HiAv. It is not necessary to configure this demonstration database.
# /opt/hpcfs/mxdb_oracle_ha/bin/mxdb -c The Configuration GUI then appears as shown below. The GUI prefills several of the fields. It uses /etc/oratab to offer known ORACLE_HOME locations and uses your UID and GID as the presumed ORACLE_OWNER and group. Oracle Home: Select from the options provided or enter your entire Oracle Home path. Oracle Owner: Your user name appears in this field. If the name is incorrect, exit the application and log in as the oracle user.
Database Name: This field contains two options: Create New Database Service, which is used to create a Virtual Oracle Service for existing databases, and Create Demo Database Service, which is used to create the PILOT demo database and Virtual Oracle Service. Select Create Demo Database Service. The GUI will then replace the text in the field with the name “PILOT.” Virtual Host Name: Enter the DNS entry that the System Administrator specified for this service. In our example, the entry is PILOT.customer.
NOTE: Do not close the window until you have configured listener.ora and tnsnames.ora. Regardless of whether you decide to use netca to configure the listener.ora and tnsnames.ora files for PILOT, you must tell MxDB-Oracle-HiAv where the files reside. The default location is $ORACLE_HOME/ network/admin. When all of the fields are complete, click Apply.
Execute the PILOT_create.sh script After you complete the Net Services configuration and click Apply, a dialog box directs you to connect to the system as the Oracle user through an xterm session. The session must be able to run dbca in silent mode. NOTE: Be sure that your $ORACLE_HOME is the same as you indicated in the GUI and change your ORACLE_SID and DBNAME environment variables to PILOT.
After logging in as the Oracle user, go to the /tmp directory and run the PILOT_create.sh script. It takes about five minutes to create the PILOT database. When the silent-mode DBCA invocation is done, the terminal session will report where the log file has been created. Be sure to examine this file. Additional logging is provided in the /tmp/MxDB-Oracle-HiAv.log file. Following is an example of running the PILOT_create.sh script.
If you want to execute the GUI from other nodes, copy the /etc/oratab entry for PILOT that you just created to the other nodes. At this point the PILOT database will be running. From now on, you will use this GUI or the corresponding CLI command to start and stop the database. (For more information, see Database operations, page 40.) Exit the GUI by clicking the X in the upper right-hand corner.
Configure an existing database Before configuring an existing database for MxDB-Oracle-HiAv, be sure to review the checklist in Appendix A, page 67 to ensure that the software is correctly installed and all preparation steps are completed. To configure High Availability for an existing database, first shut the database down and then start the configuration GUI by executing the following command from an xterm session.
In the Database Name field, select Create New Database Service. The GUI then clears the text from the field and you should enter the name of your database (ORACLE_SID). In the following screens, an existing Oracle database called PROD is placed under MxDB-Oracle-HiAv control. The Virtual Oracle Service created in this example is called PROD.customer.com. A fully qualified Virtual Oracle Service name must be specified in the Virtual Host Name field.
Select the nodes for the Virtual Oracle Service. The Available column lists the cluster nodes that can be used for the Virtual Oracle Service. To select the nodes that you want to support the database, highlight a node and use the Right arrow to move it to the Selected Nodes column. The top node listed in the Selected Nodes column will be the Primary node, the second node will be the first Backup, the third node will be the second Backup, and so forth.
In the “Net services” section of the window, either allow MxDB-Oracle-HiAv to configure Net Services or do so with your choice of Oracle database management utility. (See Create the demonstration database, page 26 for more information about this section of the window.) NOTE: If the listener.ora and tnsnames.ora were previously generated by the Oracle Net Services configuration utilities such as netca, netmgr, or dbca, MxDB-Oracle-HiAv cannot configure listener.ora and tnsnames.ora.
Be sure to verify the TNS_ADMIN path in the TNS_ADMIN Location field. While the default is generally correct, it is important that MxDB-Oracle-HiAv know where the SQL*Net files are. (Note that if the path is too long to fit in the field, it will right justify.) In the example, the PROD database uses the default TNS_ADMIN directory and the listener will listen on port 1621. Click Apply. The status bar at the bottom of the screen shows the steps taken to create the Virtual Oracle Service.
The GUI now refreshes its information regarding the new Virtual Oracle Service. In the following example, the green Running indicator shows that the PROD database is up and is being probed by MxDB-Oracle-HiAv.
To exit the GUI, check the X in the upper right-hand corner of the window. Error handling with the GUI If an error occurs while the Virtual Oracle Service is being created, MxDB-Oracle-HiAv will display a window describing the error. Click on the message and then click Details for more information about the error. For additional hints when troubleshooting, see the various logs described under Logging, page 23.
Database operations Database administration changes slightly when a database is monitored by MxDB-Oracle-HiAv. For example, if a database fails the monitor probe, MxDB-Oracle-HiAv will attempt to restart the database before moving it to a backup server. Therefore, if a DBA connects to a database through SQL*Plus or Enterprise Manager and shuts the database down, it will be immediately restarted by MxDB-Oracle-HiAv.
GUI procedure To shut down (or later restart) a High Availability database through the MxDB-Oracle-HiAv GUI, click Stop Database (or Start Database). The following series of screens show stopping the PROD Virtual Oracle Service and then restarting it. Notice the status bar indicating the task being performed. NOTE: When using this procedure, the $ORACLE_HOME and $ORACLE_SID environment variables should be set to the oracle home and database SID of the database you are operating on.
Click Refresh. The Application window now shows that the Virtual Oracle Service is stopped and the Stop Database button is now labeled “Start Database.
To restart the database and the Virtual Oracle Service, click Start Database.
Rehost a database MxDB-Oracle-HiAv makes it easy to move a database service within the cluster. When the service is moved from the primary node to a backup node, the database will be shut down on the old primary node and restarted on the new primary node. GUI procedure The following screens show an example of moving the PROD database service role from the primary node to the 1st Backup node. The primary is currently node 10.10.60.5 and the first backup is 10.10.60.6.
To change the Primary and Backup assignments, select the appropriate node in the Selected Nodes column and use the up or down arrows to move its position in the list. In the example, the original primary node 10.10.60.18 was moved to the bottom of the list. Click Apply when you have finished reordering the list. MxDB-Oracle-HiAv now shuts down the database on the old primary node (10.10.60.18) and starts it on the new primary node (10.10.60.17).
the old primary node (10.10.60.18) and starts it on the new primary node (10.10.60.17). mxdb -d PROD -m “10.10.60.17:10.10.60.16:10.10.60.18” The -m option specifies the IP addresses of the primary and backup servers. The first server listed is the primary, the next server is the first backup, and so on. Use colons to separate the server addresses.
As shown on the following screen, node 10.10.60.16 is now the second backup for the Virtual Oracle Server.
CLI procedure Use a command such as the following to add or remove nodes from the configuration of a Virtual Oracle Service. The -m option specifies the nodes that should now be included in the configuration. New nodes specified with -m will be added to the configuration, and any nodes omitted from the -m list will be dropped from the configuration. The following command performs the same action as the GUI example above. Node 10.10.60.
You can also use the -a option to add a node to the backup list. For example, the following command adds node 10.10.60.16 as the third backup. mxdb –d PROD -a “10.10.60.16” Maintenance Mode While a database is under the control of MxDB-Oracle-HiAv, you must start and stop the database by stopping the entire Virtual Oracle Service. If you attempt a shutdown using sqlplus / shutdown immediate, for example, the Virtual Oracle Service will detect a down database and will immediately attempt to restart it.
When your activities are complete, click Stop Maintenance to take the Virtual Oracle Service out of Maintenance Mode. If the database is down when Stop Maintenance is clicked, the database will be started on the primary node. If the database is already running, the monitor associated with the Virtual Oracle Service will resume its probe operations.
CLI procedure To place a Virtual Oracle Service into Maintenance Mode, use this command: mxdb -d -M start To stop Maintenance Mode, use this command: mxdb -d -M stop MxDB-Oracle-HiAv installation and administration guide 51
Delete a Virtual Oracle Service Deleting a Virtual Oracle Service does not affect the database itself. The virtual host and service monitor are removed, but the database remains intact. The database will be shut down during this operation. NOTE: The entries for the Virtual Oracle Service will remain in the SQL*Net configuration files. The DBA will need to edit the files and restart the listener to allow access through a simple IP as opposed to the virtual host.
Connect through vhost: Primary Node is: Currently active on Backups: BACKUP-3 10.10.60.15 10.10.60.82 10.10.60.18 10.10.60.18 BACKUP-1 10.10.60.17 BACKUP-2 10.10.60.16 Enabled Nodes: Disabled Nodes: Maintenance Mode: 10.10.60.15 10.10.60.16 10.10.60.17 10.10.60.18 OFF Success The mxdb command line interface In addition to the –c option, which invokes the GUI, the mxdb command can be used for operations.
Option Procedure -M [start|stop] Start or stop Maintenance Mode. -a "IP:IP:IP" Add servers to a Virtual Oracle Service. The option is a list of quoted, colon-delimited IP addresses. -m "IP:IP:IP:IP" Modify the host roles for the Virtual Oracle Service and if possible (or necessary) move the service to the new primary node. The option is a quoted list of IP addresses that represent the Primary, 1st Backup, 2nd Backup, and so on.
shops vary widely in their usage of the product. To that end, it is nearly impossible to provide a default set of start, stop and probe scripts that meet the needs of all sites. For instance, it may be desirable to have probe actions that ensure access to a specific application table or to send a message to a DBA based on certain criteria. To address this issue, MxDB-Oracle-HiAv supports the concept of site-specific start, stop and probe scripts.
It is advisable to copy the default scripts to a working directory before deploying them into the ${ORACLE_BASE}/admin//scripts directory. Once you are ready to deploy your custom scripts, simply move them into the ${ORACLE_BASE}/admin//scripts directory and the next probe cycle will use the scripts—if the Virtual Oracle Service is already running. The mxdb_probe.sql script is well documented with pointers on where to place any desired custom probe actions.
CLI or Management Console as described in the HP Scalable NAS File Serving Software administration guide. Supplemental startup scripts MxDB-Oracle-HiAv supports supplemental startup and shutdown scripts that are customized at your site. MxDB-Oracle-HiAv ensures that the database instance and SQL*Net listener processes are available in the cluster. You might also want supplemental processes to be started and stopped each time the database is restarted or failed over to another node.
• Permissions. The execution bit must be set on supplemental scripts. This can be done with the chmod command, such as: # chmod 755 /opt/hpcfs/mxdb_oracle_ha/etc/custom_{ORACLE_SID}.sh To aid in implementing a supplemental script, MxDB-Oracle-HiAv ships with a fully functional sample script that starts the Intelligent Agent. This sample script is ready for the PILOT database and is called custom_PILOT.sh.sample. It is located in the lib directory of MxDB-Oracle-HiAv.
Configure MxDB-Oracle-HiAv for a Data Guard standby database The default start/stop and probe scripts included in the product are functional with a Data Guard Primary without change. However, Data Guard Standby databases require different start/stop and probe actions. To that end, the product ships with a set of scripts that can be used for a Data Guard Standby.
approach to deploying MxDB-Oracle-HiAv with Data Guard is to get Data Guard fully functional before you create a Virtual Oracle Service for the database. Once Data Guard is fully functional, it is quite simple to change the SQL*Net files so that the HOST assignments use the Virtual Oracle Service instead of the real host name, and then use the MxDB-Oracle-HiAv GUI to create the Virtual Oracle Service.
4 Implement Oracle Enterprise Manager DB Console with the Virtual Oracle Service Implementing dbconsole Oracle Enterprise Manager (OEM) for 10g and 11g in a clustered environment requires that dbconsole work properly with virtual oracle hosts. The individual OEM services and processes on each node can then manage the Oracle instance when the instance is running on that node. You must install dbconsole for each database that you want to manage using dbconsole.
5. Using emca, create the OEM repository for this database and verify you can connect via a browser to this server. 6. Take the database out of Maintenance Mode. 7. On all of the other servers in the cluster, open a session as in step 3 with the proper environments. Using emctl, start dbconsole on those systems. 8. Connect to dbconsole via a browser.
ORACLE_SID=OLTP1 ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1 ORACLE_HOSTNAME=OLTP1.customer.com Unlock dbconsole users for this database In the session you created with the proper environment variables, unlock the dbsnmp and sysman user accounts. Use the Oracle sqlplus command line utility to run the commands.
IMPORTANT: Note the URL and port number reference when the repository creation is complete. You will use the specified URL with that port number within a browser to access dbconsole after the agent starts.
2. Activate dbconsole on the server using the emctl command: emctl start dbconsole Access dbconsole via a browser Using the URL and port number returned when the OEM dbconsole repository was created, connect to dbconsole. After agreeing with the disclaimer page, log into the database using one of the dbconsole users (sysman, dbsnmp). NOTE: Version 6.0 of Internet Explorer (IE) does not work properly with Oracle Enterprise Manager 10g in the environments tested while developing this methodology.
emca -reconfig ports -DBCONTROL_HTTP_PORT • The EMCA configuration is very difficult to repair or undo. Although some troubleshooting techniques, including those mentioned in this document exist, sometimes the best course of action is to re-install Oracle on that node.
A Configuration checklist Before attempting to create a Virtual Oracle Service, use the following checklist to ensure that MxDB-Oracle-HiAv is installed correctly and that the cluster and your Oracle databases are configured properly for use with MxDB-Oracle-HiAv. Remember that only the GUI (not the CLI) can be used to create a new Virtual Oracle Service. 1.
ACLE_BASE|$ORACLE_HOME]/oradata to an oradata subdirectory on this dboptimized filesystem. See 1. Create PSFS Filesystems , page 14. Responsibility: System administrator with root privileges 5. Ensure that each node has a .matrixrc file for root. On each node, the file must contain an entry corresponding to that node (for example, “tmr13 root password” exists in the .matrixrc file on node tmr13.) See 3. Set up .matrixrc files, page 17. Responsibility: System administrator with root privileges 6.
11. Ensure that $ORACLE_HOME is set to a valid Oracle install location before launching either the GUI or the command line interface. Remember this (or set the oracle user profile accordingly) on all nodes where you plan to manage Virtual Oracle Services. Responsibility: Oracle DBA 12. On all nodes, ensure that the /etc/oratab entry for databases under MxDBOracle-HiAv control is not set to automatically start itself upon reboot.
Configuration checklist
Index G C CLI add or remove nodes, 48 command options, 53 delete Virtual Oracle Service, 52 mxdb command access, 22 operations, 22 rehost database, 45 start or stop database, 43 start or stop Maintenance Mode, 51 configuration checklist, 67 D Data Guard, 58 database add or remove nodes, 46 configure, 33 Data Guard support, 58 demonstration database, create, 26 Maintenance Mode, 49 rehost, 44 scripts, supplemental, 57 start or stop, 40 status, query, 52 DBOPTIMIZE mount option, 14 demonstration database cr
mxdb command, 22 options, 53 MxDB-Oracle-HiAv configuration file, 23 Data Guard, configure for, 59 install, 13 log files, 23 Maintenance Mode, 49 management applications, 21 parameters, configurable, 23 prerequisites, 13 service loss, action taken, 24 upgrade, 13 mxdbha_.
