Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Second Edition (June 2001) Part Number 225083-002 Compaq Computer Corporation Compaq Confidential – Need to Know Required Writer: John Blackburn Project: Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Comments: Part Number: 225083-002 File Name: a-frnt.
Notice © 2001 Compaq Computer Corporation Compaq, the Compaq logo, Compaq Insight Manager, SmartStart, Rompaq, ProLiant, and StorageWorks Registered in U.S. Patent and Trademark Office. ActiveAnswers is a trademark of Compaq Information Technologies Group, L.P. in the United States and other countries. Microsoft, Windows, and Windows NT are trademarks of Microsoft Corporation in the United States and other countries. All other product names mentioned herein may be trademarks of their respective companies.
Contents About This Guide Purpose .................................................................................................................... xiii Audience.................................................................................................................. xiii Scope ........................................................................................................................xiv Referenced Manuals ..............................................................................
iv Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Chapter 2 Architecture Compaq ProLiant Servers ........................................................................................ 2-2 High-Availability Features of ProLiant Servers ............................................... 2-3 Shared Storage Components .................................................................................... 2-3 RA4000 Array..........................................
Contents Chapter 3 Cluster Software Components Overview of the Cluster Software............................................................................ 3-1 Microsoft Windows 2000 Advanced Server............................................................ 3-2 Compaq Software .................................................................................................... 3-2 Compaq SmartStart and Support Software.......................................................
vi Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Cluster Planning continued Disk Drive Planning............................................................................................... 4-18 Nonshared Disk Drives ................................................................................... 4-18 Shared Disk Drives ......................................................................................... 4-18 Network Planning ..................
Contents Installation and Configuration continued Verifying the Hardware and Software Installation ................................................ 5-38 Cluster Communications ................................................................................ 5-38 Access to Shared Storage from All Nodes...................................................... 5-38 OSDs .............................................................................................................. 5-38 Other Verification Tasks ...
viii Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Cluster Management continued Adding a Cluster Node........................................................................................... 6-21 Preparing the New Node................................................................................. 6-22 Preparing the Existing Cluster Nodes .............................................................
Contents Appendix A Diagnosing and Resolving Shared Disk Problems Introduction .............................................................................................................A-1 Run Object Link Manager On All Nodes ................................................................A-3 Restart All Affected Nodes in the Cluster ...............................................................A-4 Rerun and Validate Object Link Manager On All Affected Nodes .........................
x Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide List of Figures Figure 1-1. Example of a six-node Compaq Parallel Database Cluster Model PDC/O1000............................................................................................. 1-2 Figure 2-1. Fibre Host Adapters in a two-node PDC/O1000 cluster....................... 2-7 Figure 2-2. Two-node PDC/O1000 cluster with one non-redundant Fibre Channel Fabric .......................................
Contents Figure 5-4. Non-redundant Ethernet cluster interconnect using a crossover cable ................................................................................................................. 5-13 Figure 5-5. Non-redundant Ethernet cluster interconnect using an Ethernet switch or hub.................................................................................................... 5-14 Figure 5-6. Redundant Ethernet cluster interconnect for a two-node PDC/O1000 cluster .........................
About This Guide Purpose This administrator guide provides information about the planning, installation, configuration, implementation, management, and troubleshooting of the Compaq Parallel Database Cluster Model PDC/O1000 running Oracle8i software on the Microsoft Windows 2000 Advanced Server operating system. Audience The expected audience of this guide consists primarily of MIS professionals whose jobs include designing, installing, configuring, and maintaining Compaq Parallel Database Clusters.
xiv Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Scope This guide offers significant background information about clusters as well as basic concepts associated with designing clusters. It also contains detailed product descriptions and installation steps.
About This Guide Some clustering topics are mentioned, but not detailed, in this guide. For example, this guide does not describe how to install and configure Oracle8i on a PDC/O1000 cluster. For information about these topics, see the documents referenced in the guide sections or refer to the documentation provided with the Oracle8i software.
xvi Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide G Oracle8i Parallel Server Concepts G Oracle8i Parallel Server Administration, Deployment, and Performance G Oracle Enterprise Manager Administrator’s Guide G Oracle Enterprise Manager Configuration Guide G Oracle Enterprise Manager Concepts Guide Supplemental Documents The following technical documents contain important supplemental information for the Compaq Parallel Database Cluster Model PD
About This Guide Text Conventions This document uses the following conventions to distinguish elements of text: User Input, GUI Selections Text a user types or enters appears in boldface. Items a user selects from a GUI, such as tabs, buttons, or menu items, also appear in boldface. User input and GUI selections can appear in uppercase and lowercase letters. File Names, Command Names, Directory Names, Drive Names These elements can appear in uppercase and lowercase letters.
xviii Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Symbols on Equipment These icons may be located on equipment in areas where hazardous conditions may exist. Any surface or area of the equipment marked with these symbols indicates the presence of electrical shock hazards. Enclosed area contains no operator serviceable parts. WARNING: To reduce the risk of injury from electrical shock hazards, do not open this enclosure.
About This Guide Rack Stability WARNING: To reduce the risk of personal injury or damage to the equipment, be sure that: ■ The leveling jacks are extended to the floor. ■ The full weight of the rack rests on the leveling jacks. ■ The stabilizing feet are attached to the rack if it is a single rack installations. ■ The racks are coupled together in multiple rack installations. ■ Only one component is extended at a time.
xx Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Compaq Website The Compaq website has information on this product as well as the latest drivers and Flash ROM images. You can access the Compaq website by logging on to the Internet at www.compaq.com Compaq Authorized Reseller For the name of your nearest Compaq Authorized Reseller: ■ In the United States, call 1-800-345-1518. ■ In Canada, call 1-800-263-5868.
Chapter 1 Clustering Overview For many years, companies have depended on clustered computer systems to fulfill two key requirements: to ensure users can access and process information that is critical to the ongoing operation of their business, and to increase the performance and throughput of their computer systems at minimal cost. These requirements are known as availability and scalability, respectively.
1-2 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i on Windows 2000 Administrator Guide Clusters Defined A cluster is an integration of software and hardware products that enables a set of loosely coupled servers and shared storage subsystem components to present a single system image to clients and to operate as a single system.
Clustering Overview 1-3 The PDC/O1000 uses non-redundant Fibre Channel Fabric Storage Area Network (SAN) and non-redundant Fibre Channel Arbitrated Loop (FC-AL) SAN topologies for its shared storage I/O data paths. These two SAN topologies support the use of multiple non-redundant fabrics and loops, respectively. In the example shown in Figure 1-1, the clustered nodes are connected to the database on the shared storage subsystems through a non-redundant Fibre Channel Fabric or non-redundant FC-AL.
1-4 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i on Windows 2000 Administrator Guide Compaq Parallel Database Cluster Overview As traditional clustering technology has moved into the open systems of client/server computing, Compaq has provided innovative, customer-focused solutions. The PDC/O1000 moves client/server computing one step closer to the capabilities found in expensive, proprietary cluster solutions, at a fraction of the cost.
Chapter 2 Architecture The Compaq Parallel Database Cluster Model PDC/O1000 (referred to here as the PDC/O1000) is an integration of a number of different hardware and software products. This chapter discusses how each of the hardware products plays a role in bringing a complete clustering solution to your computing environment.
2-2 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide ■ G Gigabit Interface Converter-Shortwave (GBIC-SW) modules G Fibre Channel cables Cluster interconnect components G Ethernet NIC adapters G Ethernet cables G Ethernet switches or hubs The software products include: ■ Microsoft Windows 2000 Advanced Server with Service Pack 1 or later ■ Compaq drivers and utilities ■ Oracle8i Enterprise Edition with the Oracle8i Parallel Server Option Re
Architecture High-Availability Features of ProLiant Servers In addition to the increased application and data availability enabled by clustering, ProLiant servers include many reliability features that provide a solid foundation for effective clustered server solutions. The PDC/O1000 is based on ProLiant servers, most of which offer excellent reliability through redundant power supplies, redundant cooling fans, and Error Checking and Correcting (ECC) memory.
2-4 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide RA4000 Array The RA4000 Array is one shared storage solution for the PDC/O1000. Each non-redundant Fibre Channel Fabric or non-redundant Fibre Channel Arbitrated Loop (FC-AL) can contain one or more RA4000 Arrays. Each RA4000 Array contains one single-port RA4000 Array Controller.
Architecture RA4000 Array Controller One single-port RA4000 Array Controller is installed in each RA4000 Array or RA4100 Array. If the array controller fails, the cluster nodes cannot access the shared storage disks in that array. From the perspective of the cluster nodes, the RA4000 Array Controller is simply another device connected to one of the cluster’s I/O paths. Consequently, each node sends its I/O requests to the RA4000 Array Controller just as it would to any SCSI device.
2-6 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide For further information, refer to these manuals provided with the Fibre Channel SAN Switch: ■ Compaq StorageWorks Fibre Channel SAN Switch 8 Installation and Hardware Guide ■ Compaq StorageWorks Fibre Channel SAN Switch 16 Installation and Hardware Guide ■ Compaq StorageWorks Fibre Channel SAN Switch Management Guide provided with the Fibre Channel SAN Switch Storage Hub One Storage Hub can be inst
Architecture FC-AL Switch One FC-AL Switch can also be installed between cluster nodes and shared storage arrays in a PDC/O1000 cluster to create a non-redundant Fibre Channel Arbitrated Loop (FC-AL). The FC-AL Switch is another device for connecting one Fibre Host Adapter in each node with the array controller in the RA4000/RA4100 Array. The Fibre Host Adapter in each node, the FC-AL Switch, and the RA4000/RA41000 Arrays to which they are connected belong to the same non-redundant FC-AL.
2-8 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Each non-redundant Fibre Channel Fabric or non-redundant FC-AL contains one dedicated Fibre Host Adapter in every cluster node. Across nodes, Fibre Host Adapters for the same Fibre Channel Fabric or FC-AL must be installed in the same server slot and connected to the same Fibre Channel SAN Switch, FC-AL Switch, or Storage Hub.
Architecture Availability Features of the Shared Storage Components An important part of a high-availability system is the ability to improve data availability, traditionally accomplished by implementing RAID technology. Hardware RAID is an important part of the RA4000/RA4100 Array storage subsystem. RAID is implemented on the RA4000 Array Controller in the RA4000/RA4100 Array. The RA4000/RA4100 Array also accepts redundant, hot-pluggable power supplies and a hot-pluggable fan module.
2-10 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Non-Redundant Fibre Channel Fabric The PDC/O1000 supports non-redundant Fibre Channel Fabrics. A non-redundant Fibre Channel Fabric provides one I/O path between each cluster node and its Fibre Channel SAN Switch and one I/O data path between the Fibre Channel SAN Switch and each of its RA4000/RA4100 Arrays.
Architecture Figure 2-2 shows a two-node PDC/O1000 with one non-redundant Fibre Channel Fabric. RA4000/4100 Array Fibre Host Adapter ProLiant Server RA4000/4100 Array RA4000/4100 Array Node 2 Node 1 Fibre Channel SAN Switch Fibre Host Adapter ProLiant Server Switch (Cluster Interconnect) Client LAN Figure 2-2.
2-12 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Adding a second non-redundant Fibre Channel Fabric to the PDC/O1000 involves duplicating the hardware components used in the first Fibre Channel Fabric. The maximum number of non-redundant Fibre Channel Fabrics you can install in a PDC/O1000 is restricted by the number of Fibre Host Adapters your Compaq ProLiant servers support. Refer to the Compaq ProLiant server documentation for this information.
Architecture Maximum Distances Between Cluster Nodes and Shared Storage Subsystem Components in a Non-Redundant Fibre Channel Fabric By using standard short-wave Fibre Channel cables and GBIC-SW modules, an RA4000/RA4100 Array can be placed up to 500 meters from the Fibre Channel SAN Switch, and the Fibre Channel SAN Switch can be placed up to 500 meters from the Fibre Host Adapter in each cluster node. See Figure 2-4.
2-14 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Fibre Host Adapter-to-Fibre Channel SAN Switch Paths Figure 2-5 highlights the I/O data paths that run between the Fibre Host Adapter in each cluster node and the Fibre Channel SAN Switch. There is one I/O path for each Fibre Host Adapter.
Architecture Fibre Channel SAN Switch-to-Array Controller Paths Figure 2-6 highlights the I/O data path that runs between the Fibre Channel SAN Switch and the RA4000 Array Controller in each RA4000/RA4100 Array of a Fibre Channel Fabric. There is one path for each array controller. Client LAN Fibre Host Adapters Fibre Host Adapters Switch (Cluster Interconnect) ProLiant Servers ProLiant Servers Fibre Channel SAN Switch RA4000/4100 Array Figure 2-6.
2-16 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide I/O Path Configurations in a Non-Redundant Fibre Channel Arbitrated Loop Overview of FC-AL SAN Topology Fibre Channel standards define a multi-layered architecture for moving data across the storage area network (SAN). This layered architecture can be implemented using the Fibre Channel Fabric SAN or the Fibre Channel Arbitrated Loop (FC-AL) SAN topology. The PDC/O1000 supports both topologies.
Architecture Each non-redundant FC-AL consists of the following hardware: ■ One Fibre Host Adapter in each node ■ One Storage Hub or FC-AL Switch ■ One or more RA4000/RA4100 Arrays, each containing one single-port RA4000 Array Controller ■ A GBIC-SW module installed in each Fibre Host Adapter, each active port on the Storage Hub or FC-AL Switch, and each array controller ■ Fibre Channel cables used to connect the Fibre Host Adapter in each node to the Storage Hub or FC-AL Switch and the Storage Hu
2-18 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Using Multiple Non-Redundant Fibre Channel Arbitrated Loops The PDC/O1000 supports the use of multiple non-redundant FC-ALs within the same cluster. Physically, this means that within each cluster node, multiple Fibre Host Adapters are used to connect the nodes to different sets of RA4000/RA4100 Arrays.
Architecture Figure 2-8 shows a four-node PDC/O1000 cluster with two non-redundant FC-ALs. Each FC-AL has its own Fibre Host Adapter in every node, one Storage Hub or FC-AL Switch, and one or more RA4000/RA4100 Arrays. In Figure 2-8, the hardware components that constitute the second non-redundant FC-AL are shaded. RA4000/4100 Arrays (8) Fibre Host Adapters Storage Hubs/ FC-AL Switches Fibre Host Adapters RA4000/4100 Arrays (4) Figure 2-8.
2-20 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Maximum Distances Between Cluster Nodes and Shared Storage Subsystem Components in a Non-Redundant FC-AL By using standard short-wave Fibre Channel cables and GBIC-SW modules, an RA4000/RA4100 Array can be placed up to 500 meters from the Storage Hub or FC-AL Switch, and the Storage Hub or FC-AL Switch can be placed up to 500 meters from the Fibre Host Adapter in each cluster node. See Figure 2-9.
Architecture Fibre Host Adapter-to-Storage Hub/FC-AL Switch Paths Figure 2-10 highlights the I/O data paths that run between the Fibre Host Adapter in each cluster node and the Storage Hub or FC-AL Switch. There is one I/O path for each Fibre Host Adapter. Client LAN Fibre Host Adapters Fibre Host Adapters Switch (Cluster Interconnect) ProLiant Servers ProLiant Servers Storage Hub/FC-AL Switch RA4000/4100 Array Figure 2-10.
2-22 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Storage Hub/FC-AL Switch-to-Array Controller Paths Figure 2-11 highlights the I/O data path that runs between the Storage Hub or FC-AL Switch and the RA4000 Array Controller in each RA4000/RA4100 Array of the FC-AL. There is one path for each array controller.
Architecture Cluster Interconnect Options The cluster interconnect is the data path over which all of the nodes in a cluster communicate. The nodes use the cluster interconnect data path to: ■ Communicate individual resource and overall cluster status. ■ Send and receive heartbeat signals. ■ Coordinate database locks through the Oracle Integrated Distributed Lock Manager. NOTE: Several terms for cluster interconnect are used throughout the industry.
2-24 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Non-Redundant Ethernet Cluster Interconnect Components The following components are used in a non-redundant Ethernet cluster interconnect: ■ One Ethernet adapter in each cluster node ■ Ethernet cables and a switch or hub G For two-node PDC/O1000 clusters, you can either use one Ethernet crossover cable or one 100-Mbit/second Ethernet switch or hub and standard Ethernet cables to connect the two serv
Architecture The Ethernet adapters must have passed Windows 2000 Advanced HCT certification. NOTE: If you are using dual-port Ethernet adapters in a non-redundant Ethernet cluster interconnect, you can use one port for the Ethernet cluster interconnect and the second port for the client LAN.
2-26 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Ethernet Cluster Interconnect Diagrams Figure 2-12 shows the non-redundant Ethernet cluster interconnect components used in a two-node PDC/O1000 cluster. These components include a dual-port Ethernet adapter in each node. The top port on each adapter connects by Ethernet crossover cable to the top port on the adapter in the other node.
Architecture Figure 2-13 shows another option for a non-redundant Ethernet cluster interconnect in a two-node PDC/O1000 cluster. These include an Ethernet adapter in each node connected by Ethernet cables to an Ethernet switch or hub. RA4000/4100 Array Storage Hub/Switch Ethernet Adapter Ethernet Adapter Storage Hub/Switch Ethernet cables Node 1 Node 2 Client LAN Figure 2-13.
2-28 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Because Ethernet switches are required in PDC/O1000 clusters with three or more nodes, using an Ethernet switch instead of a crossover cable makes it easier to upgrade the cluster interconnect if more servers are added to the cluster.
Architecture Local Area Network NOTE: For the PDC/O1000, the client LAN and the cluster interconnect must be treated as separate networks. Do not use either network to handle the other network’s traffic. Every client/server application requires a local area network, or LAN, over which client machines and servers communicate. In the case of a cluster, the hardware components of the client LAN are no different than in a standalone server configuration.
Chapter 3 Cluster Software Components Overview of the Cluster Software The Compaq Parallel Database Cluster Model PDC/O1000 (referred to here as the PDC/O1000) combines software from several leading computer vendors. The integration of these components creates a stable cluster management environment in which the Oracle database can operate.
3-2 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Microsoft Windows 2000 Advanced Server This version of the PDC/O1000 supports only Microsoft Windows 2000 Advanced Server with Service Pack 1 or later. NOTE: The PDC/O1000 does not work in conjunction with Microsoft Cluster Server. Do not install Microsoft Cluster Server on any of the cluster nodes.
Cluster Software Components Compaq System Configuration Utility The SmartStart and Support Software CD also contains the Compaq System Configuration Utility. This utility is the primary means to configure hardware devices within your servers, such as I/O addresses, boot order of disk controllers, and so on. For information about the System Configuration Utility, see the Compaq Server Setup and Management pack.
3-4 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Compaq Insight Manager Compaq Insight Manager, loaded from the Compaq Management CD, is a software utility used to collect information about the servers in the cluster.
Cluster Software Components Compaq Operating System Dependent Modules Compaq supplies low-level services, called operating system dependent modules (OSDs), which are required by Oracle8i Parallel Server. The OSD layer monitors critical clustering hardware components, constantly relaying cluster state information to Oracle8i Parallel Server. Oracle8i Parallel Server monitors this information and takes pertinent action as needed.
3-6 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Oracle8i Server Oracle8i Server is the database application software and must be installed on each node in the PDC/O1000. Refer to the documentation provided with the Oracle8i Server software for additional information. Oracle8i Parallel Server Option Oracle8i Parallel Server Option is the key component in the Oracle8i clustering architecture.
Cluster Software Components Oracle8i Certification To ensure that Oracle8i Parallel Server is used in a compatible hardware environment, Oracle has established a certification process, which is a series of test suites designed to stress an Oracle8i Parallel Server implementation and verify stability and full functionality.
Chapter 4 Cluster Planning Before connecting any cables or powering on any hardware on your Compaq Parallel Database Cluster Model PDC/O1000 (referred to here as the PDC/O1000), it is important that you understand how all the various cluster components fit together to meet your operational requirements.
4-2 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Site Planning You must carefully select and prepare the site to ensure a smooth installation and a safe and efficient work environment.
Cluster Planning Capacity Planning for Cluster Hardware Capacity planning determines how much computer hardware is needed to support the applications and data on your clustered servers. Given the size of your database and the performance you expect, you must decide how many servers and shared storage arrays the cluster needs. Compaq ProLiant Servers The number of servers you install in a PDC/O1000 cluster should take into account the levels of availability and scalability your site requires.
4-4 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Planning Shared Storage Components for Non-Redundant Fibre Channel Fabrics Several key components make up the shared storage subsystem for the PDC/O1000.
Cluster Planning Planning Shared Storage Components for Non-Redundant Fibre Channel Arbitrated Loops Each non-redundant Fibre Channel Arbitrated Loop (FC-AL) in a PDC/O1000 uses these hardware components: ■ One Fibre Host Adapter in each node ■ One Compaq StorageWorks Storage Hub (Storage Hub) or Compaq StorageWorks FC-AL Switch (FC-AL Switch) ■ One or more RA4000 Arrays or RA4100 Arrays ■ One single-port RA4000 Array Controller installed in each RA4000/RA4100 Array NOTE: For more information about
4-6 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Planning Cluster Interconnect and Client LAN Components PDC/O1000 clusters running Oracle8i Parallel Server can use a redundant or non-redundant Ethernet cluster interconnect. A redundant cluster interconnect is recommended because it provides fault tolerance along the entire cluster interconnect path.
Cluster Planning Planning the Client LAN Every client/server application requires a local area network (LAN) over which client machines and servers communicate. In the case of a cluster, the hardware components of the client LAN are no different than in a stand-alone server configuration. In keeping with the redundant architecture of a redundant cluster interconnect, you may choose to install a redundant client LAN, with redundant Ethernet adapters and redundant Ethernet switches or hubs.
4-8 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Sample Midsize Cluster with One Non-Redundant Fibre Channel Fabric Figure 4-1 shows an example of a midsize PDC/O1000 cluster with one non-redundant Fibre Channel Fabric that contains four cluster nodes and three RA4000/RA4100 Arrays. RA4000/4100 Array Node 3 RA4000/4100 Array RA4000/4100 Array Node 2 Node 1 Node 4 Fibre Channel SAN Switch Switch (Cluster Interconnect) Client LAN Figure 4-1.
Cluster Planning Sample Large Cluster with One Non-Redundant Fibre Channel Fabric Figure 4-2 shows an example of a larger PDC/O1000 cluster with one non-redundant Fibre Channel Fabric that contains six cluster nodes and six RA4000/RA4100 Arrays. RA4000/4100 Arrays (6) Fibre Channel SAN Switch Node 5 Node 3 Node 2 Node 1 Node 4 Switch (Cluster Interconnect) Node 6 Client LAN Figure 4-2.
4-10 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Planning Cluster Configurations for Non-Redundant Fibre Channel Arbitrated Loops This section describes sample configurations for midsize and larger clusters that use non-redundant Fibre Channel Arbitrated Loops (FC-ALs). IMPORTANT: Use the Oracle documentation identified in the front matter of this guide to obtain detailed information about planning for the Oracle software.
Cluster Planning The sample midsize cluster configuration shown in Figure 4-3 contains these hardware components: ■ Four ProLiant servers (cluster nodes) ■ One Fibre Host Adapter in each cluster node ■ One Storage Hub or FC-AL Switch installed between the Fibre Host Adapters and the shared storage arrays ■ Three RA4000/RA4100 Arrays ■ One RA4000 Array Controller in each RA4000/RA4100 Array ■ Ethernet NIC adapters, cables, and Ethernet switches or hubs for the Ethernet cluster interconnect (not s
4-12 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide The larger configuration shown in Figure 4-4 contains these hardware components: ■ Six ProLiant servers (cluster nodes) ■ One Fibre Host Adapter in each cluster node ■ One Storage Hub or FC-AL Switch to connect the RA4000/RA4100 Arrays to the Fibre Host Adapters ■ Six RA4000/RA4100 Arrays ■ One RA4000 Array Controller in each RA4000/RA4100 Array ■ Ethernet NIC adapters, cables, and Ethernet s
Cluster Planning In a PDC/O1000 cluster, each node is connected to shared storage disk drives housed in RA4000/RA4100 Arrays. When planning the amount of shared storage for your cluster, you must consider the following: ■ The maximum allowable number of shared storage arrays in one cluster.
4-14 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Supported RAID Levels RAID provides several fault-tolerant options to protect your cluster’s shared data. However, each RAID level offers a different mix of performance, reliability, and cost.
Cluster Planning Raw Data Storage and Database Size Raw data storage is the amount of storage available before any RAID levels have been configured. It is called raw data storage because RAID volumes require some overhead. The maximum size of a database stored in a RAID system will always be less than the amount of raw data storage available. To calculate the amount of raw data storage in a PDC/O1000 cluster, determine the total amount of shared storage space available to the cluster.
4-16 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Selecting the Appropriate RAID Levels Many factors affect which RAID levels you select for your cluster database. These include the specific availability, performance, reliability, and recovery capabilities required from the database. Each cluster must be evaluated individually by qualified personnel.
Cluster Planning Planning the Grouping of Physical Disk Storage Space Figure 4-5 shows how the physical storage space in one RA4000/RA4100 Array that contains eight physical disk drives might be grouped for an Oracle8i Parallel Server database.
4-18 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Use Disk Management to define one extended partition per RAID logical drive. Also using Disk Management, divide the extended partitions into logical partitions, each having its own drive letter. (Windows 2000 Advanced Server logical partitions are called “logical drives” in the Oracle documentation.) IMPORTANT: To performing partitioning, all required drivers must already be installed for each server.
Cluster Planning Network Planning Windows 2000 Advanced Server Hosts Files for an Ethernet Cluster Interconnect When an Ethernet cluster interconnect is installed between cluster nodes, the Compaq operating system dependent modules (OSDs) require a unique entry in the hosts and lmhosts files located at %SystemRoot%\system32\drivers\etc for each network port on each node.
4-20 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Client LAN Physically, the structure of the client network is no different than that used for a nonclustered configuration. To ensure continued access to the database when a cluster node is evicted from the cluster, each network client should have physical network access to all of the cluster nodes.
Chapter 5 Installation and Configuration This chapter provides instructions for installing and configuring the Compaq Parallel Database Cluster Model PDC/O1000 (referred to here as the PDC/O1000) for use with Oracle8i software. A PDC/O1000 is a combination of several individually available products. As you set up your cluster, have the following materials available during installation. You will find references to them throughout this chapter.
5-2 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide ■ Compaq StorageWorks Fibre Channel SAN Switch 16 Installation and Hardware Guide ■ Compaq StorageWorks Fibre Channel SAN Switch Management Guide ■ Compaq SmartStart Installation poster ■ Compaq SmartStart and Support Software CD ■ Microsoft Windows 2000 Advanced Server Administrator’s Guide ■ Microsoft Windows 2000 Advanced Server CD with Service Pack 1 or later ■ Compaq Parallel Database C
Installation and Configuration ■ ■ Installing and configuring operating system software, including: G SmartStart 4.
5-4 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Set up the hardware on one node completely, then set up the rest of the nodes identically to the first one. Do not load any software on any cluster node until all the hardware has been installed in all cluster nodes. Before loading software, read “Installing Operating System Software and Configuring the RA4000/RA4100 Arrays” in this chapter to understand the idiosyncrasies of configuring a cluster.
Installation and Configuration 5-5 To install GBIC-SW modules: 1. Verify that a GBIC-SW module has been installed in each Fibre Host Adapter in a server. 2. Insert a GBIC-SW module into the port for the Fibre Host Adapter on its Storage Hub, FC-AL Switch, or Fibre Channel SAN Switch. Cabling the Fibre Host Adapters to the Storage Hub, FC-AL Switch, or Fibre Channel SAN Switch Each non-redundant Fibre Channel Fabric requires on Fibre Channel SAN Switch.
5-6 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Figure 5-1 shows the Fibre Host Adapters in four nodes connected to one Storage Hub, FC-AL Switch, or Fibre Channel SAN Switch. Client LAN Fibre Host Adapters Fibre Host Adapters Switch (Cluster Interconnect) ProLiant Servers ProLiant Servers Storage Hub/Switch RA4000/4100 Array Figure 5-1.
Installation and Configuration 5-7 Ethernet Cluster Interconnect Adapters For a non-redundant Ethernet cluster interconnect, install one single-port or dual-port Ethernet adapter into each cluster node. For a redundant Ethernet cluster interconnect, install two single-port or two dual-port Ethernet adapters into each cluster node. For recommended dual-port and single-port Ethernet adapters, see the Compaq Parallel Database Cluster Model PDC/O1000 Certification Matrix for Windows 2000 at www.compaq.
5-8 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Setting Up the RA4000/RA4100 Arrays Unless otherwise indicated in this guide, follow the instructions in the Compaq StorageWorks RAID Array 4000 User Guide or the Compaq StorageWorks RAID Array 4100 User Guide to set up shared storage subsystem components.
Installation and Configuration 5-9 Installing GBIC-SW Modules for the RA4000 Array Controller For the PDC/O1000, each RA4000/RA4100 Array contains one single-port Compaq StorageWorks RAID Array 4000 Controller (RA4000 Array Controller) and ships with GBIC-SW modules. Insert one module into the RA4000 Array Controller and the other module into the Storage Hub, FC-AL Switch, or Fibre Channel SAN Switch port for the controller. To install GBIC-SW modules for an RA4000 Array Controller: 1.
5-10 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Installing Additional Fibre Channel Fabrics or FC-ALs At this point, you have installed the hardware for one non-redundant Fibre Channel Fabric or one non-redundant FC-AL.
Installation and Configuration 5-11 Figure 5-3 shows a PDC/O1000 with two non-redundant Fibre Channel Fabrics or two non-redundant FC-ALs. The hardware for the second Fibre Channel Fabric or FC-AL is shaded. NOTE: You can mix non-redundant Fibre Channel Fabrics and non-redundant FC-ALs in the same PDC/O1000 cluster. RA4000/4100 Arrays (8) Fibre Host Adapters Storage Hubs/Switches Fibre Host Adapters RA4000/4100 Arrays (4) Figure 5-3.
5-12 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide The following components are used in a redundant Ethernet cluster interconnect: ■ Two Ethernet adapters in each cluster node ■ Ethernet cables and switches or hubs G For two-node PDC/O1000 clusters, you can use two 100-Mbit/sec Ethernet switches or hubs with cables to connect the servers.
Installation and Configuration 5-13 Figure 5-4 shows the non-redundant Ethernet cluster interconnect components used in a two-node PDC/O1000 cluster. These components include a dual-port Ethernet adapter in each node. The top port on each adapter connects by Ethernet crossover cable to the top port on the adapter in the other node. The bottom port on each adapter connects by Ethernet cable to the client LAN switch or hub.
5-14 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Figure 5-5 shows another option for a non-redundant Ethernet cluster interconnect in a two-node PDC/O1000 cluster. These include an Ethernet adapter in each node connected by Ethernet cables to an Ethernet switch or hub. RA4000/4100 Array Storage Hub/Switch Ethernet Adapter Ethernet Adapter Storage Hub/Switch Ethernet cables Node 1 Node 2 Client LAN Figure 5-5.
Installation and Configuration 5-15 Figure 5-6 shows the redundant Ethernet cluster interconnect components used in a PDC/O1000 with two or more nodes. Ethernet Switch/Hub #1 for Cluster Interconnect Ethernet Switch/Hub #2 for Cluster Interconnect Crossover Cable Dual-port Ethernet Adapters (2) Dual-port Ethernet Adapters (2) Node 1 Crossover Cable Node 2 Client LAN Hub/Switch #1 Client LAN Hub/Switch #2 Figure 5-6.
5-16 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Cabling the Client LAN You can use any TCP/IP network to connect to a client LAN. The following procedure contains instructions for cabling an Ethernet client LAN. To cable an Ethernet client LAN: 1. Insert one end of an Ethernet cable into an Ethernet adapter port designated for the client LAN in a cluster node.
Installation and Configuration 5-17 Installing Operating System Software and Configuring the RA4000/RA4100 Arrays You will follow an automated procedure using Compaq SmartStart to install the operating system software and configure the shared storage on the RA4000/RA4100 Arrays. Guidelines for Clusters Installing clustering software requires several specific steps and guidelines that might not be necessary when installing software on a single server.
5-18 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide ■ ■ Be sure to set up unique IP addresses and node names for each node in the hosts and lmhosts files at %SystemRoot%\system32\drivers\etc. G For an Ethernet cluster interconnect, one IP address and node name is for the cluster interconnect, and the other IP address and node name is for the client LAN. Both entries are required for each node in the cluster.
Installation and Configuration 5-19 Automated Installation Steps You will perform the following automated installation steps to install operating system software on every node in the cluster. 1. Power up the following cluster components in this order: G RA4000/RA4100 Arrays G Storage Hubs, FC-AL Switches, or Fibre Channel SAN Switches G Ethernet hubs or switches 2. Power up a cluster node and put the SmartStart and Support Software CD into the CD-ROM drive. 3.
5-20 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide 9. Enter unique IP addresses and node names for each node in the hosts and lmhosts files located at %SystemRoot%\system32\drivers\etc. Record this information: G For the Ethernet cluster interconnect, one IP address and node name is for the redundant cluster interconnect, and the other IP address and node name is for the client LAN.
Installation and Configuration 5-21 e. With the client LAN NIC selected, click Move Up to position this NIC to the top of the list. f. Click OK on the dialog box and restart the node when prompted. IMPORTANT: Record the cluster interconnect node name, the client LAN node name, and the IP addresses assigned to them. You will need this information later when installing Compaq OSDs. 12.
5-22 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Installing Compaq OSDs Use the Oracle Universal Installer (OUI) program to install the Compaq operating system dependent modules (OSDs) for Oracle8i Parallel Server. Compaq supplies a software package with the OUI that installs Compaq OSDs for an Ethernet cluster interconnect.
Installation and Configuration 5-23 If you are using an Ethernet cluster interconnect, a cluster interconnect name and IP address were assigned to each cluster node. Ping the Ethernet cluster interconnect name of every node in the cluster. When using the client LAN name with the ping command, the IP address displayed by the ping utility should be the client LAN IP address.
5-24 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide To mount drives on remote nodes, enter a net use command from a C: prompt on the installing machine. Enter one instance of the command for each remote cluster node. The following example mounts the C: drive on a remote node. C: net use \\machine_name\C$ * /User: The “*” results in the system prompting you for a password, and the password is not echoed.
Installation and Configuration 5-25 3. If autorun is enabled, the Oracle Universal Installer Welcome screen appears. If autorun is disabled, enter d:\setup at a C: prompt. Substitute the letter assigned to the CD drive for d. The Oracle Universal Installer Welcome screen appears. NOTE: The OUI screens presented here might indicate earlier revision levels for the OSDs.
5-26 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide 4. Click Next to continue. The License screen appears. 5. Click Accept to accept the terms of the license agreement. The File Locations screen appears.
Installation and Configuration 5-27 6. Click Next to continue. The Installation Types screen appears. Compaq Confidential – Need to Know Required Writer: Carol Tatick Project: Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Comments: Part Number: 225083-002 File Name: f-ch5 Installation and Configuration.
5-28 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide 7. Select Typical to install the Ethernet OSDs into the default location, C:\Compaq\OPS. Skip to step 12. 8. Select Custom to install the Ethernet OSDs into a location of your choice. The Component Locations screen appears.
Installation and Configuration 5-29 9. If you want to accept the default location, skip to step 13. 10. If you want to specify an alternate location and you know the exact directory path, specify it by typing over the default location and skip to step 12. 11. If you want to specify an alternate location but do not know the exact directory path, click Change Location. The Choose Directory screen appears.
5-30 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide 12. Browse to the location where you want to install the OSDs. Click OK. The Component Locations screen appears indicating the alternate location you specified.
Installation and Configuration 5-31 13. Click Next to continue. The Cluster Members screen appears. Compaq Confidential – Need to Know Required Writer: Carol Tatick Project: Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Comments: Part Number: 225083-002 File Name: f-ch5 Installation and Configuration.
5-32 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide 14. The Cluster Members screen indicates the host name of the installing machine. Define the node names for the client LAN by adding the node names you specified in the hosts and lmhosts files for the client LAN for all other cluster nodes. Do not use special characters. Use a space to separate the node names. 15. Click Next to continue. The Cluster Members screen reappears.
Installation and Configuration 5-33 16. The Cluster Members screen now shows the node names you specified on the first Cluster Members screen with “_san” appended to each name, indicating it as a cluster interconnect node name. If necessary, retype the cluster interconnect node names to match the names you entered into the hosts and lmhosts files for the Ethernet cluster interconnect. Do not use special characters. Separate the node names with a space. 17. Click Next to continue.
5-34 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide 18. If the information on the OSD Summary screen is not correct, click Previous to modify the previous screens. When the information is correct, click Next to continue. The Summary screen appears.
Installation and Configuration 5-35 19. Click Install to install the components listed in the summary. At this point, the OUI verifies that the installing node can communicate with other cluster nodes through both the client LAN and cluster interconnect node names. If the OUI cannot communicate with all nodes, this process can take awhile. If the node is not accessible, an error message will appear asking you to check the hostname and the network connection. 20.
5-36 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Installing Oracle Software Using the Oracle8i Parallel Server Setup and Configuration Guide, follow the steps to install Oracle8i Enterprise Edition software on all cluster nodes, including: ■ Oracle8i Server ■ Oracle8i Parallel Server Option ■ Oracle8i Parallel Server Manager ■ Oracle8i Enterprise Manager NOTE: If the Oracle Universal Installer does not offer the option to install Oracle8i Paral
Installation and Configuration 5-37 powered on in a different order than they were when the cluster was initially configured, the order and numbering of the shared disk drives will change for all nodes in the cluster. Object Link Manager simplifies the creation and maintenance of symbolic links between disk partitions and Oracle data files by placing the symbolic link names directly into the disk partitions, thereby tracking the symbolic links dynamically.
5-38 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Verifying the Hardware and Software Installation Cluster Communications Use the ping utility to verify that each node in the cluster can communicate with every other node over the client LAN. Run the ping utility from the installing cluster node. Verify the installing node can communicate with all cluster nodes by pinging the client LAN name.
Installation and Configuration 5-39 Other Verification Tasks Perform these other verification tasks: ■ Verify that Object Link Manager has been installed and is running. ■ Verify that the OPS_CMDISK symbolic link has actually been set. To verify the Object Link Manager has been installed and is running, select the Services icon in the Control Panel. Once there, look for Oracle Object Service in the list of services and see if it is shown as “Started.
5-40 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Server Power Distribution Figure 5-7 shows an example of server power distribution for a three-node PDC/O1000 cluster. Power supply #1 of each server is connected to PDU #1. Power supply #2 of each server is connected to PDU #2. PDU #1 is connected to UPS #1, and PDU #2 is connected to UPS #2. Each UPS is connected to building power.
Installation and Configuration 5-41 Power Sequencing Be sure to power up the cluster components in the following order: 1. RA4000/RA4100 Arrays 2. Storage Hubs, FC-AL Switches, or Fibre Channel SAN Switches (Power is applied to a Storage Hub when the AC power cord is plugged in.) 3. Ethernet hubs or switches 4. ProLiant servers Be sure to power down the cluster components in the following order: 1. ProLiant servers 2. Ethernet hubs or switches 3.
Chapter 6 Cluster Management Throughout the life of your cluster, you might need to improve its performance, upgrade hardware components, upgrade software, increase shared storage capacity, and monitor ongoing activities. This chapter describes these management activities for the Compaq Parallel Database Cluster Model PDC/O1000 (referred to here as the PDC/O1000).
6-2 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Cluster Management Concepts Powering Off a Node Without Interrupting Cluster Services At some time during the life of your cluster you will need to perform an operation on a cluster node that will require it to be powered off. Physically moving the cluster node, removing a hardware device, and adding a hardware device commonly require the node to be powered off.
Cluster Management 4. Determine how critical it is to repair the problem. G If the problem is not considered to be critical, wait until a non-peak time to service the problem. G If the problem is critical but does not affect all cluster nodes, shut down the Oracle instances on the affected nodes and wait until a non-peak time to service the problem. G If the problem is critical and affects all the cluster nodes, shut down the database instances on all cluster nodes and correct the problem.
6-4 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide The following log files are used: ■ The Node Manager (NM), provided by Compaq, logs messages to C:\compaq\ops\nm.log ■ By default the OracleCMService sends messages to C:\compaq\ops\cmsrvr.log Management Applications Monitoring Server and Network Hardware Compaq Insight Manager is used to manage the hardware components of your cluster.
Cluster Management Compaq Insight Manager XE is a web-based management system that can also be used to monitor cluster hardware components. Compaq Insight Manager XE is an optional CD available upon request from the Compaq System Management website at www.compaq.com/sysmanage Managing Shared Drives There are two levels of management with shared drives in RA4000 Arrays or RA4100 Arrays. The physical disks and the drive arrays created from them are managed with the Array Configuration Utility (ACU).
6-6 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide ■ Gigabit Interface Converter-Shortwave (GBIC-SW) modules ■ Fibre Channel cables Monitoring Non-Redundant Fibre Channel Arbitrated Loops Each non-redundant FC-AL in a PDC/O1000 contains this hardware: ■ One or more RA4100 Arrays or RA4000 Arrays ■ One single-port RA4000 Array Controller installed in each RA4000 Array or RA4100 Array ■ One Compaq StorageWorks Storage Hub (Storage Hub) or Compaq S
Cluster Management Several steps are required to properly configure the system for Oracle8i Enterprise Manager. Additional steps are required to integrate the management of parallel databases. For example, you must do the following: ■ Create an Enterprise Manager Repository on the console node. ■ Obtain sufficient rights on the console machine as well as on the managed cluster nodes. ■ Install Oracle agents on each managed node.
6-8 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Software Maintenance for Oracle8i Deinstalling the OSDs At some point, you might want to deinstall the operating system dependent modules (OSDs).
Cluster Management 4. If autorun is disabled, enter d:\setup at a C: prompt. Substitute the letter assigned to the CD-ROM drive for d. The Oracle Universal Installer Welcome screen appears. Compaq Confidential – Need to Know Required Writer: John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Comments: Part Number: 225083-002 File Name: g-ch6 Cluster Management.
6-10 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide 5. Click Deinstall Products. Next, a window containing an inventory of the installed components appears. The following shows the Inventory screen that is displayed when Ethernet is used for the cluster interconnect. 6. To deinstall OSDs, select Compaq Ethernet OSDs from its Inventory screen and click Remove. NOTE: Your screen might display later revision levels for the OSDs. 7.
Cluster Management Upgrading Oracle8i Server The design of the PDC/O1000 is tightly integrated with Oracle8i Server. Significant changes in Oracle8i Server will likely affect the operation of the cluster. Before upgrading to any new release of Oracle8i Server, consult your Compaq and Oracle service representatives. Make sure that the new release is certified and supported on the PDC/O1000, and get assistance to determine which procedure you should follow to perform the upgrade properly.
6-12 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Upgrading the Ethernet OSDs To upgrade the Ethernet OSDs: 1. Insert the CD containing the new revision of the OSDs into the CD-ROM drive on the primary node. (This is the node that was used for the previous installation of the OSDs.) The Oracle Universal Installer Welcome screen is displayed. 2. Click Deinstall Products. The Inventory screen displays a list of installed components.
Cluster Management 3. Select Compaq Ethernet OSDs and click Remove. The Oracle Universal Installer removes all Ethernet OSDs. 4. After the deinstallation has finished, restart all nodes. 5. Install the new Ethernet OSDs from the new by following the procedure described under “Installing Compaq OSDs” in Chapter 5, “Installation and Configuration.” 6. After installing the new Ethernet OSDs, restart OracleCMService and OracleNMService on all nodes. 7. Restart the Oracle instances on all nodes.
6-14 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide 4. Run the uninstall utility by entering: uninstallOSD 5. Restart the node. 6. Repeat steps 1 through 5 on all other nodes in the PDC/O1000 cluster. Upgrading Oracle8i Server The design of the PDC/O1000 is tightly integrated with Oracle8i Server. Significant changes in Oracle8i Server will likely affect the operation of the cluster.
Cluster Management The removal of a drive in an RA4000 Array or RA4100 Array must follow certain rules are that interpreted by reading the LEDs on the disk drive. Before proceeding, locate the information in the Compaq StorageWorks RAID Array 4000 User Guide or the Compaq StorageWorks RAID Array 4100 User Guide that describes the conditions under which a drive can be removed.
6-16 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Adding an RA4000/RA4100 Array You must shut down the PDC/O1000 cluster to add an RA4000/RA4100 Array. The following steps summarize the procedure: 1. Shut down the Oracle instance on each cluster node. 2. Stop the Oracle services running on each node. 3. Shut down Windows 2000 Advanced Server on each node and power off all the cluster nodes. 4. Power off all of the RA4000/RA4100 Arrays in the cluster.
Cluster Management Replacing a Failed Fibre Host Adapter To replace a failed Fibre Host Adapter in a cluster node: 1. Shut down the Oracle instance on the node. 2. Shut down the Oracle services running on the node. 3. Shut down the server. 4. Power off the node. 5. From the Fibre Host Adapter, disconnect the Fibre Channel cable installed between the Fibre Host Adapter and its Fibre Channel SAN Switch, FC-AL Switch, or Storage Hub. 6. Remove the failed Fibre Host Adapter and insert a new one. 7.
6-18 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Replacing a Cluster Node Before replacing a node, verify that the new configuration is supported by the PDC/O1000 cluster. To determine if your new server is supported, see the Compaq Parallel Database Cluster Model PDC/O1000 Certification Matrix for Windows 2000 at www.compaq.com/solutions/enterprise/ha-pdc.
Cluster Management 7. Disconnect the cluster interconnect cables from the evicted node and label them as the cluster interconnect cables. 8. Disconnect the client LAN cables from the evicted node and label them as the client LAN cables. 9. If adapters from the evicted node will be used in the new node, move the adapters to the new node.
6-20 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide 7. Install the Fibre Host Adapter or Fibre Host Adapters in the replacement node in the same server PCI slot locations used in the other cluster nodes. 8. Enter unique IP addresses and node names for each node in the hosts and lmhosts files located at %SystemRoot%\system32\drivers\etc. 9.
Cluster Management 2. If the replacement node is the node on which you originally installed the OSDs, do the following on all nodes: a. Shut down the Oracle instance. b. Stop all Oracle services. c. Uninstall the OSDs from each node by running uninstallOSD, located in the Utilities directory of the installation CD. d. Restart all nodes to complete the deinstallation process. 3. Use the OUI to reinstall the OSDs on each cluster node, including the original nodes and the replacement node. 4.
6-22 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide This can be a time-consuming and complex procedure. Be sure to read through all of the following directions as well as those in the Oracle documentation before starting. Adding a new node involves the following activities: ■ Preparing the new node ■ Preparing the existing cluster nodes ■ Installing the cluster software These subsections contain a high-level outline of the steps to follow.
Cluster Management 8. Using Fibre Channel cables, connect each Fibre Host Adapter in the new node to its Fibre Channel SAN Switch, FC-AL Switch, or Storage Hub. 9. Restart the new node. 10. Verify TCP/IP connectivity between the new node and the existing cluster nodes. Run the ping utility from the new node, pinging the client LAN adapters on all other nodes. If Ethernet is used as the cluster interconnect, also ping the cluster interconnect adapters of all other nodes. 11.
6-24 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide To install the cluster software, perform the following steps: 1. To add a node to the cluster, you must disassemble the existing cluster and then reform it to include the new node. Do the following on all original cluster nodes (not the new node): a. Shut down the Oracle instance. b. Stop all Oracle services. c. Deinstall the OSDs by running the OUI on the node on which the OUI was originally run. d.
Cluster Management Monitoring Cluster Operation Tools Overview Several tools exist to monitor the operation of your cluster. Use the tools to regularly monitor the operation of each node and each Oracle instance. The gathered information might identify a disparity in the workload being performed by each node, which signifies a need to adjust the distribution of the workload so that the overall cluster performance is maximized.
Chapter 7 Troubleshooting This chapter addresses potential problems you might encounter as you install, configure, test, and operate your Compaq Parallel Database Cluster Model PDC/O1000 (referred to here as the PDC/O1000).
7-2 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Basic Troubleshooting Tips Power Verify that the following are all powered on: ■ ProLiant servers (cluster nodes) ■ StorageWorks RAID Array 4000s (RA4000 Arrays) or StorageWorks RAID Array 4100s (RA4100 Arrays) ■ Compaq StorageWorks Fibre Channel SAN Switches (Fibre Channel SAN Switches), Compaq StorageWorks Storage Hubs (Storage Hubs), or Compaq StorageWorks FC-AL Switches (FC-AL Switches) ■ Swit
Troubleshooting Access to Cluster Components Once the physical connections have been checked, ensure that communication is occurring between all the cluster components. ■ To test Ethernet cluster interconnect and client LAN connections, use the ping utility. From each cluster node, ping the Ethernet cluster interconnect adapter or port located in the other nodes. Similarly, from each cluster node, ping the name, then the address, of the client LAN Ethernet adapter or port located in the other nodes.
7-4 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide For information about currently supported software revisions for the PDC/O1000, refer to the Compaq Parallel Database Cluster Model PDC/O1000 Certification Matrix for Windows 2000 at www.compaq.com/solutions/enterprise/ha-pdc.html Depending on your specific configuration, other software (for example, customer applications such as packet utilities) might exist that should be of the same revision level.
Troubleshooting Troubleshooting Oracle8i and OSD Installation Problems and Error Messages Potential Difficulties Installing the OSDs with the Oracle Universal Installer ■ The Oracle Universal Installer (OUI) uses an inventory file to track which components of the OSDs have been installed on each of the cluster nodes. A single inventory file is maintained on the node from which the OUI was initially run. All subsequent uses of the OUI must be made from that same installing node.
7-6 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide ■ Determine whether or not you need to deinstall a partial OSD installation. An incomplete installation can result in a partial installation of the OSDs on the cluster. If this occurs, you must clean up after the partial installation (by removing it) before you can successfully run the OUI again.
Troubleshooting Unable to Start OracleNMService ■ Verify that the OracleNMService has properly started by checking the error log. The default location for the error log is C:\compaq\ops\nm.log. If not found in the default location, look in the same directory in which the OSD modules were installed. ■ Verify that the Oracle8i software is correctly installed on all the nodes. Refer to Chapter 5, “Installation and Configuration,” for an installation overview.
7-8 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Initialization of the Dynamic Link Library NM.DLL Failed If this failure occurs, stop the currently running OracleCMService and OracleNMService. Restart the OracleNMService, then the OracleCMService. From Administrative Tools, open the Services screen. Locate each service in the displayed list, highlight it, then stop it. Finally, highlight the service and restart it.
Troubleshooting When Using an Ethernet Cluster Interconnect ■ Make sure that the Ethernet cluster interconnect components (adapters, cables, switches or hubs) are properly installed as specified in Chapter 5, “Installation and Configuration.” ■ Make sure that the Ethernet cluster interconnect and the client LAN are connected to different Ethernet switches or hubs. ■ Use the ping utility to test the Ethernet cluster interconnect connections.
7-10 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Node or Nodes Unable to Rejoin the Cluster ■ If you have an Ethernet cluster interconnect, use ping to ensure that the rejoined node or nodes can communicate with other nodes in the cluster. For information about using the ping utility to test redundant Ethernet cluster interconnect connections, see “Verifying Cluster Communications” in Chapter 5, “Installation and Configuration.
Troubleshooting ■ Make sure that the TNSNames.ora file is properly configured on client and cluster nodes as defined by the Oracle Net8 Config Assistant utility. ■ Make sure that the Oracle TNSListener is running on the cluster nodes. From the Services screen, verify that Oracle TNSListener is started. ■ Make sure that the Oracle Listener files are set up on the cluster nodes by using the Oracle TNSPing utility.
7-12 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Verifying Connectivity to a Non-Redundant Fibre Channel Arbitrated Loop To ensure that all I/O path connections in a non-redundant FC-AL are intact and functioning, verify the following: ■ For each redundant FC-AL, one Fibre Host Adapter has been installed in the same PCI slot across all servers in the cluster.
Troubleshooting ■ Determine whether or not the Compaq servers support the physical SCSI disks installed in the RA4000/RA4100 Arrays. Check the event log for disk I/O error messages or indications of problems with the communications transport. ■ If the drives are still not recognized, it is recommended that you start investigating the problem from the lowest level of drive configuration. G Run the Compaq Array Configuration Utility.
7-14 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide A Cluster Node Cannot Connect to the Shared Drives ■ Verify that a physical connection exists between each Fibre Host Adapter and its Fibre Channel SAN Switch, FC-AL Switch, or Storage Hub. G Verify that a GBIC-SW module is properly installed in each Fibre Host Adapter, all active ports on each Fibre Channel SAN Switch, FC-AL Switch, or Storage Hub, and each RA4000 Array Controller.
Appendix A Diagnosing and Resolving Shared Disk Problems Introduction When one or more nodes in a Compaq Parallel Database Cluster Model PDC/O1000 (referred to here as the PDC/O1000) cannot properly join the cluster, you need to consider two possible causes: ■ A problem related to the shared disk storage in the cluster’s Compaq StorageWorks RAID Array 4000s (RA4000 Arrays) or Compaq StorageWorks RAID Array 4100s (RA4100 Arrays) is preventing one or more nodes from accurately viewing or successfully acce
A-2 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Figure A-1 identifies, in a flowchart format, the sequence of tasks you should perform to diagnose, isolate, and resolve shared disk problems in a PDC/O1000 cluster. A detailed description of each task is provided in the following sections of this appendix. Entry Point: Potential Shared Disk-Related Problem Detected Run Object Link Manager On All Nodes OK Problem Persists Exit.
Diagnosing and Resolving Shared Disk Problems Run Object Link Manager On All Nodes Object Link Manager simplifies the creation and maintenance of symbolic links between disk partitions and Oracle data files by placing the symbolic link names directly onto the disk partitions, thereby tracking the symbolic links dynamically. This means that if the order in which disk drives are brought online changes, Oracle8i Server can find the correct data files and can properly start the Oracle database.
A-4 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Restart All Affected Nodes in the Cluster If Object Link Manager does not run properly on one or more nodes or does not provide consistent information about Oracle partitions and symbolic links from all nodes, perform these steps: 1. Identify the node or nodes on which Object Link Manager either does not run properly or does not present the correct information about Oracle partitions and symbolic links.
Diagnosing and Resolving Shared Disk Problems Run Disk Management On All Nodes Perform these steps from each node: 1. From Disk Management, verify that each RA4000 Array or RA4100 Array is visible. 2. Verify that all shared drive information (drive letters, volume labels, number of partitions, partition sizes, etc.) is consistent across all nodes. 3. Repeat steps 1 and 2 for all nodes in the cluster.
A-6 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Perform Cluster Software and Firmware Checks Use the Compaq SmartStart and Support Software CD to perform these checks on the cluster software and firmware: 1. Verify that all necessary drivers have been installed. For example, verify that the Compaq StorageWorks Fibre Channel Host Adapter (Fibre Host Adapter) driver for Windows 2000 Advanced Server is installed and running on all cluster nodes. 2.
Diagnosing and Resolving Shared Disk Problems 4. Verify that the Gigabit Interface Converter-Shortwave (GBIA-SW) modules are all the correct brand or model type and are properly seated in the Fibre Host Adapters, Fibre Channel SAN Switches, Storage Hubs, or FA-AL Switches, and RA4000 Array Controllers. 5. Inspect all Fibre Channel cables to verify that they are in good condition and have no breaks or excessive bends. 6.
Glossary Availability A measure of how well a computer system can deliver services to its clients continuously. Availability is typically expressed as a percentage, with 100 percent being the best possible rating. Client LAN A local area network dedicated to servicing requests between network client machines and between the cluster nodes and the client machines. This is different than a cluster interconnect, which is dedicated to servicing information requests to and from cluster nodes only.
2 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Fibre Channel An IEEE standard for providing high-speed data transfer among workstations, mainframes, supercomputers, desktop computers, storage devices, and display devices. Fibre Channel Arbitrated Loop (FC-AL) A storage area network (SAN) topology that uses the Storage Hub or FC-AL Switch to connect each cluster node with the shared storage resources.
Glossary 3 IP Address Internet Protocol Address. An address assigned to a network interface card, which computer entities use to locate and communicate with each other. IP addresses can be statically or dynamically assigned. LAN Local Area Network. Load balancing To balance the amount of work (processing) undertaken by each node in a cluster. NIC Network Interface Controller.
4 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Reliability The continuous integrity of a system (server, storage, network, or cluster). Resource A software or hardware entity upon which a client/server application or service is dependent. As it pertains to the PDC/O1000, a cluster resource must have the ability to be managed by the cluster and must reside on one of the cluster nodes.
Index A C ACU See Compaq Array Configuration Utility adding disk drives in RA4000/RA4100 Arrays 6-15 new nodes 6-21 RA4000/RA4100 Arrays 6-16 replacement nodes 6-19 administrator privileges when installing OSDs for Oracle8i 5-23 application failover software for Oracle8i 3-7 Array Configuration Utility See Compaq Array Configuration Utility automated installation for Windows 2000 Advanced Server 5-18 availability defined 1-3 features of Compaq ProLiant Servers 2-3 cabling client LAN 5-16 Ethernet cluste
2 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide redundant 5-7, 5-16 transmissions allowed 5-7 verifying communications 5-22 verifying IP addresses 5-20 client reconnection software for Oracle8i 3-7 cluster communications verifying before installing OSDs for Oracle8i 5-22 verifying over client LAN 5-38 verifying over Ethernet cluster interconnect 5-38 cluster configuration planning sample large cluster with nonredundant FC-AL 4-11 sample large cluster wi
Index cluster troubleshooting basic tips 7-2 client to cluster connectivity problems 7-10 cluster cannot communicate with client 7-10 cluster connectivity 7-9 Ethernet cluster interconnect 7-9 FC-AL connectivity guidelines 7-12 Fibre Channel Fabric connectivity guidelines 7-11 NM.
4 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Compaq Options ROMPaq, overview 3-4 Compaq Parallel Database Cluster, overview 1-4 Compaq SmartStart and Support software 3-2 Compaq SmartStart and Support Software CD 3-2 Compaq software overview Compaq Array Configuration Utility 3-3 Compaq Insight Manager 3-4 Compaq Insight Manager XE 3-4 Compaq operating system dependent modules for Oracle8i 3-5 Compaq Options ROMPaq 3-4 Compaq SmartStart and Support s
Index Disk Management running to check for shared disk problems A-5 verifying shared storage 5-21 distributed lock manager 2-23 downtime 1-3 E error logs for Oracle8i 6-4 Ethernet adapters planning 4-6 Ethernet client LAN cabling 5-16 Ethernet cluster interconnect 2-23 adapter installation 5-7 adapters 2-24 adapters shared by cluster interconnect and client LAN 2-26 cabling 5-11, 5-15 Ethernet switch 2-25 non-redundant Ethernet cluster interconnect 2-24 redundant 5-7 redundant Ethernet cluster interconnec
6 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide Fibre Host Adapters See Compaq StorageWorks Fibre Channel Host Bus Adapters replacing 6-17 firmware revisions 7-4 G GBICs See Gigabit Interface Converter-Shortwave modules getting help xix Gigabit Interface ConverterShortwave modules description 2-8 installing 5-4, 5-9 grouping physical disk storage space 4-17 H hardware installation verification 5-38 help additional sources xix Compaq authorized reselle
Index local area network (LAN) 2-29 log files for Oracle8i 5-38, 6-4 M managing network clients 6-3 Microsoft Windows 2000 Advanced Server installing for the PDC/O1000 5-18 overview 3-2 versions supported 3-2 multiple FC-ALs overview 2-18 multiple Fibre Channel Fabrics overview 2-11 multiple non-redundant FC-ALs example 2-19 overview 2-18 multiple non-redundant Fibre Channel Fabrics example 2-12 overview 2-11 N net use command 5-23 network clients database access 6-3 network protocols TCP/IP 5-17 nonshar
8 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide OSDs See Compaq operating system dependent modules OUI See Oracle Universal Installer overview cluster management 6-1 cluster software for Oracle8i 3-1 Compaq Parallel Database Cluster 1-4 Compaq software 3-2 installation 5-2 Oracle8i Server Enterprise Edition 3-5 Oracle8i software 3-5 shared disk problems A-1 shared storage components for non-redundant Fibre Channel Fabrics 4-4 site planning 4-2 supporte
Index redundant client LAN configuration 5-7 configuring 5-16 redundant Ethernet client LAN configuring 5-16 Redundant Ethernet client LAN 5-16 redundant Ethernet cluster interconnect adapter installation 5-7 cabling 5-11, 5-15 configuring 5-15 replacing cluster nodes 6-18 disk drives in RA4000/RA4100 Arrays 6-14 Fibre Host Adapters 6-17 RJ-45 receptacle xviii ROMPaq See Compaq Options Rompaq S SCU See Compaq System Configuration Utility Service Packs for Windows 2000 Advanced Server supported for the PDC
10 Compaq Parallel Database Cluster Model PDC/O1000 for Oracle8i and Windows 2000 Administrator Guide cluster degradation 6-2 cluster interconnect 7-8 Ethernet cluster interconnect 7-9 FC-AL connectivity guidelines 7-12 Fibre Channel Fabric connectivity guidelines 7-11 NM.
