Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Second Edition (June 2001) Part Number 225082-002 Compaq Computer Corporation Compaq Confidential – Need to Know Required Writer: John Blackburn Project: Parallel Database Cluster Model PDC/02000 for Oracle8i and Windows 2000 Comments: Part Number: 225082-002 File Name: a-frnt.
Notice © 2001 Compaq Computer Corporation Compaq, the Compaq logo, Compaq Insight Manager, SmartStart, ROMPaq, ProLiant, and StorageWorks Registered in U.S. Patent and Trademark Office. ActiveAnswers is a trademark of Compaq Information Technologies Group, L.P. in the United States and other countries. Microsoft, Windows, and Windows NT are trademarks of Microsoft Corporation in the United States and other countries. All other product names mentioned herein may be trademarks of their respective companies.
Contents About This Guide Purpose .................................................................................................................... xiii Audience.................................................................................................................. xiii Scope ........................................................................................................................xiv Referenced Manuals ..............................................................................
iv Compaq Parallel Database Cluster Model PDC/02000 for Oracle8i and Windows 2000 Administrator Guide Chapter 2 Cluster Architecture Compaq ProLiant Servers ........................................................................................ 2-2 High-Availability Features of ProLiant Servers ............................................... 2-3 Shared Storage Components .................................................................................... 2-3 RA4000 Array..................................
Contents Cluster Architecture continued Cluster Interconnect Options ................................................................................. 2-70 Ethernet Cluster Interconnect ......................................................................... 2-70 Local Area Network ....................................................................................... 2-76 Chapter 3 Cluster Software Components Overview of the Cluster Software..............................................................
vi Compaq Parallel Database Cluster Model PDC/02000 for Oracle8i and Windows 2000 Administrator Guide Cluster Planning continued Planning Cluster Configurations for Redundant Fibre Channel Arbitrated Loops ..................................................................................................................... 4-11 Sample Small Configuration in a Redundant FC-AL ..................................... 4-11 Sample Large Configuration in a Redundant FC-AL .....................................
Contents Installing Compaq OSDs ....................................................................................... 5-30 Verifying Cluster Communications ................................................................ 5-31 Mounting Remote Drives and Verifying Administrator Privileges ................ 5-32 Installing the Ethernet OSDs .......................................................................... 5-33 Installing Oracle Software ...............................................................
viii Compaq Parallel Database Cluster Model PDC/02000 for Oracle8i and Windows 2000 Administrator Guide Cluster Management continued Replacing a Cluster Node ...................................................................................... 6-17 Removing the Node ........................................................................................ 6-17 Adding the Replacement Node ....................................................................... 6-18 Adding a Cluster Node.....................
Contents Troubleshooting continued Troubleshooting Secure Path ................................................................................. 7-14 Secure Path Guidelines for Windows 2000 Advanced Server........................ 7-14 Secure Path Manager Cannot Start With Hosts That Use Hyphenated Host Names............................................................................................................. 7-15 Secure Path Manager Is Delayed In Reporting Path Failure Information.......
x Compaq Parallel Database Cluster Model PDC/02000 for Oracle8i and Windows 2000 Administrator Guide List of Figures Figure 1-1. Example of a two-node Compaq Parallel Database Model PDC/O2000 cluster ................................................................................. 1-2 Figure 2-1. Two-node PDC/O2000 with a two-fabric redundant Fibre Channel Fabric ................................................................................................. 2-10 Figure 2-2.
Contents Figure 2-25. Active/active configuration with two RA4000/RA4100 Arrays ...... 2-60 Figure 2-26. Active/active configuration with three RA4000/RA4100 Arrays............................................................................................................... 2-61 Figure 2-27. Active/active configuration with four RA4000/RA4100 Arrays............................................................................................................... 2-62 Figure 2-28.
xii Compaq Parallel Database Cluster Model PDC/02000 for Oracle8i and Windows 2000 Administrator Guide List of Tables Table 2-1 High-Availability Components of ProLiant Servers................................ 2-3 Table 2-2 Features of Active/Standby and Active/Active Configurations for Redundant Fibre Channel Fabrics ....................................................................
About This Guide Purpose This administrator guide provides information about the planning, installation, configuration, implementation, management, and troubleshooting of the Compaq Parallel Database Cluster Model PDC/O2000 running Oracle8i software on the Microsoft Windows 2000 Advanced Server operating system. Audience The expected audience of this guide consists primarily of MIS professionals whose jobs include designing, installing, configuring, and maintaining Compaq Parallel Database Clusters.
xiv Compaq Parallel Database Cluster Model PDC/02000 for Oracle8i and Windows 2000 Administrator Guide Scope This guide offers significant background information about clusters as well as basic concepts associated with designing clusters. It also contains detailed product descriptions and installation steps.
About This Guide Some clustering topics are mentioned, but not detailed, in this guide. For example, this guide does not describe how to install and configure Oracle8i on a cluster. For information about these topics, see the documents referenced in the guide sections or refer to the documentation provided with the Oracle software. Referenced Manuals For additional information, refer to documentation related to the specific hardware and software components of the Compaq Parallel Database Cluster.
xvi Compaq Parallel Database Cluster Model PDC/02000 for Oracle8i and Windows 2000 Administrator Guide ■ Oracle8i documentation, including: G Oracle8i Parallel Server Setup and Configuration Guide G Oracle8i Parallel Server Concepts G Oracle8i Parallel Server Administration, Deployment, and Performance G Oracle Enterprise Manager Administrator’s Guide G Oracle Enterprise Manager Configuration Guide G Oracle Enterprise Manager Concepts Guide Supplemental Documents The following technical docum
About This Guide Text Conventions This document uses the following conventions to distinguish elements of text: User Input, GUI Selections Text a user types or enters appears in boldface. Items a user selects from a GUI, such as tabs, buttons, or menu items, also appear in boldface. User input and GUI selections can appear in uppercase and lowercase letters. File Names, Command Names, Directory Names, Drive Names These elements can appear in uppercase and lowercase letters.
xviii Compaq Parallel Database Cluster Model PDC/02000 for Oracle8i and Windows 2000 Administrator Guide Symbols on Equipment These icons may be located on equipment in areas where hazardous conditions may exist. Any surface or area of the equipment marked with these symbols indicates the presence of electrical shock hazards. Enclosed area contains no operator serviceable parts. WARNING: To reduce the risk of injury from electrical shock hazards, do not open this enclosure.
About This Guide Rack Stability WARNING: To reduce the risk of personal injury or damage to the equipment, be sure that: ■ The leveling jacks are extended to the floor. ■ The full weight of the rack rests on the leveling jacks. ■ The stabilizing feet are attached to the rack if it is a single rack installations. ■ The racks are coupled together in multiple rack installations. ■ Only one component is extended at a time.
xx Compaq Parallel Database Cluster Model PDC/02000 for Oracle8i and Windows 2000 Administrator Guide Compaq Website The Compaq website has information on this product as well as the latest drivers and Flash ROM images. You can access the Compaq website by logging on to the Internet at www.compaq.com Compaq Authorized Reseller For the name of your nearest Compaq Authorized Reseller: ■ In the United States, call 1-800-345-1518. ■ In Canada, call 1-800-263-5868.
Chapter 1 Clustering Overview For many years, companies have depended on clustered computer systems to fulfill two key requirements: to ensure users can access and process information that is critical to the ongoing operation of their business, and to increase the performance and throughput of their computer systems at minimal cost. These requirements are known as availability and scalability, respectively.
1-2 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Clusters Defined A cluster is an integration of software and hardware products that enables a set of loosely coupled servers and shared storage subsystem components to present a single system image to clients and to operate as a single system.
Clustering Overview 1-3 Availability When computer systems experience outages, the amount of time the system is unavailable is referred to as downtime. Downtime has several primary causes: hardware faults, software faults, planned service, operator error, and environmental factors. Minimizing downtime is a primary goal of a cluster. Simply defined, availability is the measure of how well a computer system can continuously deliver services to clients. Availability is a system-wide endeavor.
1-4 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Compaq Parallel Database Cluster Overview As traditional clustering technology has moved into the open systems of client/server computing, Compaq has provided innovative, customer-focused solutions. The PDC/O2000 moves client/server computing one step closer to the capabilities found in expensive, proprietary cluster solutions, at a fraction of the cost.
Chapter 2 Cluster Architecture The Compaq Parallel Database Cluster Model PDC/O2000 (referred to here as the PDC/O2000) is an integration of a number of different hardware and software products. This chapter discusses how these products play a role in bringing a complete clustering solution to your computing environment.
2-2 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide ■ Cluster interconnect components G Ethernet NIC adapters G Ethernet cables G Ethernet switches/hubs The software products include: ■ Microsoft Windows 2000 Advanced Server with Service Pack 1 or later ■ Compaq drivers and utilities ■ Oracle8i Enterprise Edition with the Oracle8i Parallel Server Option Refer to Chapter 3, “Cluster Software Components,” for a description of the software produ
Cluster Architecture High-Availability Features of ProLiant Servers In addition to the increased application and data availability enabled by clustering, ProLiant servers include many reliability features that provide a solid foundation for effective clustered server solutions. The PDC/O2000 is based on ProLiant servers, most of which offer excellent reliability through redundant power supplies, redundant cooling fans, and Error Checking and Correcting (ECC) memory.
2-4 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide ■ Storage Hubs or FC-AL Switches for each redundant Fibre Channel Arbitrated Loop (FC-AL) ■ Fibre Host Adapters ■ Gigabit Interface Converter-Shortwave (GBIC-SW) modules ■ Fibre Channel cables RA4000 Array The RA4000 Array is one shared storage solution for the PDC/O2000. Each redundant Fibre Channel Fabric or redundant FC-AL supports one or more RA4000 Arrays.
Cluster Architecture RA4000 Array Controllers To ensure redundant I/O paths, two single-port RA4000 Array Controllers are installed in each RA4000 Array or RA4100 Array. Only one array controller can be active at any given time. One array controller is configured as the active controller, and the other is the standby controller.
2-6 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Access to the same logical disks is provided to both RA4000 Array Controllers to allow for successful failovers. In this configuration, both the active and standby array controllers are configured to receive and transmit data for the same logical disks.
Cluster Architecture FC-AL Switches IMPORTANT: For detailed information about cascading two FC-AL Switches, refer to the latest Compaq StorageWorks documentation. This guide does not document cascaded configurations for the FC-AL Switch. FC-AL Switches can be installed between cluster nodes and shared storage subsystems in PDC/O2000 clusters to create redundant Fibre Channel Arbitrated Loops (FC-ALs).
2-8 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide The PDC/O2000 allows the use of either the Storage Hub 7 (with 7 ports) or the Storage Hub 12 (with 12 ports). Using the Storage Hub 7 limits the size of the PDC/O2000 cluster. For example, a cluster with four cluster nodes and four RA4000/RA4100 Arrays requires Storage Hubs with at least 8 ports (Storage Hub 12s).
Cluster Architecture Fibre Channel Cables Shortwave (multi-mode) fibre optic Fibre Channel cables are used to connect the nodes, the Storage Hubs, FC-AL Switches, or Fibre Channel SAN Switches, and RA4000/RA4100 Arrays in a PDC/O2000 cluster. I/O Path Configurations for Redundant Fibre Channel Fabrics Overview of Fibre Channel Fabric SAN Topology Fibre Channel standards define a multi-layered architecture for moving data across the storage area network (SAN).
2-10 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide ■ One or more RA4000/RA4100 Arrays, each containing two single-port RA4000 Array Controllers ■ GBIC-SW modules installed in the Fibre Host Adapters, Fibre Channel SAN Switches, and array controllers ■ Fibre Channel cables used to connect the Fibre Host Adapters to the Fibre Channel SAN Switches and the Fibre Channel SAN Switches to the array controllers IMPORTANT: For detailed information about cas
Cluster Architecture Multiple Redundant Fibre Channel Fabrics The PDC/O2000 supports the use of multiple redundant Fibre Channel Fabrics within the same cluster. You would install additional redundant Fibre Channel Fabrics in a PDC/O2000 to: ■ Increase the amount of shared storage space available to the cluster’s nodes. Each redundant Fibre Channel Fabric can connect to a finite number of RA4000/RA4100 Arrays.
2-12 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Figure 2-2 shows a two-node PDC/O2000 that contains two redundant Fibre Channel Fabrics. In this example, each redundant Fibre Channel Fabric has its own pair of Fibre Host Adapters in each node, a pair of Fibre Channel SAN Switches, and two RA4000/RA4100 Arrays. In Figure 2-2, the hardware components that constitute the second redundant Fibre Channel Fabric are shaded.
Cluster Architecture Maximum Distances Between Nodes and Shared Storage Components in a Redundant Fibre Channel Fabric By using standard short-wave Fibre Channel cables with Gigabit Interface Converter-Shortwave (GBIC-SW) modules, the following maximum distances apply: ■ Each RA4000/RA4100 Array can be placed up to 500 meters from the Fibre Channel SAN Switches to which it is cabled. ■ Each Fibre Channel SAN Switch can be placed up to 500 meters from the Fibre Host Adapters to which it is cabled.
2-14 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide I/O Data Paths in a Redundant Fibre Channel Fabric A distinct I/O path connection exists between each Fibre Host Adapter in a redundant Fibre Channel Fabric and every array controller port in each RA4000/RA4100 Array in that redundant Fibre Channel Fabric.
Cluster Architecture Fibre Channel SAN Switch-to-Array Controller Paths Figure 2-5 highlights the I/O data paths that run between the Fibre Channel SAN Switches and the two single-port RA4000 Array Controllers in each RA4000/RA4100 Array. Fibre Host Adapters (2) Fibre Host Adapters (2) Node 1 Node 2 Fibre Channel SAN Switch #2 Fibre Channel SAN Switch #1 RA4000/4100 Array #1 RA4000/4100 Array #2 Figure 2-5.
2-16 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide In every redundant Fibre Channel Fabric, at least two Fibre Host Adapters must be installed in each node (one adapter for each Fibre Channel SAN Switch in that redundant Fibre Channel Fabric). In the active/standby configuration, only one of the two Fibre Host Adapters in each node is active at any one time; the other Fibre Host Adapter is in the standby state.
Cluster Architecture Table 2-2 identifies the features of the active/standby and active/active configurations for redundant Fibre Channel Fabrics. Table 2-2 Features of Active/Standby and Active/Active Configurations for Redundant Fibre Channel Fabrics I/O Path Configuration Advantage Active/standby with one RA4000/RA4100 Array Active/standby is the only I/O path configuration you can use in a redundant Fibre Channel Fabric that contains just one RA4000/RA4100 Array.
2-18 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Table 2-2 Features of Active/Standby and Active/Active Configurations for Redundant Fibre Channel Fabrics continued I/O Path Configuration Advantage Disadvantage Active/active with two or more RA4000/RA4100 Arrays Provides a small but measurable improvement in I/O performance over the active/standby configuration because both Fibre Host Adapters in each node and both Fibre Channel SAN Switches are s
Cluster Architecture I/O Path Configuration Rules for Redundant Fibre Channel Fabrics The following rules must be observed in I/O path configurations for PDC/O2000 clusters with redundant Fibre Channel Fabrics: ■ Each redundant Fibre Channel Fabric must use either an active/standby or active/active configuration. The active/standby or active/active configuration is confined to that redundant Fibre Channel Fabric.
2-20 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Active/Standby Configuration Examples for Redundant Fibre Channel Fabrics This section describes examples of active/standby configurations when one, two, three, four, and five RA4000/RA4100 Arrays are present in one redundant Fibre Channel Fabric of a four-node PDC/O2000. These examples represent one method for configuring active/standby configurations.
Cluster Architecture ■ In each RA4000/RA4100 Array, the bottom (rack model) or left rear (tower model) array controller is always the standby controller. ■ The odd-numbered Fibre Channel SAN Switch (Fibre Channel SAN Switch #1) is connected to the active array controller in each RA4000/RA4100 Array. ■ The even-numbered Fibre Channel SAN Switch (Fibre Channel SAN Switch #2) is connected to the standby array controller in each RA4000/RA4100 Array.
2-22 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Active/Standby Configuration with One RA4000/4100 Array Figure 2-6 shows an active/standby I/O path configuration for a redundant Fibre Channel Fabric in a four-node cluster with one RA4000/RA4100 Array.
Cluster Architecture Active/Standby Configuration with Two RA4000/4100 Arrays Figure 2-7 shows an active/standby I/O path configuration for a redundant Fibre Channel Fabric in a four-node cluster with two RA4000/RA4100 Arrays. Fibre Host Adapter (Active) Node 1 Fibre Host Adapter (Standby) ProLiant Servers (4) Node 2 Node 3 Node 4 Fibre Channel SAN Switch #1 RA4000/4100 Active Array #1 Array Controller Standby Array Controller Fibre Channel SAN Switch #2 RA4000/4100 Array #2 Figure 2-7.
2-24 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Active/Standby Configuration with Three RA4000/RA4100 Arrays Figure 2-8 shows an active/standby I/O path configuration for a redundant Fibre Channel Fabric in a four-node cluster with three RA4000/RA4100 Arrays.
Cluster Architecture Active/Standby Configuration with Four RA4000/RA4100 Arrays Figure 2-9 shows an active/standby I/O path configuration for a redundant Fibre Channel Fabric in a four-node cluster with four RA4000/RA4100 Arrays.
2-26 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Active/Standby Configuration with Five RA4000/RA4100 Arrays Figure 2-10 shows an active/standby I/O path configuration for a redundant Fibre Channel Fabric in a four-node cluster with five RA4000/RA4100 Arrays.
Cluster Architecture Active/Active Configuration Examples for Redundant Fibre Channel Fabrics This section describes examples of active/active configurations when two, three, four, and five RA4000/RA4100 Arrays are present in one redundant Fibre Channel Fabric of a four-node PDC/O2000 cluster. These examples represent one method for configuring active/active configurations.
2-28 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide ■ The even-numbered Fibre Channel SAN Switch (Fibre Channel SAN Switch #2) is connected to the active array controller in each even-numbered RA4000/RA4100 Array (2 and 4). ■ The even-numbered Fibre Channel SAN Switch is connected to the standby array controller in each odd-numbered RA4000/RA4100 Array (1, 3, and 5). NOTE: The following active/active configurations are examples only.
Cluster Architecture Active/Active Configuration with Two RA4000/RA4100 Arrays Figure 2-11 shows an active/active I/O path configuration for a redundant Fibre Channel Fabric in a four-node cluster with two RA4000/RA4100 Arrays. Fibre Host Adapters (2) Node 1 ProLiant Servers (4) Node 2 Node 3 Node 4 Fibre Channel SAN Switch #1 Active RA4000/4100 Array Controller Array #1 Standby Array Controller Fibre Channel SAN Switch #2 RA4000/4100 Array #2 Figure 2-11.
2-30 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Active/Active Configuration with Three RA4000/RA4100 Arrays Figure 2-12 shows an active/active I/O path configuration for a redundant Fibre Channel Fabric in a four-node cluster with three RA4000/RA4100 Arrays.
Cluster Architecture Active/Active Configuration with Four RA4000/RA4100 Arrays Figure 2-13 shows an active/active I/O path configuration for a redundant Fibre Channel Fabric in a four-node cluster with four RA4000/RA4100 Arrays.
2-32 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Active/Active Configuration with Five RA4000/RA4100 Arrays Figure 2-14 shows an active/active I/O path configuration for a redundant Fibre Channel Fabric in a four-node cluster with five RA4000/RA4100 Arrays.
Cluster Architecture Summary of I/O Path Failure and Failover Scenarios for Redundant Fibre Channel Fabrics Table 2-3 identifies possible I/O path failure events in redundant Fibre Channel Fabrics for active/standby configurations with one RA4000/RA4100 Array and the failover response, if any, implemented by Secure Path for each failure.
2-34 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Table 2-3 I/O Path Failure and Failover Scenarios in Redundant Fibre Channel Fabrics for Active/Standby Configurations With One RA4000/RA4100 Array continued Description of Failure Failover Response The Fibre Channel SAN Switch connected to the standby array controller fails.
Cluster Architecture Table 2-4 identifies possible I/O path failure events for active/standby configurations in redundant Fibre Channel Fabrics with two or more RA4000/RA4100 Arrays and the failover response, if any, implemented by Secure Path for each failure.
2-36 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Table 2-4 I/O Path Failure and Failover Scenarios in Redundant Fibre Channel Fabrics for Active/Standby Configurations With Two or More RA4000/RA4100 Arrays continued Description of Failure Failover Response The Fibre Channel cable connection between the standby array controller in one RA4000/RA4100 Array and its Fibre Channel SAN Switch is broken.
Cluster Architecture Table 2-5 identifies possible I/O path failure events in redundant Fibre Channel Fabrics for active/active configurations with two or more RA4000/RA4100 Arrays and the failover response, if any, implemented by Secure Path for each failure.
2-38 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Table 2-5 I/O Path Failure and Failover Scenarios in Redundant Fibre Channel Fabrics for Active/Active Configurations With Two or More RA4000/RA4100 Arrays continued Description of Failure Failover Response The Fibre Channel cable connection between the standby array controller in one RA4000/RA4100 Array and its Fibre Channel SAN Switch is broken. None A Fibre Channel SAN Switch fails.
Cluster Architecture I/O Path Configurations for Redundant Fibre Channel Arbitrated Loops Overview of FC-AL SAN Topology Fibre Channel standards define a multi-layered architecture for moving data across the storage area network (SAN). This layered architecture can be implemented using the Fibre Channel Fabric or the Fibre Channel Arbitrated Loop (FC-AL) topology. The PDC/O2000 supports both topologies.
2-40 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Redundant Fibre Channel Arbitrated Loops A redundant FC-AL refers to the redundant hardware implemented to connect Fibre Host Adapters to a particular set of shared storage devices in a PDC/O2000.
Cluster Architecture Figure 2-15 shows a two-node PDC/O2000 with a redundant FC-AL that contains two loops, one for each Fibre Host Adapter per node. The components and cable paths for the first loop are shaded to distinguish them from the components for the other loop. Fibre Host Adapters (2) Node 1 FC-AL Switch/Storage Hub #1 RA4000/4100 Array #1 Fibre Host Adapters (2) Node 2 FC-AL Switch/Storage Hub #2 RA4000/4100 Array #2 Figure 2-15.
2-42 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Multiple Redundant Fibre Channel Arbitrated Loops The PDC/O2000 supports the use of multiple redundant FC-ALs within the same cluster. You would install additional redundant FC-ALs in a PDC/O2000 to: ■ Increase the amount of shared storage space available to the cluster’s nodes. Each redundant FC-AL can connect to a finite number of RA4000/RA4100 Arrays.
Cluster Architecture Figure 2-16 shows a two-node PDC/O2000 that contains two redundant FC-ALs. In this example, each redundant FC-AL has its own pair of Fibre Host Adapters in each node, a pair of Storage Hubs or FC-AL Switches, and two RA4000/RA4100 Arrays. In Figure 2-2, the hardware components that constitute the second redundant FC-AL are shaded.
2-44 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Maximum Distances Between Nodes and Shared Storage Components in a Redundant FC-AL By using standard short-wave Fibre Channel cables with Gigabit Interface Converter-Shortwave (GBIC-SW) modules, the following maximum distances apply: ■ Each RA4000/RA4100 Array can be placed up to 500 meters from the Storage Hubs or FC-AL Switches to which it is cabled.
Cluster Architecture I/O Data Paths in a Redundant FC-AL A distinct I/O path connection exists between each Fibre Host Adapter in a cluster node and every array controller port in each RA4000/RA4100 Array in the redundant FC-AL. Fibre Host Adapter-to-Storage Hub/Switch Paths Figure 2-18 highlights the I/O data paths that run between the Fibre Host Adapters and the Storage Hubs or FC-AL Switches in a redundant FC-AL.
2-46 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Storage Hub/Switch-to-Array Controller Paths Figure 2-19 highlights the I/O data paths that run between the Storage Hubs or FC-AL Switches and the two single-port RA4000 Array Controllers in each RA4000/RA4100 Array. Fibre Host Adapters (2) Fibre Host Adapters (2) Node 1 Node 2 FC-AL Switch/Storage Hub #1 RA4000/4100 Array #1 FC-AL Switch/Storage Hub #2 RA4000/4100 Array #2 Figure 2-19.
Cluster Architecture I/O Path Configuration Guidelines for Redundant Fibre Channel Arbitrated Loops You can use either of two I/O path configurations for every redundant FC-AL in your PDC/O2000 cluster: ■ Active/standby configuration ■ Active/active configuration In every redundant FC-AL, at least two Fibre Host Adapters must be installed in each node (one Fibre Host Adapter for each Storage Hub or FC-AL Switch in that redundant FC-AL).
2-48 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide See “Active/Standby Configuration Examples for Redundant FC-ALs” in this chapter for a detailed description of active/standby configuration examples for a redundant FC-AL with from one to five RA4000/RA4100 Arrays. See “Active/Active Configuration Examples for Redundant FC-ALs” for a detailed description of active/active configuration examples when two to five RA4000/RA4100 Arrays are present.
Cluster Architecture Table 2-6 Features of Active/Standby and Active/Active Configurations for Redundant Fibre Channel Arbitrated Loops continued I/O Path Configuration Advantage Disadvantage Active/active with two or more RA4000/RA4100 Arrays Provides a small but measurable improvement in I/O performance over the active/standby configuration because both Fibre Host Adapters in each node and both Storage Hubs or FC-AL Switches are simultaneously active.
2-50 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide I/O Path Configuration Rules for Redundant FC-ALs The following rules must be observed in I/O path configurations for PDC/O2000 clusters with redundant FC-ALs: ■ Each redundant FC-AL must use either an active/standby or active/active configuration. The active/standby or active/active configuration is confined to that redundant FC-AL.
Cluster Architecture Active/Standby Configuration Examples for Redundant FC-ALs This section describes examples of active/standby configurations when one, two, three, four, and five RA4000/RA4100 Arrays are present in one redundant FC-AL of a four-node PDC/O2000. These examples represent one method for configuring active/standby configurations. They are presented here to provide a relatively simple and consistent method for building active/standby configurations.
2-52 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide ■ In each RA4000/RA4100 Array, the bottom (rack model) or left rear (tower model) array controller is always the standby controller. ■ The odd-numbered Storage Hub or FC-AL Switch (Storage Hub/FC-AL Switch #1) is connected to the active array controller in each RA4000/RA4100 Array.
Cluster Architecture Active/Standby Configuration with One RA4000/4100 Array Figure 2-20 shows an active/standby I/O path configuration for a redundant FC-AL in a four-node cluster with one RA4000/RA4100 Array. Fibre Host Adapter (Active) Fibre Host Adapter (Standby) Node 1 ProLiant Servers (4) Node 2 Node 3 FC-AL Switch/Storage Hub #1 Standby Array Controller Node 4 FC-AL Switch/Storage Hub #2 Active Array Controller RA4000/4100 Array #1 Figure 2-20.
2-54 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Active/Standby Configuration with Two RA4000/4100 Arrays Figure 2-21 shows an active/standby I/O path configuration for a redundant FC-AL in a four-node cluster with two RA4000/RA4100 Arrays.
Cluster Architecture Active/Standby Configuration with Three RA4000/RA4100 Arrays Figure 2-22 shows an active/standby I/O path configuration for a redundant FC-AL in a four-node cluster with three RA4000/RA4100 Arrays.
2-56 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Active/Standby Configuration with Four RA4000/RA4100 Arrays Figure 2-23 shows an active/standby I/O path configuration for a redundant FC-AL in a four-node cluster with four RA4000/RA4100 Arrays.
Cluster Architecture Active/Standby Configuration with Five RA4000/RA4100 Arrays Figure 2-24 shows an active/standby I/O path configuration for a redundant FC-AL in a four-node cluster with five RA4000/RA4100 Arrays.
2-58 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Active/Active Configuration Examples for Redundant FC-ALs This section describes examples of active/active configurations when two, three, four, and five RA4000/RA4100 Arrays are present in one redundant FC-AL of a four-node PDC/O2000 cluster. These examples represent one method for configuring active/active configurations.
Cluster Architecture ■ The even-numbered Storage Hub or FC-AL Switch (Storage Hub/FC-AL Switch #2) is connected to the active array controller in each even-numbered RA4000/RA4100 Array (2 and 4). ■ The even-numbered Storage Hub or FC-AL Switch is connected to the standby array controller in each odd-numbered RA4000/RA4100 Array (1, 3, and 5). NOTE: The following active/active configurations are examples only. You are not required to follow these configurations.
2-60 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Active/Active Configuration with Two RA4000/RA4100 Arrays Figure 2-25 shows an active/active I/O path configuration for a redundant FC-AL in a four-node cluster with two RA4000/RA4100 Arrays.
Cluster Architecture Active/Active Configuration with Three RA4000/RA4100 Arrays Figure 2-26 shows an active/active I/O path configuration for a redundant FC-AL in a four-node cluster with three RA4000/RA4100 Arrays. Fibre Host Adapters (2) Node 1 ProLiant Servers (4) Node 2 Node 3 Node 4 FC-AL Switch/Storage Hub #1 RA4000/4100 Array #1 Active Array Controller RA4000/4100 Array #3 FC-AL Switch/Storage Hub #2 RA4000/4100 Array #2 Standby Array Controller Figure 2-26.
2-62 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Active/Active Configuration with Four RA4000/RA4100 Arrays Figure 2-27 shows an active/active I/O path configuration for a redundant FC-AL in a four-node cluster with four RA4000/RA4100 Arrays.
Cluster Architecture Active/Active Configuration with Five RA4000/RA4100 Arrays Figure 2-28 shows an active/active I/O path configuration for a redundant FC-AL in a four-node cluster with five RA4000/RA4100 Arrays.
2-64 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Summary of I/O Path Failure and Failover Scenarios for Redundant FC-ALs Table 2-7 identifies possible I/O path failure events in redundant FC-ALs for active/standby configurations with one RA4000/RA4100 Array and the failover response, if any, implemented by Secure Path for each failure.
Cluster Architecture Table 2-7 I/O Path Failure and Failover Scenarios in Redundant FC-ALs for Active/Standby Configurations With One RA4000/RA4100 Array continued Description of Failure Failover Response The Fibre Channel cable connection between a Fibre Host Adapter and the Storage Hub or FC-AL Switch connected to the standby array controller is broken. None A Fibre Host Adapter connected to the Storage Hub or FC-AL Switch that connects to the active array controller fails.
2-66 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Table 2-8 identifies possible I/O path failure events for active/standby configurations with two or more RA4000/RA4100 Arrays and the failover response, if any, implemented by Secure Path for each failure.
Cluster Architecture Table 2-8 I/O Path Failure and Failover Scenarios in Redundant FC-ALs for Active/Standby Configurations With Two or More RA4000/RA4100 Arrays continued Description of Failure Failover Response The Fibre Channel cable connection between the Storage Hub or FC-AL Switch that is connected to the active array controllers and a Fibre Host Adapter is broken.
2-68 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Table 2-9 identifies possible I/O path failure events in redundant FC-ALs for active/active configurations with two or more RA4000/RA4100 Arrays and the failover response, if any, implemented by Secure Path for each failure.
Cluster Architecture Table 2-9 I/O Path Failure and Failover Scenarios in Redundant FC-ALs for Active/Active Configurations With Two or More RA4000/RA4100 Arrays continued Description of Failure Failover Response A Storage Hub or FC-AL Switch fails. Secure Path makes each active array controller to which the failed Storage Hub or FC-AL Switch is connected inactive. The standby array controller in each affected RA4000/RA4100 Array becomes the active array controller.
2-70 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Cluster Interconnect Options The cluster interconnect is the data path over which all of the nodes in a cluster communicate. The nodes use the cluster interconnect data path to: ■ Communicate individual resource and overall cluster status. ■ Send and receive heartbeat signals. ■ Coordinate database locks through the Oracle Integrated Distributed Lock Manager.
Cluster Architecture Non-Redundant Ethernet Cluster Interconnect Components The following components are used in a non-redundant Ethernet cluster interconnect: ■ One Ethernet adapter in each cluster node ■ Ethernet cables and a switch or hub G For two-node PDC/O2000 clusters, you can either use one Ethernet crossover cable or one 100-Mbit/second Ethernet switch or hub and standard Ethernet cables to connect the two servers.
2-72 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Ethernet Cluster Interconnect Adapters To implement the Ethernet cluster interconnect, each cluster node must be equipped with Ethernet adapters capable of 100-Mbit/second transfer rates. Some adapters may be capable of operating at both 10-Mbit/second and 100-Mbit/second; however, Ethernet adapters used for the cluster interconnect must run at 100-Mbit/second.
Cluster Architecture Ethernet Cluster Interconnect Diagrams Figure 2-29 shows the non-redundant Ethernet cluster interconnect components used in a two-node PDC/O2000 cluster. These components include a dual-port Ethernet adapter in each node. The top port on each adapter connects by Ethernet crossover cable to the top port on the adapter in the other node. The bottom port on each adapter connects by Ethernet cable to the client LAN switch or hub.
2-74 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Figure 2-30 shows another option for a non-redundant Ethernet cluster interconnect in a two-node PDC/O2000 cluster. These include an Ethernet adapter in each node connected by Ethernet cables to an Ethernet switch or hub. RA4000/4100 Array Storage Hub/Switch Ethernet Adapter Ethernet Adapter Storage Hub/Switch Ethernet cables Node 1 Node 2 Client LAN Figure 2-30.
Cluster Architecture Figure 2-31 shows the redundant Ethernet cluster interconnect components used in a two-node PDC/O2000 cluster. Ethernet Switch/Hub #1 for Cluster Interconnect Ethernet Switch/Hub #2 for Cluster Interconnect Crossover Cable Dual-port Ethernet Adapters (2) Dual-port Ethernet Adapters (2) Node 1 Crossover Cable Node 2 Client LAN Hub/Switch #1 Client LAN Hub/Switch #2 Figure 2-31.
2-76 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Local Area Network NOTE: For the PDC/O2000, the client LAN and the cluster interconnect must be treated as separate networks. Do not use either network to handle the other network’s traffic. Every client/server application requires a local area network, or LAN, over which client machines and servers communicate.
Chapter 3 Cluster Software Components Overview of the Cluster Software The Compaq Parallel Database Cluster Model PDC/O2000 (referred to here as the PDC/O2000) combines software from several leading computer vendors. The integration of these components creates a stable cluster management environment in which the Oracle database can operate.
3-2 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Compaq Software Compaq offers an extensive set of features and optional tools to support effective configuration and management of the PDC/O2000: ■ Compaq SmartStart™ and Support Software ■ Compaq System Configuration Utility ■ Compaq Array Configuration Utility ■ Fibre Channel Fault Isolation Utility ■ Compaq Insight Manager™ ■ Compaq Insight Manager XE ■ Compaq Options ROMPaq™ ■ Compaq S
Cluster Software Components Compaq System Configuration Utility The SmartStart and Support Software CD also contains the Compaq System Configuration Utility. This utility is the primary means to configure hardware devices within your servers, such as I/O addresses, boot order of disk controllers, and so on. For information about the System Configuration Utility, see the Compaq Server Setup and Management pack.
3-4 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Compaq Insight Manager Compaq Insight Manager, loaded from the Compaq Management CD, is a software utility used to collect information about the servers in the cluster.
Cluster Software Components Compaq StorageWorks Secure Path for Windows 2000 Compaq StorageWorks Secure Path for Windows 2000 (Secure Path) must be installed on each server (node) in the PDC/O2000. Secure Path monitors all I/O paths to the storagesets in the shared storage subsystems. If any component along an active I/O path fails, Secure Path detects this failure and automatically reroutes I/O activity through the defined backup path. Failovers are transparent and non-disruptive to applications.
3-6 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Oracle Software The PDC/O2000 supports Oracle8i software. If you are using a release other than Oracle8i Release 8.1.7, confirm that the release has been certified for the PDC/O2000 on the Compaq website at www.compaq.com/solutions/enterprise/ha-pdc.
Cluster Software Components For additional information, refer to: ■ Oracle8i Parallel Server Setup and Configuration Guide ■ Other Oracle documentation for Oracle8i Server and Oracle8i Parallel Server provided with the Oracle software Oracle8i Enterprise Manager Oracle8i Enterprise Manager is responsible for monitoring the state of both the database entities and the cluster members. It primarily manages the software components of the cluster. Hardware components are managed with Compaq Insight Manager.
3-8 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Application Failover and Reconnection Software When a network client computer operates in a clustered environment, it must be more resilient than when operating with a stand-alone server. Because a client can access the database through any of the cluster nodes, the failure of the connection to a node does not have to prevent the client from reattaching to the cluster and continuing its work.
Chapter 4 Cluster Planning Before connecting any cables or powering on any hardware on your Compaq Parallel Database Cluster Model PDC/O2000 (referred to here as the PDC/O2000), it is important that you understand how all the various cluster components fit together to meet your operational requirements.
4-2 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Site Planning You must carefully select and prepare the site to ensure a smooth installation and a safe and efficient work environment.
Cluster Planning 4-3 Capacity Planning for Cluster Hardware Capacity planning determines how much computer hardware is needed to support the applications and data on your clustered servers. Given the size of your database and the performance you expect, you must decide how many servers and shared storage arrays the cluster needs. Compaq ProLiant Servers The number of servers you install in a PDC/O2000 cluster should take into account the levels of availability and scalability your site requires.
4-4 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide ■ One or more Compaq StorageWorks RAID Array 4000s (RA4000 Arrays) or Compaq StorageWorks RAID Array 4100s (RA4100 Arrays) ■ Two single-port Compaq StorageWorks RAID Array 4000 Controllers (RA4000 Array Controllers) installed in each RA4000/RA4100 Array. Only one array controller in the RA4000/RA4100 Array is active at any one time; the other is the standby array controller.
Cluster Planning 4-5 Planning Shared Storage Components for Redundant Fibre Channel Arbitrated Loops Several key components make up the shared storage subsystem for the PDC/O2000.
4-6 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Planning Cluster Interconnect and Client LAN Components PDC/O2000 clusters running Oracle8i Parallel Server can use a redundant or non-redundant Ethernet cluster interconnect. Redundant cluster interconnects are recommended because they provide fault tolerance along the entire cluster interconnect path.
Cluster Planning 4-7 Planning Cluster Configurations for Redundant Fibre Channel Fabrics Once you have investigated your requirements with respect to particular parts of the cluster (ProLiant servers, shared storage components, cluster interconnect components, client LAN components), you need to plan the configuration of the entire PDC/O2000. This section describes sample configurations for small and larger clusters using redundant Fibre Channel Fabrics.
4-8 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide The small redundant Fibre Channel Fabric cluster configuration shown in Figure 4-1 contains these key cluster components: ■ Two ProLiant servers (nodes) ■ Two Fibre Host Adapters in each server ■ Two Fibre Channel SAN Switches ■ One RA4000/RA4100 Array. Each RA4000 Array holds up to eight 1.6-inch disk drives or twelve 1-inch disk drives. Each RA4100 Array holds up to twelve 1-inch disk drives.
Cluster Planning 4-9 Sample Large Configuration in a Redundant Fibre Channel Fabric Figure 4-2 shows an example of a larger PDC/O2000 cluster: a six-node cluster with five RA4000/RA4100 Arrays in one redundant Fibre Channel Fabric. Fibre Host Adapters (2) Node 5 Node 1 Node 6 Node 2 Node 3 Node 4 Fibre Channel SAN Switch #2 Fibre Channel SAN Switch #1 RA4000/4100 Array #2 RA4000/4100 Array #1 RA4000/4100 Array #3 RA4000/4100 Array #4 RA4000/4100 Array #5 Figure 4-2.
4-10 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide The larger redundant Fibre Channel Fabric configuration shown in Figure 4-2 contains these key cluster components: ■ Six ProLiant servers (nodes) ■ Two Fibre Host Adapters in each server node ■ Two Fibre Channel SAN Switches ■ Five RA4000/RA4100 Arrays. Each RA4000 Array holds up to eight 1.6-inch disk drives or twelve 1-inch disk drives. Each RA4100 Array holds up to twelve 1-inch disk drives.
Cluster Planning 4-11 Planning Cluster Configurations for Redundant Fibre Channel Arbitrated Loops Once you have investigated your requirements with respect to particular parts of the cluster (ProLiant servers, shared storage components, cluster interconnect components, client LAN components), you need to plan the configuration of the entire PDC/O2000. This section describes sample configurations for small and large clusters using redundant FC-ALs.
4-12 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide The small redundant FC-AL configuration shown in Figure 4-3 contains these key cluster components: ■ Two ProLiant servers (nodes) ■ Two Fibre Host Adapters in each server ■ Two Storage Hubs or FC-AL Switches ■ One RA4000/RA4100 Array. Each RA4000 Array holds up to eight 1.6inch disk drives or twelve 1-inch disk drives. Each RA4100 Array holds up to twelve 1-inch disk drives.
Cluster Planning 4-13 Sample Large Configuration in a Redundant FC-AL Figure 4-4 shows an example of a larger PDC/O2000 cluster: a six-node cluster with five RA4000/RA4100 Arrays in one redundant FC-AL. Fibre Host Adapters (2) Node 5 Node 1 Node 6 Node 2 Node 3 Node 4 FC-AL Switch/Storage Hub #2 FC-AL Switch/Storage Hub #1 RA4000/4100 Array #2 RA4000/4100 Array #1 RA4000/4100 Array #3 RA4000/4100 Array #4 RA4000/4100 Array #5 Figure 4-4.
4-14 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide The larger redundant FC-AL configuration shown in Figure 4-4 contains these key cluster components: ■ Six ProLiant servers (nodes) ■ Two Fibre Host Adapters in each server node ■ Two Storage Hub 12s or 11-port FC-AL Switches ■ Five RA4000/RA4100 Arrays. Each RA4000 Array holds up to eight 1.6-inch disk drives or twelve 1-inch disk drives. Each RA4100 Array holds up to twelve 1-inch disk drives.
Cluster Planning 4-15 In a PDC/O2000 cluster, each node is connected to shared storage disk drives housed in RA4000/RA4100 Arrays. When planning the amount of shared storage for your cluster, you must consider the following: ■ The maximum allowable number of shared storage arrays in one cluster.
4-16 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Supported RAID Levels RAID provides several fault-tolerant options to protect your cluster’s shared data. However, each RAID level offers a different mix of performance, reliability, and cost.
Cluster Planning 4-17 Raw Data Storage and Database Size Raw data storage is the amount of storage available before any RAID levels have been configured. It is called raw data storage because RAID volumes require some overhead. The maximum size of a database stored in a RAID system will always be less than the amount of raw data storage available. To calculate the amount of raw data storage in a PDC/O2000 cluster, determine the total amount of shared storage space available to the cluster.
4-18 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide The following general guidelines apply to RAID selection for a cluster with RA4000/RA4100 Arrays using Oracle8i Parallel Server: ■ Oracle recommends that some form of disk fault tolerance be implemented in the cluster.
Cluster Planning 4-19 Using the Compaq Array Configuration Utility, group the RA4000/RA4100 Array disk drives into RAID disk arrays at specific RAID levels. This example shows four disk drives grouped into one RAID disk array at RAID level 5. It also shows two RAID level 1 disk arrays containing two disk drives each. A logical drive is what you see labeled as Disk 1, Disk 2, and so on, from the Disk Management utility.
4-20 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Shared Disk Drives Shared disk drives are contained in the RA4000/RA4100 Arrays and are accessible to each node in the PDC/O2000. If a logical drive is configured with a RAID level that does not support fault tolerance (for example, RAID 0), then the failure of the shared disk drives in that logical drive will disrupt service to all Oracle databases that are dependent on that disk drive.
Cluster Planning 4-21 Client LAN Physically, the structure of the client network is no different than that used for a nonclustered configuration. To ensure continued access to the database when a cluster node is evicted from the cluster, each network client should have physical network access to all of the cluster nodes. Software used by the client to communicate to the database must be able to reconnect to another cluster node in the event of a node eviction.
Chapter 5 Installation and Configuration This chapter provides instructions for installing and configuring the Compaq Parallel Database Cluster Model PDC/O2000 (referred to here as the PDC/O2000) for use with Oracle8i software. A PDC/O2000 is a combination of several individually available products. As you set up your cluster, have the following materials available during installation. You will find references to them throughout this chapter.
5-2 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide ■ Compaq StorageWorks Fibre Channel SAN Switch Management Guide ■ Compaq SmartStart Installation poster ■ Compaq SmartStart and Support Software CD ■ Microsoft Windows 2000 Advanced Server Administrator’s Guide ■ Microsoft Windows 2000 Advanced Server CD with Service Pack 1 or later ■ Compaq Parallel Database Cluster Clustering Software for Oracle8i on Microsoft Windows 2000 CD ■ Oracle8i En
Installation and Configuration ■ ■ ■ Installing and configuring operating system software, including: G SmartStart 4.
5-4 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Installing the Hardware Setting Up the Nodes Physically preparing the nodes (servers) for a cluster is not very different than preparing them for individual use. You will install all necessary adapters and insert all internal hard disks. You will attach network cables and plug in SCSI and Fibre Channel cables. The primary difference is in setting up the shared storage subsystem.
Installation and Configuration 5-5 Installing GBIC-SW Modules for the Fibre Host Adapters Each Fibre Host Adapter ships with two GBIC-SW modules. Verify that one module is installed in the Fibre Host Adapter and the other in its Storage Hub, FC-AL Switch, or Fibre Channel SAN Switch. Each end of the Fibre Channel cable connecting a Fibre Host Adapter to a Storage Hub, FC-AL Switch, or Fibre Channel SAN Switch plugs into a GBIC-SW module. To install GBIC-SW modules: 1.
5-6 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Figure 5-1 shows the Fibre Host Adapters in two servers connected to two Storage Hubs, FC-AL Switches, or Fibre Channel SAN Switches. Fibre Host Adapters (2) Fibre Host Adapters (2) Node 1 Storage Hub/Switch #1 Node 2 Storage Hub/Switch #2 Figure 5-1.
Installation and Configuration 5-7 Ethernet Cluster Interconnect Adapters For a non-redundant Ethernet cluster interconnect, install one single-port or dual-port Ethernet adapter into each cluster node. For a redundant Ethernet cluster interconnect, install two single-port or two dual-port Ethernet adapters into each cluster node. For recommended dual-port and single-port Ethernet adapters, see the Compaq Parallel Database Cluster Model PDC/O2000 Certification Matrix for Windows 2000 at www.compaq.
5-8 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Figure 5-2 shows two RA4000/RA4100 Arrays connected to two clustered servers through one redundant Fibre Channel Fabric or redundant FC-AL. The Fibre Channel Fabric or FC-AL is redundant because there are two paths from each node to each RA4000/RA4100 Array.
Installation and Configuration 5-9 For detailed information about configuring the drives using the Array Configuration Utility, see the Compaq StorageWorks RAID Array 4000 User Guide or the Compaq StorageWorks RAID Array 4100 User Guide. For information about configuring your shared storage subsystem with RAID, see Chapter 4, “Cluster Planning.
5-10 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide In an active/active configuration, both Fibre Host Adapters in each node are connected to an active array controller. The number of array controllers each Fibre Host Adapter is connected to depends on the number of RA4000/RA4100 Arrays in the redundant Fibre Channel Fabric or redundant FC-AL. For more information, see Chapter 2, “Cluster Architecture.
Installation and Configuration 5-11 Figure 5-3 shows two Storage Hubs, FC-AL Switches, or Fibre Channel SAN Switches connected to the RA4000 Array Controllers in two RA4000/RA4100 Arrays. Storage Hub/Switch #1 Active RA4000/4100 Array Controller Array #1 Standby Array Controller Storage Hub/Switch #2 RA4000/4100 Array #2 Figure 5-3.
5-12 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Table 5-1 summarizes the active/active cabling methods. Select the method that is best for your site. Table 5-1 Active/Active Cabling Methods Method Active Array Controller Location Advantage Disadvantage 1 Top array controller slot Consistency in active array controller definition. All active array controllers are defined to be in the top array controller slot in each RA4000/RA4100 Array.
Installation and Configuration 5-13 Figure 5-4 shows an example of using method 1 cabling to connect two Storage Hubs, FC-AL Switches, or Fibre Channel SAN Switches to RA4000 Array Controllers in two RA4000/RA4100 Arrays. Storage Hub/Switch #1 Active RA4000/4100 Array Controller Array #1 Standby Array Controller Storage Hub/Switch #2 RA4000/4100 Array #2 Figure 5-4.
5-14 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Figure 5-5 shows an example of using method 2 cabling to connect two Storage Hubs, FC-AL Switches, or Fibre Channel SAN Switches to RA4000 Array Controllers in two RA4000/RA4100 Arrays. Storage Hub/Switch #1 RA4000/4100 Array #1 Active Array Controller RA4000/4100 Array #3 Storage Hub/Switch #2 RA4000/4100 Array #2 Standby Array Controller RA4000/4100 Array #4 Figure 5-5.
Installation and Configuration 5-15 Installing Additional Redundant Fibre Channel Fabrics At this point, you have installed the hardware for one redundant Fibre Channel Fabric or one redundant FC-AL.
5-16 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Cabling the Ethernet Cluster Interconnect A PDC/O2000 running Oracle8i software uses an Ethernet cluster interconnect.
Installation and Configuration 5-17 Figure 5-6 shows the non-redundant Ethernet cluster interconnect components used in a two-node PDC/O2000 cluster. These components include a dual-port Ethernet adapter in each node. The top port on each adapter connects by Ethernet crossover cable to the top port on the adapter in the other node. The bottom port on each adapter connects by Ethernet cable to the client LAN switch or hub.
5-18 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide To connect two or more nodes in a non-redundant Ethernet cluster interconnect: 1. Install a standard Ethernet cable between an Ethernet adapter in one node and one Ethernet hub or switch. 2. Repeat step 1 for every other node in the cluster. Figure 5-7 shows another option for a non-redundant Ethernet cluster interconnect in a two-node PDC/O2000 cluster.
Installation and Configuration 5-19 Figure 5-8 shows the redundant Ethernet cluster interconnect components used in a PDC/O2000 with two or more nodes. Ethernet Switch/Hub #1 for Cluster Interconnect Ethernet Switch/Hub #2 for Cluster Interconnect Crossover Cable Dual-port Ethernet Adapters (2) Dual-port Ethernet Adapters (2) Node 1 Crossover Cable Node 2 Client LAN Hub/Switch #1 Client LAN Hub/Switch #2 Figure 5-8.
5-20 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Cabling the Client LAN You can use any TCP/IP network to connect to a client LAN. The following procedure contains instructions for cabling an Ethernet client LAN. To cable an Ethernet client LAN: 1. Insert one end of an Ethernet cable into an Ethernet adapter port designated for the client LAN in a cluster node.
Installation and Configuration 5-21 Installing the Operating System Software and Configuring the RA4000/RA4100 Arrays You will follow an automated procedure using Compaq SmartStart to install the operating system software and configure the shared storage disks on the RA4000/RA4100 Arrays. Guidelines for Clusters Installing clustering software requires several specific steps and guidelines that might not be necessary when installing software on a single server.
5-22 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide ■ When you set up an Ethernet cluster interconnect, be sure to select TCP/IP as the network protocol. The Ethernet cluster interconnect should be on its own subnet. IMPORTANT: The IP addresses of the Ethernet cluster interconnect must be static, not dynamically assigned by DHCP.
Installation and Configuration 5-23 The one difference is that through the Array Configuration Utility, SmartStart gives you the opportunity to configure the shared drives on all servers. For cluster configuration, you should configure the drives on the first server, then accept the same settings for the shared drives when given the option on the other servers.
5-24 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide 9. Enter unique IP addresses and node names for each node in the hosts and lmhosts files located at %SystemRoot%\system32\drivers\etc. Record this information: G For an Ethernet cluster interconnect, one IP address and node name is for the redundant cluster interconnect, and the other IP address and node name is for the client LAN.
Installation and Configuration 5-25 e. With the client LAN NIC selected, click Move Up to position this NIC to the top of the list. f. Click OK on the dialog box and restart the node when prompted. IMPORTANT: Record the cluster interconnect node name, the client LAN node name, and the IP addresses assigned to them. You will need this information later when installing Compaq OSDs. 12.
5-26 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Installing Secure Path Software for Windows 2000 The Secure Path software for Windows 2000 is installed on both the server and the client. The Secure Path server software is installed on the Windows host system to which the RAID storage system is connected. TCP/IP must be installed on the host system. Secure Path software must be installed on each member of the cluster.
Installation and Configuration 5-27 Installing the Secure Path Client Software Install the Secure Path Client software on either the same Windows host system as the Secure Path Server software or any Windows workstation. The workstation must be TCP/IP-capable. To install the Secure Path Client software: 1. Insert the Compaq SANWorks Secure Path Software CD into your CD-ROM drive. 2. If you have AutoRun enabled on your server, the Secure Path setup program will start automatically.
5-28 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide 3. If you double-click “Storage System X”, you should get two more branches that represent the two RA4000 Array Controllers in that array. These controllers are named “ControllerA” and “ControllerB,” which correspond to the Active controller and the Standby controller, respectively. 4.
Installation and Configuration 5-29 Table 5-2 Active Array Controller Locations Configuration Active Array Controller Location Active/Standby Top array controller slot Active/Active: Cabling Method 1 Top array controller slot Active/Active: Cabling Method 2 Top array controller slot in odd-numbered RA4000/RA4100 Arrays Bottom array controller slot in even-numbered RA4000/RA4100 Arrays NOTE: If Secure Path Manager indicates that the active array controllers are already located in the proper slots,
5-30 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide 1. On any cluster node, click Start, Programs. Under Secure Path, click SPM. The Secure Path Manager (Fibre Channel) screen is displayed. 2. After clicking on the logical drives (LUNS) you would like to move to the other array controller, right-click the first controller and a pop-up message appears that states “Move to Other Controller.
Installation and Configuration 5-31 From one node in the cluster, the OUI installs and configures OSDs on the other nodes in the cluster, provided that: ■ The servers are communicating through a local area network using a TCP/IP protocol. ■ The user running the OUI program has administrator privileges on every node. Verifying Cluster Communications IMPORTANT: The successful completion of the OUI program depends on TCP/IP connectivity being set up correctly.
5-32 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide For example, the command ping node1 should return the following type of information. Pinging node1.loc1.yoursite.com [127.0.0.1] with 32 bytes of data: Reply from 127.0.0.1: bytes=32 time<10ms TTL=128 Reply from 127.0.0.1: bytes=32 time<10ms TTL=128 Reply from 127.0.0.1: bytes=32 time<10ms TTL=128 Reply from 127.0.0.
Installation and Configuration 5-33 If successful, the net use command returns: “The command completed successfully.” If the command is not successful, make sure the administrative shares on remote node drives are present. Installing the Ethernet OSDs To install the OSDs for an Ethernet cluster interconnect: 1. From the installing node, log in as a user who has administrator privileges on all nodes in the cluster.
5-34 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide 3. If autorun is enabled, the Oracle Universal Installer Welcome screen appears. If autorun is disabled, enter d:\setup at a C: prompt. Substitute the letter assigned to the CD drive for d. The Oracle Universal Installer Welcome screen appears. NOTE: The OUI screens presented here might indicate earlier revision levels for the OSDs.
Installation and Configuration 5-35 4. Click Next to continue. The License screen appears. 5. Click Accept to accept the terms of the license agreement. The File Locations screen appears. Compaq Confidential – Need to Know Required Writer: Carol Tatick Project: Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Comments: Part Number: 225082-002 File Name: f-ch5 Installation and Configuration.
5-36 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide 6. Click Next to continue. The Installation Types screen appears. Compaq Confidential – Need to Know Required Writer: Carol Tatick Project: Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Comments: Part Number: 225082-002 File Name: f-ch5 Installation and Configuration.
Installation and Configuration 5-37 7. Select Typical to install the Ethernet OSDs into the default location, C:\Compaq\OPS. Skip to step 12. 8. Select Custom to install the Ethernet OSDs into a location of your choice. The Component Locations screen appears. 9. If you want to accept the default location, skip to step 13. 10. If you want to specify an alternate location and you know the exact directory path, specify it by typing over the default location and skip to step 12.
5-38 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide 11. If you want to specify an alternate location but do not know the exact directory path, click Change Location. The Choose Directory screen appears.
Installation and Configuration 5-39 12. Browse to the location where you want to install the OSDs. Click OK. The Component Locations screen appears indicating the alternate location you specified. Compaq Confidential – Need to Know Required Writer: Carol Tatick Project: Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Comments: Part Number: 225082-002 File Name: f-ch5 Installation and Configuration.
5-40 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide 13. Click Next to continue. The Cluster Members screen appears. 14. The Cluster Members screen indicates the host name of the installing machine. Define the node names for the client LAN by adding the node names you specified in the hosts and lmhosts files for the client LAN for all other cluster nodes. Do not use special characters. Use a space to separate the node names.
Installation and Configuration 5-41 15. Click Next to continue. The Cluster Members screen reappears. 16. The Cluster Members screen now shows the node names you specified on the first Cluster Members screen with “_san” appended to each name, indicating it as a cluster interconnect node name. If necessary, retype the cluster interconnect node names to match the names you entered into the hosts and lmhosts files for the Ethernet cluster interconnect. Do not use special characters.
5-42 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide 17. Click Next to continue. The OSD Summary screen appears. Compaq Confidential – Need to Know Required Writer: Carol Tatick Project: Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Comments: Part Number: 225082-002 File Name: f-ch5 Installation and Configuration.
Installation and Configuration 5-43 18. If the information on the OSD Summary screen is not correct, click Previous to modify the previous screens. When the information is correct, click Next to continue. The Summary screen appears. 19. Click Install to install the components listed in the summary. At this point, the OUI verifies that the installing node can communicate with other cluster nodes through both the client LAN and cluster interconnect node names.
5-44 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide 21. When the installation completes, the End of Installation screen appears. 22. Click Exit to quit the OUI or Next Install to install another product. 23. Start the cluster manager, OracleCMService, on all nodes in the cluster. This service must be started before installing Oracle software. See the Oracle documentation for information on starting the cluster manager.
Installation and Configuration ■ Oracle8i Parallel Server Manager ■ Oracle8i Enterprise Manager 5-45 NOTE: If the Oracle Universal Installer does not offer the option to install Oracle8i Parallel Server, it could mean that the OSDs did not install successfully or that the OracleCMService is not running. After installing Oracle software, the Oracle8i documentation instructs you to install vendor-supplied operating system dependent modules (OSDs). You have already completed this step.
5-46 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Object Link Manager simplifies the creation and maintenance of symbolic links between disk partitions and Oracle data files by placing the symbolic link names directly into the disk partitions, thereby tracking the symbolic links dynamically.
Installation and Configuration 5-47 Verifying the Hardware and Software Installation Cluster Communications Use the ping utility to verify that each node in the cluster can communicate with every other node over the client LAN. Run the ping utility from the installing cluster node. Verify the installing node can communicate with all cluster nodes by pinging the client LAN name. The IP address displayed by the ping utility should be the client LAN IP address.
5-48 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Other Verification Tasks Perform these other verification tasks: ■ Verify that Object Link Manager has been installed and is running. ■ Verify that the OPS_CMDISK symbolic link has actually been set. To verify the Object Link Manager has been installed and is running, select the Services icon in the Control Panel.
Installation and Configuration 5-49 Server Power Distribution Figure 5-9 shows an example of server power distribution for a three-node cluster. Power supply #1 of each server is connected to PDU #1. Power supply #2 of each server is connected to PDU #2. PDU #1 is connected to UPS #1, and PDU #2 is connected to UPS #2. Each UPS is connected to building power. Proliant Servers PDU #1 UPS #1 PDU #2 UPS #2 Building Power Power Supply #1 Power Supply #2 Figure 5-9.
5-50 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Power Sequencing Be sure to power up the cluster components in the following order: 1. RA4000/RA4100 Arrays 2. Storage Hubs, FC-AL Switches, or Fibre Channel SAN Switches (Power is applied to Storage Hub when the AC power cord is plugged in.) 3. Ethernet hubs or switches 4. ProLiant servers Be sure to power down the cluster components in the following order: 1. ProLiant servers 2.
Chapter 6 Cluster Management Throughout the life of your cluster, you might need to improve its performance, upgrade hardware components, upgrade software, increase storage capacity, and monitor ongoing activities. This chapter describes these management activities for the Compaq Parallel Database Cluster Model PDC/O2000 (referred to here as the PDC/O2000).
6-2 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Cluster Management Concepts Powering Off a Node Without Interrupting Cluster Services At some time during the life of your cluster you will need to perform an operation on a cluster node that will require it to be powered off. Physically moving the cluster node, removing a hardware device, and adding a hardware device commonly require the node to be powered off.
Cluster Management 6-3 4. Determine how critical it is to repair the problem. G If the problem is not considered to be critical, wait until a non-peak time to service the problem. G If the problem is critical but does not affect all cluster nodes, shut down the Oracle instances on the affected nodes and wait until a non-peak time to service the problem. G If the problem is critical and affects all the cluster nodes, shut down the database instances on all cluster nodes and correct the problem.
6-4 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide The following log files are used: ■ The Node Manager (NM), provided by Compaq, logs messages to C:\compaq\ops\nm.log ■ By default the OracleCMService sends messages to C:\compaq\ops\cmsrvr.log Management Applications Monitoring Server and Network Hardware Compaq Insight Manager is used to manage the hardware components of your cluster.
Cluster Management 6-5 Managing Shared Drives There are two levels of management with shared drives in RA4000 Arrays or RA4100 Arrays. The physical disks and the drive arrays created from them are managed with the Array Configuration Utility (ACU). The logical drives are created and managed with the Disk Management utility.
6-6 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Compaq Secure Path software can detect failures in redundant Fibre Channel Fabric components. When a failure occurs on an active path, Secure Path reroutes I/O through the standby path, allowing applications to continue processing. The Secure Path product CD is included in the cluster kit for the Compaq Parallel Database Model PDC/O2000 for Oracle8i on Microsoft Windows 2000.
Cluster Management 6-7 Monitoring the Database Oracle8i Enterprise Manager uses agents to continuously monitor database activities and offer a graphical console allowing administrators to manage the database. From the Enterprise Manager Console, the administrator can administer, diagnose, and tune multiple databases. Jobs can also be scheduled, software can be distributed, and objects and events can be monitored from the console.
6-8 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Before installing Compaq Insight Manager, Compaq Insight Manager XE, or Oracle8i Enterprise Manager, it is recommended that you read through the corresponding documentation to determine how to set up and configure each of these programs to run remotely. Software Maintenance for Oracle8i Deinstalling the OSDs At some point, you might want to deinstall the operating system dependent modules (OSDs).
Cluster Management 6-9 4. If autorun is disabled, enter d:\setup at a C: prompt. Substitute the letter assigned to the CD-ROM drive for d. The Oracle Universal Installer Welcome screen appears. Compaq Confidential – Need to Know Required Writer: John Blackburn Project: Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Comments: Part Number: 225082-002 File Name: g-ch6 Cluster Management.
6-10 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide 5. Click Deinstall Products. Next, a window containing an inventory of the installed components appears. The following shows the Inventory screen that is displayed when Ethernet is used for the cluster interconnect. 6. Select Compaq Ethernet OSDs and click Remove. NOTE: Your screen might display later revision levels for the OSDs. 7.
Cluster Management 6-11 Upgrading Oracle8i Server The design of the PDC/O2000 is tightly integrated with Oracle8i Server. Significant changes in Oracle8i Server will likely affect the operation of the cluster. Before upgrading to any new release of Oracle8i Server, consult your Compaq and Oracle service representatives. Make sure that the new release is certified and supported on the PDC/O2000, and get assistance to determine which procedure you should follow to perform the upgrade properly.
6-12 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Upgrading the Ethernet OSDs To upgrade the Ethernet OSDs: 1. Insert the CD containing the new revision of the OSDs into the CD-ROM drive on the primary node. (This is the node that was used for the previous installation of the OSDs.) The Oracle Universal Installer window is displayed. 2. Click Deinstall Products. The Inventory screen displays a list of installed components. 3.
Cluster Management 6-13 6. After installing the new Ethernet OSDs, restart OracleCMService and OracleNMService on all nodes. 7. Restart the Oracle instances on all nodes. Deinstalling a Partial OSD Installation While not a common occurrence, the Oracle Universal Installer might not be able to complete the installation of the Compaq Ethernet OSDs. An incomplete OSD installation could result for various reasons, for example: ■ A client LAN cable is no longer seated tightly into its connector.
6-14 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Upgrading Oracle8i Server The design of the PDC/O2000 is tightly integrated with Oracle8i Server. Significant changes in Oracle8i Server will likely affect the operation of the cluster. Before upgrading to any new release of Oracle8i Server, consult your Compaq and Oracle service representatives.
Cluster Management 6-15 Adding Disk Drives to Increase Storage Capacity During the life of your PDC/O2000 cluster, you might need to expand the capacity of an RA4000 Array or RA4100 Array. The following steps describe how to add drives to an RA4000/RA4100 Array and how to allocate the added disk capacity to Oracle. The drive can be added and allocated to Windows 2000 Advanced Server while the database is online. The following summarizes the procedure. 1.
6-16 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide 5. Insert drives into the added RA4000/RA4100 Array. 6. Using the Fibre Channel cables, physically connect the RA4000 Array Controllers in the added RA4000/RA4100 Array to the Fibre Channel SAN Switches, FC-AL Switches, or Storage Hubs. 7. Connect all of the RA4000/RA4100 Arrays to a power source and restart them. 8. Restart the primary cluster node. 9.
Cluster Management 6-17 5. From the Fibre Host Adapter, disconnect the Fibre Channel cable installed between the Fibre Host Adapter and its Fibre Channel SAN Switch, FC-AL Switch, or Storage Hub. 6. Remove the failed Fibre Host Adapter and insert a new one. 7. From the new Fibre Host Adapter, connect the Fibre Channel cable that runs to its Fibre Channel SAN Switch, FC-AL Switch, or Storage Hub. 8. Power on the node. 9. Restart the Oracle services on the node. 10. Start up the Oracle instance on the node.
6-18 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide 4. Stop all Oracle services on the node to be removed. 5. Shut down the server on that node and power it off. 6. Disconnect the Fibre Channel cables between the evicted node and the Fibre Channel SAN Switches, FC-AL Switches, or Storage Hubs. 7. Disconnect the cluster interconnect cables from the evicted node and label them as the cluster interconnect cables. 8.
Cluster Management 6-19 5. Install and configure Windows 2000 Advanced Server with Service Pack 1 or later (whichever is installed on the other cluster nodes). 6. Install necessary Compaq drivers and utilities from the Compaq SmartStart and Support Software CD. 7. Install Fibre Host Adapters in the replacement node in the same slot locations used in the other cluster nodes. 8.
6-20 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide 1. To add a node to the cluster, you must disassemble the existing cluster and then reform it to include the replacement node. Do the following on all original cluster nodes (not the replacement node): a. Shut down the Oracle instance. b. Stop all Oracle services. c. Deinstall the OSDs using the OUI from the original installing node. d. Restart all nodes to complete the deinstallation process. 2.
Cluster Management 6-21 Adding a Cluster Node Before adding a node, verify the new configuration is supported by the PDC/O2000. To determine if your new server is supported, see the Compaq Parallel Database Cluster Model PDC/O2000 Certification Matrix for Windows 2000 at www.compaq.com/solutions/enterprise/ha-pdc.html IMPORTANT: With some possible exceptions, the servers in the cluster must be set up identically. For example, this means using one model server for all nodes.
6-22 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Preparing the New Node To minimize downtime of the cluster, several preparation steps should be performed on the new node before it is integrated into the cluster. Prior to integrating the new node into the cluster, perform the following steps: 1. Connect the server for the new node to the cluster interconnect components and to the client LAN. Power on the new node.
Cluster Management 6-23 10. Restart the new node. 11. Verify TCP/IP connectivity between the replacement node and the existing cluster nodes. Run the ping utility from the replacement node, pinging the client LAN adapters on all other nodes. If Ethernet is used as the cluster interconnect, also ping the cluster interconnect adapters of all other nodes. 12. Verify that the new node can access the shared storage.
6-24 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide To install the cluster software, perform the following steps: 1. To add a node to the cluster, you must disassemble the existing cluster and then reform it to include the new node. Do the following on all original cluster nodes (not the new node): a. Shut down the Oracle instance. b. Stop all Oracle services. c. Deinstall the OSDs by running the OUI on the node on which the OUI was originally run. d.
Cluster Management 6-25 Monitoring Cluster Operation Tools Overview Several tools exist to monitor the operation of your cluster. Use the tools to regularly monitor the operation of each node and each Oracle instance. The gathered information might identify a disparity in the workload being performed by each node, which signifies a need to adjust the distribution of the workload so that the overall cluster performance is maximized.
6-26 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Secure Path Manager is the cluster management component of Secure Path. Secure Path Manager provides a Graphical User Interface (GUI) utility that: ■ Reports the status of the I/O paths in each redundant FC-AL or redundant Fibre Channel Fabric. ■ Reports disk status (path assignment, failover, and failback activity). ■ Enables automatic drive failback upon path restoration.
Cluster Management 6-27 Manipulating Storagesets and I/O Paths You can use Secure Path Manager to perform the following actions on the storagesets and I/O paths that are managed by Secure Path Manager: ■ Move a storageset from one array controller to another. Use this action when you want to change the ownership of a storageset from one array controller to the other. ■ Make an I/O path Alternate.
6-28 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Note that Secure Path Manager client is not required to be running in order for Secure Path to protect path availability. The RaiDisk device driver running on cluster nodes handles Secure Path’s protection capability. Refer to the Secure Path documentation for descriptions of the icons used to indicate changes in path and controller status.
Chapter 7 Troubleshooting This chapter addresses potential problems you might encounter as you install, configure, test, and operate your Compaq Parallel Database Cluster Model PDC/O2000 (referred to here as the PDC/O2000).
7-2 Compaq Parallel Database Cluster Model PDC/02000 for Oracle8i and Windows 2000 Administrator Guide Basic Troubleshooting Tips Power Verify that the following are all powered on: ■ Cluster nodes ■ StorageWorks RAID Array 4000s (RA4000 Arrays) or StorageWorks RAID Array 4100s (RA4100 Arrays) ■ Compaq StorageWorks Fibre Channel SAN Switches (Fibre Channel SAN Switches), Compaq StorageWorks Storage Hubs (Storage Hubs), or Compaq StorageWorks FC-AL Switches (FC-AL Switches) ■ Switches or hubs for th
Troubleshooting 7-3 Access to Cluster Components Once the physical connections have been checked, ensure that communication is occurring between all the cluster components. ■ To test the redundant Ethernet cluster interconnect and client LAN connections, use the ping utility. From each cluster node, ping the Ethernet cluster interconnect adapter or port located in the other nodes.
7-4 Compaq Parallel Database Cluster Model PDC/02000 for Oracle8i and Windows 2000 Administrator Guide For information about currently supported software revisions for the PDC/O2000, refer to the Compaq Parallel Database Cluster Model PDC/O2000 Certification Matrix for Windows 2000 at www.compaq.com/solutions/enterprise/ha-pdc.html Depending on your specific configuration, other software (for example, customer applications such as packet utilities) might exist that should be of the same revision level.
Troubleshooting 7-5 Troubleshooting Oracle8i and OSD Installation Problems and Error Messages Potential Difficulties Installing the OSDs with the Oracle Universal Installer ■ The Oracle Universal Installer (OUI) uses an inventory file to track which components of the OSDs have been installed on each of the cluster nodes. A single inventory file is maintained on the node from which the OUI was initially run. All subsequent uses of the OUI must be made from that same installing node.
7-6 Compaq Parallel Database Cluster Model PDC/02000 for Oracle8i and Windows 2000 Administrator Guide ■ Determine whether or not you need to deinstall a partial OSD installation. An incomplete installation can result in a partial installation of the OSDs on the cluster. If this occurs, you must clean up after the partial installation (by removing it) before you can successfully run the OUI again.
Troubleshooting 7-7 Unable to Start OracleNMService ■ Verify that the OracleNMService has properly started by checking the error log. The default location for the error log is C:\compaq\ops\nm.log. If not found in the default location, look in the same directory in which the OSD modules were installed. ■ Verify that the Oracle8i software is correctly installed on all the nodes. Refer to Chapter 5, “Installation and Configuration,” for an installation overview.
7-8 Compaq Parallel Database Cluster Model PDC/02000 for Oracle8i and Windows 2000 Administrator Guide Initialization of the Dynamic Link Library NM.DLL Failed If this failure occurs, stop the currently running OracleCMService and OracleNMService. Restart the OracleNMService, then the OracleCMService. From Administrative Tools, open the Services screen. Locate each service in the displayed list, highlight it, then stop it. Finally, highlight the service and restart it.
Troubleshooting 7-9 Unable to Ping the Cluster Interconnect or the Client LAN If you are using the recommended method of using static IP addresses, ensure that the hosts file and lmhosts files are properly set up. When Using the Ethernet Cluster Interconnect ■ Make sure that the Ethernet cables are properly connected and that the Ethernet switches or hubs are powered on.
7-10 Compaq Parallel Database Cluster Model PDC/02000 for Oracle8i and Windows 2000 Administrator Guide ■ Make sure the OracleCMService and all other Oracle services have started without error. From the Services screen, scan the list of services and check to see if the Oracle services have started. From the Application log, check for error messages about the Oracle services.
Troubleshooting 7-11 Troubleshooting Shared Storage Problems This section addresses potential problems arising from the use of RA4000 Arrays or RA4100 Arrays as the shared storage devices for the PDC/O2000. This section does not address RA4000/RA4100 Array problems that are specific to the product itself or to the use of the RA4000/RA4100 Array in a stand-alone server configuration.
7-12 Compaq Parallel Database Cluster Model PDC/02000 for Oracle8i and Windows 2000 Administrator Guide Verifying Connectivity to a Redundant Fibre Channel Arbitrated Loop To ensure that all I/O path connections in the redundant FC-AL are intact and functioning, verify the following: ■ For each redundant FC-AL, at least two Fibre Host Adapters have been installed in the same PCI slots across all servers in the cluster.
Troubleshooting ■ Determine whether or not the Compaq servers support the physical SCSI disks installed in the RA4000/RA4100 Arrays. Check the event log for disk I/O error messages or indications of problems with the communications transport. ■ If the drives are still not recognized, it is recommended that you start investigating the problem from the lowest level of drive configuration. G Run the Compaq Array Configuration Utility.
7-14 Compaq Parallel Database Cluster Model PDC/02000 for Oracle8i and Windows 2000 Administrator Guide A Cluster Node Cannot Connect to the Shared Drives ■ Verify that a physical connection exists from each node to each Fibre Channel SAN Switch, FC-AL Switch, or Storage Hub. G Verify the shortwave Gigabit Interface Converter (GBIC) modules are properly seated in the affected Fibre Channel SAN Switch, FC-AL Switch, or Storage Hub, and RA4000 Array Controller.
Troubleshooting ■ The same Secure Path Manager password must be used for all of the hosts in a profile. ■ Before you change the Secure Path Agent’s list of authorized clients or passwords, you must first manually stop the Secure Path Agent. Be sure to restart the Secure Path Agent after the changes have been completed. ■ Before you modify device volume labels with the Disk Management, you must first manually stop the Secure Path Agent.
7-16 Compaq Parallel Database Cluster Model PDC/02000 for Oracle8i and Windows 2000 Administrator Guide Secure Path Manager Is Delayed In Reporting Path Failure Information Due to extended error checking features present in the Secure Path software, there may be some delay with Secure Path Manager in reporting failed path information. By default, the Secure Path Manager screen is refreshed every 60 seconds. To perform a manual refresh, press the F5 key.
Appendix A Diagnosing and Resolving Shared Disk Problems Introduction When one or more nodes in a Compaq Parallel Database Cluster Model PDC/O2000 (referred to here as the PDC/O2000) cannot properly join the cluster, you need to consider two possible causes: ■ A problem related to the shared disk storage in the cluster’s Compaq StorageWorks RAID Array 4000s (RA4000 Arrays) or Compaq StorageWorks RAID Array 4100s (RA4100 Arrays) is preventing one or more nodes from accurately viewing or successfully acce
A-2 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Figure A-1 identifies, in a flowchart format, the sequence of tasks you should perform to diagnose, isolate, and resolve shared disk problems in a PDC/O2000. A detailed description of each task is provided in the following sections of this appendix. Entry Point: Potential Shared Disk-Related Problem Detected Run Object Link Manager On All Nodes OK Problem Persists Exit.
Diagnosing and Resolving Shared Disk Problems A-3 Run Object Link Manager on All Nodes Object Link Manager simplifies the creation and maintenance of symbolic links between disk partitions and Oracle data files by placing the symbolic link names directly onto the disk partitions, thereby tracking the symbolic links dynamically. This means that if the order in which disk drives are brought online changes, Oracle8i Server can find the correct data files and can properly start the Oracle database.
A-4 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Restart All Affected Nodes in the Cluster If Object Link Manager does not run properly on one or more nodes or does not provide consistent information about Oracle partitions and symbolic links from all nodes, perform these steps: 1. Identify the node or nodes on which Object Link Manager either does not run properly or does not present the correct information about Oracle partitions and symbolic links.
Diagnosing and Resolving Shared Disk Problems A-5 Run and Validate Secure Path Manager On All Nodes The Secure Path Manager screens provide information about the shared storage resources in the cluster, including RA4000/RA4100 Array storagesets, array controller path configurations, drive lettering, and volume labels. To run and validate Secure Path Manager, perform these steps from each node: 1.
A-6 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Run and Validate the Array Configuration Utility On All Nodes The Array Configuration Utility is used to configure the hardware aspects of disk drives connected to the RA4000 Array Controllers. The Array Configuration Utility allows you to configure RAID levels and to add disk drives or RA4000/RA4100 Arrays to an existing configuration.
Diagnosing and Resolving Shared Disk Problems A-7 Perform Cluster Hardware Checks Validate the cluster hardware by performing these steps: 1. Determine whether or not hardware errors or communications transport failures might be causing the problem. Use the Event Viewer to verify that no hardware errors or transport problems exist. Check the event log for disk I/O error messages or indications of problems with the communications transport. 2.
A-8 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Contact Your Compaq Support Representative If you are unable to resolve the problem, contact your Compaq support representative for assistance. In North America, call the Compaq Technical Support Phone Center at 1-800-OK-COMPAQ. Outside North America, call the nearest Compaq Technical Support Phone Center. Telephone numbers for worldwide Technical Support Centers are listed on the Compaq website at www.
Glossary Availability A measure of how well a computer system can deliver services to its clients continuously. Availability is typically expressed as a percentage, with 100 percent being the best possible rating. Client LAN A local area network dedicated to servicing requests between network client machines and between the cluster nodes and the client machines. This is different than a cluster interconnect, which is dedicated to servicing information requests to and from cluster nodes only.
2 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Fibre Channel An IEEE standard for providing high-speed data transfer among workstations, mainframes, supercomputers, desktop computers, storage devices, and display devices. Fibre Channel Arbitrated Loop (FC-AL) A data path from the Fibre Host Adapters in the cluster nodes to the array controllers in the RA4000 Arrays or RA4100 Arrays. A PDC/O2000 requires one or more redundant FC-ALs.
Glossary 3 Load balancing To balance the amount of work (processing) undertaken by each node in a cluster. NIC Network Interface Controller. A board that allows a computer to be connected with a network, and works with the network operating system to control the flow of information over the network. Node An individual server in a cluster. NTFS New Technology File System. A file organization system by which data is stored and accessed in a Microsoft Windows NT operating system.
4 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Scalability The ability of a system (server, storage, network, or cluster) to grow or expand in performance, throughput, or capacity. Secure Path The Compaq software that provides cluster nodes with continuous data access to RA4000/RA4100 Array disks drives.
Index A active/active configurations cabling for Oracle8i 5-11 active/active configurations for redundant FC-ALs definition 2-47 example with five RA4000/RA4100 Arrays 2-63 example with four RA4000/RA4100 Arrays 2-62 example with three RA4000/RA4100 Arrays 2-61 example with two RA4000/RA4100 Arrays 2-60 examples overview 2-58 active/active configurations for redundant Fibre Channel Fabrics definition 2-16 example with five RA4000/RA4100 Arrays 2-32 example with four RA4000/RA4100 Arrays 2-31 example with
2 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide active/standby configurations for redundant Fibre Channel Fabrics definition 2-16 example with five RA4000/RA4100 Arrays 2-26 example with four RA4000/RA4100 Arrays 2-25 example with one RA4000/RA4100 Array 2-22 example with three RA4000/RA4100 Arrays 2-24 example with two RA4000/RA4100 Arrays 2-23 examples overview 2-20 ACU See Compaq Array Configuration Utility adding disk drives in RA4000/RA4100 Arrays
Index client reconnection software for Oracle8i 3-8 cluster communications verifying before installing OSDs for Oracle8i 5-31 verifying over client LAN for Oracle8i 5-47 verifying over Ethernet cluster interconnect for Oracle8i 5-47 cluster configuration planning sample large PDC/O2000 cluster 4-9, 4-13 sample small PDC/O2000 cluster 4-7, 4-11 cluster connectivity, troubleshooting 7-9 cluster degradation, troubleshooting 6-2 cluster interconnect adapters installation for Oracle8i 5-6 defined 2-70 distribut
4 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide node cannot connect to shared drives 7-14 nodes cannot communicate 7-8 nodes unable to rejoin cluster 7-9 Oracle8i installation 7-5 OracleCMService 7-6 OracleNMService 7-7 RA4000/RA4100 Arrays not recognized 7-12 Secure Path 7-14 shared storage 7-11 summary of potential problems 7-1 unable to start database for Oracle8i 7-7 clusters availability in 1-3 defined 1-2 overview 1-1 PDC/O2000 example 1-2 scalabi
Index Compaq software overview Compaq Array Configuration Utility 3-3 Compaq Insight Manager 3-4 Compaq Insight Manager XE 3-4 Compaq operating system dependent modules for Oracle8i 3-5 Compaq Options ROMPaq 3-4 Compaq SmartStart and Support software 3-2 Compaq SmartStart and Support Software CD 3-2 Compaq System Configuration Utility 3-3 Fibre Channel Fault Isolation Utility 3-3 Compaq StorageWorks 64-Bit/66 MHz Fibre Channel Host Bus Adapters installing for Oracle8i 5-4 Compaq StorageWorks Channel Host B
6 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide E error logs for Oracle8i 6-4 Ethernet adapters planning 4-6 Ethernet client LAN cabling for Oracle8i 5-20 Ethernet cluster interconnect 2-70 adapter installation for Oracle8i 5-7 adapters 2-72 adapters shared by cluster interconnect and client LAN 2-73 cabling for Oracle8i 5-16, 5-19 Ethernet switch 2-72 lmhosts and hosts files entries for Oracle8i 5-22 non-redundant Ethernet cluster interconnect 2-71 red
Index Gigabit Interface ConverterShortwave modules description 2-8 installing for Oracle8i 5-5, 5-9 grouping physical disk storage space 4-18 H hardware installation verification for Oracle8i 5-47 help additional sources xix Compaq authorized resellers, telephone numbers xx Compaq website xx technical support telephone numbers xix hosts files for Ethernet cluster interconnect 4-20 for Oracle8i 5-24, 5-31 I I/O data paths Fibre Channel SAN Switches to array controllers 2-15 Fibre Host Adapters to Fibre Ch
8 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide installation additional Fibre Channel Arbitrated Loops for Oracle8i 5-15 automated using SmartStart for Oracle8i 5-22 client LAN adapters for Oracle8i 5-7 cluster interconnect adapters for Oracle8i 5-6 Ethernet cluster interconnect for Oracle8i 5-7 Fibre Host Adapters for Oracle8i 5-4 GBIC-SW modules for Oracle8i 5-5, 5-9 Object Link Manager for Oracle8i 5-45 operating system software for Oracle8i 5-21 Ora
Index Oracle software certification for Oracle8i 3-7 configuring for Oracle8i 5-45 installation troubleshooting for Oracle8i 7-5 installing for Oracle8i 5-44 troubleshooting the Oracle Universal Installer 7-5 Oracle Universal Installer deinstalling OSDs 6-8 installing OSDs for Oracle8i 5-30 Oracle8 Enterprise Manager monitoring the database with 6-7 Oracle8 Performance Manager and Oracle8i Performance Manager, monitoring cluster operation with 6-25 Oracle8i Enterprise Edition, overview 3-6 Oracle8i Enterpr
10 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide shared storage components for redundant Fibre Channel Fabrics 4-3 site planning 4-2 supported RAID levels 4-16 power distribution for Oracle8i 5-48 power distribution units for Oracle8i 5-48 power sequencing for Oracle8i 5-50 ProLiant servers high-availability features 2-3 power distribution for Oracle8i 5-49 supported servers 2-2 R RA4000 Array Controllers See Compaq StorageWorks RAID Array 4000 Array C
Index Secure Path Manager monitoring cluster operation with 6-25 running to check for shared disk problems A-5 verifying shared storage resources 5-27 Service Packs for Windows 2000 Advanced Server versions supported 3-1 Service Packs for Windows NT Server supported for Oracle8i 5-22 shared disk problems contacting your Compaq support representative A-8 introduction A-1 performing cluster hardware checks A-7 performing cluster software and firmware checks A-6 rerunning and validating Object Link Manager A-
12 Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i and Windows 2000 Administrator Guide Fibre Channel Fabric connectivity guidelines 7-11 NM.
