Serviceguard Extended Distance Cluster (EDC) with VxVM/CVM Mirroring on HP-UX, May 2008

ManualsBrandsHP ManualsSoftwareHP-UX 11i Volume Management (LVM/VxVM) Software

HP Serviceguard Extended Distance Cluster

(EDC) with VxVM/CVM mirroring on HP-UX

Best practices

Table of contents

Introduction............................................................................................................................................................ 2

Management Summary........................................................................................................................................ 2

Terms and Definitions.......................................................................................................................................... 2

Related documents.............................................................................................................................................. 3

Scope of this document........................................................................................................................................ 3

Extended Distance Cluster Architecture...................................................................................................................... 4

Architecture........................................................................................................................................................ 4

Example hardware configuration.......................................................................................................................... 5

Example VxVM/CVM configuration...................................................................................................................... 6

VxVM/CVM features for an EDC.............................................................................................................................. 7

Disk naming scheme ........................................................................................................................................... 7

Disk Access Name and Disk Media Name............................................................................................................. 8

Disk Group configuration database on disk ........................................................................................................... 9

Mirror (plex) configuration................................................................................................................................... 9

Data Change Object (DCO) volume for Dirty Region Log (DRL) and FastMirrorResync (FMR)...................................... 10

Choosing between the Dirty Region Logging (DRL) types.................................................................................... 12

SmartSync Recovery Accelerator..................................................................................................................... 13

FastMirrorResync (FMR)................................................................................................................................. 13

Read Policy for Mirrored Volumes....................................................................................................................... 14

Hot Relocation not recommended for shared volumes in an EDC ............................................................................ 14

Detach policy and Disk Group Failure policy........................................................................................................ 15

Storage Multi-Pathing configuration..................................................................................................................... 15

License information for selected VxVM features..................................................................................................... 15

Administrative VxVM/CVM tasks in an EDC ............................................................................................................ 16

Taking a site offline........................................................................................................................................... 16

Adding shared storage...................................................................................................................................... 17

Adding LUNs to existing shared disk groups .................................................................................................... 17

Extending existing volumes and creating new volumes ......................................................................................17

EDC specific failure scenarios................................................................................................................................. 18

Complete site failure ......................................................................................................................................... 18

Site failure simulation and reaction ................................................................................................................. 18

Restoring a failed site.................................................................................................................................... 19

Inter-site link failure ........................................................................................................................................... 20

Inter-Site Link failure simulation and reaction .................................................................................................... 20

Restore the Inter-Site Link and restart the detached site....................................................................................... 21

Special considerations for Inter-Site Link failures ............................................................................................... 21

Summary of Best Practices...................................................................................................................................... 23

Attachment A: Extended Distance Cluster for RAC..................................................................................................... 24

Oracle Cluster Registry and Vote Disk ................................................................................................................. 25

Oracle Clusterware and Oracle Database Binaries............................................................................................... 25

Oracle Database Datafiles................................................................................................................................. 25

Database Recovery Data ................................................................................................................................... 25

For more information............................................................................................................................................. 26

Summary of content (26 pages)

PAGE 1
HP Serviceguard Extended Distance Cluster (EDC) with VxVM/CVM mirroring on HP-UX Best practices Table of contents Introduction ............................................................................................................................................................ 2 Management Summary........................................................................................................................................ 2 Terms and Definitions ...............................................
PAGE 2
Introduction Management Summary An Extended Distance Cluster (EDC) is a “special” Serviceguard cluster architecture that protects against multiple failures, such as a complete datacenter failure, by extending a single Serviceguard cluster across two datacenters. A key component is the data replication method between the two datacenters. Since data replication is based on volume manager functionality, an EDC is independent from the type of shared storage. Serviceguard supports two volume managers.
PAGE 3
Related documents The following documents provide valuable information on the technology discussed in this white paper. • http://docs.hp.com → 11i v2 (under “Core HP-UX”) → VxVM – VERITAS Volume Manager 4.1 Administrator's Guide – VERITAS Volume Manager 4.1 Troubleshooting Guide – VERITAS Volume Manager 5.0 Administrator’s Guide – VERITAS Volume Manager 5.0 Troubleshooting Guide – VERITAS Flashsnap Point-In-Time-Copy Solutions Administrator’s Guide • http://docs.hp.
PAGE 4
Extended Distance Cluster Architecture Architecture Local Serviceguard clusters are implemented to reduce single points of failure (SPOF) by deploying redundant nodes, network switches, storage switches in the cluster. This leaves the data center itself as a SPOF when a disaster such as fire, flood, or a power outage occurs.
PAGE 5
Example hardware configuration Figure 1.
PAGE 6
Example VxVM/CVM configuration The disk group and volume configuration must follow the same general guidelines as the hardware configuration. To survive a data center outage, each volume must have one copy of the data located in each data center. VxVM/CVM provides a mirroring functionality that can store one plex of each volume in each data center.
PAGE 7
VxVM/CVM features for an EDC VxVM/CVM has a number of features that are useful in an EDC. The following sections explain these features and make recommendations on how to use them in an EDC. Disk naming scheme VxVM 4.1 offers two different disk naming schemes—operating system naming (OSN) and enclosurebased naming (EBN).
PAGE 8
Disk Access Name and Disk Media Name When VxVM recognizes a disk device for the first time, (either after the installation of VxVM/CVM or when attaching new devices to the system), the disk device is configured in the internal DMP and VxVM configuration database. At this time the device files (referred to as DMP meta nodes in VxVM documentation) are also created. They are located in “/dev/vx/dmp/rdsk” and “/dev/vx/dmp/dsk”. These device files are persistent across reboot by default.
PAGE 9
Disk Group configuration database on disk VxVM creates and maintains configuration information in the private region of several disks belonging to a disk group. Multiple disks in different enclosures are automatically selected to store persistent copies of the VxVM object information. There is normally no need to manually change the choices that VxVM makes. To verify that disks at both sites are configured to hold configuration information, use the vxdg command described in figure 5. Figure 5.
PAGE 10
Figure 7 shows an example of the vxassist command with all of the recommended options for creating an EDC volume—including the mirror configuration. Data Change Object (DCO) volume for Dirty Region Log (DRL) and FastMirrorResync (FMR) When large mirrored volumes get out of sync due to a failure condition like system crash or storage/link failure, or due to a maintenance operation, the resynchronization can take a long time.
PAGE 11
Figure 6 shows the objects of a mirrored data volume, including its attached DCO volume: Figure 6.
PAGE 12
• dcomirror=2 This option specifies the number of plexes for the attached DCO volume. The DCO volume should have the same number of plexes as its parent volume. Figure 7 shows an example of the vxassist command with all of the recommended options for creating an EDC volume, including how to specify the creation of a version 20 DCO volume for DRL and FMR. Choosing between the Dirty Region Logging (DRL) types A DRL is a bitmap of regions that is used to track write I/Os that are in flight.
PAGE 13
SmartSync Recovery Accelerator For certain applications/databases, VxVM offers a feature which is called SmartSync Recovery Accelerator. If supported by the database vendor, the SmartSync feature uses an interface between VxVM volumes and the database software to avoid unnecessary work during mirror resynchronization. In an Oracle environment, VxVM/CVM and Oracle exchange information about which volumes are used as online “redo-log” volumes. For those volumes, DRL should be set to sequential.
PAGE 14
Read Policy for Mirrored Volumes VxVM offers different read policies for mirrored volumes. The default policy is “select” which reads in a round-robin fashion if none of the plexes are striped. In an EDC, it might be beneficial to read from a specific plex that is local to the data center where the reading node resides, thus reducing the load on the inter-site storage link. With the VxVM 5.
PAGE 15
Detach policy and Disk Group Failure policy There are two policies that control the behavior following I/O errors or failures. These are the detach policy and the disk group failure policy. The detach policy has two possible settings: • Global—This is the default policy. When the detach policy is set to global, a plex will be detached from a volume on a cluster-wide basis if a CVM slave node experiences an I/O error on that plex. • Local—This policy was introduced in VxVM 3.
PAGE 16
Administrative VxVM/CVM tasks in an EDC Taking a site offline The main purpose of an EDC is to provide a highly available environment which can continue to provide computing resources to applications even if an entire data center becomes unavailable, either due to a disaster or scheduled maintenance.
PAGE 17
The same vxsnap commands can be used to break off a mirror for off-host processing for example, to perform a backup, or to refresh data for decision support systems. After the off-host processing is completed, vxsnap can be used to re-attach the mirror. The VERITAS FlashSnap Point-In-Time Copy Solutions Administrator’s Guide listed in the “Related Documents” section provides further examples and details for off-host processing.
PAGE 18
This command changes the size of vol1 to 20 GB. At the same time, it will also change the file system size to 20GB, if vol1 contains a file system. The associated DCL volume will also be resized to accommodate the larger FMR. The vxresize command can also be used to shrink a volume. For more details on resizing VxVM/CVM volumes, please see the VERITAS Volume Manager Administrator’s Guides referred to in the Related Documents section.
PAGE 19
Depending on where the application ran prior to the site failure, the following behavior can be expected when a site completely fails: • Applications which ran on the failing site are expected to fail over automatically to the remaining site. • The applications that ran at the remaining site are expected to keep running without manual intervention or interruption. A short pause or hang might be experienced. A site failure like this leaves each volume with just one mirror (plex) available.
PAGE 20
Inter-site link failure By extending a single Serviceguard cluster to span multiple data centers to protect against data center outages, a new component is added to the cluster—an inter-site link. To keep the cluster from splitting into halves, the inter-site link needs to be made redundant. This applies to the heartbeat network as well as to the storage link between the sites.
PAGE 21
• Applications that run on nodes of the failing site (nodes that were halted by Serviceguard) are expected to fail over automatically to the remaining site. • The applications that run at the remaining site are expected to keep running without manual intervention or interruption. A short pause or hang might be experienced. • If, for any reason, a volume has no active, enabled plex at the remaining site, I/O operations to that volume will be blocked.
PAGE 22
• If the storage ISL remains unavailable, choose the site at which the last complete mirror of each volume resides as the remaining site, and halt the nodes at the other site. See the steps in Restore the Inter-Site Link and restart the detached site for procedures to make the LUNs visible to the remaining nodes at the OS and DMP level. • Each disabled volume needs to be recovered individually so the nodes from the remaining site will work with the storage from the same site.
PAGE 23
Summary of Best Practices This section summarizes the best practices for EDC with VxVM/CVM which have been discussed in this document - it can be used as a checklist. For information about why and how these practices are recommended, the entire document should be read. General best practices, which also apply for non VxVM/CVM EDC: • Use a common redundant link for heartbeat and storage traffic. • Use two fault tolerant DWDM, CWDM, SONET, or SDH boxes per site instead of one.
PAGE 24
Attachment A: Extended Distance Cluster for RAC Figure A-1 illustrates a basic configuration that includes common high availability (HA) components in SGeRAC clusters. Figure A-1.
PAGE 25
Oracle Cluster Registry and Vote Disk Oracle 10g Release 2 supports up to two copies of the Oracle Cluster Registry (OCR) and up to three vote disks. For EDC, each copy of OCR and each vote disk are required to be physically mirrored by software (CVM mirroring) between the two datacenters. In the event of a loss of communication between the datacenters, the quorum server serves as a tiebreaker for Serviceguard.
PAGE 26
For more information Please visit: www.hp.com/go/serviceguard © Copyright 2008 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Linux is a U.S.