Veritas Storage Foundation™ Cluster File System Installation Guide HP-UX 5.
Veritas Storage Foundation Cluster File System Installation Guide Copyright © 2006 Symantec Corporation. All rights reserved. SFCFS 5.0 Symantec, the Symantec logo, Veritas, and Veritas Storage Foundation Cluster File System are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners.
Third-party legal notices Third-party software may be recommended, distributed, embedded, or bundled with this Veritas product. Such third-party software is licensed separately by its copyright holder. All third-party copyrights associated with this product are listed in the accompanying release notes. HP-UX is a registered trademark of Hewlett-Packard Development Company, L.P. Licensing and registration Veritas Storage Foundation Cluster File System is a licensed product.
Contents Chapter 1 Installing and configuring the product Hardware overview ..............................................................................................10 Shared storage ..............................................................................................11 Fibre channel switch ...................................................................................11 Cluster platforms .........................................................................................
Chapter 2 Upgrading the product Preparing to upgrade the product ..................................................................... 36 Planning the upgrade .................................................................................. 36 Upgrade paths .............................................................................................. 36 Upgrade Overview ............................................................................................... 38 Phased upgrade ....................
Chapter 1 Installing and configuring the product This chapter describes how to install the Veritas Storage Foundation Cluster File System (SFCFS). SFCFS requires several Veritas software packages to configure a cluster and to provide messaging services.
10 Installing and configuring the product Hardware overview Hardware overview VxFS cluster functionality runs optimally on a Fibre Channel fabric. Fibre Channel technology provides the fastest, most reliable, and highest bandwidth connectivity currently available. By employing Fibre Channel technology, SFCFS can be used in conjunction with the latest Veritas Storage Area Network (SAN) applications to provide a complete data storage and retrieval solution.
Installing and configuring the product Hardware overview Shared storage Shared storage can be one or more shared disks or a disk array connected either directly to the nodes of the cluster or through a Fibre Channel Switch. Nodes can also have non-shared or local devices on a local I/O channel. It is advisable to have /, /usr, /var and other system partitions on local devices. Fibre channel switch Each node in the cluster must have a Fibre Channel I/O channel to access shared storage devices.
12 Installing and configuring the product Software components Software components Storage Foundation for Cluster File System is the name of the Veritas Cluster File System product and its supporting software packages. Storage Foundation Cluster File System HA (SFCFS HA) provides support for application failover functionality of Veritas Cluster Server (VCS) in addition to other SFCFS features.
Installing and configuring the product Software components Package Contents VRTSvcs Veritas Cluster Server VRTSacclib Veritas ACC Library VRTSvcsag Veritas Cluster Server Bundled Agents VRTSvcsmg Veritas Cluster Server Message Catalogs VRTSjre Veritas Java Runtime Environment Redistribution VRTSjre15 Veritas Java Runtime Environment Redistribution VRTScutil Veritas Cluster Utilities VRTSweb Veritas Java Web Server VRTScscw Veritas Cluster Server Configuration Wizards VRTSvxvm Veritas V
14 Installing and configuring the product Software components Optional packages for SFCFS and SFCFS HA Packages Contents VRTScfsdc Veritas Cluster File System Documentation VRTScmccc Veritas Cluster Management Console Cluster Connector VRTScmcs Veritas Cluster Management Console (Single Cluster Mode) VRTScscm Veritas Cluster Server Cluster Manager VRTScssim Veritas Cluster Server Simulator VRTSfsdoc Veritas File System Documentation VRTSvcsdc Veritas Cluster Server Documentation VRTSvcsmn
Installing and configuring the product Required HP-UX patches Required HP-UX patches HP-UX required patches include the following: HP-UX Patch ID Description PHCO_32385 Enables fscat(1M). PHCO_32387 Enables getext(1M). PHCO_32388 Enables setext(1M). PHCO_32389 Enables vxdump(1M). PHCO_32390 Enables vxrestore(1M). PHCO_32391 Enables vxfsstat(1M). PHCO_32392 Enables vxtunefs(1M). PHCO_32393 Enables vxupgrade(1M). PHCO_32488 Enables LIBC for VxFS 4.1 and later file systems.
16 Installing and configuring the product Required HP-UX patches HP-UX Patch ID Description PHKL_32430 Changes to separate vxfs symbols from libdebug.a, so that symbols of VxFS 4.1and later are easily available in q4/p4. PHKL_32431 Changes to disallow mounting of a file system on a vnode having VNOMOUNT set. Enhancements for supporting quotas on large uids. PHKL_33312 LVM Cumulative Patch. PHKL_34010 Cumulative VM Patch.
Installing and configuring the product Preinstallation Preinstallation Release Notes Read the Release Notes for all products included with this product. Portable Document Format (.pdf) versions of the Release Notes are included on the software disc in the storage_foundation_cluster_file_system/release_notes directory and on the documentation disc that came with your software.
18 Installing and configuring the product Preinstallation Also, you can get the patches from Hewlett-Packard’s Patch Database offered under the Maintenance and Support section of the HP Services & Support - IT Resource Center. HP’s Patch Database provides fast, accurate searches for the latest recommended and superseded patches available for Veritas File System or Veritas Volume Manager.
Installing and configuring the product Prerequisites Veritas Enterprise Administrator The Veritas Enterprise Administrator (VEA) client can be installed and run on any machine that supports the Java Runtime Environment. VEA is required to access the graphical user interface (GUI) for Veritas Storage Foundation. You can use the GUI to administer disks, volumes, file systems, and database functionality on local or remote machines.
20 Installing and configuring the product Installing the product ■ The host names of the cluster nodes. ■ The device names of the network interface cards (NICs) used for the private networks among nodes. ■ Establishing communication between nodes is required to install Veritas software from a remote system, or to install and configure a cluster.
Installing and configuring the product Installing the product To install the product 1 Log in as superuser. 2 Insert the appropriate media disc into your system’s DVD-ROM drive connected to your system. 3 Determine the block device file for the DVD drive: # ioscan -fnC disk Make a note of the device file as it applies to your system. 4 Create a directory in which to mount the software disc and mount the disc using the appropriate drive name.
22 Installing and configuring the product Configuring the Components Configuring the Components This sections describes the configuration of SFCFS components. To configure the components 1 Log in as superuser. 2 Run the installer command to install the SFCFS. For example: # cd /cdrom # ./installer 3 From the Installation menu, choose the C option for Configuration and select 6 the Veritas Storage Foundation Cluster File System. You are prompted to enter one or more system names.
Installing and configuring the product Configuring the Components 7 Answer the prompts to configure VCS for SFCFS. You are prompted to configure SFCFS to use Veritas Security Services. Would you like to configure SFCFS to use Veritas Security Services? [y,n,q] (n) 8 Enter y or n to configure SFCFS to use Veritas Security Services. You are prompted to add Storage Foundation Cluster File System users.
24 Installing and configuring the product Using the log files 16 Enter y or n if the VxVM default disk group information is correct. You are prompted to enable centralized management. Enable Centralized Management? [y,n,q] (y) n 17 Enter y or n to enable centralized management. You are prompted to verify the fully qualified domain name for system01. Is the fully qualified host name system01.domain_name? [y,n,q] (y) 18 Enter y or n to verify the fully qualified domain name for system01.
Installing and configuring the product Verifying the configuration files Verifying the configuration files You can inspect the contents of the configuration files that were installed and modified after a successful installation process. These files reflect the configuration based on the information you supplied. To verify the configuration files 1 Log in as superuser to any system in the cluster. 2 Set up your environment PATH variable.
26 Installing and configuring the product Verifying the configuration files Checking Low Latency Transport operation Use the lltstat command to verify that links are active for LLT. This command returns information about the links for LLT for the system on which it is typed. See the lltstat(1M) manual page. In the following example, lltstat -n is typed on each system in the cluster. To check LLT operation 1 Log into system01.
Installing and configuring the product Verifying the configuration files . . . 31 CONNWAIT lan1 lan2 DOWN DOWN Note: The output lists 32 nodes. It reports on the two cluster nodes, system01 and system02, plus non-existent nodes. For each correctly configured system, the information shows a state of OPEN, a status for each link of UP, and an address for each link.
28 Installing and configuring the product Verifying the configuration files Group Membership and Atomic Broadcast configuration files The following files are required by the VCS communication services for Group Membership and Atomic Broadcast (GAB). /etc/gabtab After installation, the file /etc/gabtab contains a gabconfig(1M) command that configures the GAB driver for use.
Installing and configuring the product Verifying the configuration files Checking cluster operation This section describes how to check cluster operation.
30 Installing and configuring the product Verifying the configuration files #System Attribute Value system01 ConfigDiskState CURRENT system01 ConfigFile /etc/VRTSvcs/conf/config system01 ConfigInfoCnt 0 system01 ConfigModDate Tues June 25 23:00:00 2006 system01 CurrentLimits system01 DiskHbStatus system01 DynamicLoad 0 system01 Frozen 0 system01 GUIIPAddr system01 LLTNodeId system01 Limits system01 LoadTimeCounter 1890 system01 LoadTimeThreshold 600 system01 LoadWarnin
Installing and configuring the product Verifying agent configuration #System Attribute system01 UserStr Value Verifying agent configuration This section describes how to verify the agent configuration.
32 Installing and configuring the product Configuring VCS In a VCS cluster, the first system to be brought online reads the configuration file and creates an internal (in-memory) representation of the configuration. Systems brought online after the first system derive their information from systems running in the cluster. You must stop the cluster while you are modifying the files from the command line. Changes made by editing the configuration files take effect when the cluster is restarted.
Installing and configuring the product VCS application failover services ) CVMVxconfigd cvm_vxconfigd ( Critical = 0 CVMVxconfigdArgs = { syslog } ) cvm_clus requires cvm_vxconfigd vxfsckd requires cvm_clus // resource dependency tree // // group cvm // { // CFSfsckd vxfsckd // { // CVMCluster // cvm_clus // { // CVMVxconfigd // cvm_vxconfigd // } // } // } SFCFS HA Only If you configured VCS Cluster Manager (Web Console), a service group, “ClusterService,” was created that includes IP, Process, and Not
34 Installing and configuring the product VCS application failover services
Chapter 2 Upgrading the product If you are running an earlier release of Veritas Storage Foundation Cluster File System, you can upgrade your product using the procedures described in this chapter. Topics covered in this chapter include: ■ Preparing to upgrade the product ■ Upgrade Overview ■ Upgrading from 3.5 to 5.0 ■ Upgrading from 4.1 to 5.
36 Upgrading the product Preparing to upgrade the product Preparing to upgrade the product This section prepares you for the Veritas Storage Foundation Cluster File System upgrade. Planning the upgrade Complete the following tasks in advance of upgrading: ■ Review the Veritas Storage Foundation Cluster File System Release Notes for any late-breaking information on upgrading your system. ■ Be sure that the administrator doing the upgrade has root access and a working knowledge of system administration.
Upgrading the product Preparing to upgrade the product From Upgrade to Tasks Storage Foundation Cluster File System 3.5 Update 3 (formerly known as, SANPoint Foundation Suite 3.5 Update 3) Storage Foundation Cluster File System 5.0 Proceed to “Upgrading from 3.5 to 5.0” on page 39. Storage Foundation Cluster File System 4.1 Storage Foundation Cluster File System 5.0 Proceed to “Upgrading from 4.1 to 5.0” on page 45. Storage Foundation Cluster File System 4.
38 Upgrading the product Upgrade Overview Upgrade Overview There are two ways to upgrade cluster nodes to the latest version of Storage Foundation Cluster File System: phased and full. Phased upgrade A phased upgrade minimizes downtime by upgrading portions of the cluster, one at a time. Although the entire cluster is offline for a shorter period than a full upgrade, this method requires command-line interaction and some manual configuration.
Upgrading the product Upgrading from 3.5 to 5.0 Upgrading from 3.5 to 5.0 SFCFS can be upgraded from 3.5 to 5.0 using phased or full upgrade procedure. Phased upgrade Following procedure assumes a 4 node cluster system01, system02, system03, system04 where system01 and system02 are initially upgraded and rest of the cluster is brought up later. To upgrade from 3.5 to 5.0 1 Log in as superuser. 2 Select one or more nodes to upgrade, say system01 and system02.
40 Upgrading the product Upgrading from 3.5 to 5.0 9 Uninstall VCS 3.5 from system01 and system02. Run the following commands from one of the nodes. See the Veritas Cluster Server Installation Guide. # cd /opt/VRTSvcs/install # ./uninstallvcs Note: Ignore any errors from the uninstallvcs script and proceed with the uninstall of VCS. Also run the following command from rest of the nodes in the cluster on which uninstallation is not performed using .
Upgrading the product Upgrading from 3.5 to 5.0 15 Change the configuration files by running the following commands on one of the upgraded nodes, say system01.
42 Upgrading the product Upgrading from 3.5 to 5.0 18 Configure SFCFS on system01 and system02. See “Using the log files”. Note: VCS configuration files are not changed during this configuration. 19 Upgrade file systems to proper disk layout version as mentioned in “Upgrading the disk layout versions” on page 50. 20 Verify that all the file systems are working properly and data is intact. See cfsmount(1M).
Upgrading the product Upgrading from 3.5 to 5.0 8 Uninstall VCS 3.5 from all the nodes. Run the following commands from one of the nodes. See the Veritas Cluster Server Installation Guide. # cd /opt/VRTSvcs/install # ./uninstallvcs Note: Ignore any errors from the uninstallvcs script and proceed with the uninstall of VCS. 9 Upgrade the operating system from HP-UX 11i Version 1 to HP-UX 11i Version 2 on all the nodes. See the HP-UX Operating System documentation.
44 Upgrading the product Upgrading from 3.5 to 5.0 13 Change the configuration files by running the following commands from one of the nodes.
Upgrading the product Upgrading from 4.1 to 5.0 16 Configure SFCFS on system01 and system02. See “Using the log files”. Note: VCS configuration files are not changed during this configuration. 17 Upgrade file systems to proper disk layout version as mentioned in “Upgrading the disk layout versions” on page 50. 18 Verify that all the file systems are working properly and data is intact. See cfsmount(1M). Upgrading from 4.1 to 5.0 SFCFS can be upgraded from 4.1 to 5.0 using phased or full upgrade procedure.
46 Upgrading the product Upgrading from 4.1 to 5.0 # mv /etc/llthosts /etc/llthosts.bak b Install all the prerequisite patches and reboot the machines. c Move /etc/llthosts to /etc/llthosts.bak on all the nodes to be upgraded. # mv /etc/llthosts.bak /etc/llthosts 8 Offline all SFCFS resources on nodes selected in step 2 by running the following commands on one of the cluster nodes.
Upgrading the product Upgrading from 4.1 to 5.0 12 Change the configuration files by running the following commands on one of the upgraded nodes. For example, system01. # # # # # # # /opt/VRTS/bin/hastart /opt/VRTS/bin/haconf -makerw hagrp -unfreeze cvm -persistent hagrp -unfreeze service_group -persistent /opt/VRTS/bin/hares -delete qlogckd /opt/VRTS/bin/haconf -dump -makero /opt/VRTS/bin/hastop -all -force where service_group is the command that displays the service groups by hagrp -dep cvm.
48 Upgrading the product Upgrading from 4.1 to 5.0 16 Configure SFCFS on system01 and system02. See “Using the log files”. Note: VCS configuration files are not changed during this configuration. 17 Upgrade file systems to proper disk layout version as mentioned in “Upgrading the disk layout versions” on page 50. 18 Verify that all the file systems are working properly and data is intact. See cfsmount(1M). 19 Run step 7, step 9, and step 16 on rest of the nodes to be upgraded.
Upgrading the product Upgrading from 4.1 to 5.0 8 Install SFCFS 5.0 and reboot all the nodes. See “Installing the product” on page 19. Note: Do not configure SFCFS after reboot. 9 Start vxfen on all the nodes. vxfen can be started either in disable or enable mode. For starting vxfen in disabled mode, run the following commands: # cat vxfen_mode=disabled > /etc/vxfenmode # /sbin/init.d/vxfen start See the Veritas Cluster Server Installation Guide for information regarding starting vxfen in enabled mode.
50 Upgrading the product Upgrading the disk layout versions 12 Verify the syntax of the /etc/VRTSvcs/conf/config/main.cf file by running the following commands on system01: # cd /etc/VRTSvcs/conf/config # /opt/VRTS/bin/hacf -verify . 13 Run the following command on all the nodes to start VCS. # /opt/VRTS/bin/hastart 14 Configure SFCFS on all the nodes. See “Using the log files”. Note: VCS configuration files are not changed during this configuration.
Upgrading the product Upgrading the disk layout versions 4 On the node selected in step 1, after the disk layout has been successfully upgraded, unmount the file system. # umount /mnt1 5 This file system can be mounted on all nodes of the cluster using cfsmount.
52 Upgrading the product Upgrading the disk layout versions
Chapter 3 Adding and removing a node This chapter provides information on how to add a node to an existing cluster and removing a node from a cluster.
54 Adding and removing a node Adding a node to a cluster Adding a node to a cluster If you want to add a new node to a multi-node cluster, first prepare the new system hardware. Physically connect the new system to the cluster using private networks and attach to any shared storage. Then install the required OS software. Install all the prerequisite patches mentioned in “Required HP-UX patches” on page 15. See “Cluster platforms” on page 11.
Adding and removing a node Adding a node to a cluster 11 Enter y or n for another license key. You are prompted to press Return to continue. Do you want to enter another license key for system03? [y,n,q,?] (n) 12 Enter 1 or 2 to be installed on all systems. Select the packages to be installed on all systems? [1-2,q,?] (2) 13 Press Return to continue. Press [Return] to continue: Note: Do not reboot the machine now. 14 Create the /etc/llttab file the same as it looks on another node in the cluster.
56 Adding and removing a node Configuring SFCFS and CVM agents on the new node Configuring SFCFS and CVM agents on the new node You must configure the SFCFS and CVM agents, after rebooting the new system. To configure SFCFS and CVM agents on the new node 1 Start the VCS server and vxfen on system03. a Use hastart on system03 for starting the VCS server. b For starting vxfen in the disable mode, run the following commands on system03: # echo vxfen_mode=disabled > /etc/vxfenmode # /sbin/init.
Adding and removing a node Removing a node from a cluster # haconf —dump -makero 9 Put the CVM resources back online, in the following order: # hagrp -online cvm -sys system01 # hagrp -online cvm -sys system02 # hagrp -online cvm -sys system03 10 Check the system status to see whether the new node is online: # hastatus —sum -- SYSTEM STATE -- System A system01 A system02 A system03 State RUNNING RUNNING RUNNING -- GROUP STATE -- Group System B cvm system01 B cvm system02 B cvm system03 Probed Y Y Y
58 Adding and removing a node Removing a node from a cluster 4 Remove system03 from the system list attribute of the CVM and SFCFS service groups: # hagrp -modify service_group SystemList -delete system03 # hagrp -modify cvm SystemListf -delete system03 where service_group is the command that displays the service groups by hagrp -dep cvm. If an error message similar to the following is displayed by either of the above commands: VCS:10456:Configuration must be ReadWrite. (’hagrp -modify ...
Adding and removing a node Removing a node from a cluster 12 From the scripts directory, run the uninstallsfcfs script and remove SFCFS on system03: # ./uninstallsfcfs If you do not want to remove the Veritas Cluster Server software, enter n when prompted to uninstall VCS. See the Veritas Cluster Server Installation Guide, on the software disc.
60 Adding and removing a node Removing a node from a cluster
Chapter 4 Uninstalling the product If you need to uninstall SFCFS software. Use the uninstallsfcfs script. To uninstall SFCFS HA 1 Log in as superuser. Note: Do not use the hastop -force command to stop VCS. 2 Change directory to /opt/VRTS/install: # cd /opt/VRTS/install 3 Run the uninstallsfcfs command to uninstall SFCFS. The uninstallsfcfs script uses ssh to communicate with remote nodes as default: # ./uninstallsfcfs If you want to use rsh you must specify on the command line: # .
62 Uninstalling the product
Appendix A Troubleshooting and recovery Installation issues If you encounter any issues installing SFCFS, refer to the following paragraphs for typical problems and their solutions. Incorrect permissions for root on remote system The permissions are inappropriate. Make sure you have remote root access permission on each system to which you are installing. Checking communication with system01 ...............
64 Storage Foundation Cluster File System problems Resource temporarily unavailable If the installation fails with the following error message on the console: fork() failed: Resource temporarily unavailable The value of nkthread tunable parameter nay not be large enough. The nkthread tunable requires a minimum value of 600 on all systems in the cluster.
65 Storage Foundation Cluster File System problems Unmount failures The umount command can fail if a reference is being held by an NFS server. Unshare the mount point and try the unmount again. Mount failures Mounting a file system can fail for the following reasons: ■ The file system is not using disk layout Version 6 or 7. ■ The mount options do not match the options of already mounted nodes.
66 Storage Foundation Cluster File System problems ■ If this error message displays: mount: slow The node may be in the process of joining the cluster.
67 Storage Foundation Cluster File System problems High availability issues Network partition/jeopardy Network partition (or split brain) is a condition where a network failure can be misinterpreted as a failure of one or more nodes in a cluster. If one system in the cluster incorrectly assumes that another system failed, it may restart applications already running on the other system, thereby corrupting data. CFS tries to prevent this by having redundant heartbeat links.
68 Storage Foundation Cluster File System problems Low memory Under heavy loads, software that manages heartbeat communication links may not be able to allocate kernel memory. If this occurs, a node halts to avoid any chance of network partitioning. Reduce the load on the node if this happens frequently. A similar situation may occur if the values in the /etc/llttab files on all cluster nodes are not correct or identical.