Veritas Storage Foundation 5.0.
Legal Notices © Copyright 2005-2009 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor’s standard commercial license. The information contained herein is subject to change without notice.
Table of Contents Preface.................................................................................................................................7 Publishing History..................................................................................................................................7 1 Technical Overview........................................................................................................9 Overview of Cluster File System Architecture...................................
Time Synchronization for Cluster File Systems..............................................................................26 Growing a Cluster File System........................................................................................................26 The fstab file....................................................................................................................................27 Distributing the Load on a Cluster...................................................................
List of Figures 2-1 4-1 Example of a Four-Node Cluster...................................................................................................19 Example of a 4-Node Cluster........................................................................................................
List of Tables 1 1-1 1-2 2-1 2-2 2-3 3-1 4-1 4-2 4-3 6 Publishing History .........................................................................................................................7 CFS Supported Features................................................................................................................10 CFS Unsupported Features...........................................................................................................11 Primary and Secondary Mount Options.......
Preface The Veritas Storage Foundation™ 5.0.1 Cluster File System Administration Guide Extracts for the HP Serviceguard Storage Management Suite contains information extracted from the Veritas Storage Foundation™ Cluster File System Administration Guide - HP-UX - 5.0.1, which has been modified to support the HP Serviceguard Storage Management Suite bundles that include the Veritas Storage Foundation™ Cluster File System by Symantec and the Veritas Storage Foundation™ Cluster Volume Manager by Symantec.
1 Technical Overview This chapter includes the following topics: • “Overview of Cluster File System Architecture” (page 9) • “VxFS Functionality on Cluster File Systems” (page 10) • “Benefits and Applications” (page 12) HP Serviceguard Storage Management Suite (SG SMS) bundles provide several options for clustering and storage. The information in this document applies to the SG SMS bundles that include the Veritas Storage Foundation™ 5.0.
NOTE: The master/slave node naming convention continues to be used when referring to Veritas Cluster Volume Manager (CVM) nodes. Cluster File System Failover If the server designated as the CFS primary node fails, the remaining nodes in the cluster elect a new primary node. The new primary node reads the intent log of the old primary node and completes any metadata updates that were in process at the time of the failure.
Table 1-1 CFS Supported Features (continued) Features and Commands Supported on CFS Freeze and Thaw Synchronizing operations, which require freezing and thawing file systems, are done on a cluster-wide basis. Snapshots Snapshots are supported with CFS. Quotas Quotas are supported with CFS. NFS Mounts You can mount cluster file systems to NFS. Memory Mapping Shared memory mapping established by the map()function is supported on CFS. See the mmap(2) manual page.
Table 1-2 CFS Unsupported Features (continued) Features and Commands Not Supported on CFS Commands that File access times may appear different across nodes because the atime file attribute is not Depend on File Access closely synchronized in a cluster file system. Utilities that depend on checking access times Times may not function reliably. Nested Mounts HP Serviceguard does not support CFS nested mounts. Benefits and Applications The following sections describe CFS benefits and some applications.
different servers are accessing data in a cluster file system, overall system I/O performance improves due to the load balancing effect of having one cluster file system on a separate underlying volume. This is automatic; no tuning or other administrative action is required. Many applications consist of multiple concurrent threads of execution that could run on different servers if they had a way to coordinate their data accesses. CFS provides this coordination.
2 Cluster File System Architecture This chapter includes the following topics: • “Role of Component Products” (page 15) • “About CFS” (page 16) • “About Veritas Cluster Volume Manager Functionality” (page 19) Role of Component Products The HP Serviceguard Storage Management Suite bundles that include CFS also include the Veritas™ Volume Manager by Symantec (VxVM) and it's cluster component, the Veritas Storage Foundation™ Cluster Volume Manager by Symantec (CVM).
About CFS If the CFS primary node fails, the remaining cluster nodes elect a new primary node. The new primary node reads the file system intent log and completes any metadata updates that were in process at the time of the failure. Application I/O from other nodes may block during this process and cause a delay. When the file system becomes consistent again, application processing resumes.
Parallel I/O Some distributed applications read and write to the same file concurrently from one or more nodes in the cluster; for example, any distributed application where one thread appends to a file and there are one or more threads reading from various regions in the file. Several high-performance computing (HPC) applications can also benefit from this feature, where concurrent I/O is performed on the same file. Applications do not require any changes to use this parallel I/O feature.
There are several characteristics of a cluster snapshot, including: • • • • • • • A snapshot for a cluster mounted file system can be mounted on any node in a cluster. The file system can be a primary, secondary, or secondary-only. A stable image of the file system is provided for writes from any node. Multiple snapshots of a cluster file system can be mounted on the same or different cluster nodes. A snapshot is accessible only on the node it is mounted on.
I/O Error Handling Policy I/O errors can occur for several reasons, including failures of FibreChannel links, host-bus adapters, and disks. CFS disables the file system on the node that is encountering I/O errors, but the file system remains available from other nodes. After the I/O error is fixed, the file system can be forcibly unmounted and the mount resource can be brought online from the disabled node to reinstate the file system.
To the cmvx daemon, all nodes are the same. VxVM objects configured within shared disk groups can potentially be accessed by all nodes that join the cluster. However, the cluster functionality of VxVM requires one node to act as the master node; all other nodes in the cluster are slave nodes. Any node is capable of being the master node, which is responsible for coordinating certain VxVM activities. NOTE: You must run commands that configure or reconfigure VxVM objects on the master node.
NOTE: Applications running on each node can access the data on the VM disks simultaneously. VxVM does not protect against simultaneous writes to shared volumes by more than one node. It is assumed that applications control consistency (by using Veritas Storage Foundation Cluster File System or a distributed lock manager, for example).
The activation-mode is one of exclusivewrite, readonly, sharedread, sharedwrite, or off. When a shared disk group is created or imported, it is activated in the specified mode. When a node joins the cluster, all shared disk groups accessible from the node are activated in the specified mode. NOTE: The activation mode of a disk group controls volume I/O from different nodes in the cluster.
3 Cluster File System Administration The following HP Serviceguard Storage Management Suite bundles include the Veritas Storage Foundation™ 5.0.1 Cluster File System (CFS) and Cluster Volume Manager (CVM) by Symantec: • Bundles T2775DB, T2777DB of the HP Serviceguard Storage Management Suite version A.03.00 for HP-UX 11i v3 • High Availability Operating Environment (HAOE) bundles T8680DB, T8681DB, and T8682DB, of the HP Serviceguard Storage Management Suite version A.03.
Cluster Communication - LLT LLT provides kernel-to-kernel communications and monitors network communications. The LLT files /etc/llttab and /etc/llthosts can be configured to set system IDs within a cluster, set cluster IDs for multiple clusters, and tune network parameters such as heartbeat frequency. LLT is implemented so events such as cluster membership changes are reflected quickly, which in turn enables fast responses.
cluster environment. A local mount cannot be remounted in shared mode and a shared mount cannot be remounted in local mode. File systems in a cluster can be mounted with different read-write options. These are called asymmetric mounts. Cluster File System Primary and Cluster File System Secondary Both primary and secondary nodes handle metadata intent logging for a cluster file system. The first node of a cluster file system to mount is called the primary node - the other nodes are called secondary nodes.
IMPORTANT: Once disk group and mount point multi-node packages are created with HP Serviceguard, it is critical to use the CFS commands, including cfsdgadm, cfsmntadm, cfsmount, and cfsumount. If the HP-UX mount and umount commands are used, serious problems such as writing to the local file system, instead of the cluster file system, could occur.
# fsclustadm –v showprimary mount_point To determine if the CFS primary is also the CVM master node, enter: # vxdctl -c mode To increase the size of the file system, run the following commands: On the CVM master node, enter: # vxassist -g shared_disk_group growto volume_name newlength On any CFS node, enter: # fsadm –F vxfs –b newsize –r device_name mount_point The fstab file In the /etc/fstab file, do not specify any cluster file systems to mount-at-boot, because mounts initiated from fstab occur before c
Performance Considerations Mounting a snapshot file system for backup increases the load on the system because of the resources used to perform copy-on-writes and to read data blocks from the snapshot. In this situation, cluster snapshots can be used to do off-host backups. Off-host backups reduce the load of a backup application on the primary server. Overhead from remote snapshots is small when compared to overall snapshot overhead.
4 Cluster Volume Manager Administration A cluster consists of a number of hosts or nodes that share a set of disks. The main benefits of cluster configurations are: • Availability—If one node fails, the other nodes can still access the shared disks. When configured with suitable software, mission-critical applications can continue running by transferring their execution to a standby node in the cluster.
IMPORTANT: The cluster functionality of VxVM is supported only when used in conjunction with the cmvx daemon. Figure 4-1: “Example of a 4-Node Cluster”, illustrates a simple cluster arrangement consisting of four nodes with similar or identical hardware characteristics (CPUs, RAM and host adapters), and configured with identical software (including the operating system).
Private and Shared Disk Groups Two types of disk groups are defined: • • Private disk groups (belong to only one node). A private disk group is only imported by one system. Disks in a private disk group may be physically accessible from one or more systems, but access is restricted to one system only. The boot disk group (usually aliased by the reserved disk group name bootdg) is always a private disk group. Shared disk groups (shared by all nodes).
NOTE: The default activation mode for shared disk groups is off. Applications such as high availability and off-host backup can use disk group activation to explicitly control volume access from different nodes in the cluster. The activation mode of a disk group controls volume I/O from different nodes in the cluster. It is not possible to activate a disk group on a cluster node, if it is activated in a conflicting mode on another node in the cluster.
If the defaults file is edited while the vxconfigd daemon is already running, the vxconfigd process must be restarted for the changes in the defaults file to take effect. If the default activation mode is anything other than off, an activation following a cluster join, or an activation following a disk group creation (or import) will fail, if another node in the cluster has activated the disk group in a conflicting mode.
Limitations of Shared Disk Groups NOTE: The boot disk group (usually aliased as bootdg) cannot be made cluster-shareable. It must be private. Only raw device access can be performed via the cluster functionality of VxVM. It does not support shared access to file systems in shared volumes unless the appropriate software, such as the HP Serviceguard Storage Management Suite, is installed and configured.
A Troubleshooting This appendix contains the following topics: • “Installation Issues” (page 35) • “Cluster File System Problems” (page 36) Installation Issues If you encounter any issues installing CFS, refer to the following paragraphs for typical problems and their solutions. You can also refer to the HP Serviceguard Storage Management Suite Release Notes and the HP Serviceguard Storage Management Suite Read Before Installing document, if you encounter an issue that is not included here.
Suggested solutions: check that swagentd is running. Check whether there is an entry for the target system in /etc/hosts. If there is no entry, then ensure the hosts file is not the primary lookup for the “hosts” entry in the /etc/nsswitch.conf file. Cluster File System Problems If there is a device failure or controller failure to a device, the file system may become disabled cluster-wide. To address this problem, unmount the file system on all of the nodes, then run a full fsck.
or cluster reservation failed for the volume Command Failures • • • Manual pages not accessible with the man command. Set the MANPATH environment variable to include the path to the Veritas manual pages. The mount, fsck, and mkfs utilities reserve a shared volume. They fail on volumes that are in use. Be careful when accessing shared volumes with other utilities such as dd, it is possible for these commands to destroy data on the disk.
