HP StoreAll 9300/9320 Storage Administrator Guide Abstract This guide describes tasks related to cluster configuration and monitoring, system upgrade and recovery, hardware component replacement, and troubleshooting for the HP 9300 Storage Gateway and the HP 9320 Storage. It does not document StoreAll file system features or standard Linux administrative tools and commands.
© Copyright 2010, 2013 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license. The information contained herein is subject to change without notice.
Contents 1 Upgrading the StoreAll software to the 6.3 release.........................................9 Online upgrades for StoreAll software.......................................................................................11 Preparing for the upgrade...................................................................................................11 Performing the upgrade......................................................................................................12 After the upgrade..........
Configuring Insight Remote Support for HP SIM 6.3 and IRS 5.6..............................................41 Testing the Insight Remote Support configuration....................................................................44 Updating the Phone Home configuration...............................................................................44 Disabling Phone Home.......................................................................................................
NDMP process management...............................................................................................78 Viewing or canceling NDMP sessions..............................................................................78 Starting, stopping, or restarting an NDMP Server..............................................................78 Viewing or rescanning tape and media changer devices.........................................................79 NDMP events........................................
Powering file serving nodes on or off.......................................................................................108 Performing a rolling reboot....................................................................................................108 Starting and stopping processes.............................................................................................109 Tuning file serving nodes and StoreAll clients............................................................................
The ibrix_auth command fails after a restore............................................................................150 16 Support and other resources...................................................................151 Contacting HP......................................................................................................................151 Related information...............................................................................................................
B Component diagrams for 9300 systems.....................................................180 Front view of file serving node................................................................................................180 Rear view of file serving node................................................................................................180 C System component and cabling diagrams for 9320 systems.........................183 System component diagrams..............................................
1 Upgrading the StoreAll software to the 6.3 release This chapter describes how to upgrade to the 6.3 StoreAll software release. You can also use this procedure for any subsequent 6.3.x patches. IMPORTANT: Print the following table and check off each step as you complete it. NOTE: (Upgrades from version 6.0.x) CIFS share permissions are granted on a global basis in v6.0.X. When upgrading from v6.0.X, confirm that the correct share permissions are in place.
Table 1 Prerequisites checklist for all upgrades (continued) Step Step completed? Description NOTE: Save a copy of the /etc/grub.conf file before you modify it. The following example shows the crash kernel set to 256M: kernel /vmlinuz-2.6.18-194.el5 ro root=/dev/vg1/lv1 crashkernel=256M@16M 5. Reboot the active FM. 6. Use SSH to access each passive FM and do the following: a. Modify the /etc/grub.conf file as described in the previous steps. b. Reboot the node. 7.
Table 1 Prerequisites checklist for all upgrades (continued) Step completed? Step Description 10 Record all host tunings, FS tunings and FS mounting options by using the following commands: 1. To display file system tunings, enter: ibrix_fs_tune -l >/local/ibrix_fs_tune-l.txt 2. To display default StoreAll tunings and settings, enter: ibrix_host_tune -L >/local/ibrix_host_tune-L.txt 3. To display all non-default configuration tunings and settings, enter: ibrix_host_tune -q >/local/ibrix_host_tune-q.
1. 2. 3. Make sure you have completed all steps in the upgrade checklist (Table 1 (page 9)). If Failed was displayed for the HA Configuration or Auto Failover columns or both, perform the steps described in the section “Configuring High Availability on the cluster” in the administrator guide for your current release.
files must be upgraded for retention features. To upgrade a file system, use the ibrix_reten_adm -u -f FSNAME command. Additional steps are required before and after you run the ibrix_reten_adm -u -f FSNAME command. For more information, see “Upgrading pre-6.1.1 file systems for data retention features” (page 161). 5. If you have an Express Query enabled file system prior to version 6.3, manually complete each file system upgrade as described in “Required steps after the StoreAll Upgrade for pre-6.
1. 2. 3. 4. 5. If your cluster nodes contain any 10Gb NICs, reboot these nodes to load the new driver. You must do this step before you upgrade the server firmware, as requested later in this procedure. Upgrade your firmware as described in “Upgrading firmware” (page 128). Mount file systems on Linux StoreAll clients. If you have a file system version prior to version 6, you might have to make changes for snapshots and data retention, as mentioned in the following list: • Snapshots.
10. Run the following command to verify that automated failover is off. In the output, the HA column should display off. /bin/ibrix_server -l 11. Unmount file systems on Linux StoreAll clients: ibrix_umount -f MOUNTPOINT 12. Stop the SMB, NFS and NDMP services on all nodes.
5. Change to the /local/ibrix/ directory, and then run the upgrade script: cd /local/ibrix/ ./ibrixupgrade —f The upgrade script automatically stops the necessary services and restarts them when the upgrade is complete. The upgrade script installs the Fusion Manager on the server. 6. 7. 8. After completing the previous steps for the server running the active Fusion Manager, repeat the steps for each of the servers running the passive Fusion Manager. Upgrade Linux StoreAll clients.
upgrade a file system, use the upgrade60.sh utility. For more information, see “Upgrading pre-6.0 file systems for software snapshots” (page 159). • Data retention. Files used for data retention (including WORM and auto-commit) must be created on StoreAll software 6.1.1 or later, or the pre-6.1.1 file system containing the files must be upgraded for retention features. To upgrade a file system, use the ibrix_reten_adm -u -f FSNAME command.
3. 4. 5. 6. 7. Launch the Windows Installer and follow the instructions to complete the upgrade. Register the Windows StoreAll client again with the cluster and check the option to Start Service after Registration. Check Administrative Tools | Services to verify that the StoreAll client service is started. Launch the Windows StoreAll client. On the Active Directory Settings tab, click Update to retrieve the current Active Directory settings. Mount file systems using the StoreAll Windows client GUI.
In this instance 6. ◦ is the share name. ◦ is the profile name. ◦ is the virtual host name Disable Express Query by entering the following command: ibrix_fs -T -D -f 7. Shut down Archiving daemons for Express Query by entering the following command: ibrix_archiving -S -F 8. delete the internal database files for this file system by entering the following command: rm -rf /.
5. Wait for the resynchronizer to complete by entering the following command until its output is : OK: ibrix_archiving -l 6. Restore your audit log data by entering the following command: MDImport -f -n /tmp/auditData.csv -t audit In this instance is the file system. 7. Restore your custom metadata by entering the following command: MDImport -f -n /tmp/custAttributes.csv -t custom In this instance is the file system.
/usr/local/ibrix/autocfg/bin/ibrixapp upgrade -f -s Offline upgrade fails because iLO firmware is out of date If the iLO2 firmware is out of date on a node, the auto_ibrixupgrade script will fail. The /usr/ local/ibrix/setup/logs/auto_ibrixupgrade.log reports the failure and describes how to update the firmware.
File system unmount issues If a file system does not unmount successfully, perform the following steps on all servers: 1. Run the following commands: chkconfig ibrix_server off chkconfig ibrix_ndmp off chkconfig ibrix_fusionmanager off 2. 3. Reboot all servers. Run the following commands to move the services back to the on state. The commands do not start the services. chkconfig ibrix_server on chkconfig ibrix_ndmp on chkconfig ibrix_fusionmanager on 4.
5. Re-enable Express Query on the file systems: ibrix_fs -T -E -f In this instance is the file system. Express Query will begin resynchronizing (repopulating) a new database for this file system. 6. Re-enable auditing if you had it running before (the default). ibrix_fs -A -f -oa audit_mode=on In this instance is the file system. 7. Restore your audit log data: MDImport -f -n /tmp/auditData.csv -t audit In this instance is the file system. 8.
2 Product description This guide provides information about configuring, monitoring, and maintaining HP StoreAll 9300 Storage Gateways and 9320 Storage. IMPORTANT: It is important to keep regular backups of the cluster configuration. 9300 Storage Gateway The 9300 Storage Gateway is a flexible, scale-out solution that brings gateway file services to HP MSA, EVA, P4000, or 3rd-party arrays or SANs.
StoreAll software is designed to operate with high-performance computing applications that require high I/O bandwidth, high IOPS throughput, and scalable configurations. Some of the key features and benefits are as follows: • Scalable configuration. You can add servers to scale performance and add storage devices to scale capacity. • Single namespace. All directories and files are contained in the same namespace. • Multiple environments. Operates in both the SAN and DAS environments.
3 Getting started IMPORTANT: Follow these guidelines when using your system: • Do not modify any parameters of the operating system or kernel, or update any part of the 9320 Storage unless instructed to do so by HP; otherwise, the system could fail to operate properly. • File serving nodes are tuned for file serving operations. With the exception of supported backup programs, do not run other applications directly on the nodes.
File systems. Set up the following features as needed: • NFS, SMB (Server Message Block), FTP, or HTTP. Configure the methods you will use to access file system data. • Quotas. Configure user, group, and directory tree quotas as needed. • Remote replication. Use this feature to replicate changes in a source file system on one cluster to a target file system on either the same cluster or a second cluster. • Data retention and validation. Use this feature to manage WORM and retained files.
• Cluster configuration (ibrix_clusterconfig) • Configuration database consistency (ibrix_dbck) • Shell task management (ibrix_shell) The following operations can be performed only from the StoreAll Management Console: • Scheduling recurring data validation scans • Scheduling recurring software snapshots • Scheduling recurring block snapshots Using the StoreAll Management Console The StoreAll Management Console is a browser-based interface to the Fusion Manager.
System Status The System Status section lists the number of cluster events that have occurred in the last 24 hours. There are three types of events: Alerts. Disruptive events that can result in loss of access to file system data. Examples are a segment that is unavailable or a server that cannot be accessed. Warnings. Potentially disruptive conditions where file system access is not lost, but if the situation is not addressed, it can escalate to an alert condition.
Statistics Historical performance graphs for the following items: • Network I/O (MB/s) • Disk I/O (MB/s) • CPU usage (%) • Memory usage (%) On each graph, the X-axis represents time and the Y-axis represents performance. Use the Statistics menu to select the servers to monitor (up to two), to change the maximum value for the Y-axis, and to show or hide resource usage distribution for CPU and memory. Recent Events The most recent cluster events.
example, you can sort the contents of the Mountpoint column in ascending or descending order, and you can select the columns that you want to appear in the display. Adding user accounts for Management Console access StoreAll software supports administrative and user roles. When users log in under the administrative role, they can configure the cluster and initiate operations such as remote replication or snapshots.
StoreAll client interfaces StoreAll clients can access the Fusion Manager as follows: • Linux clients. Use Linux client commands for tasks such as mounting or unmounting file systems and displaying statistics. See the HP StoreAll Storage CLI Reference Guide for details about these commands. • Windows clients. Use the Windows client GUI for tasks such as mounting or unmounting file systems and registering Windows clients.
Configuring ports for a firewall IMPORTANT: To avoid unintended consequences, HP recommends that you configure the firewall during scheduled maintenance times. When configuring a firewall, you should be aware of the following: • SELinux should be disabled. • By default, NFS uses random port numbers for operations such as mounting and locking. These ports must be fixed so that they can be listed as exceptions in a firewall configuration file. For example, you will need to lock specific ports for rpc.
Port Description 20/tcp, 20/udp Between file serving nodes and FTP clients (user network) 21/tcp, 21/udp 7777/tcp Between GUI and clients that need to access the GUI 8080/tcp 5555/tcp, 5555/udp Dataprotector 631/tcp, 631/udp Internet Printing Protocol (IPP) 1344/tcp, 1344/udp ICAP Configuring NTP servers When the cluster is initially set up, primary and secondary NTP servers are configured to provide time synchronization with an external time source.
Prerequisites The required components for supporting StoreAll systems are preinstalled on the file serving nodes. You must install HP Insight Remote Support on a separate Windows system termed the Central Management Server (CMS): • HP Insight Manager (HP SIM). This software manages HP systems and is the easiest and least expensive way to maximize system uptime and health. • Insight Remote Support Advanced (IRSA). This version is integrated with HP Systems Insight Manager (SIM).
IMPORTANT: You must compile and manually register the StoreAll MIB file by using HP Systems Insight Manager: 1. Download ibrixMib.txt from /usr/local/ibrix/doc/. 2. Rename the file to ibrixMib.mib. 3. In HP Systems Insight Manager, complete the following steps: a. Unregister the existing MIB by entering the following command: \mibs>mxmib -d ibrixMib.mib b. Copy the ibrixMib.mib file to the \mibs directory, and then enter the following commands: \mibs>mcompile ibrixMib.
Click Enable to configure the settings on the Phone Home Settings dialog box. Skip the Software Entitlement ID field; it is not currently used. The time required to enable Phone Home depends on the number of devices in the cluster, with larger clusters requiring more time.
ibrix_phonehome -c -i -P Country Name [-z Software Entitlement ID] [-r Read Community] [-w Write Community] [-t System Contact] [-n System Name] [-o System Location] For example: ibrix_phonehome -c -i 99.2.4.75 -P US -r public -w private -t Admin -n SYS01.US -o Colorado Next, configure Insight Remote Support for the version of HP SIM you are using: • HP SIM 7.1 and IRS 5.7. See “Configuring Insight Remote Support for HP SIM 7.1 and IRS 5.7” (page 38).
Use the following commands to entitle devices from the CLI. The commands must be run for each device present in the cluster. Entitle a server: ibrix_phonehome -e -h -b -g Enter the Host Name parameter exactly as it is listed by the ibrix_fm -l command.
Enter the read community string on the Credentials > SNMP tab. This string should match the Phone Home read community string. If the strings are not identical, the Fusion Manager IP might be discovered as “Unknown.
Devices are discovered as described in the following table.
Enter the read community string on the Credentials > SNMP tab. This string should match the Phone Home read community string. If the strings are not identical, the device will be discovered as “Unknown.” The following example shows discovered devices on HP SIM 6.3. File serving nodes are discovered as ProLiant server. Configuring device Entitlements Configure the CMS software to enable remote support for StoreAll systems.
Enter the following custom field settings in HP SIM: • Custom field settings for 9300/9320 Servers are discovered with their IP addresses. When a server is discovered, edit the system properties on the HP Systems Insight Manager.
NOTE: For storage support on 9300 systems, do not set the Custom Delivery ID. (The MSA is an exception; the Custom Delivery ID is set as previously described.) Verifying device entitlements To verify the entitlement information in HP SIM, complete the following steps: 1. Go to Remote Support Configuration and Services and select the Entitlement tab. 2. Check the devices discovered. NOTE: If the system discovered on HP SIM does not appear on the Entitlement tab, click Synchronize RSE. 3. 4. 5.
Configuration in the upper Navigator, select Phone Home in the lower Navigator, and click Rescan on the Phone Home Setup panel. On the CLI, run the following command: ibrix_phonehome -s Disabling Phone Home When Phone Home is disabled, all Phone Home information is removed from the cluster and hardware and software are no longer monitored. To disable Phone Home on the GUI, click Disable on the Phone Home Setup panel.
2. Install the credentials for a single MSA by running the following four commands once for each MSA that the hpsp_couplet_info command lists in step 1. Replace with one of the UUIDs as provided by the output of the hpsp_couplet_info command.
4 Configuring virtual interfaces for client access StoreAll software uses a cluster network interface to carry Fusion Manager traffic and traffic between file serving nodes. This network is configured as bond0 when the cluster is installed. To provide failover support for the Fusion Manager, a virtual interface is created for the cluster network interface.
3. To assign the IFNAME a default route for the parent cluster bond and the user VIFS assigned to FSNs for use with SMB/NFS, enter the following ibrix_nic command at the command prompt: # ibrix_nic -r -n IFNAME -h HOSTNAME-A -R 4. Configure backup monitoring, as described in “Configuring backup servers” (page 48). Creating a bonded VIF NOTE: The examples in this chapter use the unified network and create a bonded VIF on bond0.
For example: # # # # ibric_nic ibric_nic ibric_nic ibric_nic -m -m -m -m -h -h -h -h node1 node2 node3 node4 -A -A -A -A node2/bond0:1 node1/bond0:1 node4/bond0:1 node3/bond0:1 Configuring automated failover To enable automated failover for your file serving nodes, execute the following command: ibrix_server -m [-h SERVERNAME] Example configuration This example uses two nodes, ib50-81 and ib50-82. These nodes are backups for each other, forming a backup pair.
NOTE: Because the backup NIC cannot be used as a preferred network interface for StoreAll clients, add one or more user network interfaces to ensure that HA and client communication work together. Configuring VLAN tagging VLAN capabilities provide hardware support for running multiple logical networks over the same physical networking hardware. To allow multiple packets for different VLANs to traverse the same physical interface, each packet must have a field added that contains the VLAN tag.
To determine whether link state monitoring is enabled on an iSCSI interface, run the following command: ibrix_nic -l Next, check the LINKMON column in the output. The value yes means that link state monitoring is enabled; no means that it is not enabled.
5 Configuring failover This chapter describes how to configure failover for agile management consoles, file serving nodes, network interfaces, and HBAs. Agile management consoles The agile Fusion Manager maintains the cluster configuration and provides graphical and command-line user interfaces for managing and monitoring the cluster. The agile Fusion Manager is installed on all file serving nodes when the cluster is installed. The Fusion Manager is active on one node, and is passive on the other nodes.
console. This Fusion Manager rebuilds the cluster virtual interface, starts Fusion Manager services locally, transitions into active mode, and take over Fusion Manager operation. Failover of the active Fusion Manager affects the following features: • User networks. The virtual interface used by clients will also fail over. Users may notice a brief reconnect while the newly active Fusion Manager takes over management of the virtual interface. • GUI.
What happens during a failover The following actions occur when a server is failed over to its backup: 1. The Fusion Manager verifies that the backup server is powered on and accessible. 2. The Fusion Manager migrates ownership of the server’s segments to the backup and notifies all servers and StoreAll clients about the migration. This is a persistent change. If the server is hosting the active FM, it transitions to another server. 3.
Use the NIC HA Setup dialog box to configure NICs that will be used for data services such as SMB or NFS. You can also designate NIC HA pairs on the server and its backup and enable monitoring of these NICs. For example, you can create a user VIF that clients will use to access an SMB share serviced by server ib69s1. The user VIF is based on an active physical network on that server. To do this, click Add NIC in the section of the dialog box for ib69s1.
On the Add NIC dialog box, enter a NIC name. In our example, the cluster uses the unified network and has only bond0, the active cluster FM/IP. We cannot use bond0:0, which is the management IP/VIF. We will create the VIF bond0:1, using bond0 as the base. When you click OK, the user VIF is created. The new, active user NIC appears on the NIC HA setup dialog box.
Next, enable NIC monitoring on the VIF. Select the new user NIC and click NIC HA. On the NIC HA Config dialog box, check Enable NIC Monitoring.
In the Standby NIC field, select New Standby NIC to create the standby on backup server ib69s2. The standby you specify must be available and valid. To keep the organization simple, we specified bond0:1 as the Name; this matches the name assigned to the NIC on server ib69s1. When you click OK, the NIC HA configuration is complete.
You can create additional user VIFs and assign standby NICs as needed. For example, you might want to add a user VIF for another share on server ib69s2 and assign a standby NIC on server ib69s1. You can also specify a physical interface such eth4 and create a standby NIC on the backup server for it. The NICs panel on the GUI shows the NICs on the selected server.
The NICs panel for the ib69s2, the backup server, shows that bond0:1 is an inactive, standby NIC and bond0:2 is an active NIC. Changing the HA configuration To change the configuration of a NIC, select the server on the Servers panel, and then select NICs from the lower Navigator. Click Modify on the NICs panel. The General tab on the Modify NIC Properties dialog box allows you change the IP address and other NIC properties.
Configuring automated failover manually To configure automated failover manually, complete these steps: 1. Configure file serving nodes in backup pairs. 2. Identify power sources for the servers in the backup pair. 3. Configure NIC monitoring. 4. Enable automated failover. 1. Configure server backup pairs File serving nodes are configured in backup pairs, where each server in a pair is the backup for the other. This step is typically done when the cluster is installed.
IMPORTANT: When configuring NIC monitoring, use the same backup pairs that you used when configuring backup servers. Identify the servers in a backup pair as NIC monitors for each other. Because the monitoring must be declared in both directions, enter a separate command for each server in the pair.
Turn off automated failover: ibrix_server -m -U [-h SERVERNAME] To specify a single file serving node, include the -h SERVERNAME option. Failing a server over manually The server to be failed over must belong to a backup pair. The server can be powered down or remain up during the procedure. You can perform a manual failover at any time, regardless of whether automated failover is in effect. Manual failover does not require the use of a programmable power supply.
does not play a role in vendor- or software-mediated HBA failover; traffic moves to the remaining functional port with no Fusion Manager involvement. HBAs use worldwide names for some parameter values. These are either worldwide node names (WWNN) or worldwide port names (WWPN). The WWPN is the name an HBA presents when logging in to a SAN fabric. Worldwide names consist of 16 hexadecimal digits grouped in pairs. In StoreAll software, these are written as dot-separated pairs (for example, 21.00.00.e0.8b.05.
For example, to turn on HBA monitoring for port 20.00.12.34.56.78.9a.bc on node s1.hp.com: ibrix_hba -m -h s1.hp.com -p 20.00.12.34.56.78.9a.bc To turn off HBA monitoring for an HBA port, include the -U option: ibrix_hba -m -U -h HOSTNAME -p PORT Deleting standby port pairings Deleting port pairing information from the configuration database does not remove the standby pairing of the ports. The standby pairing is either built in by the HBA vendor or implemented by software.
For each High Availability feature, the summary report returns status for each tested file serving node and optionally for their standbys: • Passed. The feature has been configured. • Warning. The feature has not been configured, but the significance of the finding is not clear. For example, the absence of discovered HBAs can indicate either that the HBA monitoring feature was not configured or that HBAs are not physically present on the tested servers. • Failed. The feature has not been configured.
Hba port 21.01.00.e0.8b.2a.0d.6d monitored Hba port 21.00.00.e0.8b.0a.0d.6d monitored FAILED FAILED Not monitored Not monitored Capturing a core dump from a failed node The crash capture feature collects a core dump from a failed node when the Fusion Manager initiates failover of the node. You can use the core dump to analyze the root cause of the node failure. When enabled, crash capture is supported for both automated and manual failover. Failback is not affected by this feature.
3. 4. 5. 6. Highlight the BIOS Serial Console & EMS option in main menu, and then press the Enter key. Highlight the BIOS Serial Console Port option and then press the Enter key. Select the COM1 port, and then press the Enter key. Highlight the BIOS Serial Console Baud Rate option, and then press the Enter key. Select the 115200 Serial Baud Rate. Highlight the Server Availability option in main menu, and then press the Enter key. Highlight the ASR Timeout option and then press the Enter key.
6 Configuring cluster event notification Cluster events There are three categories for cluster events: Alerts. Disruptive events that can result in loss of access to file system data. Warnings. Potentially disruptive conditions where file system access is not lost, but if the situation is not addressed, it can escalate to an alert condition. Information. Normal events that change the cluster. The following table lists examples of events included in each category.
Associating events and email addresses You can associate any combination of cluster events with email addresses: all Alert, Warning, or Info events, all events of one type plus a subset of another type, or a subset of all types. The notification threshold for Alert events is 90% of capacity. Threshold-triggered notifications are sent when a monitored system resource exceeds the threshold and are reset when the resource utilization dips 10% below the threshold.
Viewing email notification settings The ibrix_event -L command provides comprehensive information about email settings and configured notifications. ibrix_event -L Email Notification SMTP Server From Reply To : : : : Enabled mail.hp.com FM@hp.com MIS@hp.com EVENT ------------------------------------asyncrep.completed asyncrep.failed LEVEL ----ALERT ALERT TYPE ----EMAIL EMAIL DESTINATION ----------admin@hp.com admin@hp.
Some SNMP parameters and the SNMP default port are the same, regardless of SNMP version. The default agent port is 161. SYSCONTACT, SYSNAME, and SYSLOCATION are optional MIB-II agent parameters that have no default values. NOTE: The default SNMP agent port was changed from 5061 to 161 in the StoreAll 6.1 release. This port number cannot be changed. The -c and -s options are also common to all SNMP versions. The -c option turns the encryption of community names and passwords on or off.
The following command creates a v3 trapsink with a named user and specifies the passwords to be applied to the default algorithms. If specified, passwords must contain at least eight characters. ibrix_snmptrap -c -h lab13-114 -v 3 -n trapsender -k auth-passwd -z priv-passwd Associating events and trapsinks Associating events with trapsinks is similar to associating events with email recipients, except that you specify the host name or IP address of the trapsink instead of an email address.
The command to create a group assigns its SNMPv3 security level, read and write views, and context name. A context is a collection of managed objects that can be accessed by an SNMP entity. A related option, -m, determines how the context is matched.
In the Email Configuration section, set the options: • Notification Level. Select the minimum severity for which the system should send notifications: Critical (only); Error (and Critical); Warning (and Error and Critical); Informational (all). The default is none, which disables email notification. • SMTP Server address. The IP address of the SMTP mail server to use for the email messages.
7 Configuring system backups Backing up the Fusion Manager configuration The Fusion Manager configuration is automatically backed up whenever the cluster configuration changes. The backup occurs on the node hosting the active Fusion Manager. The backup file is stored at /tmp/fmbackup.zip on that node. The active Fusion Manager notifies the passive Fusion Manager when a new backup file is available. The passive Fusion Manager then copies the file to /tmp/fmbackup.
hard quota limit for the directory tree has been exceeded, NDMP cannot create a temporary file and the restore operation fails. Configuring NDMP parameters on the cluster Certain NDMP parameters must be configured to enable communications between the DMA and the NDMP Servers in the cluster. To configure the parameters on the GUI, select Cluster Configuration from the Navigator, and then select NDMP Backup. The NDMP Configuration Summary shows the default values for the parameters.
To configure NDMP parameters from the CLI, use the following command: ibrix_ndmpconfig -c [-d IP1,IP2,IP3,...] [-m MINPORT] [-x MAXPORT] [-n LISTENPORT] [-u USERNAME] [-p PASSWORD] [-e {0=disable,1=enable}] -v [{0=10}] [-w BYTES] [-z NUMSESSIONS] NDMP process management All NDMP actions are usually controlled from the DMA.
Viewing or rescanning tape and media changer devices To view the tape and media changer devices currently configured for backups, select Cluster Configuration from the Navigator, and then select NDMP Backup > Tape Devices. If you add a tape or media changer device to the SAN, click Rescan Device to update the list. If you remove a device and want to delete it from the list, reboot all of the servers to which the device is attached.
8 Creating host groups for StoreAll clients A host group is a named set of StoreAll clients. Host groups provide a convenient way to centrally manage clients. You can put different sets of clients into host groups and then perform the following operations on all members of the group: • Create and delete mount points • Mount file systems • Prefer a network interface • Tune host parameters • Set allocation policies Host groups are optional.
To create one level of host groups beneath the root, simply create the new host groups. You do not need to declare that the root node is the parent. To create lower levels of host groups, declare a parent element for host groups. Do not use a host name as a group name. To create a host group tree using the CLI: 1. Create the first level of the tree: ibrix_hostgroup -c -g GROUPNAME 2.
For example, to add the domain rule 192.168 to the finance group: ibrix_hostgroup -a -g finance -D 192.168 Viewing host groups To view all host groups or a specific host group, use the following command: ibrix_hostgroup -l [-g GROUP] Deleting host groups When you delete a host group, its members are reassigned to the parent of the deleted group.
9 Monitoring cluster operations This chapter describes how to monitor the operational state of the cluster and how to monitor cluster health. Monitoring 9300/9320 hardware The GUI displays status, firmware versions, and device information for the servers, virtual chassis, and system storage included in 9300 and 9320 systems. Monitoring servers To view information about the server and chassis included in your system. 1. Select Servers from the Navigator tree.
Select the server component that you want to view from the lower Navigator panel, such as NICs.
The following are the top-level options provided for the server: NOTE: Information about the Hardware node can be found in “Monitoring hardware components” (page 87). • HBAs.
• • • • ◦ Monitoring ◦ State NICs. The NICs panel shows all NICs on the server, including offline NICs. The NICs panel displays the following information: ◦ Name ◦ IP ◦ Type ◦ State ◦ Route ◦ Standby Server ◦ Standby Interface Mountpoints. The Mountpoints panel displays the following information: ◦ Mountpoint ◦ Filesystem ◦ Access NFS. The NFS panel displays the following information: ◦ Host ◦ Path ◦ Options CIFS.
• • Events. The Events panel displays the following information: ◦ Level ◦ Time ◦ Event Hardware. The Hardware panel displays the following information: ◦ The name of the hardware component. ◦ The information gathered in regards to that hardware component. See “Monitoring hardware components” (page 87) for detailed information about the Hardware panel. Monitoring hardware components The Management Console provides information about the server hardware and its components.
• Message1 • Diagnostic Message1 1 Column dynamically appears depending on the situation. Obtain detailed information for hardware components in the server by clicking the nodes under the Server node.
Table 2 Obtaining detailed information about a server Panel name Information provided CPU • Status • Type • Name • UUID • Model • Location ILO Module • Status • Type • Name • UUID • Serial Number • Model • Firmware Version • Properties Memory DiMM • Status • Type • Name • UUID • Location • Properties NIC • Status • Type • Name • UUID • Properties Power Management Controller • Status • Type • Name • UUID • Firmware Version Storage Cluster • Status • Type • Name • UUID Monitoring 9300/9320 hardw
Table 2 Obtaining detailed information about a server (continued) Panel name Information provided Drive: Displays information about each drive in a storage cluster. • Status • Type • Name • UUID • Serial Number • Model • Firmware Version • Location • Properties Storage Controller (Displayed for a server) • Status • Type • Name • UUID • Serial Number • Model • Firmware Version • Location • Message • Diagnostic message Volume: Displays volume information for each server.
Table 2 Obtaining detailed information about a server (continued) Panel name Information provided Temperature Sensor: Displays information for each temperature sensor. • Status • Type • Name • UUID • Locations • Properties Monitoring storage and storage components Select Vendor Storage from the Navigator tree to display status and device information for storage and storage components.
The Management Console provides a wide-range of information in regards to vendor storage. Drill down into the following components in the lower Navigator tree to obtain additional details: • Servers. The Servers panel lists the host names for the attached storage. • LUNs. The LUNs panel provides information about the LUNs in a storage cluster. See “Managing LUNs in a storage cluster” (page 92) for more information.
Monitoring the status of file serving nodes The dashboard on the GUI displays information about the operational status of file serving nodes, including CPU, I/O, and network performance information. To view this information from the CLI, use the ibrix_server -l command, as shown in the following sample output: ibrix_server -l SERVER_NAME ----------node1 node2 STATE -----------Up, HBAsDown Up, HBAsDown CPU(%) -----0 0 NET_IO(MB/s) -----------0.00 0.00 DISK_IO(MB/s) ------------0.00 0.
Viewing events The GUI dashboard specifies the number of events that have occurred in the last 24 hours. Click Events in the GUI Navigator to view a report of the events. You can also view events that have been reported for specific file systems or servers. On the CLI, use the ibrix_event command to view information about cluster events.
Monitoring cluster health To monitor the functional health of file serving nodes and StoreAll clients, execute the ibrix_health command. This command checks host performance in several functional areas and provides either a summary or a detailed report of the results. Health checks The ibrix_health command runs these health checks on file serving nodes: • Pings remote file serving nodes that share a network with the test hosts.
The following is an example of the output from the ibrix_health -l command: [root@bv18-03 ~]# ibrix_health -l Overall Health Checker Results - PASSED ======================================= Host Summary Results ==================== Host Result Type State Network ------- ------ ------ ----- ---------bv18-03 PASSED Server Up 10.10.18.3 bv18-04 PASSED Server Up 10.10.18.
------------ ------ -----------------bv18-03 engine uuid matches on Iad and Fusion Manager bv18-03 IP address matches on Iad and Fusion Manager bv18-03 network protocol matches on Iad and Fusion Manager bv18-03 engine connection state on Iad is up bv18-04 engine uuid matches on Iad and Fusion Manager bv18-04 IP address matches on Iad and Fusion Manager bv18-04 network protocol matches on Iad and Fusion Manager bv18-04 engine connection state on Iad is up ibrixFS file system uuid matches on Iad and Fusion Ma
lab12-10.hp.com 261 3 355 2 ---------Mem-----------HOST MemTotal(MB) MemFree(MB) SwapTotal(MB) SwapFree(MB) lab12-10.hp.com 1034616 703672 2031608 2031360 ---------CPU----------HOST User System Nice Idle IoWait Irq SoftIrq lab12-10.hp.com 0 0 0 0 97 1 0 ---------NFS v3-------HOST Null Getattr Setattr Lookup Access Readlink Read Write lab12-10.hp.com 0 0 0 0 0 0 0 0 98 HOST lab12-10.hp.com Create Mkdir Symlink Mknod Remove Rmdir Rename 0 0 0 0 0 0 0 HOST lab12-10.hp.
10 Using the Statistics tool The Statistics tool reports historical performance data for the cluster or for an individual file serving node. You can view data for the network, the operating system, and the file systems, including the data for NFS, memory, and block devices. Statistical data is transmitted from each file serving node to the Fusion Manager, which controls processing and report generation.
Upgrading the Statistics tool from StoreAll software 6.0 The statistics history is retained when you upgrade to version 6.1 or later. The Statstool software is upgraded when the StoreAll software is upgraded using the ibrix_upgrade and auto_ibrixupgrade scripts. Note the following: • If statistics processes were running before the upgrade started, those processes will automatically restart after the upgrade completes successfully.
The Time View lists the reports in chronological order, and the Table View lists the reports by cluster or server. Click a report to view it.
Generating reports To generate a new report, click Request New Report on the StoreAll Management Console Historical Reports GUI. To generate a report, enter the necessary specifications and click Submit. The completed report appears in the list of reports on the statistics home page. When generating reports, be aware of the following: • A report can be generated only from statistics that have been gathered. For example, if you start the tool at 9:40 a.m. and ask for a report from 9:00 a.m. to 9:30 a.m.
Maintaining the Statistics tool Space requirements The Statistics tool requires about 4 MB per hour for a two-node cluster. To manage space, take the following steps: • Maintain sufficient space (4 GB to 8 GB) for data collection in the /usr/local/statstool/ histstats directory. • Monitor the space in the /local/statstool/histstats/reports/ directory. For the default values, see “Changing the Statistics tool configuration” (page 103).
The following actions occur after a successful failover: • If Statstool processes were running before the failover, they are restarted. If the processes were not running, they are not restarted. • The Statstool passive management console is installed on the StoreAll Fusion Manager in maintenance mode. • Setrsync is run automatically on all cluster nodes from the current active Fusion Manager. • Loadfm is run automatically to present all file system data in the cluster to the active Fusion Manager.
Troubleshooting the Statistics tool Testing access To verify that ssh authentication is enabled and data can be obtained from the nodes without prompting for a password, run the following command: # /usr/local/ibrix/stats/bin/stmanage testpull Other conditions • Data is not collected. If data is not being gathered in the common directory for the Statistics Manager (/usr/local/statstool/histstats/ by default), restart the Statistics tool processes on all nodes.
11 Maintaining the system Shutting down the system To shut down the system completely, first shut down the StoreAll software, and then power off the system hardware. Shutting down the StoreAll software Use the following procedure to shut down the StoreAll software. Unless noted otherwise, run the commands from the node hosting the active Fusion Manager. 1. Stop any active remote replication, data tiering, or rebalancer tasks.
7. Unmount all file systems on the cluster nodes: ibrix_umount -f To unmount file systems from the GUI, select Filesystems > unmount. 8. Verify that all file systems are unmounted: ibrix_fs -l If a file system fails to unmount on a particular node, continue with this procedure. The file system will be forcibly unmounted during the node shutdown. 9. Shut down all StoreAll Server services and verify the operation: # pdsh -a /etc/init.d/ibrix_server stop | dshbak # pdsh -a /etc/init.
Starting the StoreAll software To start the StoreAll software, complete the following steps: 1. Power on the node hosting the active Fusion Manager. 2. Power on the file serving nodes (*root segment = segment 1; power on owner first, if possible). 3. Monitor the nodes on the GUI and wait for them all to report UP in the output from the following command: ibrix_server -l 4. Mount file systems and verify their content.
To perform the rolling reboot, complete the following steps on each file serving node: 1. Reboot the node directly from Linux. (Do not use the "Power Off" functionality in the GUI, as it does not trigger failover of file serving services.) The node will fail over to its backup. 2. Wait for the GUI to report that the rebooted node is Up. 3. From the GUI, failback the node, returning services to the node from its backup.
The IAD Tunings dialog box configures the StoreAll administrative daemon.
The Module Tunings dialog box adjusts various advanced parameters that affect server operations. On the Servers dialog box, select the servers to which the tunings should be applied.
Tuning file serving nodes from the CLI All Fusion Manager commands for tuning hosts include the -h HOSTLIST option, which supplies one or more host groups. Setting host tunings on a host group is a convenient way to tune a set of clients all at once. To set the same host tunings on all clients, specify the clients host group. CAUTION: Changing host tuning settings alters file system performance. Contact HP Support before changing host tuning settings.
The values that are restored depend on the values specified for the -h HOSTLIST command: • ◦ File serving nodes. The default file serving node host tunings are restored. ◦ StoreAll clients. The host tunings that are in effect for the default clients host group are restored. ◦ Hostgroups. The host tunings that are in effect for the parent of the specified host groups are restored. To list host tuning settings on file serving nodes, StoreAll clients, and host groups, use the following command.
Occasionally you may need to manage the segments manually: • Migrate segments. This operation transfers ownership of segments to other servers. For example, if a server is overloaded or unavailable, you can transfer its segments to another server that can see the same storage. • Rebalance segments. This operation redistributes files across segments and can be used if certain segments are filling up and affecting file system performance. See “Maintaining file systems” in the for more information.
The new owner of the segment must be able to see the same storage as the original owner. The Change Segment Owner dialog box lists the servers that can see the segment you selected. Select one of these servers to be the new owner.
The Summary dialog box shows the segment migration you specified. Click Back to make any changes, or click Finish to complete the operation. To migrate ownership of segments from the CLI, use the following commands. Migrate ownership of specific segments: ibrix_fs -m -f FSNAME -s LVLIST -h HOSTNAME [-M] [-F] [-N] To force the migration, include -M. To skip the source host update during the migration, include -F. To skip server health checks, include -N.
For example, to migrate ownership of all segments in file system ifs1 from server1 to server2: ibrix_fs -m -f ifs1 -H server1,server2 Evacuating segments and removing storage from the cluster Before removing storage used for a StoreAll software file system, you will need to evacuate the segments (or logical volumes) storing file system data. This procedure moves the data to other segments in the file system and is transparent to users or applications accessing the file system.
The Summary dialog box lists the source and destination segments for the evacuation. Click Back to make any changes, or click Finish to start the evacuation. The Active Tasks panel reports the status of the evacuation task. When the task is complete, it will be added to the Inactive Tasks panel. 4. When the evacuation is complete, run the following command to retire the segment from the file system: ibrix_fs -B -f FSNAME -n BADSEGNUMLIST The segment number associated with the storage is not reused.
After obtaining the name of the file, use a command such as cp to move the file manually. Then run the segment evacuation process again. The analyzer log lists the chunks that were left on segments.
4. Remove the High Availability configuration between FSN4 and FSN3: a. Stop NIC monitoring for the user NICs: ibrix_nic -m -h FSN3 -D FSN4/bond0:1 ibrix_nic -m -h FSN4 -D FSN3/bond0:1 b. Remove the backup NICs: ibrix_nic -b -u FSN4/bond0:2 ibrix_nic -b -u FSN3/bond0:2 c. Remove the backup server: ibrix_server -b -U -h FSN4 ibrix_server -b -U -h FSN3 5.
14. Uninstall the StoreALL OS software from FSN4: /usr/local/ibrix/local/installation/ibrix/ibrixinit -u -F NOTE: If the same StoreAll OS version will be reinstalled on FSN4, use the following command instead: /usr/local/ibrix/local/installation/ibrix/ibrixinit -U The node is no longer in the cluster. Maintaining networks Cluster and user network interfaces StoreAll software supports the following logical network interfaces: • Cluster network interface.
Identifying a user network interface for a file serving node To identify a user network interface for specific file serving nodes, use the ibrix_nic command. The interface name (IFNAME) can include only alphanumeric characters and underscores, such as eth1. ibrix_nic -a -n IFNAME -h HOSTLIST If you are identifying a VIF, add the VIF suffix (:nnnn) to the physical interface name.
when you prefer a network interface, you can force clients to query the Fusion Manager by executing the command ibrix_lwhost --a on the client or by rebooting the client. Preferring a network interface for a file serving node or Linux StoreAll client The first command prefers a network interface for a File Server Node; the second command prefers a network interface for a client.
ibrix_hostgroup -n -g HOSTGROUP -A DESTHOST/IFNAME The destination host (DESTHOST) cannot be a host group. For example, to prefer network interface eth3 for traffic from all StoreAll clients (the clients host group) to file serving node s2.hp.com: ibrix_hostgroup -n -g clients -A s2.hp.com/eth3 Unpreferring network interfaces To return file serving nodes or StoreAll clients to the cluster interface, unprefer their preferred network interface.
Changing the cluster interface If you restructure your networks, you might need to change the cluster interface. The following rules apply when selecting a new cluster interface: • The Fusion Manager must be connected to all machines (including standby servers) that use the cluster network interface. Each file serving node and StoreAll client must be connected to the Fusion Manager by the same cluster network interface. A Gigabit (or faster) Ethernet port must be used for the cluster interface.
The following table describes the fields in the output. Field Description BACKUP HOST File serving node for the standby network interface. BACKUP-IF Standby network interface. HOST File serving node. IFNAME Network interface on this file serving node. IP_ADDRESS IP address of this NIC. LINKMON Whether monitoring is on for this NIC. MAC_ADDR MAC address of this NIC. ROUTE IP address in routing table used by this NIC. STATE Network interface state. TYPE Network type (cluster or user).
12 Licensing This chapter describes how to view your current license terms and how to obtain and install new StoreAll software product license keys. NOTE: For licensing features such as block snapshots on the HP P2000 G3 MSA Array System or HP 2000 Modular Smart Array, see the array documentation. Viewing license terms The StoreAll software license file is stored in the installation directory. To view the license from the GUI, select Cluster Configuration in the Navigator and then select License.
13 Upgrading firmware The Firmware Management Tool (FMT) is a utility that scans the StoreAll system for outdated firmware and provides a comprehensive report that provides the following information: • Device found • Active firmware found on the discovered device • Qualified firmware for the discovered device • Proposed action — Users are told whether an upgrade is recommended • Severity — How severe an upgrade is required • Reboot required on flash • Device information • Parent device ID Co
Steps for upgrading the firmware IMPORTANT: On the StoreAll 9320 Storage platform the prerequisite measures must be followed before performing the firmware update operation on enclosure hard disk drives: • Storage disk drive update is an OFFLINE process. Ensure that all host and array I/O must be stopped prior to the update. • Make sure all the file systems are unmounted. Failure to comply may result in an OS crashing.
2. 3. Do the following based on the Proposed Action and Severity: Status in Proposed Action column Status in Severity column Go to UPGRADE MANDATORY Step 3 UPGRADE RECOMMENDED Step 3 is optional. However, it is recommended to perform step 3 for system stability and to avoid any known issues. NONE or DOWNGRADE MANDATORY Step 4 NONE or DOWNGRADE RECOMMENDED Step 4 is optional. However, it is recommended to perform step 4 for system stability and to avoid any known issues.
a. Determine whether the node to be flashed is the active Fusion Manager by enter the following command: ibrix_fm -i b. Perform a manual FM failover on the local node by entering the following command from the active Fusion Manager: ibrix_fm -m nofmfailover server1 The FM failover will take approximately one minute. c. d. If server1 is not the active Fusion Manager, proceed to step e to fail over server1 to server2.
6. If you are upgrading to 6.3, you must complete the steps provided in the “After the upgrade” section for your type of upgrade, as shown in the following table: Type of upgrade Complete the steps in this section Online upgrades “After the upgrade” (page 12) Automated offline upgrades “After the upgrade” (page 13) Manual offline upgrades “After the upgrade” (page 16) Finding additional information on FMT You can find additional information on FMT from the online help for FMT.
14 Troubleshooting Collecting information for HP Support with the IbrixCollect Ibrix Collect is a log collection utility that allows you collect relevant information for diagnosis by HP Support when system issues occur. The collection can be triggered manually using the GUI or CLI, or automatically during a system crash.
4. Click Okay. To collect logs and command results using the CLI, use the following command: ibrix_collect -c -n NAME NOTE: • Only one manual collection of data is allowed at a time. • When a node restores from a system crash, the vmcore under /var/crash/ directory is processed. Once processed, the directory will be renamed /var/crash/ _PROCESSED. HP Support may request that you send this information to assist in resolving the system crash.
To specify more than one collection to be deleted at a time from the CLI, provide the names separated by a semicolon. To delete all data collections manually from the CLI, use the following command: ibrix_collect -d -F Configuring Ibrix Collect You can configure data collection to occur automatically upon a system crash. This collection will include additional crash digester output. The archive filename of the system crash-triggered collection will be in the format _crash_.tgz.
ibrix_collect -C -m [-s ] [-f ] [-t ] NOTE: More than one email ID can be specified for -t option, separated by a semicolon. The “From” and “To” command for this SMTP server are Ibrix Collect specific. Obtaining custom logging from ibrix_collect add-on scripts You can create add-on scripts that capture custom StoreAll and operating system commands and logs.
2. Place the added-on script in the following directory: /usr/local/ibrix/ibrixcollect/ibrix_collect_add_on_scripts/ The following example shows several add-on scripts stored in the ibrix_collect_add_on_scripts directory: root@host2 /]# ls -l /usr/local/ibrix/ibrixcollect/ibrix_collect_add_on_scripts/ total 8 -rwxr-xr-x 1 root root 93 Dec 7 13:39 60_addOn.sh -rwxrwxrwx 1 root root 48 Dec 20 09:22 63_AddOnTest.sh 3.
-rw-r--r-- 1 root root 2021895 Dec 20 12:41 addOnCollection.tgz 3. Extract the tar file, containing the output of the add-on script. The tar file containing the output of the add-on script has the name of the collection by entering the following command: [root@host2 archive]#tar -xvf addOnCollection.tgz In this instance, addOnCollection.tgz is the tar file containing the output of the add-on script. The tar command displays the following: ./ ./host2_addOnCollection_2012-12-20-12-38-36.tgz 4.
Viewing data collection information To view data collection history from the CLI, use the following command: ibrix_collect -l To view data collection details such as date (of creation), size, description, state and initiator, use the following command: ibrix_collect -v -n Adding/deleting commands or logs in the XML file To add or change the logs that are collected or commands that are executed during data collection, you can modify the Ibrix Collect XML files that are stored in the directory /usr/lo
To permanently disable SELinux, edit its configuration file (/etc/selinux/config) and set SELINUX=parameter to either permissive or disabled. SELinux will be stopped at the next boot. For StoreAll clients, the client might not be registered with the Fusion Manager. For information on registering clients, see the HP StoreAll Storage Installation Guide.
Mounted drive not visible when using Terminal Server Refresh the browser's view of the system by logging off and then logging back on. StoreAll client auto-startup interferes with debugging The StoreAll client is set to start automatically, which can interfere with debugging a Windows StoreAll client problem. To prevent this, reboot the machine in safe mode and change the Windows StoreAll client service mode to manual, which enables you to reboot without starting the client. 1.
In some very rare cases, a database corruption might occur, as a result of these external events or from some internal dysfunction. Express Query contains a recovery mechanism that tries to rebuild the database from information Express Query is keeping specifically for that critical situation. Express Query might be unable to recover from internal database corruption.
5. Cluster and file system health checks have an OK status but Express Query is yet in a MIF condition for one or several specific file systems. This unlikely situation occurs when some data has been corrupted and it cannot be recovered. To solve this situation: a. If there is a full backup of the file system involved, do a restore. b. If there is no full backup: 1. Disable Express Query for the file system, by entering the following command: ibrix_fs -T -D -f 2.
15 Recovering a file serving node Use the following procedure to recover a failed file serving node. You will need to create a QuickRestore DVD or USB key, as described later, and then install it on the affected node. This step installs the operating system and StoreAll software on the node and launches a configuration wizard. CAUTION: The Quick Restore DVD or USB key restores the file serving node to its original factory state.
3. Enter the information for the node being restored on the Network Configuration dialog box and click OK. 4. Confirm that the information displayed in the Configuration Summary dialog box is correct and click Commit.
5. 146 On the X9000 Installation — Network Setup Complete dialog box, select Join this IBIRX server to an existing cluster and click OK.
6. The wizard scans the network for existing clusters. On the Join Cluster dialog box, select the management console (Fusion Manager) for your cluster, and then click OK. If your cluster does not exist in the list of choices, click Cancel so that you can provide the IP address of the FM to which this node has to be registered.
7. If you clicked the Cancel button in the previous dialog box, enter the management console IP of the desired cluster on the Management Console IP dialog box and click OK. 8. On the Replace Existing Server dialog box, click Yes when you are asked if you want to replace the existing server.
Completing the restore on a file serving node Complete the following steps: 1. Ensure that you have root access to the node. The restore process sets the root password to hpinvent, the factory default. 2. Verify information about the node you restored: ibrix_server -f [-p] [-M] [-N] -h SERVERNAME 3. If you disabled NIC monitoring before using the QuickRestore, re-enable the monitor: ibrix_nic -m -h MONITORHOST -A DESTHOST/IFNAME For example: ibrix_nic -m -h titan16 -A titan15/eth2 4. 5.
ibrix_cifs -d -s SHARENAME 4. Run the following command to verify that the original share information is on the restored node: ibrix_cifs -i -h SERVERNAME Restore HTTP services. Complete the following steps: 1. Take the appropriate actions: 2. • If Active Directory authentication is used, join the restored node to the AD domain manually. • If Local user authentication is used, create a temporary local user on the GUI and apply the settings to all servers.
16 Support and other resources Contacting HP For worldwide technical support information, see the HP support website: http://www.hp.
Using HP MSA Disk Arrays • HP 2000 G2 Modular Smart Array Reference Guide • HP 2000 G2 Modular Smart Array CLI Reference Guide • HP P2000 G3 MSA System CLI Reference Guide • Online help for HP Storage Management Utility (SMU) and Command Line Interface (CLI) To find these documents, go the Manuals page (http://www.hp.com/support/manuals) and select storage >Disk Storage Systems > MSA Disk Arrays >HP 2000sa G2 Modular Smart Array or HP P2000 G3 MSA Array Systems.
17 Documentation feedback HP is committed to providing documentation that meets your needs. To help us improve the documentation, send any errors, suggestions, or comments to Documentation Feedback (docsfeedback@hp.com). Include the document title and part number, version number, or the URL when submitting your feedback.
A Cascading Upgrades If you are running a StoreAll version earlier than 5.6, do incremental upgrades as described in the following table. If you are running StoreAll 5.6, upgrade to 6.1 before upgrading to 6.3. If you are upgrading from Upgrade to Where to find additional information StoreAll version 5.4 StoreAll version 5.5 “Upgrading the StoreAll software to the 5.5 release” (page 167) StoreAll version 5.5 StoreAll version 5.6 “Upgrading the StoreAll software to the 5.
NOTE: • Verify that the root partition contains adequate free space for the upgrade. Approximately 4 GB is required. • Be sure to enable password-less access among the cluster nodes before starting the upgrade. • Do not change the active/passive Fusion Manager configuration during the upgrade. • Linux StoreAll clients must be upgraded to the 6.x release. Online upgrades for StoreAll software 6.x to 6.1 Online upgrades are supported only from the StoreAll 6.x release.
4. Change to the /local/ibrix/ directory. cd /local/ibrix/ 5. Run the upgrade script and follow the on-screen directions: ./auto_online_ibrixupgrade 6. 7. Upgrade Linux StoreAll clients. See “Upgrading Linux StoreAll clients” (page 17). If you received a new license from HP, install it as described in “Licensing” (page 127). After the upgrade Complete these steps: 1. Start any remote replication, rebalancer, or data tiering tasks that were stopped before the upgrade. 2.
If a node is configured on the user network, see “Node is not registered with the cluster network ” (page 21) for a workaround. 6. 7. Stop all client I/O to the cluster or file systems. On the Linux client, use lsof to show open files belonging to active processes. Unmount file systems on Linux StoreAll clients: ibrix_umount -f MOUNTPOINT 8. On all nodes hosting the passive Fusion Manager, place the Fusion Manager into maintenance mode: /bin/ibrix_fm -m maintenance -A 9.
3. Change to the /local/ibrix/ directory. cd /local/ibrix/ 4. Run the following upgrade script: ./auto_ibrixupgrade The upgrade script automatically stops the necessary services and restarts them when the upgrade is complete. The upgrade script installs the Fusion Manager on all file serving nodes. The Fusion Manager is in active mode on the node where the upgrade was run, and is in passive mode on the other file serving nodes.
3. Run the upgrade script: ./ibrixupgrade -f The upgrade software automatically stops the necessary services and restarts them when the upgrade is complete. 4. Execute the following command to verify the client is running StoreAll software: /etc/init.d/ibrix_client status IBRIX Filesystem Drivers loaded IBRIX IAD Server (pid 3208) running... The IAD service should be running, as shown in the previous sample output. If it is not, contact HP Support.
• The upgrade takes place in parallel across all file serving nodes owning segments in the file system, with at least one thread running on each node. For a system with multiple controllers, the utility will run a thread for each controller if possible. • Files up to 3.8 TB in size can be upgraded. To enable snapshots on larger files, they must be migrated after the upgrade is complete (see “Migrating large files” (page 160). • In general, the upgrade takes approximately three hours per TB of data.
Upgrading pre-6.1.1 file systems for data retention features Data retention was automatically enabled for file systems created with StoreAll 6.1.1 or later. If you want to enable data retention for file systems created with StoreAll 6.0 or earlier, run the ibrix_reten_adm -u command, as described in this section. To enable data retention: 1. If you have a pre-6.0 file system, run the upgrade60.sh utility, as described in Section (page 161). 2.
Manual upgrade Check the following: • If the restore script fails, check /usr/local/ibrix/setup/logs/restore.log for details. • If configuration restore fails, look at /usr/local/ibrix/autocfg/logs/appliance.log to determine which feature restore failed. Look at the specific feature log file under /usr/ local/ibrix/setup/logs/ for more detailed information.
ib51-101 10.10.51.101 ib51-102 10.10.51.102 [root@ib51-102 ~]# ibrix_fm -i FusionServer: ib51-102 (active, quorum is running) ================================================== File system unmount issues If a file system does not unmount successfully, perform the following steps on all servers: 1. Run the following commands: chkconfig ibrix_server off chkconfig ibrix_ndmp off chkconfig ibrix_fusionmanager off 2. 3. Reboot all servers. Run the following commands to move the services back to the on state.
The upgrade takes approximately 45 minutes for 9320 systems with a standard configuration. NOTE: If you are upgrading from a StoreAll 5.x release, any support tickets collected with the ibrix_supportticket command will be deleted during the upgrade. Download a copy of the archive files (.tgz) from the /admin/platform/diag/supporttickets directory. Automatic upgrades All file serving nodes and management consoles must be up when you perform the upgrade.
5. 6. If file systems are mounted from a Windows StoreAll client, unmount the file systems using the Windows client GUI. Unmount all StoreAll file systems: /bin/ibrix _umount -f Saving the node configuration Complete the following steps on each node, starting with the node hosting the active management console: 1. Run /usr/local/ibrix/setup/save_cluster_config. This script creates a tgz file named _cluser_config.tgz, which contains a backup of the node configuration. 2.
5. From the node hosting the active management console, perform a manual backup of the upgraded configuration: /bin/ibrix_fm -B 6. Verify that all version indicators match for file serving nodes. Run the following command from the active management console: /bin/ibrix_version -l If there is a version mismatch, run the /ibrix/ibrixupgrade -f script again on the affected node, and then recheck the versions. The installation is successful when all version indicators match.
Manual upgrade Check the following: • If the restore script fails, check /usr/local/ibrix/setup/logs/restore.log for details. • If configuration restore fails, look at /usr/local/ibrix/autocfg/logs/appliance.log to determine which feature restore failed. Look at the specific feature log file under /usr/ local/ibrix/setup/logs/ for more detailed information.
1. 2. Check the dashboard on the management console GUI to verify that all nodes are up. Verify that you have an even number of FSNs configured in a couplet pair high availability architecture by running the following command: ibrix_server -l 3. On the current active management console, move the /ibrix directory used in the previous release installation to ibrix.old.
Standard online upgrade The management console must be upgraded first. You can then upgrade file serving nodes and StoreAll clients in any order. Upgrading the management console Complete the following steps on the Management Server machine or blade: 1. Disable automated failover on all file serving nodes: /bin/ibrix_server -m -U 2. Verify that automated failover is off: /bin/ibrix_server -l In the output, the HA column should display off. 3.
6. Verify that the ibrix and ipfs services are running: lsmod|grep ibrix ibrix 2323332 0 (unused) lsmod|grep ipfs ipfs1 102592 0 (unused) If either grep command returns empty, contact HP Support. 7. From the management console, verify that the new version of StoreAll software FS/IAS is installed on the file serving node: /bin/ibrix_version -l -S 8. If the upgrade was successful, failback the file serving node: /bin/ibrix_server -f -U -h HOSTNAME 9.
Verify that all likewise services are down on all file serving nodes: ps -ef | grep likewise Use kill -9 to kill any likewise services that are still running. 4. From the management console, unmount all StoreAll file systems: /bin/ibrix_umount -f Upgrading the management console Complete the following steps on the management console: 1. Force a backup of the configuration: /bin/ibrix_fm -B The output is stored at /usr/local/ibrix/tmp/fmbackup.zip.
ibrix 2323332 0 (unused) lsmod|grep ipfs ipfs1 102592 0 (unused) If either grep command returns empty, contact HP Support. 6. From the management console, verify that the new version of StoreAll software FS/IAS has been installed on the file serving nodes: /bin/ibrix_version -l -S Completing the upgrade 1. Remount all file systems: /bin/ibrix_mount -f -m 2. From the management console, turn automated failover back on: /bin/ibrix_server -m 3.
1. On the node hosting the active management console, force a backup of the management console configuration: /bin/ibrix_fm -B The output is stored at /usr/local/ibrix/tmp/fmbackup.zip. Be sure to save this file in a location outside of the cluster. 2. On the active management console node, disable automated failover on all file serving nodes: /bin/ibrix_server -m -U 3.
The installer upgrades both the management console software and the file serving node software on this node. 13. Verify the status of the management console: /etc/init.d/ibrix_fusionmanager status The status command confirms whether the correct services are running. Output will be similar to the following: Fusion Manager Daemon (pid 18748) running... Also run the following command, which should report that the console is Active: /bin/ibrix_fm -i 14. Check /usr/local/ibrix/log/fusionserver.
24. Verify that the agile management console software and the file serving node software are now upgraded on the two nodes hosting the agile management console: /bin/ibrix_version -l -S Following is some sample output: Fusion Manager version: 5.5.XXX =============================== Segment Servers =============== HOST_NAME FILE_SYSTEM IAD/IAS IAD/FS OS KERNEL_VERSION ARCH --------- ---------------------- ------- ------- --------- -------------- ---ib50-86 5.5.205(9000_5_5) 5.5.XXX 5.5.
Completing the upgrade 1. From the node hosting the active management console, turn automated failover back on: /bin/ibrix_server -m 2. Confirm that automated failover is enabled: /bin/ibrix_server -l In the output, the HA column should display on. 3. Verify that all version indicators match for file serving nodes.
1. On the node hosting the active management console, force a backup of the management console configuration: /bin/ibrix_fm -B The output is stored at /usr/local/ibrix/tmp/fmbackup.zip. Be sure to save this file in a location outside of the cluster. 2. On the node hosting the passive management console, place the management console into maintenance mode: /bin/ibrix_fm -m maintenance -A 3.
The upgrade automatically stops services and restarts them when the process is complete. 4. When the upgrade is complete, verify that the StoreAll software services are running on the node: /etc/init.d/ibrix_server status The output should be similar to the following example. If the IAD service is not running on your system, contact HP Support. IBRIX Filesystem Drivers loaded ibrcud is running.. pid 23325 IBRIX IAD Server (pid 23368) running... 5.
ibrixupgrade hangs The installation can hang because the RPM database is corrupted. This is caused by inconsistencies in the Red Hat Package Manager. Rebuild the RPM database using the following commands and then attempt the installation again. Note that rm is followed by a space and then two underscores, and rpm is followed by a space and then two dashes: cd /var/lib/rpm rm __* rpm --rebuilddb On the management console, ibrixupgrade may also hang if the NFS mount points are stale.
B Component diagrams for 9300 systems Front view of file serving node Item Description 1 Quick-release levers (2) 2 HP Systems Insight Manager display 3 Hard drive bays 4 SATA optical drive bay 5 Video connector 6 USB connectors (2) Rear view of file serving node Item Description 1 PCI slot 5 2 PCI slot 6 3 PCI slot 4 4 PCI slot 2 5 PCI slot 3 6 PCI slot 1 7 Power supply 2 (PS2) 8 Power supply 1 (PS1) 9 USB connectors (2) 10 Video connector 180 Component diagrams for 93
Item Description 11 NIC 1 connector 12 NIC 2 connector 13 Mouse connector 14 Keyboard connector 15 Serial connector 16 iLO 2 connector 17 NIC 3 connector 18 NIC 4 connector Rear view of file serving node 181
Server PCIe card PCI slot HP SC08Ge 3Gb SAS Host Bus Adapter 1 NC364T Quad 1Gb NIC 2 empty 3 empty 4 empty 5 empty 6 HP SC08Ge 3Gb SAS Host Bus Adapter 1 empty 2 empty 3 NC522SFP dual 10Gb NIC 4 empty 5 empty 6 HP SC08Ge 3Gb SAS Host Bus Adapter 1 NC364T Quad 1Gb NIC 2 empty 3 HP SC08Ge 3Gb SAS Host Bus Adapter 4 empty 5 empty 6 HP SC08Ge 3Gb SAS Host Bus Adapter 1 HP SC08Ge 3Gb SAS Host Bus Adapter 2 empty 3 NC522SFP dual 10Gb NIC 4 empty 5 empty 6 SATA 1G
C System component and cabling diagrams for 9320 systems System component diagrams Front view of 9300c array controller or 9300cx 3.
Rear view of 9300c array controller Item Description 1 Power supplies 2 Power switches 3 Host ports 4 CLI port 5 Network port 6 Service port (used by service personnel only) 7 Expansion port (connects to drive enclosure) Rear view of 9300cx 3.
Front view of file serving node Item Description 1 Quick-release levers (2) 2 HP Systems Insight Manager display 3 Hard drive bays 4 SATA optical drive bay 5 Video connector 6 USB connectors (2) Rear view of file serving node Item Description 1 PCI slot 5 2 PCI slot 6 3 PCI slot 4 4 PCI slot 2 5 PCI slot 3 6 PCI slot 1 7 Power supply 2 (PS2) 8 Power supply 1 (PS1) 9 USB connectors (2) 10 Video connector 11 NIC 1 connector 12 NIC 2 connector System component diagrams
Item Description 13 Mouse connector 14 Keyboard connector 15 Serial connector 16 iLO 2 connector 17 NIC 3 connector 18 NIC 4 connector 186 System component and cabling diagrams for 9320 systems
Server PCIe card PCI slot HP SC08Ge 3Gb SAS Host Bus Adapter 1 NC364T Quad 1Gb NIC 2 empty 3 empty 4 empty 5 empty 6 HP SC08Ge 3Gb SAS Host Bus Adapter 1 empty 2 empty 3 NC522SFP dual 10Gb NIC 4 empty 5 empty 6 HP SC08Ge 3Gb SAS Host Bus Adapter 1 NC364T Quad 1Gb NIC 2 empty 3 HP SC08Ge 3Gb SAS Host Bus Adapter 4 empty 5 empty 6 HP SC08Ge 3Gb SAS Host Bus Adapter 1 HP SC08Ge 3Gb SAS Host Bus Adapter 2 empty 3 NC522SFP dual 10Gb NIC 4 empty 5 empty 6 SATA 1G
Cabling diagrams Cluster network cabling diagram 188 System component and cabling diagrams for 9320 systems
SATA option cabling Line Description SAS I/O pathController A SAS I/O pathController B Cabling diagrams 189
SAS option cabling Line Description SAS I/O pathArray 1: Controller A SAS I/O pathArray 1: Controller B SAS I/O pathArray 2: Controller A SAS I/O pathArray 2: Controller B 190 System component and cabling diagrams for 9320 systems
Drive enclosure cabling Item Description 1 SAS controller in 9300c controller enclosure 2 I/O modules in four 9300cx drive enclosures Cabling diagrams 191
D Warnings and precautions Electrostatic discharge information To prevent damage to the system, be aware of the precautions you need to follow when setting up the system or handling parts. A discharge of static electricity from a finger or other conductor could damage system boards or other static-sensitive devices. This type of damage could reduce the life expectancy of the device.
Equipment symbols If the following symbols are located on equipment, hazardous conditions could exist. WARNING! Any enclosed surface or area of the equipment marked with these symbols indicates the presence of electrical shock hazards. Enclosed area contains no operator serviceable parts. To reduce the risk of injury from electrical shock hazards, do not open this enclosure. WARNING! Any RJ-45 receptacle marked with these symbols indicates a network interface connection.
WARNING! Verify that the AC power supply branch circuit that provides power to the rack is not overloaded. Overloading AC power to the rack power supply circuit increases the risk of personal injury, fire, or damage to the equipment. The total rack load should not exceed 80 percent of the branch circuit rating. Consult the electrical authority having jurisdiction over your facility wiring and installation requirements.
CAUTION: Protect the installed solution from power fluctuations and temporary interruptions with a regulating Uninterruptible Power Supply (UPS). This device protects the hardware from damage caused by power surges and voltage spikes, and keeps the system in operation during a power failure. CAUTION: To properly ventilate the system, you must provide at least 7.6 centimeters (3.0 inches) of clearance at the front and back of the device.
E Regulatory information For important safety, environmental, and regulatory information, see Safety and Compliance Information for Server, Storage, Power, Networking, and Rack Products, available at http:// www.hp.com/support/Safety-Compliance-EnterpriseProducts.
Glossary ACE Access control entry. ACL Access control list. ADS Active Directory Service. ALB Advanced load balancing. BMC Baseboard Management Configuration. CIFS Common Internet File System. The protocol used in Windows environments for shared folders. CLI Command-line interface. An interface comprised of various commands which are used to control operating system responses. CSR Customer self repair. DAS Direct attach storage.
SELinux Security-Enhanced Linux. SFU Microsoft Services for UNIX. SID Secondary controller identifier number. SNMP Simple Network Management Protocol. TCP/IP Transmission Control Protocol/Internet Protocol. UDP User Datagram Protocol. UID Unit identification. VACM SNMP View Access Control Model. VC HP Virtual Connect. VIF Virtual interface. WINS Windows Internet Name Service. WWN World Wide Name. A unique identifier assigned to a Fibre Channel device. WWNN World wide node name.
Index Symbols /etc/sysconfig/i18n file, 27 9300 system components, 24 configuration, 26 features, 24 management interfaces, 27 shut down, 106 software, 24 start, 107 9320 system components, 24 configuration, 26 features, 24 management interfaces, 27 shutdown, 106 software, 24 start, 107 A agile Fusion Manager, 52 AutoPass, 127 B backups file systems, 76 Fusion Manager configuration, 76 NDMP applications, 76 Belarus Kazakhstan Russia EAC marking, 196 C CLI, 31 clients access virtual interfaces, 49 cluster
view process status, 109 file system migrate segments, 114 firewall configuration, 33 firmware, upgrade, 128 Fusion Manager agile, 52 back up configuration, 76 failover, 52 G grounding methods, 192 GUI add users, 31 change password, 32 customize, 30 Details panel, 30 Navigator, 30 open, 28 view events, 94 H hardware, power off, 107 hazardous conditions symbols on equipment, 193 HBAs display information, 65 monitor for high availability, 63 health check reports, 95 help obtaining, 151 High Availability agi
GUI password, 32 Phone Home, 36 ports, open, 33 power sources, server, 61 pre-6.
HP ProLiant and X86 Servers and Options, 196 HP Storage products, 196 websites HP, 152 HP Subscriber's Choice for Business, 152 product manuals, 151 spare parts, 152 Windows StoreAll clients, upgrade, 17, 159 202 Index
