HP StoreEver Tape Libraries Failover User Guide HP Part Number: 5697-3519 Published: December 2014 Edition: Fourth
© Copyright 2014 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license. The information contained herein is subject to change without notice.
Contents 1 Overview..................................................................................................6 Traditional library control path and drive connections without failover..............................................7 Basic path failover....................................................................................................................8 Basic data path port failover details.......................................................................................
Enabling data path failover.................................................................................................29 Verifying data path failover.................................................................................................30 7 Configuring failover for other HP StoreEver MSL Tape Libraries.......................31 Configuring data path port failover...........................................................................................31 Enabling data path port failover.....
Report cached reservation information..................................................................................61 Debug output control..........................................................................................................62 Reading the trace log.........................................................................................................62 Library path failover events......................................................................................................
1 Overview With today’s high dependency on access to business information, safe-guarded data and limited backup windows, the reliability of the backup hardware and software is vital. Additionally, backup operations are usually automated, often done at night, and any first pass operator intervention is done remotely.
◦ Is available for the ESL G3 and MSL6480 Tape Libraries. ◦ Is not available for the 1/8 G2 Tape Autoloader nor the MSL2024, MSL4048, MSL8048, or MSL8096 Tape Libraries. Traditional library control path and drive connections without failover A typical connection for a Fibre Channel tape library using the drive to bridge commands to the library controller in a two drive tape library is shown in “Typical bridged library controller connection” (page 7).
Figure 2 Logical view of traditional configuration 1. HP Ultrium Dual Port Fibre Channel Tape Drives 2. Lun 0 at the WWPN of this drive 3. Lun 1 at the WWPN of this drive 4. Lun 0 at the WWPN of this drive 5. Host connection Basic path failover Basic path failover uses features in the tape drive and library firmware to recover paths following a link failure.
Figure 3 Data path port failover example configuration 1. Primary data path 2. Secondary data path 3. Host connection 4. HP Ultrium dual port Fibre Channel tape drive Basic control path failover details Library basic control path port failover (see Figure 4 (page 10)) may be configured with one drive hosting the active logical link to the library controller and a second drive configured as the passive logical link to the library controller.
NOTE: The MSL libraries monitor drive removal events and if the active control path drive is removed, the library will automatically move the control path to the standby drive. However, if the control path drive is not removed but SAN connectively is lost, the administrator might need to use the library remote management interface to move the control path to the standby drive.
Technology for basic control path failover HP LTO-5 and LTO-6 Fibre Channel tape drives use a technology called N-Port Identifier Virtualization (NPIV) which is defined as part of the Fibre Channel standards maintained by the INCITS/T11 working group (see clause 6) in the FC-LS-2 specification. NPIV allows a single physical port to connect to a Fibre Channel switch multiple times using multiple node and port names.
Table 1 Example path list SAN Addressed Logical Unit Port Example SCSI address Example Logical Unit Worldwide Identifier 1 Tape drive 1 Port A ID 1 LUN 0 50:01:10:a0:00:00:00:01 1 Library Controller Port A ID 1 LUN 1 50:01:10:a0:00:00:00:02 2 Tape drive 1 Port B ID 2 LUN 0 50:01:10:a0:00:00:00:01 2 Library Controller Port B ID 2 LUN 1 50:01:10:a0:00:00:00:02 1 Tape drive 2 Port A ID 3 LUN 0 50:01:10:a0:00:00:00:03 1 Library Controller Port A ID 3 LUN 1 50:01:10:a0:00:00:00:
Driver operation during normal communications The advanced path failover driver passes commands through without any command translation and with minimal additional processing in normal operation to retain the maximum possible performance. During normal communication the advanced path failover driver does not introduce additional commands that would cause delays (for example, commands to determine position) during reading and writing.
tape library user interface will show which control path drive is the current active control path drive. Library controller path failure recovery In most cases the library controller path failure recovery is the same as the tape drive path failure recovery. If all paths to the drive hosting the library controller have failed, the advanced path failover driver is capable of opening a connection to the library controller through another tape drive.
2 Enabling basic path failover To successfully enable basic path failover, complete the following: • Verify that the switch and host prerequisites are met. See “Preparing the switches and hosts for failover” (page 18).
3 Installing advanced path failover To successfully install advanced path failover, complete the following: • Prepare the SAN. See “Preparing the SAN” (page 16). • Prepare the library. See “Preparing the library” (page 16). • Prepare the host. See “Preparing the host” (page 17). Preparing the SAN To prepare the SAN, review the following: • “Minimize host access” (page 16) • “Configure redundant connections” (page 16).
that is not using an advanced path failover driver, the drive or library will reject the command with an additional sense code of 82h/93h, FAILOVER SESSION SEQUENCE ERROR. That error code should only be reported to an application if the advanced path failover drivers are not installed on that host.
4 Preparing the switches and hosts for failover Prerequisites for using basic data and control path failover: • The library drive FC ports must be attached to a Fibre Channel SAN that supports NPIV and NPIV must be enabled. LTO-5 and LTO-6 drives need an 8G connection with the fill word set to arb(ff). With 4G connections, set the fill word to idle. • The switch is running a supported version of software. For a list of supported software, see the compatibility matrix on the HP website: http://www.hp.
To enable NPIV on a Brocade switch, navigate to port administration and execute an advance view. Cisco switches Some Cisco switches that support NPIV do not have NPIV enabled by default. The Cisco MDS 9148 may disable NPIV when power cycled. To enable NPIV on a Cisco switch use Cisco_Device_Manager > Admin > Feature_Control or use the Cisco CLI commands show NPIV status and NPIV enable. Hardware-specific requirements for advanced path failover The NPIV settings are not needed for advanced path failover.
5 Configuring failover for the HP StoreEver ESL G3 Tape Libraries Configuring data path failover When basic data path failover is enabled, the drive is presented on the first drive Fibre Channel port, by default, and the second Fibre Channel port is a passive port. If the primary port goes down, the library transfers various settings over to the passive port and then enables the failover port.
NOTE: Any drives that belong to an access group and have hosts mapped to the second port are considered to be custom mapped. BDPF cannot be enabled on these drives until the custom mapping is removed. Advanced Data Path Failover can only be enabled on LTO-6 drives. Basic Data Path Failover can be enabled on both LTO-5 and LTO-6 drives. See the HP StoreEver Enterprise Systems Library (ESL) G3 Tape Library User Guide for more information on modifying partitions. 5.
To verify that basic or advanced data path failover is working: 1. From the Fibre Channel switch's management console, down the main Fibre Channel port to a drive. 2. From the host computers confirm that the drive is still accessible. After confirming that the drive is still accessible the host logs should show a failover event. 3. To verify that failover occurred, navigate to Monitor→Drives to launch the Drive Status dialog.
Click Drives to display the Event List, and then select the event and click Details to see more information about the failover event. The Repair tab provides detailed instructions on how to troubleshoot and fix the problem. Disabling data path failover To disable data path failover: 1. Log into the Management Console as a user with Administrator privileges. 2. Select the Physical managed view. 3. Select Setup→Partitions→Data Path Failover. 4.
For both ACPF and BCPF, you must configure a secondary control path drive and the ports on that drive will be standby control path failover ports that can be activated in the event of complete loss of connection to the primary control path drive. Standby control path drive ports are still active data path ports and can be used for tape drive access even when the drive is configured as a passive control path drive.
Configuration requirements after enabling control path failure Library configuration If Secure Manager is enabled, hosts that have been granted access to the control path device must be mapped to the first Fibre Channel port of the drive for basic control path failover and must be granted access to both ports for advanced path failover.
Verifying control path failover Before testing control path failover, verify that the configuration is correct in the Drive Status dialog, which is accessed from the Monitor→Drives menu. The Control Path column identifies the primary and secondary control path drives. The drive currently hosting control path access is marked Active. To test basic control path failover: 1. From the Physical managed view, select Monitor→Partitions→Control Path. 2.
6 Configuring failover for HP StoreEver MSL6840 Tape Libraries The MSL6480 Tape Library supports both basic and advanced path failover, though basic and advanced path failover cannot both be used in a partition at the same time. For example, advanced data path failover can be used with advanced control path failover or without control path failover, but not with basic control path failover.
Figure 7 Selecting the basic control path failover settings Configuration requirements after enabling basic control path failure When basic control path failover is enabled, the library is no longer presented as a logical unit behind the tape drive and is assigned a new Fibre Channel node name. You can find the new world wide node name on the Status > Library Status screen.
Figure 8 Status of the Partitions Configuration to verify basic control path failover After verifying library connectivity using each of the library control paths, the library control path can be moved back to the original drive if needed by power cycling the alternate control path drive. Configuring data path failover Basic and advanced data path failover can be used with the drive ports configured in loop mode or fabric mode.
Verifying data path failover After data path failover is enabled: 1. Navigate to the Status > Drive Status screen, expanding the section for the drive to see the details. 2. Expand the section for each drive to see the details. The Data Path Failover field will show the type of data path failover configured. Verify that the interface status for both ports is Login complete. 3. 30 From the host, verify connectivity to the drive.
7 Configuring failover for other HP StoreEver MSL Tape Libraries The HP StoreEver MSL2024, MSL4048, MSL8048, and MSL8096 Tape Libraries with dual-port LTO-5 or LTO-6 FC tape drives support basic data path failover. The HP StoreEver MSL2024, MSL4048, MSL8048, and MSL8096 Tape Libraries with multiple LTO-5 or LTO-6 FC tape drives support basic control path failover.
Figure 11 Standby and active ports After verifying that the configuration change was successful, verify that the hosts with access to the tape drive are still able to communicate with the drive. After verifying host access via the initial path, force a data path port failover by disconnecting the cable from the active port on the drive or using an FC switch management interface to down the port connected to the tape drive.
Figure 12 Configure library, basic control path failover configuration disabled Figure 13 (page 33) and Figure 14 (page 34) show the System configuration pane. In Figure 13 (page 33) the option for Enable Library Control Path Failover is not selectable and the option for Alternate Drive for Library Control Path Failover is set to N.A.. When the options for enabling basic control path failover are not available the configuration requirements have not been met.
Figure 14 Configure library, basic control path failover configuration available Configuration requirements after enabling control path failure When basic control path failover is enabled, the library is no longer presented as a logical unit behind the tape drive and is assigned a new Fibre Channel node name. Verifying basic control path failover After enabling basic control path failover, verify the configuration of both paths.
8 Installing and using Windows advanced path failover drivers After the host has been configured and booted, install both the HP tape and the HP changer drivers. Both the HP tape and the HP changer drivers must be installed before the advanced path failover drivers are installed. The tape and changer drivers are in the HP StoreEver Tape Drivers for Windows driver bundle available from http://www.hp.com/storage/tapecompatibility.
Installing the Windows advanced path failover drivers “Application path list ” (page 36) shows the device handles as presented to a Windows application when using the advanced path failover driver for the same hardware configuration shown in “Advanced path failover” (page 11). For the drives, both ports are mapped to the same device handle and one port is active while the other is the standby port.
NOTE: The installation requires Windows Installer version 4.5 or later. If it is not present, the installer reports the following errors: • Error 0x80070643: Failed to install MSI package • Error 0x80070643: Failed to execute MSI package. • Error 0x80070643: Failed to configure per-machine MSI package. • Error 0x80070643: Failed to configure per-machine MSI package. If these errors appear, download the Windows Installer appropriate for your version of Windows using the following link: http://www.
Figure 15 (page 38) shows the organization of the Windows drivers and the system components when an Emulex LPe12002 host bus adapter is installed. In this example, the tape drive presents both SSC (tape) and SMC (media changer) device servers. Not all drives in the library will present an SMC device server.
Multi-path intermediate class driver The intermediate class (IC) driver, HPtapeFailover_mpio, has two halves. The upper half is a virtual bus (VB) driver. This VB driver provides a virtual device which manages the paths to all of the devices that support failover. The lower half is a multi-path function driver for the devices using the hardware ID created by the lower level filter driver. Windows calls these hardware paths physical device objects (PDOs).
Figure 16 Drivers and device objects Device manager view The Windows Device Manager shows the instances of the failover paths which are configured Figure 18 (page 42). In the system shown in Figure 17 the library has two tape drives and each tape drive has two primary ports attached to a different switch. Each drive port presents both tape drive (SSC) and tape library (SMC) devices.
Figure 17 Library configuration example In this example, there are two physical tape drives in the library; each of these is shown under “Tape drives” as “Hewlett Packard LTO Ultrium-6 drive.” At boot time, the drivers discovered eight paths to supported SCSI devices in the system, although not all paths were in active use. Each drive has an SSC device (i.e., tape drive) and an SMC device (i.e., media changer).
Figure 18 Device manager window Device firmware These two Windows drivers work in conjunction with firmware in the library and tape devices. The special firmware ensures that the host computer and devices stay synchronized on state and position information. The advanced path failover device drivers hide the details of this from the rest of the system.
9 Installing and using Linux advanced path failover drivers The advanced path failover drivers for Linux replace the normal SCSI Tape and SCSI Generic drivers. The advanced path failover drivers for Linux pass all SCSI commands for devices that do not support advanced path failover through the same code path that is followed when the standard drivers are loaded and route commands for devices that do support failover through the new pfo driver.
Command line user interface The advanced path failover drivers support several configuration and diagnostic functions through a command line interface. Most of the functions are accessed by directing commands to a specific path. The following example shows the command for a specific path and the commands may be directed at any path by replacing the device identifier with the identifier from the sg or st device. For example, a command executed on pfo3 will report information for the device at sg3 and st3.
Example 1 Example paths # cat /sys/class/pfo/*/paths sg=/dev/sg0 st=/dev/st0 wwnn=50014382c6c2f001 type=tape 0:0:3:0 up wwpn=50014382c6c2f003 0:0:0:0 up wwpn=50014382c6c2f002 1:0:0:0 up wwpn=50014382c6c2f002 1:0:2:0 up wwpn=50014382c6c2f003 sg=/dev/sg1 st=none wwnn=50014382c6c2f800 type=changer session_key=cae82d14 0:0:3:1 up active_drive 0:0:0:1 up last prefer active_drive 0:0:1:1 up 0:0:2:1 up 1:0:0:1 up active_drive 1:0:1:1 up 1:0:2:1 up active_drive 1:0:3:1 up sg=/dev/sg2 st=/dev/st1 wwnn=50014382c6c2f0
echo prefer=8:0:1:0 > /sys/class/pfo/pfo5/ctrl You can disable the configuration of a preferred path by creating an empty prefer entry in the crtl file. For example: echo prefer > /sys/class/pfo/pfo5/ctrl Path rotation exercise function Path rotation is a test function that will cause the driver to change the path it uses each time the device file is closed and reopened. To cause the path to change with every open() of the /dev file, set rotate=1.
Load balancing is only applied when a tape drive device file is opened by the first thread; the paths are not reevaluated at other times. For example: • The failover algorithms do not consider other tape drive paths when selecting a path. • The path is not changed if another thread already has the tape drive device file open. • The path is not changed during data transfers.
10 Installing and using HP-UX advanced path failover drivers Advanced path failover for HP-UX is implemented by updating HP-UX drivers to support advanced path failover with the LTO-6 tape drives. The drivers function as both failover and non-failover drivers.
Example 2 Using ioscan (1M) to view tape devices # ioscan -knNfC tape Class I H/W Path Driver S/W State H/W Type Description ============================================================================= tape 18 64000/0xfa00/0xd estape CLAIMED DEVICE HP Ultrium 6-SCSI /dev/rtape/tape18_BEST /dev/rtape/tape18_BESTb /dev/rtape/tape18_BESTn /dev/rtape/tape18_BESTnb tape 20 64000/0xfa00/0xf estape CLAIMED DEVICE /dev/rtape/tape20_BEST /dev/rtape/tape20_BESTb HP Ultrium 6-SCSI /dev/rtape/tape20_BESTn /dev/rtape
Example 4 Finding HP-UX paths for drives listed on the library interface This example shows how the drive and port information displayed in the ESL G3 library web interface can be seen from the HP-UX host. On the Control Path (CP) Selection screen for a partition, the drive with serial number 82C6E4F007 is the drive with the control path for the partition.
Similarly, to find the special files for tape drives, use ioscan -knNfC tape to list the tape drives and then use ioscan -m lun -H to see information for the drive.
1. On the HP-UX host, run rmsf -H on the lunpath hardware paths that are in NO_HW state. For example: rmsf –H 0/4/0/0/0/1.0x50014380023560d4.0x1000000000000 2. Run ioscan -kfNH . For example: ioscan -kfNH 0/4/0/0/0 HP recommends only enabling or disabling advanced path failover when the library is not opened by any applications.
11 Troubleshooting failover operation To find diagnostic information about the operation of basic and advanced path failover: • Examine the library event log or RAS tickets. See “Library path failover events” (page 62) and “Additional tape drive sense codes” (page 65). • Examine the Windows Device Manager view of tape drives and media changers. See “Confirming path failover on Windows” (page 54). • Tape drive support tickets contain more detailed information that may be required by HP support.
1. 2. 3. Examine the library event or RAS ticket indicating that failover has occurred. The Serial # is that of the tape drive with the failed path and the tape drive coordinates indicate the physical location of the drive. (See information about Tape Drive Location Coordinates in the library user guide.) Use host utilities to confirm the failure. • On Windows, use the diagnostic application. 1. Launch the application, or if it is already running go to Actions→Refresh Cached Driver Data (F5). 2.
Windows system log The drivers create entries in the Windows system log when they are first started and whenever any exceptional conditions occur. These log entries are summarized in Table 3 (page 55). The symbols beginning with ‘%’ are strings filled in by the operating system when the log entry is generated.
Diagnostic application The Advanced Failover Diagnostic application communicates with the drivers to obtain the state of failover sessions. The diagnostic application is installed with the driver and can be found in the driver installation directory: C:\Program Files\Hewlett-Packard\Advanced Failover Drivers\Utils or C:\Program Files (x86)\Hewlett-Packard\Advanced Failover Drivers\ HPAdvFoDgn. The application consists of two files, HPAdvFoDgn.exe and HPtapeFailover_mpio.dll.
All other paths are passive, i.e., they pass through a passive control path drive (see “Active and passive control path drives” (page 13)). If the selected device is a tape drive, the capital A indicates the active data path, and there will be no lower case a. The Cached Driver Data pane shows the last state of the selected device that was reported by the failover drivers to the diagnostic application.
Table 4 Device information Menu Item Data Enabled Reports whether advanced path failover is enabled for the SSC device server (i.e., for the tape drive data path), and for the SMC device server (i.e., for the media changer controller path) Session Status Expected FSC: A command indicator used for management of commands that require synchronization following a link failure. PFSE: 1: The device server is using failover via this device server.
Table 5 Local SMC device server information Menu item Data Legacy Reservations SET: 1: An initiator has a legacy reservation established. 0: No initiator has a legacy reservation established. NOIV: 1: The following fields are valid. 0: The following are not valid and should be ignored.
Before the action is performed, the alert shown below is displayed. Figure 24 Clear Nexus Settings warning Enable Logging The Enable Logging action provides a mechanism for capturing run-time diagnostic information from the drivers. Figure 25 Enable Logging menu item Figure 26 Enable Logging dialog The Enable selection immediately begins capturing driver diagnostic information after OK is clicked. The Enable setting is not persistent across a system reboot.
In these file names, YYYYMMDDhhmmss is the date and time OK was clicked. After a reboot there also will be two files in the specified location: • tapeapfousbf_YYYYMMDDhhmmss.bin.NNN • tapeapfompic_YYYYMMDDhhmmss.bin.NNN In addition to the date and time, in these file names NNNN will be a number from 1 - 16. The File Size Limit value constrains the size of the log files and if the size limit is reached, the oldest log entries are replaced with the new log entries.
Legacy reservations: echo legacy > /sys/class/pfo/pfo4/ctrl Persistent reservations: echo persistent > /sys/class/pfo/pfo4/ctrl Prevent allow medium removal conditions: echo prevent > /sys/class/pfo/pfo4/ctrl Debug output control You can control the level of debug messages being sent to the console. The default output level is minimum.
Table 7 MSL6480 events related to path failover Event code Message and description Details and solution 4037 Loss of redundant data path. Verify that both FC ports are correctly cabled to the SAN. 4038 The drive configuration failed because of unsupported ADPF features selected. 4039 The drive configuration failed because of unsupported ACPF features selected. Advanced path failover, ADPF and ACPF, are only supported on LTO-6 tape drives.
Table 8 MSL library events related to path failover Event code 64 Description Details and Solution EC Unable to negotiate for NPIV. Library control path The port of the FC switch connected to the library failover has been disabled. master drive must be in NPIV mode. Verify the FC switch configuration. ED Common control or data path failover issue. Failover has been disabled. EF A data path failover-related error has occurred. FF The master drive was removed without being powered off.
Additional tape drive sense codes Tape drives supporting advanced path failover use vendor-specific SCSI additional sense codes (ASCs) to report certain conditions to the failover drivers. These additional sense codes are handled by the device driver in normal operation and should not be visible to the applications. Because these ASCs may be visible in device logs and diagnostic tools they are listed in Table 9 (page 65).
12 Advanced troubleshooting techniques The techniques in this chapter assume an advanced level of experience. Windows driver debug dump setup Generating a driver debug dump log file requires installing checked drivers and the DebugView application. Drivers with debug capability are available from HP technical support (http://www.hp.com/ support). 1. Uninstall the release drivers as explained in “Uninstalling drivers” (page 37). 2.
13 Support and other resources HP technical support Telephone numbers for worldwide technical support are listed on the HP support website: www.hp.com/support.
Table 10 Document conventions (continued) Convention Element Monospace text • File and directory names • System output • Code • Commands, their arguments, and argument values Monospace, italic text • Code variables • Command variables Monospace, bold text WARNING! CAUTION: Indicates that failure to follow directions could result in bodily harm or death. Indicates that failure to follow directions could result in damage to equipment or data.
A Known issues and workarounds Path failover issues are observed on Windows Server 2012 Symptom: On Windows Server 2012, path failover issues are observed, even after installing the advanced path failover driver and rebooting the server twice. The installation file used was: HP_StoreEver_High_Availability_Path_Failover_Driver_Windows_2012_2012_R2_Z7550-01470.
The driver does not perform speed or performance comparisons of all available paths to a device to select the best one as a primary. Symptom: The device driver does not connect to the highest performance path after a system boot or device rescan. Workaround: The driver prefers the first path in the hardware scan performed by the operating system. If a particular path is preferred, load the HBA for that path in the lower numbered hardware location.
B Data Protector device discovery with advanced path failover When using the Data Protector Autoconfigure functionality, Data Protector will discover both active and passive control and data paths to the library robot control and drive paths respectively. Data Protector will associate all active data paths with the active control path, and can be identified by the library in the list that includes the selection box. All other listed libraries will correspond to passive control paths.
Data Protector device discovery with advanced path failover
Index A advanced path failover, 6 Data Protector device discovery, 71 details, 11 installing, 16 preparing the host, 17 preparing the library, 16 prerequisites, 18 B-series switches, 19 C-Series switches, 19 H-Series switches, 19 uninstalling Windows, 37 advanced path failover drivers downloading Linux, 43 Windows, 36 installing HP-UX, 48 Linux, 43 Windows, 35 updating Linux, 43 user interface HP-UX, 48 Linux, 44 B basic control path failover details, 9 technology, 11 basic data path failover details, 8 ba
configuring data path failover, 20 diagnosing data path failover, 22 disabling control path failover, 26 disabling data path failover, 23 enabling control path failover, 24 enabling data path failover, 20 failover events, 62 verifying control path failover, 26 verifying data path failover, 21 F failover events ESL G3, 62 MSL2024, MSL4048, MSL8048, MSL8096, 64 MSL6480, 63 tape drive sense codes, 65 H help obtaining, 67 HP technical support, 67 HP-UX command line user interface, 48 drivers supporting advanc
ESL G3, 26 MSL2024, MSL4048, MSL8048, MSL8096, 34 MSL6480, 28 verifying data path failover ESL G3, 21 MSL2024, MSL4048, MSL8048, MSL8096, 31 MSL6480, 30 W websites HP, 67 HP Subscriber's Choice for Business, 68 Windows confirming failover operation, 54 downloading advanced path failover drivers, 36 driver debug dump setup, 66 driver theory of operation, 37 installing advanced path failover drivers, 35 uninstalling advanced path failover drivers, 37 Windows diagnostic application, 56 Windows diagnostic dump
