HP StoreEver Tape Libraries Failover User Guide HP Part Number: 5697-3402 Published: August 2014 Edition: Second
© Copyright 2014 Hewlett-Packard Development Company, L.P.
Contents 1 Overview..................................................................................................6 Traditional library control path and drive connections without failover..............................................7 Basic path failover....................................................................................................................8 Basic data path port failover details.......................................................................................
Enabling data path failover.................................................................................................29 Verifying data path failover.................................................................................................30 7 Configuring failover for other HP StoreEver MSL Tape Libraries.......................31 Configuring data path port failover...........................................................................................31 Enabling data path port failover.....
Report cached reservation information..................................................................................59 Debug output control..........................................................................................................59 Reading the trace log.........................................................................................................60 Library path failover events......................................................................................................
1 Overview With today’s high dependency on access to business information, safe-guarded data and limited backup windows, the reliability of the backup hardware and software is vital. Additionally, backup operations are usually automated, often done at night, and any first pass operator intervention is done remotely.
◦ Is available for the ESL G3 and MSL6480 Tape Libraries. ◦ Is not available for the 1/8 G2 Tape Autoloader nor the MSL2024, MSL4048, MSL8048, or MSL8096 Tape Libraries. Traditional library control path and drive connections without failover A typical connection for a Fibre Channel tape library using the drive to bridge commands to the library controller in a two drive tape library is shown in “Typical bridged library controller connection” (page 7).
Figure 2 Logical view of traditional configuration 1. HP Ultrium Dual Port Fibre Channel Tape Drives 2. Lun 0 at the WWPN of this drive 3. Lun 1 at the WWPN of this drive 4. Lun 0 at the WWPN of this drive 5. Host connection Basic path failover Basic path failover uses features in the tape drive and library firmware to recover paths following a link failure.
Figure 3 Data path port failover example configuration 1. Primary data path 2. Secondary data path 3. Host connection 4. HP Ultrium dual port Fibre Channel tape drive Basic control path failover details Library basic control path port failover (see Figure 4 (page 10)) may be configured with one drive hosting the active logical link to the library controller and a second drive configured as the passive logical link to the library controller.
NOTE: The MSL libraries monitor drive removal events and if the active control path drive is removed, the library will automatically move the control path to the standby drive. However, if the control path drive is not removed but SAN connectively is lost, the administrator might need to use the library remote management interface to move the control path to the standby drive.
Technology for basic control path failover HP LTO-5 and LTO-6 Fibre Channel tape drives use a technology called N-Port Identifier Virtualization (NPIV) which is defined as part of the Fibre Channel standards maintained by the INCITS/T11 working group (see clause 6) in the FC-LS-2 specification. NPIV allows a single physical port to connect to a Fibre Channel switch multiple times using multiple node and port names.
Table 1 Example path list SAN Addressed Logical Unit Port Example SCSI address Example Logical Unit Worldwide Identifier 1 Tape drive 1 Port A ID 1 LUN 0 50:01:10:a0:00:00:00:01 1 Library Controller Port A ID 1 LUN 1 50:01:10:a0:00:00:00:02 2 Tape drive 1 Port B ID 2 LUN 0 50:01:10:a0:00:00:00:01 2 Library Controller Port B ID 2 LUN 1 50:01:10:a0:00:00:00:02 1 Tape drive 2 Port A ID 3 LUN 0 50:01:10:a0:00:00:00:03 1 Library Controller Port A ID 3 LUN 1 50:01:10:a0:00:00:00:
Driver operation during normal communications The advanced path failover driver passes commands through without any command translation and with minimal additional processing in normal operation to retain the maximum possible performance. During normal communication the advanced path failover driver does not introduce additional commands that would cause delays (for example, commands to determine position) during reading and writing.
tape library user interface will show which control path drive is the current active control path drive. Library controller path failure recovery In most cases the library controller path failure recovery is the same as the tape drive path failure recovery. If all paths to the drive hosting the library controller have failed, the advanced path failover driver is capable of opening a connection to the library controller through another tape drive.
2 Enabling basic path failover To successfully enable basic path failover, complete the following: • Verify that the switch and host prerequisites are met. See “Preparing the switches and hosts for failover” (page 18).
3 Installing advanced path failover To successfully install advanced path failover, complete the following: • Prepare the SAN. See “Preparing the SAN” (page 16). • Prepare the library. See “Preparing the library” (page 16). • Prepare the host. See “Preparing the host” (page 17). Preparing the SAN To prepare the SAN, review the following: • “Minimize host access” (page 16) • “Configure redundant connections” (page 16).
that is not using an advanced path failover driver, the drive or library will reject the command with an additional sense code of 82h/93h, FAILOVER SESSION SEQUENCE ERROR. That error code should only be reported to an application if the advanced path failover drivers are not installed on that host.
4 Preparing the switches and hosts for failover Prerequisites for using basic data and control path failover: • The library drive FC ports must be attached to a Fibre Channel SAN that supports NPIV and NPIV must be enabled. LTO-5 and LTO-6 drives need an 8G connection with the fill word set to arb(ff). With 4G connections, set the fill word to idle. • The switch is running a supported version of software. For a list of supported software, see the EBS matrix on the HP website: http://www.hp.
To enable NPIV on a Brocade switch, navigate to port administration and execute an advance view. Cisco switches Some Cisco switches that support NPIV do not have NPIV enabled by default. The Cisco MDS 9148 may disable NPIV when power cycled. To enable NPIV on a Cisco switch use Cisco_Device_Manager > Admin > Feature_Control or use the Cisco CLI commands show NPIV status and NPIV enable. Hardware-specific requirements for advanced path failover The NPIV settings are not needed for advanced path failover.
5 Configuring failover for the HP StoreEver ESL G3 Tape Libraries Configuring data path failover When basic data path failover is enabled, the drive is presented on the first drive Fibre Channel port, by default, and the second Fibre Channel port is a passive port. If the primary port goes down, the library transfers various settings over to the passive port and then enables the failover port.
NOTE: Any drives that belong to an access group and have hosts mapped to the second port are considered to be custom mapped. BDPF cannot be enabled on these drives until the custom mapping is removed. Advanced Data Path Failover can only be enabled on LTO-6 drives. Basic Data Path Failover can be enabled on both LTO-5 and LTO-6 drives. See the HP StoreEver Enterprise Systems Library (ESL) G3 Tape Library User Guide for more information on modifying partitions. 5.
To verify that basic or advanced data path failover is working: 1. From the Fibre Channel switch's management console, down the main Fibre Channel port to a drive. 2. From the host computers confirm that the drive is still accessible. After confirming that the drive is still accessible the host logs should show a failover event. 3. To verify that failover occurred, navigate to Monitor→Drives to launch the Drive Status dialog.
Click Drives to display the Event List, and then select the event and click Details to see more information about the failover event. The Repair tab provides detailed instructions on how to troubleshoot and fix the problem. Disabling data path failover To disable data path failover: 1. Log into the Management Console as a user with Administrator privileges. 2. Select the Physical managed view. 3. Select Setup→Partitions→Data Path Failover. 4.
For both ACPF and BCPF, you must configure a secondary control path drive and the ports on that drive will be standby control path failover ports that can be activated in the event of complete loss of connection to the primary control path drive. Standby control path drive ports are still active data path ports and can be used for tape drive access even when the drive is configured as a passive control path drive.
Configuration requirements after enabling control path failure Library configuration If Secure Manager is enabled, hosts that have been granted access to the control path device must be mapped to the first Fibre Channel port of the drive for basic control path failover and must be granted access to both ports for advanced path failover.
Verifying control path failover Before testing control path failover, verify that the configuration is correct in the Drive Status dialog, which is accessed from the Monitor→Drives menu. The Control Path column identifies the primary and secondary control path drives. The drive currently hosting control path access is marked Active. To test basic control path failover: 1. From the Physical managed view, select Monitor→Partitions→Control Path. 2.
6 Configuring failover for HP StoreEver MSL6840 Tape Libraries The MSL6480 Tape Library supports both basic and advanced path failover, though basic and advanced path failover cannot both be used in a partition at the same time. For example, advanced data path failover can be used with advanced control path failover or without control path failover, but not with basic control path failover.
Figure 7 Selecting the basic control path failover settings Configuration requirements after enabling basic control path failure When basic control path failover is enabled, the library is no longer presented as a logical unit behind the tape drive and is assigned a new Fibre Channel node name. You can find the new world wide node name on the Status > Library Status screen.
Figure 8 Status of the Partitions Configuration to verify basic control path failover After verifying library connectivity using each of the library control paths, the library control path can be moved back to the original drive if needed by power cycling the alternate control path drive. Configuring data path failover Basic and advanced data path failover can be used with the drive ports configured in loop mode or fabric mode.
Verifying data path failover After data path failover is enabled: 1. Navigate to the Status > Drive Status screen, expanding the section for the drive to see the details. 2. Expand the section for each drive to see the details. The Data Path Failover field will show the type of data path failover configured. Verify that the interface status for both ports is Login complete. 3. 30 From the host, verify connectivity to the drive.
7 Configuring failover for other HP StoreEver MSL Tape Libraries The HP StoreEver 1/8 G2 Tape Autoloader and MSL2024, MSL4048, MSL8048, and MSL8096 Tape Libraries with dual-port LTO-5 and later generation FC tape drives support basic data path failover. The HP StoreEver MSL2024, MSL4048, MSL8048, and MSL8096 Tape Libraries with multiple LTO-5 or later generation FC tape drives support basic control path failover.
Figure 11 Standby and active ports After verifying that the configuration change was successful, verify that the hosts with access to the tape drive are still able to communicate with the drive. After verifying host access via the initial path, force a data path port failover by disconnecting the cable from the active port on the drive or using an FC switch management interface to down the port connected to the tape drive.
Figure 12 Configure library, basic control path failover configuration disabled Figure 13 (page 33) and Figure 14 (page 34) show the System configuration pane. In Figure 13 (page 33) the option for Enable Library Control Path Failover is not selectable and the option for Alternate Drive for Library Control Path Failover is set to N.A.. When the options for enabling basic control path failover are not available the configuration requirements have not been met.
Figure 14 Configure library, basic control path failover configuration available Configuration requirements after enabling control path failure When basic control path failover is enabled, the library is no longer presented as a logical unit behind the tape drive and is assigned a new Fibre Channel node name. Verifying basic control path failover After enabling basic control path failover, verify the configuration of both paths.
8 Installing and using Windows advanced path failover drivers After the host has been configured and booted, install both the HP tape and the HP changer drivers. Both the HP tape and the HP changer drivers must be installed before the advanced path failover drivers are installed. The tape and changer drivers are in the HP StoreEver Tape Drivers for Windows driver bundle available from http://www.hp.com/storage/tapecompatibility.
Table 2 Application path list (continued) Addressed Logical Unit Device Handle Path SAN Port Example Logical Unit Worldwide Identifier Passive 2 Port B 50:01:10:a0:00:00:00:02 Passive 1 Port A 50:01:10:a0:00:00:00:02 Downloading and installing the drivers 1. Locate the drivers on the HP website: a. Navigate to http://www.hp.com/support/storage. b. Click Tape Storage. c. Click Enterprise Class Tape Libraries (for ESL G3) or Tape Libraries (for MSL6480). d.
Windows driver theory of operation The advanced path failover drivers support data path failover and control path failover on HP LTO-6 drives integrated into HP tape libraries. A failover device driver must identify devices accessible over the different paths available and associate paths with devices so that it is able to mask duplicate paths and present a single path to an ISV application.
Figure 15 Driver and system organization Storage bus filter driver The storage bus filter driver is named HPtapeFailover_filt and manages the OS notifications indicating that devices have been added or removed. The storage bus filter driver bus enumerator is usually "PCI". It attaches as an upper filter to all HBA drivers. It then monitors the creation of raw devices, looking for supported failover capable devices. Requests to all other devices are passed through unchanged.
"SCSI" bus and the device associated with the virtual device that manages the paths is shown as a "ROOT" device. The lower half of this driver acts like a class driver and it attaches to the devices presented by the lower driver. The upper half acts like a virtual bus driver, presenting abstractions of devices to the layers above.
Figure 16 Drivers and device objects Device manager view The Windows Device Manager shows the instances of the failover paths which are configured Figure 18 (page 42). In the system shown in Figure 17 the library has two tape drives and each tape drive has two primary ports attached to a different switch. Each drive port presents both tape drive (SSC) and tape library (SMC) devices.
Figure 17 Library configuration example In this example, there are two physical tape drives in the library; each of these is shown under “Tape drives” as “Hewlett Packard LTO Ultrium-6 drive.” At boot time, the drivers discovered eight paths to supported SCSI devices in the system, although not all paths were in active use. Each drive has an SSC device (i.e., tape drive) and an SMC device (i.e., media changer).
Figure 18 Device manager window Device firmware These two Windows drivers work in conjunction with firmware in the library and tape devices. The special firmware ensures that the host computer and devices stay synchronized on state and position information. The advanced path failover device drivers hide the details of this from the rest of the system.
9 Installing and using Linux advanced path failover drivers Installing advanced path failover drivers The advanced path failover drivers for Linux replace the normal SCSI Tape and SCSI Generic drivers. The advanced path failover drivers for Linux pass all SCSI commands for devices that do not support advanced path failover through the same code path that is followed when the standard drivers are loaded and route commands for devices that do support failover through the new pfo driver. 1.
Command line user interface The advanced path failover drivers support several configuration and diagnostic functions through a command line interface. Most of the functions are accessed by directing commands to a specific path. The following example shows the command for a specific path and the commands may be directed at any path by replacing the device identifier with the identifier from the sg or st device. For example, a command executed on pfo3 will report information for the device at sg3 and st3.
Example 1 Example paths # cat /sys/class/pfo/*/paths sg=/dev/sg0 st=/dev/st0 wwnn=50014382c6c2f001 type=tape 0:0:3:0 up wwpn=50014382c6c2f003 0:0:0:0 up wwpn=50014382c6c2f002 1:0:0:0 up wwpn=50014382c6c2f002 1:0:2:0 up wwpn=50014382c6c2f003 sg=/dev/sg1 st=none wwnn=50014382c6c2f800 type=changer session_key=cae82d14 0:0:3:1 up active_drive 0:0:0:1 up last prefer active_drive 0:0:1:1 up 0:0:2:1 up 1:0:0:1 up active_drive 1:0:1:1 up 1:0:2:1 up active_drive 1:0:3:1 up sg=/dev/sg2 st=/dev/st1 wwnn=50014382c6c2f0
echo prefer=8:0:1:0 > /sys/class/pfo/pfo5/ctrl You can disable the configuration of a preferred path by creating an empty prefer entry in the crtl file. For example: echo prefer > /sys/class/pfo/pfo5/ctrl Path rotation exercise function Path rotation is a test function that will cause the driver to change the path it uses each time the device file is closed and reopened. To cause the path to change with every open() of the /dev file, set rotate=1.
10 Installing and using HP-UX advanced path failover drivers Advanced path failover for HP-UX is implemented by updating HP-UX drivers to support advanced path failover with the LTO-6 tape drives. The drivers function as both failover and non-failover drivers.
Example 2 Using ioscan (1M) to view tape devices # ioscan -knNfC tape Class I H/W Path Driver S/W State H/W Type Description ============================================================================= tape 18 64000/0xfa00/0xd estape CLAIMED DEVICE HP Ultrium 6-SCSI /dev/rtape/tape18_BEST /dev/rtape/tape18_BESTb /dev/rtape/tape18_BESTn /dev/rtape/tape18_BESTnb tape 20 64000/0xfa00/0xf estape CLAIMED DEVICE /dev/rtape/tape20_BEST /dev/rtape/tape20_BESTb HP Ultrium 6-SCSI /dev/rtape/tape20_BESTn /dev/rtape
Example 4 Finding HP-UX paths for drives listed on the library interface This example shows how the drive and port information displayed in the ESL G3 library web interface can be seen from the HP-UX host. On the Control Path (CP) Selection screen for a partition, the drive with serial number 82C6E4F007 is the drive with the control path for the partition.
Similarly, to find the special files for tape drives, use ioscan -knNfC tape to list the tape drives and then use ioscan -m lun -H to see information for the drive.
1. On the HP-UX host, run rmsf -H on the lunpath hardware paths that are in NO_HW state. For example: rmsf –H 0/4/0/0/0/1.0x50014380023560d4.0x1000000000000 2. Run ioscan -kfNH . For example: ioscan -kfNH 0/4/0/0/0 HP recommends only enabling or disabling advanced path failover when the library is not opened by any applications.
11 Troubleshooting failover operation To find diagnostic information about the operation of basic and advanced path failover: • Examine the library event log or RAS tickets. See “Library path failover events” (page 60) and “Additional tape drive sense codes” (page 61). • Examine the Windows Device Manager view of tape drives and media changers. See “Confirming path failover on Windows” (page 53). • Tape drive support tickets contain more detailed information that may be required by HP support.
2. 3. Use host utilities to confirm the failure. • On Windows, use the diagnostic application. 1. Launch the application, or if it is already running go to Actions→Refresh Cached Driver Data (F5). 2. In the Devices pane, locate the drive whose serial number matches that from the support ticket. Select the device. 3. In the Paths pane, confirm that not all of the ports are available.
Table 3 Event Log Entries (continued) Event Severity Message Path failure detected Error Path failure detected on SCSI Port %s Bus %3 Target Id %4 Logical Unit %5 Failover session established Informational Failover session established on SCSI Port %2 SCSI Bus %3 Target Id %4 Logical Unit %5 Failover session establishment error Error Failover session establishment error on SCSI Port %2 SCSI Bus %3 Target Id %4 Logical Unit %5. Reason code %6.
Figure 19 Advanced failover diagnostic application The Devices pane lists the tape drives and media changers that are associated with the path failover drivers. This panel lists all devices that are capable of supporting advanced path failover regardless of whether they are configured to use advanced path failover. In the example above Changer0 and Changer2 are paths to an ESL G3 library partition that is not using advanced path failover so two entries are shown for the same changer.
The bottom pane shows failover session information that has been requested for the selected device. Actions There are several actions that can be performed by the diagnostic application. • Display worldwide port and node names (WWNs). • Refresh failover drivers’ state information held by the diagnostic application. • Request current failover information from tape and changer devices. • Clear reservations on changer devices which had been set by hosts that are now offline.
Table 4 Device information Menu Item Data Enabled Reports whether advanced path failover is enabled for the SSC device server (i.e., for the tape drive data path), and for the SMC device server (i.e., for the media changer controller path) Session Status Expected FSC: A command indicator used for management of commands that require synchronization following a link failure. PFSE: 1: The device server is using failover via this device server.
Table 5 Local SMC device server information (continued) Menu item Data SET: 1: A persistent reservation is established. 0: No persistent reservation is established. PTPL A: 1: Persist through power loss capability is activated. 0: Persist through power loss capability Full status descriptors (see SPC-4) Prevent Medium Removal SET: 1: An initiator has a prevent medium removal established. 0: No initiator has a prevent medium removal established.
Memory dump setup To configure Windows to produce a memory dump, see http://support.microsoft.com/kb/254649. The configuration process includes specifying the name and location for the dump file, as well as the type of the dump. The type should be a kernel memory dump, although the larger complete memory dump is acceptable. Do not select the small memory dump. If a bug check occurs, then after a reboot the file can be retrieved and sent for diagnosis.
Reading the trace log The failover driver maintains an internal buffer of the most recent significant events. To read the trace log, cat the ‘trace’ file. For example: cat /sys/bus/scsi/drivers/pfo/trace Library path failover events When the library detects a path failover, it generates an appropriate event. Path failover events do not always occur immediately following a path break.
Table 7 MSL6480 events related to path failover (continued) Event code Message and description Details and solution 4048 The drive configuration failed because of unsupported BDPF feature selected. Disable basic path failover for this drive or replace the drive with one supporting this feature. 4049 The drive configuration failed because of unsupported BCPF feature selected. 4050 Basic data path failover occurred.
Because these ASCs may be visible in device logs and diagnostic tools they are listed in Table 9 (page 62). Table 9 Advanced failover vendor specific additional sense codes ASC ASCQ Description Meaning 82h1 93h FAILOVER SESSION SEQUENCE ERROR Failover is enabled but the device driver has not opened a failover capable connection to the device. Check that the failover driver is installed and attempt to reopen the device.
12 Advanced troubleshooting techniques The techniques in this chapter assume an advanced level of experience. Windows driver debug dump setup Generating a driver debug dump log file requires installing checked drivers and the DebugView application. Drivers with debug capability are available from HP technical support (http://www.hp.com/ support). 1. Uninstall the release drivers as explained in “Uninstalling drivers” (page 36). 2.
13 Support and other resources HP technical support Telephone numbers for worldwide technical support are listed on the HP support website: www.hp.com/support.
Table 10 Document conventions (continued) Convention Element Monospace text • File and directory names • System output • Code • Commands, their arguments, and argument values Monospace, italic text • Code variables • Command variables Monospace, bold text WARNING! CAUTION: Indicates that failure to follow directions could result in bodily harm or death. Indicates that failure to follow directions could result in damage to equipment or data.
A Data Protector device discovery with advanced path failover When using the Data Protector Autoconfigure functionality, Data Protector will discover both active and passive control and data paths to the library robot control and drive paths respectively. Data Protector will associate all active data paths with the active control path, and can be identified by the library in the list that includes the selection box. All other listed libraries will correspond to passive control paths.
Index Symbols 1/8 G2 autoloader configuring data path failover, 31 enabling data path failover, 31 failover events, 61 verifying data path failover, 31 A advanced path failover, 6 Data Protector device discovery, 66 details, 11 installing, 16 preparing the host, 17 preparing the library, 16 prerequisites, 18 B-series switches, 19 C-Series switches, 19 H-Series switches, 19 uninstalling Windows, 36 advanced path failover drivers downloading Linux, 43 Windows, 36 installing HP-UX, 47 Linux, 43 Windows, 35 up
enabling control path failover ESL G3, 24 MSL2024, MSL4048, MSL8048, MSL8096, 32 MSL6480, 27 enabling data path failover 1/8 G2 autoloader, 31 ESL G3, 20 MSL2024, MSL4048, MSL8048, MSL8096, 31 MSL6480, 29 ESL E-Series Tape Libraries, 6 ESL G3 configuring control path failover, 23, 25 configuring data path failover, 20 diagnosing data path failover, 22 disabling control path failover, 26 disabling data path failover, 23 enabling control path failover, 24 enabling data path failover, 20 failover events, 60 ve
library path failover events, 60 Linux cached reservation information, 59 Linux debug output control, 59 Linux path rotation exercise function, 46 Linux trace log, 60 locating a failed path, 52 Windows diagnostic application, 54 Windows diagnostic dumps, 58 Windows driver debug dump setup, 63 Windows logs and tools, 53 V verifying control path failover ESL G3, 26 MSL2024, MSL4048, MSL8048, MSL8096, 34 MSL6480, 28 verifying data path failover 1/8 G2 autoloader, 31 ESL G3, 21 MSL2024, MSL4048, MSL8048, MSL80
