The Next Generation Mass Storage Stack HP-UX 11i v3 Abstract ..............................................................................................................................................3 Publication History ............................................................................................................................3 Background .........................................................................................................................................
Legacy Device Special Files and Hardware Paths ...........................................................................26 Tunable Kernel Parameters ...........................................................................................................26 Changes to Existing Procedures........................................................................................................26 Replacing a Disk..................................................................................................
Abstract This white paper discusses the next generation mass storage subsystem in HP-UX 11i v3. It is intended for system administrators or operators who have experience with HP-UX and manage mass storage devices such as disks and tape drives.
In addition, there are four appendixes: Summary of Changes A short summary of the user-visible changes to commands, device file names, and tunables. Using ioscan A sample output of the ioscan command using several of the new options. Using scsimgr The most common examples of setting device tunables. Interpreting lunpath hardware paths How to interpret the address elements for the new lunpath hardware paths.
Feature Number of I/O busses Number of LUNs supported LUN size Number of I/O paths to a single LUN File system size supported HP-UX 11i v3 no limit 16384 (architectural limit of 16 M) > 2 TB (subject to driver support) 32 32 TB at initial release (subject to volume manager support) (architectural limit of 8 EB) HP-UX 11i v2 256 8192 active 2 TB 8 32 TB In addition, the mass storage stack has been enhanced to take advantage of large multi-CPU server configurations for greater parallelism.
binding, the DSF name and minor number no longer have hardware path information encoded in them. Instead, they have a simple device instance number such as /dev/disk/disk3. Rather than mapping to the LUN hardware path, the persistent DSF maps to the LUN WorldWide Identifier (WWID), a resource on the LUN similar to a serial number. The WWID is unique to a LUN and does not change for that LUN. If any of a LUN lunpaths change, the DSF bound to the LUN does not change – hence, the DSF has agile addressing.
For information on the interaction between native multi-pathing and third-party multi-pathing products such as SecurePath and VERITAS DMP, see the impacts section on Third-Party Multi-Pathing Products. If you are using LVM alternate links to manage multi-pathed LVM disks, see the impacts section on Logical Volume Manager. Command changes include the following: ioscan –m lun ioscan –P health scsimgr Displays the multiple paths to a LUN. Indicates whether a lunpath is disabled.
Binds a specific driver to a hardware path. This new command accepts a hardware path, a driver name, and an optional instance number to assign. The drivers involved must support online rebinding; otherwise, the rebinding can be deferred to next reboot. iobind ioscan ioscan ioscan ioscan -b -B -r -P property lssf -s rmsf -x The ioscan command with the existing –M and –H options also binds a driver to a hardware path, but does not attempt to unbind any existing driver at the specified hardware path.
Several new command options provide performance information: scsimgr get_stat glance –U sar sar sar sar –H -L -t -R Displays mass storage statistics, either at a global level or for a particular LUN, target, or controller. Displays HBA port-level statistics. The motif version of glance (gpm) has a new IO by HBA report under the Disk details menu. Displays HBA port-level activity. Displays separate activity for each lunpath. Displays tape activity.
Figure 1: Legacy View # ioscan -fkn Class I H/W Path Driver S/W State H/W Type Description ============================================================================= ext_bus 2 0/1/1/0 mpt CLAIMED INTERFACE SCSI Ultra320 target 0 0/1/1/0.0 tgt CLAIMED DEVICE disk 1 0/1/1/0.0.0 sdisk CLAIMED DEVICE HP 36.4GST336753LC /dev/dsk/c2t0d0 /dev/rdsk/c2t0d0 target 1 0/1/1/0.1 tgt CLAIMED DEVICE disk 0 0/1/1/0.1.0 sdisk CLAIMED DEVICE HP 36.
Agile View The agile view includes a new persistent type of disk and tape DSF, and represents hardware pathing to disk and tape devices in ways that support larger configurations and enable transparent multi-pathing. The naming of DSFs and hardware paths is shown in Figure 2. Like the legacy view diagram, the HBAs are on the left, disks are on the right, with connections in between them.
Figure 2: Agile View # ioscan –N -fkn Class I H/W Path Driver S/W State H/W Type Description ============================================================================ ext_bus 2 0/1/1/0 mpt CLAIMED INTERFACE SCSI Ultra320 tgtpath 4 0/1/1/0.0x0 estp CLAIMED TGT_PATH parallel_scsi target lunpath 1 0/1/1/0.0x0.0x0 eslpt CLAIMED LUN_PATH LUN path for disk48 tgtpath 3 0/1/1/0.0x1 estp CLAIMED TGT_PATH parallel_scsi target lunpath 0 0/1/1/0.0x1.
Hardware Path Details This section contains more information about the hardware paths shown in the Legacy View and Agile View. In HP-UX 11i v3, there are three different types of paths to a device: legacy hardware path, lunpath hardware path, and LUN hardware path. All three are numeric strings of hardware components, with each number typically representing the location of a hardware component on the path to the device. Legacy hardware path This is the format used in releases prior to HP-UX 11i v3.
The following three formats are three different representations of the same LUN. A single LUN can have all of the following addresses. • • • 0/2/1/0.1.6.0.0.2.1 0/4/1/0.1.6.0.0.2.1 0/2/1/0.0x500805f300083891.0x4011000000000000 0/4/1/0.0x500805f300083891.0x4011000000000000 64000/0xfa00/0x3 In the sample configuration, the LUN is multi-pathed with two hardware paths.
DSF Details There are two types of DSFs for mass storage: legacy DSFs and persistent DSFs. Both can be used to access a given mass storage device independently, and both can coexist on a system. Legacy DSF The only type of mass storage DSF available in releases prior to HP-UX 11i v3, so it is associated with the legacy view. It is locked to a particular lunpath, and does not support agile addressing. Each lunpath requires a different DSF, so a multi-pathed LUN has multiple DSFs, one for each lunpath.
instance number refers to exactly one device. Note: The instance numbers might not be sequentially ordered based on hardware path, as instance numbers are assigned in the order of discovery in HP-UX 11i v3. DSFs for disks are now under the /dev/[r]disk directory, and DSFs for tape devices are under the /dev/rtape directory. This avoids any confusion with existing legacy DSFs, which reside in /dev/[r]dsk and /dev/rmt.
Changes to I/O and Mass Storage Subsystems This section describes the user-visible changes to the mass storage commands and configuration interfaces. Changes to Existing Commands Existing I/O commands are backward compatible; next generation features are controlled by new options to existing commands or by the mode of the command arguments. The changes are described in detail in the command manpages, and are summarized in the following sections.
–P property_name Displays the value of the given property for all entries. Use the –d, -C, -I or –H options or a device file name to limit the ioscan output to particular entries.
/dev/dsk/c11t0d1 –m hwpath [-H hardware_path] Displays the mappings between a LUN legacy, lunpath, and LUN hardware paths. Any type of hardware path can be specified. If no hardware path is specified, the mapping is displayed for all entries. For example: # ioscan -m hwpath -H 0/4/1/1.124.2.0.0.0.1 Lun H/W Path Lunpath H/W Path Legacy H/W Path ==================================================================== 64000/0xfa00/0x1 0/4/1/1.0x50001fe15008a738.0x4001000000000000 0/4/1/1.124.2.0.0.0.
–L Creates legacy DSFs and enables the support of legacy DSFs. When used with the–v option, insf –L reports whether the legacy mode is enabled or disabled. For example: # insf –Lv insf: Legacy mode enabled lssf(1M) The lssf command displays information about a DSF. For persistent DSFs, the output shows the LUN hardware path. For example: # lssf /dev/dsk/c2t0d0 sdisk card instance 2 SCSI target 0 SCSI LUN 0 section 0 at address 0/1/1/0.0.
–x Removes stale entries from the I/O configuration (devices for which there appears to be no hardware) and associated DSFs. –L Removes legacy DSFs and their entries in the I/O configuration. This option is typically used during migration in conjunction with insf. The rmsf -L command disables legacy addressing support by removing all legacy DSFs and legacy configuration information, while insf –L re-enables legacy addressing support by recreating the legacy DSFs and configuration information.
–v ioconfig_file Checks an I/O configuration file for corruption. The ioconfig_file argument specifies the file to check. -A With the –f option, performs Critical Resource Analysis (CRA) on all the devices corresponding to the hardware paths specified in infile. You must specify the -f option with this option. Instance numbers are reassigned as specified in infile, only if CRA of all these devices report SUCCESS.
Syntax: mediainit –S [–c scrub_character] [-t scrub_count] special_file -S Scrubs the disk. You must specify the -c and -t options with the –S option. If you do not specify both options, the device is scrubbed three times. -c scrub_character Specifies the character to use to scrub the disk. You can specify any character between 0-9, a-z, or AZ. You must specify this option with the -S option. -t scrub_count Specifies the number of times to scrub the disk. You must specify this option with the -S option.
iobind(1M) The iobind command unbinds the driver from an existing LUN and binds a new driver to it. You can use iobind to override a default driver selection made by the operating system. In releases prior to HP-UX 11i v3, forcing a particular driver to claim a specified device was done through driver statements in the /stand/system file. This method is supported in HP-UX 11 v3, but the iobind command supersedes it.
scsimgr(1M) The scsimgr command provides a single command line interface to manage and diagnose the mass storage stack. It is designed to work with persistent DSFs, but also works with legacy DSFs for a limited set of operations.
Legacy Device Special Files and Hardware Paths As of HP-UX 11i v3, the legacy view is deprecated, and all its components—naming conventions, DSFs, and hardware paths—will be obsoleted in future versions of HP-UX. Tunable Kernel Parameters The following tunable kernel parameters related to mass storage are obsolete in HP-UX 11i v3: scsi_max_qdepth Controls the maximum number of I/O operations that a LUN can queue up for execution. It has been replaced with the max_q_depth scsimgr attribute.
• • • Replacing a Disk Force-Binding a Driver Disabling Multi-Pathing on Legacy DSFs Replacing a Disk Replacing an internal disk or a disk in a JBOD, even at the same hardware path, changes its WWID because the WWID is tied to the physical disk. Since persistent DSFs are tied to the LUN WWID, the mass storage stack considers the replacement a new disk and creates new DSFs for it.
# ioscan –r -H device_hw_path Disabling Multi-Pathing on Legacy DSFs By default, the multi-pathing feature of the next generation mass storage stack enables I/O requests and I/O control operations to be processed along any hardware path to a LUN. Even if legacy DSFs are used for I/O, requests can still be routed through a different hardware path. This maximizes availability and parallelism.
fcmsutil(1M) The replace_dsk option to the fcmsutil command is no longer available. It is replaced with a combination of the replace_wwid and replace_leg_dsf options to the scsimgr command. The replace_wwid option validates the change of a LUN associated with a lunpath. The replace_leg_dsf option validates the change of a LUN associated with a legacy DSF. For example, to validate the LUN association of a lunpath with hardware path 0/1/1/0.1.1, enter the following command: # scsimgr replace_wwid -H 0/1/1/0.
Figure 3: SMH Toggle Global Device View Screen The setting of this toggle persists between HP SMH sessions by creating a small file when the user selects the agile view for a system. This file is used by various HP SMH applications including pdweb and fsweb. For additional information on HP SMH, see the HP System Management Homepage white paper in For more information. Ignite-UX The Ignite-UX product operates in the agile view to take advantage of the mass storage stack multi-pathing and failover features.
Figure 4: Ignite-UX Root Disk Selection Screen Note: Starting with Ignite-UX version C.7.2.x, the Ignite-UX variables inventory_block_path and inventory_block_protocols enable you to hide disks from the I/O configuration process, based respectively on the disk hardware path or the I/O protocol used to access them. This inventory blocking reduces the number of disks displayed in the selection screens while improving the I/O inventory performance.
Figure 5: Ignite-UX More Info Screen During an installation, Ignite-UX configures the root, dump, and swap devices with their agile addressing representation. This means that after installing, the commands that display the boot, dump, and swap devices show them using the agile form. In previous releases, Ignite-UX associated the same DSFs to the same I/O components for any identically configured systems. This is not true in HP-UX 11i v3, given the parallelism of the I/O system.
disk using its agile form. If the system was updated rather than installed, lvlnboot displays the root disk in its legacy form. A new LVM utility, vgdsf, converts existing volume group configuration information from legacy to persistent DSFs. You can manage alternate links to a multi-pathed device outside of LVM using the next generation mass storage stack. By default, the mass storage stack balances the I/O load across all available paths to a disk.
• When using the boot loader, you must use the 32-bit format. For example, to boot from a device with a lunpath hardware path of 1/2/0.0x51ef000012003400.0x4000000000000000, enter the following: ISL> hpux boot (1/2/0.0x51ef0000.0x12003400.0x40000000.0x0;0)/stand/vmunix To display hardware paths in BCH format while the operating system is running, use the ioscan command with the –e option. The BCH hardware paths are printed in both hexadecimal and decimal. For example: # ioscan –N –k –e –H 0/4/1/1.
PowerPath PowerPath does not manage multi-pathing on HP-UX 11i v3. Native multi-pathing is used for EMC devices. For information on migrating from PowerPath to native multi-pathing, see the HP-UX 11i v2 to 11i v3 Mass Storage Stack Update Guide in For More Information. Dynamic Multipathing VERITAS Dynamic Multipathing (DMP) does not support the agile view. See the VERITAS Volume Manager 4.1 Administrator's Guide for information about the coexistence of DMP with native multi-pathing in HP-UX 11i v3.
Performance • • • Improved I/O bandwidth through native multi-pathing with choice of load balancing algorithms Faster I/O scan time through parallelized scanning of mass storage devices Faster dump time by writing concurrently to multiple dump devices Backward Compatibility HP-UX 11i v3 provides full backward compatibility when using legacy mode, except in very limited cases. Legacy DSFs remain on the system and continue to work as before, unless you explicitly disable legacy mode.
DSFs. For example: legacy DSFs.
Glossary Agile Addressing The ability to address a LUN with the same device special file regardless of the physical location of the LUN or the number of paths leading to it. The DSF for a LUN remains the same even if the LUN is moved from one HBA to another, moved from one switch or hub port to another, presented using a different target port to the host, or configured with multiple hardware paths. Also referred to as persistent LUN binding.
Lunpath Hardware Path The representation of a hardware path for a mass storage device. It is identical in format to a legacy hardware path up to the HBA. After the HBA, additional addressing is represented in hexadecimal format. The string 0/2/1/0.0x50001fe1500170ac.0x4017000000000000 is an example of a lunpath hardware path. Multi-pathing The detection, correlation, and coordinated usage of multiple hardware paths leading to the same LUN. Pass-Through DSF A DSF that enables direct access to SCSI devices.
Appendix A: Summary of Changes This appendix summarizes the changes to the mass storage stack in HP-UX 11i v3. For details on command changes, see the command manpages. New Commands Command scsimgr(1M) Usage Manages and troubleshoots mass storage.
iofind(1M) iobind(1M) io_redirect_dsf(1M) Finds references to legacy view in system files, and optionally updates them to agile view.
insf(1M) lssf(1M) mksf(1M) rmsf(1M) ioinit(1M) mediainit(1M) Installs special (device) files. -L Enables legacy addressing by restoring legacy DSFs (if legacy view was disabled with rmsf –L) -Lv Displays whether the legacy mode is enabled or disabled Lists special files. -s Lists stale special files -c dsf Performs a critical resource analysis on a DSF Makes a DSF. -P Creates a pass-through persistent DSF for the esdisk, estape, and eschgr drivers Removes a DSF.
max_retries Configures the maximum number of times to retry a failing I/O request, subject to finite retry policy.
Appendix B: Using ioscan Note: The output from the following examples are trimmed for readability. To display the agile view, use ioscan with the –N option.
lunpath 6 0/2/1/0.0x500805f300083891.0x4011000000000000 eslpt NO_HW LUN_PATH LUN path for disk49 tgtpath 5 0/2/1/0.0x500805f300083899 estp CLAIMED TGT_PATH fibre_channel target served by td driver lunpath 2 0/2/1/0.0x500805f300083899.0x0 eslpt NO_HW LUN_PATH LUN path for ctl4 lunpath 22 0/2/1/0.0x500805f300083899.0x4007000000000000 eslpt NO_HW LUN_PATH LUN path for disk59 lunpath 21 0/2/1/0.0x500805f300083899.0x4008000000000000 eslpt NO_HW LUN_PATH LUN path for disk58 lunpath 19 0/2/1/0.0x500805f300083899.
tty 1 graphics processor processor ba ipmi 0 0 1 7 0 tty 2 tty 3 acpi_node 0 esvroot 0 disk 47 disk 48 ctl 4 disk 49 disk 50 disk 57 disk 58 disk 59 disk 60 /dev/cul0p0 /dev/mux0 /dev/ttyd0p0 /dev/diag/mux0 /dev/tty0p0 0/6/1/1 asio0 CLAIMED INTERFACE PCI Serial (103c1048) /dev/MPdiag1 /dev/mux1 /dev/tty1p4 /dev/cul1p0 /dev/tty1p0 /dev/ttyd1p0 /dev/diag/mux1 /dev/tty1p2 0/6/2/0 gvid_core CLAIMED INTERFACE PCI Display (10025159) 120 processor CLAIMED PROCESSOR Processor 121 processor
To show the mapping of LUN hardware paths to lunpath hardware paths, use ioscan with the –m lun option. For example: # ioscan -m lun Class I Lun H/W Path Driver S/W State H/W Type Health Description ======================================================================= disk 47 64000/0xfa00/0x0 esdisk CLAIMED DEVICE online HP 36.4GST336753LC 0/1/1/0.0x1.
To show the mapping of a persistent DSF to its equivalent legacy paths, and vice versa, use ioscan with the –m dsf option.
Appendix C: Using scsimgr This appendix presents the syntax of the scsimgr command, a partial list of available operations, and some use cases. It is not intended to be a complete description of the scsimgr command. For information on the scsimgr command capabilities, see scsimgr(1M) and the white paper described in For More Information. Introduction to scsimgr The scsimgr command manages and troubleshoots SCSI objects and the mass storage subsystem.
This construct allows attributes to be set or retrieved for a set of objects. For example, a scope of "/escsi/esdisk" refers to all disks on a system and a scope of "/escsi/esdisk/0x0/HP " refers to all HP disks on a system. Keyword Provides additional scoping information, such as selecting all lunpaths for a LUN. Argument Provides additional information specific to the command. For example, when using the get_attr command, argument specifies which attribute to print.
STATUS INFORMATION FOR LUN : 64000/0xfa00/0x3 Generic Status Information SCSI services internal state Device type EVPD page 0x83 description code EVPD page 0x83 description association EVPD page 0x83 description type World Wide Identifier (WWID) af Serial number Vendor id Product id Product revision Other properties SPC protocol revision Open count (includes chr/blk/pass-thru/class) Raw open count (includes class/pass-thru) Pass-thru opens LUN path count Active LUN paths Standby LUN paths Failed LUN paths M
LUN path probe failures Target path offline events from I/F driver Target path online events from I/F driver Port id change events from I/F driver Target Warm Reset events Target Cold Reset events Target Warm reset failures Target Cold Reset failures Invalid port id changes Total I/Os processed Last time cleared = = = = = = = = = = = 0 0 0 0 0 0 0 0 0 533 N/A I/F Common Statistics: Offline events WARM/COLD target resets Time of last WARM/COLD target reset Bytes read Bytes written Outstanding I/Os = = =
I/Os flushed Check condition status Busy status Queue full status Reservation conflicts Invalid Request status Select Timeout status Incomplete status No Resource status Target Path Offline status IO Timeout status IO Aborted status Reset Occurred status Unrecognized CDB status Bad Sense data status Deferred errors Recovered errors NOT READY sense status Medium errors Hardware errors Illegal request sense status Unit Attentions Data protect sense status Blank checks Vendor specific sense status Copy aborted
LUN path : lunpath4 Class Instance Hardware path SCSI transport protocol State Last Open or Close state = = = = = = lunpath 4 0/2/1/0.0x500805f300083899.0x4010000000000000 fibre_channel UNOPEN ACTIVE LUN path : lunpath7 Class Instance Hardware path SCSI transport protocol State Last Open or Close state = = = = = = lunpath 7 0/2/1/0.0x500805f300083891.
SCSI ATTRIBUTES FOR LUN PATH : 0/2/1/0.0x500805f300083899.0x4011000000000000 name = lunid current =0x4011000000000000 (LUN # 17, Flat Space Addressing) default = saved = Selecting Load Balancing Algorithms By setting the load_bal_policy attribute, you can select one of the following load balancing algorithms: least_cmd_load round_robin cl_round_robin weighted_rr preferred_path pref_tport closest_path path_lockdown Directs I/O requests through the hardware path with the least outstanding I/O requests.
Assigning Aliases to LUNs There are two methods of assigning a user-friendly name to a LUN. The first method is to set the alias attribute for a LUN.
Appendix D: Interpreting Lunpath Hardware Paths This appendix describes how to interpret address elements in lunpath hardware paths. It also shows how to map between the legacy hardware path and lunpath hardware path formats. Components of Lunpath Hardware Paths The lunpath hardware path has three parts: the HBA path, the target address, and the LUN address. • • • The HBA path is composed of a series of bus-nexus addresses separated by a slash (/).
# fcmsutil /dev/td0 get remote all Target N_Port_id is Target state Symbolic Name Port Type SLER Capable (supports Retry & TRID) Target Port World Wide Name Target Node World Wide Name = = = = = = = 0x010700 DVS_READY N_PORT NO 0x500805f300083899 0x500805f300083890 … The LUN address for a Fibre Channel device is a 64-bit LUNid.
To convert a lunpath hardware path to a legacy hardware path, use the scsimgr get_info command on the target hardware path to display the port_id, which contains the domain, area, and port values. Extract the controller, target, and lun values from the LUN id. The following example uses lunpath hardware path 0/2/1/0.0x500805f300083899.0x4011000000000000. # scsimgr get_info -H 0/2/1/0.0x500805f300083899 STATUS INFORMATION FOR TARGET PATH : 0/2/1/0.
name = lunid current = 0x0 (LUN # 0, Peripheral Addressing) default = saved = The legacy hardware path format for SAS disks has the following format: HBA_path.target.virtual_bus.virtual_target.lun You can use the sasmgr command to correlate SAS addresses with legacy hardware paths, by using the lun qualifier without the –N option.
For more information To learn more about the agile view and mass storage on HP-UX, see the following documents on the HP Business Support Center: http://www.hp.com/go/hpux-core-docs Click HP-UX 11i v3.
© 2009 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.
