Planning Guide hp StorageWorks SAN High Availability Product Version: FW v06.xx/HAFM SW v08.02.00 Fourth Edition (July 2004) Part Number: AA–RS2DD–TE/623–000005–001 This guide introduces HP Fibre Channel switching products, storage area networks (SANs), and Fibre Channel technologies. It describes HP StorageWorks directors and edge switches and the High Availability Fabric Manager (HAFM) application.
© Copyright 2001-2004 Hewlett-Packard Development Company, L.P. Hewlett-Packard Company makes no warranty of any kind with regard to this material, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. Hewlett-Packard shall not be liable for errors contained herein or for incidental or consequential damages in connection with the furnishing, performance, or use of this material.
Contents Contents About this Guide. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Intended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Related Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Contents 2 Product Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39 Product Management Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 HAFM Appliance Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 HAFM Appliance Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Ethernet Hub .
Contents Core-to-Edge Fabric. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 Fabric Island. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 Planning a Fibre Channel Fabric Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Fabric Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Contents Data Transmission Distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Cost-Effectiveness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Device or Cable Restrictions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Extended-Distance Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . High-Availability Considerations . . .
Contents SANtegrity Binding Planning Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . Open Trunking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Full Volatility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . CNT WAN Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Element Manager Application. . .
Contents 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 8 Edge Switch 2/32 (front view) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Edge Switch 2/32 (rear view). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Out-of-band product management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Contents 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 iSCSI WAN Extension. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Port Properties Dialog Box . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Node List View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Enterprise Fabric Mode Dialog Box . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Contents 10 SAN High Availability Planning Guide
About This Guide About this Guide This planning guide provides information to help you plan the acquisition and installation of one or more of the following Hewlett-Packard (HP) products: About this Guide ■ HP StorageWorks Director 2/64 ■ HP StorageWorks Director 2/140 ■ HP StorageWorks Edge Switch 2/12 ■ HP StorageWorks Edge Switch 2/16 ■ HP StorageWorks Edge Switch 2/24 ■ HP StorageWorks Edge Switch 2/32 ■ High Availability Fabric Manager (HAFM) application “About this Guide” topics includ
About this Guide Overview This section covers the following topics: ■ Intended Audience ■ Related Documentation Intended Audience This book is intended for use by configuration and installation planners who are experienced with the following: ■ System administration. ■ Customer engineering. ■ Project management. Related Documentation For a list of corresponding documentation, see the “Related Documents” section of the Release Notes that came with the product.
About this Guide Conventions Conventions consist of the following: ■ Document Conventions ■ Text Symbols ■ Equipment Symbols Document Conventions This document follows the conventions in Table 1.
About this Guide Tip: Text in a tip provides additional help to readers by providing nonessential or optional techniques, procedures, or shortcuts. Note: Text set off in this manner presents commentary, sidelights, or interesting points of information. Equipment Symbols The following equipment symbols may be found on hardware for which this guide pertains.
About this Guide Power supplies or systems marked with these symbols indicate the presence of multiple sources of power. WARNING: To reduce the risk of personal injury from electrical shock, remove all power cords to completely disconnect power from the power supplies and systems. Any product or assembly marked with these symbols indicates that the component exceeds the recommended weight for one individual to handle safely.
About this Guide Rack Stability Rack stability protects personnel and equipment. WARNING: To reduce the risk of personal injury or damage to the equipment, be sure that: ■ The leveling jacks are extended to the floor. ■ The full weight of the rack rests on the leveling jacks. ■ In single rack installations, the stabilizing feet are attached to the rack. ■ In multiple rack installations, the racks are coupled. ■ Only one rack component is extended at any time.
About this Guide Getting Help If you still have a question after reading this guide, contact an HP authorized service provider or access our web site: http://www.hp.com. HP Technical Support Telephone numbers for worldwide technical support are listed on the following HP web site: http://www.hp.com/support/. From this web site, select the country of origin. Note: For continuous quality improvement, calls may be recorded or monitored.
About this Guide 18 SAN High Availability Planning Guide
Introduction to HP Fibre Channel Products 1 This chapter introduces Hewlett-Packard (HP) Fibre Channel switching products that allow deployment and implementation of a storage area network (SAN) topology in a Fibre Channel Protocol (FCP) or IBM fiber connection (FICON) environment. HP offers several switch alternatives to build a robust and scalable SAN infrastructure that meets the customer’s data center requirements.
Introduction to HP Fibre Channel Products Product Overview HP provides three broad classes of Fibre Channel switching products, as follows: ■ Directors — A director is a high port count, high-bandwidth switch designed with fully redundant, hot-swappable field replaceable units (FRUs) that provide an availability of 99.999% (approximately five minutes of down time per year). HP offers the 64-port StorageWorks Director 2/64 and 140-port StorageWorks Director 2/140.
Introduction to HP Fibre Channel Products Managed products and the HAFM appliance communicate on a local area network (LAN) through one or more HP-supplied 10/100 Base-T Ethernet hubs. Hubs are daisy-chained as required to provide additional Ethernet connections as more directors or switches are installed on a customer network. Refer to “Product Management” on page 39 for information about managing products through the HAFM appliance.
Introduction to HP Fibre Channel Products Directors Directors provide high-performance, dynamic connections between end devices such as servers, mass storage devices, and peripherals in a Fibre Channel switched network. Directors also support mainframe and open systems interconnection (OSI) computing environments and provide data transmission and flow control between device node ports (N_Ports) as dictated by the Fibre Channel Physical and Signaling Interface (FC-PH 4.3).
Introduction to HP Fibre Channel Products ■ Low latency — The latency is less than 2.5 microseconds between transmission of a frame at a source port to receipt of the frame at the corresponding destination port (with no port contention). ■ Local control — Actions taking place at a device N_Port seldom affect operation of other ports; therefore, servers need to maintain little or no information about other connected devices in a SAN.
Introduction to HP Fibre Channel Products — Class F transmission service that is used by multiple directors to communicate across ISLs to configure, control, and coordinate the behavior of a multi-switch fabric. Director 2/64 The Director 2/64 is a second-generation, enterprise-class switch that provides switched fabric connectivity for up to 64 Fibre Channel devices. Figure 1 illustrates the front of the director. Each UPM card provides four 2.
Introduction to HP Fibre Channel Products 1 2 Redundant fan modules Power module assembly with AC power switch 3 Redundant serial crossbar (SBAR) assemblies Figure 2: Director 2/64 (rear view) The director provides a modular design that enables quick removal and replacement of FRUs. The power module assembly at the rear of the director also provides a 9-pin, D-type subminiature (DSUB) maintenance port for connection to a local terminal or remote terminal.
Introduction to HP Fibre Channel Products 1 2 Front bezel Power and system error LEDs 3 4 UPM cards (32) CTP cards Figure 3: Director 2/140 (front view) 26 SAN High Availability Planning Guide
Introduction to HP Fibre Channel Products Figure 4 illustrates the rear of the director. 1 2 3 Fan modules UPM cards (3) Maintenance port 4 5 6 SBAR assemblies AC modules Power supplies Figure 4: Director 2/140 (rear view) The director provides a modular design that enables quick removal and replacement of FRUs. The rear of the assembly provides a 9-pin, D-type subminiature (DSUB) maintenance port for connection to a local terminal or remote terminal.
Introduction to HP Fibre Channel Products Edge Switches Like directors, edge switches also provide high-performance, dynamic connections between end devices in a Fibre Channel switched network. Edge switches also support mainframe and OSI computing environments. Through non-blocking architecture and limited FRU redundancy, edge switches also offer high availability and high-performance bandwidth.
Introduction to HP Fibre Channel Products Shortwave laser transceivers are available for transferring data over multimode fiber-optic cable. Longwave laser transceivers are available for transferring data over single-mode fiber-optic cable. Fiber-optic cables attach to small form factor pluggable transceivers (SFP) with duplex LC connectors. Green and amber status light-emitting diodes (LEDs) are associated with each port.
Introduction to HP Fibre Channel Products Shortwave laser transceivers are available for transferring data over multimode fiber-optic cable. Longwave laser transceivers are available for transferring data over single-mode fiber-optic cable. Fiber-optic cables attach to switch port transceivers with duplex LC connectors. Green and amber status light-emitting diodes (LEDs) are associated with each port.
Introduction to HP Fibre Channel Products Edge Switch 2/24 The Edge Switch 2/24 provides 2.125 Gbps fabric connectivity for to up to 24 Fibre Channel devices. Figure 9 illustrates the front of the switch. Shortwave laser transceivers are available for transferring data over multimode fiber-optic cable. Longwave laser transceivers are available for transferring data over single-mode fiber-optic cable. Fiber-optic cables attach to switch port transceivers with duplex LC connectors.
Introduction to HP Fibre Channel Products 1 Power supplies with internal cooling fans 2 Maintenance Port Figure 10: Edge Switch 2/24 (rear view) Edge Switch 2/32 The Edge Switch 2/32 provides 2.125 Gbps fabric connectivity for to up to 32 Fibre Channel devices. Figure 11 illustrates the front of the switch. Shortwave laser transceivers are available for transferring data over multimode fiber-optic cable.
Introduction to HP Fibre Channel Products 1 2 3 Initial machine load (IML) button Ethernet LAN connector SFP transceivers 4 5 6 Port LEDs Error LED (amber) Power LED (green) Figure 11: Edge Switch 2/32 (front view) Figure 12 illustrates the rear of the switch. The FRUs on the rear panel include two power supplies and four individual cooling fan FRUs.
Introduction to HP Fibre Channel Products Product Features In addition to the characteristics and performance features described in this chapter, directors and switches managed by HP also provide a variety of: ■ Connectivity Features ■ Security Features ■ Serviceability Features Connectivity Features Directors, switches, and the associated HAFM and Element Manager applications support the following Fibre Channel connectivity features: 34 ■ Any-to-any connectivity — Director and switch software con
Introduction to HP Fibre Channel Products ■ State change notification — Directors and switches support a state change notification function that allows attached N_Ports to request notification when other N_Ports change operational state. ■ Port binding — Directors and switches support a feature that binds an attached Fibre Channel device to a specified port through the device’s World Wide Name (WWN).
Introduction to HP Fibre Channel Products ■ SANtegrity™ Binding - This feature enhances data security in large and complex SANs that have numerous fabrics and devices provided by multiple OEMs. The feature allows or prohibits director or switch attachment to fabrics (fabric binding) and Fibre Channel device attachment to directors or switches (switch binding).
Introduction to HP Fibre Channel Products Note: For directors and switches installed in some legacy environments, call-home notification requires installation of HP Proactive Service software. This service is offered at no additional charge for subsystems covered under an on-site warranty or on-site storage hardware support contract. To register or order Proactive Service software, contact your HP customer service representative.
Introduction to HP Fibre Channel Products ■ 38 SNMP management using the Fibre Channel Fabric Element MIB (Version 3.1), Transmission Control Protocol/Internet Protocol (TCP/IP), MIB-II definition (RFC 1213), or a product-specific MIB that runs on each director or switch. Up to six authorized management workstations can be configured through the associated Element Manager application to receive unsolicited SNMP trap messages. The trap messages indicate operational state changes and failure conditions.
Product Management 2 This chapter describes management of HP directors and edge switches.
Product Management Product Management Overview Out-of-band (non-Fibre Channel) management access to HP products is provided through two Ethernet LAN connections to director control processor (CTP) cards or a single connection to a director front panel. The following out-of-band management access methods are provided: ■ Management through the HAFM application.
Product Management ■ Management through a PC-based Telnet session using the CLI. Any platform that supports Telnet client software can be used. Figure 13 illustrates an example of out-of-band product management. In the figure, the managed product is a Director 2/64. The customer intranet could be an HP Ethernet hub providing device connectivity.
Product Management ■ Management through the product’s Fibre Connection (FICON) management server (FMS) that communicates with one of the following: — IBM® System Automation for OS/390™ (SA OS/390™) operating system resident on a System/390® (S/390) Parallel Enterprise Server™ Generation 5 or Generation 6. — IBM z/OS® operating system resident on an eServer™ zSeries® 800 (z800), zSeries 900 (z900), or zSeries 990 (z990) processor.
Product Management HAFM Appliance Description The HAFM appliance is a one rack unit (1u) high, LAN-accessed, rack-mount unit that provides a central point of control for up to 48 LAN-connected directors or switches. However, note that the maximum number of switches per storage area network (SAN) fabric is different. For the latest supported topology limits, contact your local HP sales representative. The HAFM appliance is accessed through a LAN-attached PC and standard Web browser.
Product Management United States English is the only language supported by the HAFM and Element Manager applications. The HAFM appliance provides two auto-detecting 10/100 Mbps Ethernet LAN connectors (RJ-45 adapters). The first adapter (LAN 1) attaches (optionally) to a public customer intranet to allow access from remote user workstations. The second adapter (LAN 2) attaches to a private LAN segment containing switches or managed HP products.
Product Management Ethernet Hub The HAFM appliance and managed directors and switches can be connected through a 10/100 Base-T Ethernet hub. Figure 16 illustrates the 12-port hub. The hub can be ordered from HP and is installed at the top front of the equipment rack.
Product Management 46 ■ Video card supporting 256 colors at 800 x 600 pixel resolution. ■ Ethernet network adapter. ■ Java-enabled Internet browser, such as Microsoft Internet Explorer (Version 4.0 or later) or Netscape Navigator (Version 4.6 or later).
Product Management Product Firmware Director or edge switch firmware provides services that manage and maintain Fibre Channel connections between ports. Although the product hardware transmits Fibre Channel frames between source and destination ports, the firmware maintains routing tables required by the hardware to perform these switching functions.
Product Management 48 ■ Operating System Services — This function includes boot and loader software, a command line monitor for engineering fault isolation, a serial maintenance port driver, and other support for the product operating system.
Product Management Backup and Restore Features The HAFM appliance provides two backup and restore features. One feature backs up (to the HAFM appliance) or restores the configuration file stored in nonvolatile random-access memory (NVRAM) on a director or switch CTP card. The other feature backs up to a backup drive or restores the entire HAFM data (HafmData) directory.
Product Management Product Software This section describes the Management Services and HAFM applications. The HAFM application includes the Element Manager application for each product (Director 2/64, Director 2/140, Edge Switch 2/16, Edge Switch 2/24, and Edge Switch 2/32). The applications provide a GUI and management services for monitoring and controlling directors and switches.
Product Management The HAFM appliance also provides hypertext transfer protocol (HTTP) server functionality. Use of this protocol with a standard Web server allows the download of client HAFM and Element Manager applications from the HAFM appliance to remote workstations. The server is configured to limit the maximum number of concurrent connections to eight.
Product Management HAFM Main Window The HAFM management application opens automatically when the management server desktop is accessed, and the HAFM application main window opens by default. Figure 17: HAFM Main Window The main window provides the following: 52 ■ Menu bar — Commands at the top of the window provide drop-down menu selections to perform functions for SAN devices, including editing, viewing, planning, discovery, configuration, and monitoring.
Product Management ■ Product list — When the View tab is selected, the product list at the left side of the window displays a list of discovered devices and associated properties. ■ Physical map — When the View tab is selected, the physical map at the right side of the window depicts the SAN topology, discovered devices, and color-coded links.
Product Management Element Manager Application The Element Manager application works in conjunction with the HAFM application, and is a Java-based GUI for managing and monitoring multiple directors or switches. The application operates locally on the HAFM appliance, or through a network connection from a remote PC or workstation.
Product Management The graphical representation of the product emulates the hardware configuration and operational status of the corresponding real product. For example, if a director or switch is fully redundant and fully populated, this configuration is reflected in the Hardware View. Colored symbols display on the graphical FRUs to represent failed or degraded status. The colors and shapes are consistent with status displays on other windows in the HAFM and Element Manager applications.
Product Management Embedded Web Server Interface With product firmware version 1.2 (or later) installed, administrators and operators with a browser-capable PC and an Internet connection can monitor and manage the director or switch through an EWS interface. The interface provides a GUI similar to the Element Manager application and supports product configuration, statistics monitoring, and basic operation.
Product Management Figure 20: View Panel (Embedded Web Server interface) Task selection tabs display at the top of the panel, a graphical representation of product hardware (front and rear) displays at the right side of the panel, and menu selections (View, Configure, Monitor, Operations, and Help) display at the left side of the panel. The task selection tabs allow users to perform director or switch-specific tasks.
Product Management Command Line Interface The CLI provides a director and switch management alternative to the HAFM application, Element Manager application, and EWS user interface. The interface allows users to access application functions by entering commands through a PC-attached telnet session. Any platform that supports telnet client software can be used. The primary purpose of the CLI is to automate management of several directors or switches using scripts.
Planning Considerations for Fibre Channel Topologies 3 A storage area network (SAN) is typically defined as a network of shared storage resources that can be allocated throughout a heterogeneous environment. This chapter describes planning considerations for incorporating Hewlett-Packard (HP) switching products into Fibre Channel SAN topologies.
Planning Considerations for Fibre Channel Topologies Fibre Channel Topologies The Director 2/64, Director 2/140, Edge Switch 2/12, Edge Switch 2/16, Edge Switch 2/24, and Edge Switch 2/32 support point-to-point and multi-switch fabric topologies. The Edge Switch 2/12 and Edge Switch 2/24 indirectly support arbitrated loop topology. A combination of these topologies (hybrid topology) is also supported.
Planning Considerations for Fibre Channel Topologies ■ Multiswitch fabric — This topology provides the ability to connect directors and edge switches through expansion ports (E_Ports) or interswitch links (ISLs) to form a Fibre Channel fabric. Director or switch elements receive data from a device, and, based on the destination N_Port address, route the data through the fabric (and possibly through multiple switch elements) to the destination device.
Planning Considerations for Fibre Channel Topologies Planning for Point-to-Point Connectivity Point-to-point Fibre Channel topology consists of two device N_Ports communicating by a direct connection through a director or edge switch. The product-operational software provides the ability to configure a dedicated point-to-point connection by binding a director or switch port to a device World Wide Name (WWN).
Planning Considerations for Fibre Channel Topologies Characteristics of Arbitrated Loop Operation When implementing Fibre Channel arbitrated loop topology, consideration must be given to switch operating mode, device connectivity, and loop configuration.
Planning Considerations for Fibre Channel Topologies Figure 21: Shared mode operation ■ Switched mode — When set to switched mode or by default, the switch bypasses full loop arbitration and enables frame transmission through logical connected device pairs. Connections can be established between H_Port pairs, or between an H_Port and fabric loop port (FL_Port).
Planning Considerations for Fibre Channel Topologies Figure 22: Switched mode operation Public Versus Private Devices Arbitrated loop switches support connection of public and private arbitrated loop devices as follows: ■ Public device — A loop device that can transmit a fabric login (FLOGI) command to the switch, receive acknowledgement from the switch’s login server, register with the switch’s name server, and communicate with fabric-attached devices is a public device.
Planning Considerations for Fibre Channel Topologies Figure 23: Public device connectivity Public devices support normal fabric operational requirements, such as fabric busy and reject conditions, frame multiplexing, and frame delivery order. ■ Private device — A loop device that cannot transmit an FLOGI command to the switch nor communicate with fabric-attached devices is a private device. As shown in Figure 24, device D2 is a private loop device that cannot communicate with any fabric-attached device.
Planning Considerations for Fibre Channel Topologies Figure 24: Private device connectivity Private devices communicate only with other devices on the same arbitrated loop, and interconnected public and private devices can communicate with each other. Such intermixed devices establish operating parameters and loop topology configuration through a port login (PLOGI) command exchange rather than through the switch’s name server. Be aware that public device-to-private device communication may cause problems.
Planning Considerations for Fibre Channel Topologies other, and public devices attached to the loop can communicate with fabric-attached devices. FL_Port operation is not affected by the switch operating mode (shared or switched). Public loop connectivity is illustrated in Figure 25. Figure 25: Public loop connectivity ■ Private loop — A private loop is not connected to a switched fabric, and the switch’s embedded E_Port and FL_Port are inactive.
Planning Considerations for Fibre Channel Topologies Planning for Private Arbitrated Loop Connectivity Private arbitrated loop topology supports the clustering of isolated servers and storage subsystems into workgroup or departmental SANs. This topology is well-suited to small-sized and mid-sized configurations where modest connectivity levels and high data transmission speeds are required.
Planning Considerations for Fibre Channel Topologies between device D1 and server S1 travels through a loop that consists of all eight H_Ports, device D1, device D2, and server S1. Each H_Port not participating in the communication pair and the NL_Port on device D2 provide a repeater function that allows frames to pass around the loop at the full switch bandwidth. Part B of Figure 27 shows the logical equivalent of this arbitrated loop.
Planning Considerations for Fibre Channel Topologies Although connection of additional devices to a loop does not impact switch bandwidth (1.0625 Gbps), it does adversely impact overall loop performance because part of the bandwidth is dedicated to overhead instead of information transmission.
Planning Considerations for Fibre Channel Topologies Although the architectural limit of a Fibre Channel arbitrated loop is 125 devices, 32 or fewer devices should be attached to the switch to avoid adversely impacting loop performance. In particular, avoid attaching an excess number of servers or high-bandwidth storage devices. Switched Mode Operation When set to switched mode (default setting), a loop switch enables frame transmission through multiple point-to-point connected pairs.
Planning Considerations for Fibre Channel Topologies Loop Switch H_Ports H_Ports Hub S1 Hub S2 D1 Hub S3 D2 Hub S4 D3 Hub S5 D4 Hub S6 D5 Hub S7 D6 Hub S8 D7 D8 Figure 30: Switched mode operation with eight independent looplets When communication within two or more looplets ceases, a device attached to one looplet can be switched to communicate with a device attached to another looplet.
Planning Considerations for Fibre Channel Topologies ■ Consider the data traffic capacity of the department or workgroup (normal and peak load) as part of the switch planning and installation process. Such capacity planning: — Ensures loop traffic is distributed and balanced across servers and storage devices. — Identifies traffic bottlenecks and provides for alternate connectivity solutions if required.
Planning Considerations for Fibre Channel Topologies Planning for Fabric-Attached Loop Connectivity Public arbitrated loop topology supports the connection of workgroup or departmental FC-AL devices to a switched fabric through a loop switch B_Port. This topology is well suited for: ■ Providing connectivity between a workgroup or departmental SAN and a switched fabric, thus implementing connectivity of FC-AL devices to fabric devices at the core of the enterprise.
Planning Considerations for Fibre Channel Topologies The switch B_Port provides a single 1.0625 Gbps ISL to an E_Port on a director or edge switch. Direct ISL connectivity between loop switches (with or without a redundant B_Port connection to a director or edge switch) is generally not supported. However, a director or edge switch does support the connection of multiple, independent switches. Figure 31 shows a configuration of two loop switches attached to a director.
Planning Considerations for Fibre Channel Topologies H_Ports Arbitration Cycle D1 (30 MBps) Idle Burst Data (D1) S1 (10 MBps) Burst Data (D2) Burst Data (D3) Burst Data (S1) D2 (30 MBps) D3 (30 MBps) ISL (1 Gbps) B_Port Figure 32: ISL bandwidth limitation Three 30-megabyte per second (MBps) tape drives and one 10-MBps server are attached to a switch. Each device uses only a portion of the bandwidth of its respective H_Port connection.
Planning Considerations for Fibre Channel Topologies Figure 33: Server consolidation Tape Device Consolidation Providing fabric connectivity for multiple FC-AL tape drives by attaching them individually to a Fibre Channel director is likewise not a cost-effective solution. A practical solution is to consolidate the tape drives on an inexpensive loop switch, and then connect the switch to a single director or edge switch E_Port.
Planning Considerations for Fibre Channel Topologies Figure 34: Tape drive consolidation SAN High Availability Planning Guide 79
Planning Considerations for Fibre Channel Topologies Planning for Multi-Switch Fabric Support A Fibre Channel topology that consists of one or more interconnected director or switch elements is called a fabric. The product operational software provides the ability to interconnect directors and switches (through E_Port connections) to form a multi-switch fabric. Support of multi-switch fabric operation is a major feature of a director or edge switch.
Planning Considerations for Fibre Channel Topologies A multi-switch fabric is typically complex and provides the facilities to maintain routing to all device N_Ports attached to the fabric, handle flow control, and satisfy the requirements of the classes of Fibre Channel service that are supported. Fabric Topology Limits Operation of multiple directors or switches in a fabric topology is subject to the following topology limits. Consider the impact of these limits when planning the fabric.
Planning Considerations for Fibre Channel Topologies Note: The hop count is equal to the number of ISL connections traversed in a single path, not the total number of ISL connections between devices. As shown in Figure 35, the number of ISL connections between Switch S1 and S2 is 4, while the number of hops is 1. Factors to Consider When Implementing a Fabric Topology Director and switch-based fabrics offer scalable, high-performance, and high-availability connectivity solutions for the enterprise.
Planning Considerations for Fibre Channel Topologies — Distance limitations can be increased by using multiple fabric elements. Each director or switch retransmits received signals, thus performing a repeater and multiplexer function. Distance limitations can also be increased by using a variety of local area network (LAN), metropolitan area network (MAN), or wide area network (WAN) extension technologies.
Planning Considerations for Fibre Channel Topologies OpenTrunking feature key. For information about the feature and managing multiple ISLs, refer to “Open Trunking” on page 167 and “Large Fabric Design Implications” on page 109.
Planning Considerations for Fibre Channel Topologies ■ Zoning — For multi-switch fabrics, zoning is configured on a fabric-wide basis. Changes to the zoning configuration apply to all directors and switches in the fabric. To ensure the zoning configuration is maintained, certain rules are enforced when two or more elements are connected through ISLs to form a fabric, or when two or more fabrics are joined. For additional information, refer to “Configuring Zones” on page 156.
Planning Considerations for Fibre Channel Topologies ■ Fabric WWN assignment — The Fabric Manager application identifies fabrics using a fabric WWN. The fabric WWN is the same as the WWN of the fabric’s principal switch. If a new principal switch is selected because of a change to the fabric topology, the fabric WWN changes to the WWN of the newly selected principal switch.
Planning Considerations for Fibre Channel Topologies ■ Path selection — Directors and switches are not manually configured with data transmission paths to each other. Participating fabric elements automatically exchange information to determine the fabric topology and resulting minimum-hop data transfer paths through the fabric. These paths route Fibre Channel frames between devices attached to the fabric and enable operation of the fabric services firmware on each director or switch.
Planning Considerations for Fibre Channel Topologies Note: Activating a preferred path can result in receipt of out-of-order frames if the preferred path differs from the current path, if input and output (I/O) is active from the source port, and if congestion is present on the current path. A rerouting delay parameter can be enabled at the Element Manager application to ensure the director or switch provides correct frame order delivery.
Planning Considerations for Fibre Channel Topologies — No response from attached switch — After a fabric is created, each element in the fabric periodically verifies operation of all attached switches and directors. An ISL segments if a director or switch does not respond to a verification request. — ELP retransmission failure timeout — A director or switch that exhibits a hardware failure or connectivity problem cannot transmit or receive Class F frames.
Planning Considerations for Fibre Channel Topologies — Fabric A zoned and Fabric B zoned — The fabrics join successfully only if the zone sets can be merged. If the fabrics cannot join, the connecting E_Ports segment and the fabrics remain independent. Zone sets for two directors or switches are compatible (the fabrics can join) only if the zone names for each fabric element are unique.
Planning Considerations for Fibre Channel Topologies Fabric Topologies Several topologies exist from which to build a Fibre Channel fabric infrastructure. This section describes the most effective fabric topologies and provides guidance on when to deploy each topology. The topologies are effective for a wide variety of applications, are extensively tested by HP, and are deployed in several customer environments.
Planning Considerations for Fibre Channel Topologies TM 10/100 RST 31 30 29 28 27 26 25 24 23 22 TM 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 5 3 1 PWR ERR 6 4 2 0 10/100 RST TM PWR TM ERR Interswitch Link Fabric Connection Figure 36: Cascaded fabric One design variation is to use more than one ISL between fabric elements. This eliminates ISLs as a single point of failure and greatly increases the fabric design reliability.
Planning Considerations for Fibre Channel Topologies TM TM TM TM TM 10/100 RST TM PWR ERR 10/100 RST 10/100 TM RST 31 30 PWR ERR 29 28 27 26 25 24 23 22 TM 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 5 3 1 PWR ERR 6 4 2 0 Interswitch Link Fabric Connection Figure 37: Ring fabric Ring fabrics are generally more expensive than cascaded fabrics, are also easy to deploy, provide a simple solution to add additional fabric devices, and can solve hop-count problems in
Planning Considerations for Fibre Channel Topologies TM TM TM TM Interswitch Link Fabric Connection Figure 38: Full mesh fabric Full-mesh fabrics provide increased resiliency over cascaded or ring fabrics and are well suited for applications that require any-to-any connectivity. If a single ISL fails, traffic is automatically routed through an alternate path. Mesh fabrics also form effective backbones to which other SAN islands can be connected.
Planning Considerations for Fibre Channel Topologies A modified or partial-mesh fabric is similar to a full-mesh fabric, but each switch does not have to be directly connected to every other switch in the fabric. The fabric is still resilient to failure but does not carry a cost premium for unused or redundant ISLs. In addition, partial-mesh fabrics scale easier than full-mesh fabrics.
Planning Considerations for Fibre Channel Topologies Tier 2 Devices TM TM TM Edge Switches 10/100 RST 10/100 RST 31 29 27 25 10/100 23 30 TM PWR ERR Tier 1 Device Core Director 28 26 24 22 31 29 27 25 23 TM 21 RST 20 TM 21 10/100 19 18 17 16 RST 15 14 13 12 30 TM 11 10 9 8 7 5 3 28 26 24 1 PWR ERR 6 4 2 22 20 19 18 17 16 15 14 13 12 11 10 9 8 7 5 3 1 PWR ERR 6 4 2 0 PWR 0 ERR Tier 1 Device Core Director TM TM TM TM TM
Planning Considerations for Fibre Channel Topologies Tier 3 Devices Edge Switches TM TM TM 10/100 RST 31 29 27 25 10/100 23 TM 21 RST 30 TM 28 26 PWR ERR 24 22 20 10/100 19 18 17 16 RST 15 14 13 12 TM 11 10 9 8 7 5 3 1 PWR ERR 6 4 2 PWR 0 ERR TM TM Core Directors Core Directors Tier 1 Devices Tier 1 Devices TM TM Edge Switches TM TM TM 10/100 RST 10/100 31 29 RST 27 10/100 25 23 RST 30 28 26 24 PWR ERR 22 31 29 27 TM 21 19 17
Planning Considerations for Fibre Channel Topologies Each edge switch connects (through at least one ISL) to each core switch but not to other edge switches. There are typically more device connections to an edge switch than ISL connections; therefore, edge switches act as consolidation points for servers and storage devices. The ratio of ISLs to device connections for each switch is a function of device performance. For additional information, refer to “FCP and FICON in a Single Fabric” on page 110.
Planning Considerations for Fibre Channel Topologies Planning a Fibre Channel Fabric Topology To be effective, the fabric topology design must: ■ Solve the customer’s business problem and provide the required level of performance. ■ Meet the customer’s requirements for high availability. ■ Be scalable to meet future requirements.
Planning Considerations for Fibre Channel Topologies However, when multiple directors or switches are connected through a fabric ISL that multiplexes traffic from several devices, significant potential for congestion arises. To minimize congestion, factors such as application I/O profiles, ISL oversubscription, and device locality must be included in the fabric design. Application I/O Profiles Understanding application I/O characteristics is essential to SAN, fabric, and ISL design.
Planning Considerations for Fibre Channel Topologies ISL Oversubscription ISL oversubscription (or congestion) occurs when multiplexed traffic from several devices is transmitted across a single ISL. When an ISL is oversubscribed, fabric elements use fairness algorithms to interleave data frames from multiple devices, thus giving fractional bandwidth to the affected devices. Although all devices are serviced, ISL and fabric performance is reduced. Figure 41 illustrates ISL oversubscription.
Planning Considerations for Fibre Channel Topologies ■ Install an additional ISL — A second ISL can be installed to balance the traffic load between fabric elements. Two ISLs are sufficient to support the bandwidth of both NT servers operating at peak load. ■ Upgrade the existing ISL — Fabric element software, firmware, and hardware can be upgraded to support a 2.125 Gbps bandwidth traffic load between fabric elements. A 2.
Planning Considerations for Fibre Channel Topologies When designing a core-to-edge fabric, servers and storage devices that support such bandwidth-intensive applications should be attached to core directors as Tier 1 devices. As a best practices policy (assuming 1.0625 Gbps ISLs), devices that generate a sustained output of 35 MBps or higher are candidates for Tier 1 connectivity. IBM FICON devices also must use Tier 1 connectivity.
Planning Considerations for Fibre Channel Topologies Performance Tuning When designing or tuning a fabric for performance, it is critical to understand application I/O characteristics so that: ■ Device output in Gbps does not oversubscribe ISLs, leading to fabric congestion. ■ Device output in IOPS does not result in a connectivity scheme that exceeds fan-out ratios, leading to port congestion.
Planning Considerations for Fibre Channel Topologies ■ Local Tier 1 devices — A video server application with I/O capabilities of 40 MBps and 2,000 IOPS must be connected to the fabric. Because the application is critical and high bandwidth (in excess of 35 MBps), the server and associated storage are directly attached to the core director as Tier 1 devices. No ISLs are used for server-to-storage connectivity.
Planning Considerations for Fibre Channel Topologies ■ Nonresilient single fabric — Directors and switches are connected to form a single fabric that contains at least one single point of failure (fabric element or ISL). Such a failure causes the fabric to fail and segment into two or more smaller fabrics. A cascaded fabric topology (Figure 36) illustrates this design.
Planning Considerations for Fibre Channel Topologies 10/100 10/100 RST 10/100 RST 10/100 RST TM RST TM TM TM PWR PWR PWR PWR ERR ERR ERR ERR Fabric “A” Fabric “B” TM TM 10/100 10/100 RST 10/100 RST RST 10/100 RST TM TM TM TM PWR ERR PWR ERR PWR PWR ERR ERR Interswitch Link Fabric Connection Figure 45: Redundant fabrics Some dual-attached devices support active-active paths, while others support only active-passive paths.
Planning Considerations for Fibre Channel Topologies Obtaining Professional Services Planning and implementing a multi-switch fabric can be a complex and difficult task. HP recommends that you obtain planning assistance from our professional services organization before implementing a fabric topology.
Planning Considerations for Fibre Channel Topologies Fabric Topology Design Considerations This section discusses additional fabric topology design considerations, including: ■ Large Fabric Design Implications ■ FCP and FICON in a Single Fabric ■ Multiple Data Transmission Speeds in a Single Fabric ■ Fibre Channel Distance Extension Large Fabric Design Implications Businesses are experiencing an unprecedented growth of information and the requirement to maintain that information online.
Planning Considerations for Fibre Channel Topologies Because of these problems, a fabric with a high ISL count is more difficult to build. Note that the fabric problem is not directly related to the large number of fabric elements but to the large number of ISLs associated with the elements. Fabric build concerns currently limit the combined number of directors and switches to about 24.
Planning Considerations for Fibre Channel Topologies Director or Switch Management When intermixing FCP and FICON protocols, it must be determined if the director or switch is to be managed through Open Systems management style or FICON management style. This setting affects only the management style used to manage the director or switch; it does not affect Fibre Channel port operation.
Planning Considerations for Fibre Channel Topologies Port Numbering Versus Port Addressing Consideration must be given to the implications of port numbering for the FCP protocol versus logical port addressing for the FICON protocol. FCP configuration attributes are implemented through zoning. Zones are configured through the associated Element Manager application by authorizing or restricting access to name server information associated with device N_Ports that attach to director or switch F_Ports.
Planning Considerations for Fibre Channel Topologies Although Figure 46 depicts a UPM card map only for the Director 2/64, physical port numbers and logical port addresses can be extrapolated for the Director 2/140 (140 ports), Edge Switch 2/12 (12 ports), Edge Switch 2/16 (16 ports), Edge Switch 2/24 (24 ports), and Edge Switch 2/32 (32 ports).
Planning Considerations for Fibre Channel Topologies allowed with another fabric switch. The director or switch reports an attempted E_Port connection as invalid and prevents the port from coming online. — For later versions of director or switch firmware (version 4.0 and later), the domain field of the destination ID is added to the Fibre Channel link address, thus specifying the link address on source and target fabric elements and enabling E_Port (ISL) connectivity.
Planning Considerations for Fibre Channel Topologies SANtegrity Binding The SANtegrity Binding feature (including both fabric binding and switch binding) allows the creation of reliable SAN configurations and provides a mechanism for attached devices to query the user-configured security level employed in a SAN. The feature significantly reduces the impacts of accidental or operator-induced errors.
Planning Considerations for Fibre Channel Topologies However, the firmware and HAFM application do not prevent FCP and FICON device configurations that may interfere with each other. A successful intermix environment requires a set of best practice conventions as follows: 1. Upgrade fabric element firmware to a common version — Ensure fabric elements are operating at a common firmware level. This reduces errors due to director or switch incompatibility. Firmware Version 4.
Planning Considerations for Fibre Channel Topologies director or switch incompatibility. In addition, the SANtegrity Binding feature (with Enterprise Fabric Mode enabled) is required to support FICON cascading. 4. Logically assign ports — To organize devices into manageable groups for zoning, director or switch ports should be logically assigned to FCP port groups and FICON port groups.
Planning Considerations for Fibre Channel Topologies — On the Configure Allow/Prohibit Matrix - Active dialog box, assigning port names to logical port addresses is another practice that should be followed. For example, the port name for all FCP devices could begin with FCP or OS to indicate the associated port addresses attached to open-systems devices. This information emphasize which ports are FCP ports and which are FICON ports, and gives a user the ability to better manage the connectivity matrix.
Planning Considerations for Fibre Channel Topologies — FICON port addressing provides the ability to swap ports for maintenance. In general, swapping ports in intermix environments does not affect the practices described. However, if a user implements zoning using a domain ID and port numbers, zoning information must be updated contiguous with the port swap operation. Multiple Data Transmission Speeds in a Single Fabric The Director 2/64, Edge Switch 2/16, and Edge Switch 2/32 support auto-sensing of 1.
Planning Considerations for Fibre Channel Topologies ■ Data networks introduce variable delay and usually support high latency. SANs require minimal delay and latency. ■ Data networks rely on a software protocol stack such as Transmission Control Protocol/Internet Protocol (TCP/IP) to provide communications. Such stacks impose prohibitive performance penalties in SANs because data traffic quickly overloads servers.
Planning Considerations for Fibre Channel Topologies Figure 47: FCIP WAN Extension FCIP supports existing Fibre Channel SAN hardware and software, while allowing SAN-connected data to be accessed over an IP network infrastructure. FCIP allows data to be accessed remotely without altering the SAN fabric and maintains critically valuable bandwidth, data integrity, and flow control.
Planning Considerations for Fibre Channel Topologies iFCP, each connected SAN fabric is maintained separately from the others, while the IP network provides connectivity, congestion control, error detection, and error recovery. Figure 48 illustrates iFCP WAN extension. Figure 48: iFCP WAN Extension iSCSI Protocol iSCSI is a TCP/IP-based protocol for establishing and managing connections between IP-based storage devices, hosts, and clients.
Planning Considerations for Fibre Channel Topologies Figure 49: iSCSI WAN Extension SAN High Availability Planning Guide 123
Planning Considerations for Fibre Channel Topologies FICON Cascading The initial FICON architecture did not permit connection of multiple directors or switches, because the protocol specified a single byte for the link (port) address definition in the input-output configuration program (IOCP). The link address only defined the Port_ID for a unique domain (director or switch).
Planning Considerations for Fibre Channel Topologies incompatible list is isolated from the fabric. Membership list data eliminates duplicate domain IDs and other address conflicts and ensures a consistent, unified behavior across the fabric. ■ Switch binding — Switch binding allows only specified devices and fabric elements to connect to specified director or switch ports.
Planning Considerations for Fibre Channel Topologies — eServer zSeries 990 (z990) processor. Note: FICON cascading is not supported for IBM S/390 Parallel Enterprise Servers (Generation 5 or Generation 6). ■ The z/OS version 1.3 or version 1.4 operating system (with service as defined in PSP Buckets for device type 2032, 2042, 2064, or 2066) must be installed on the IBM server. ■ Licensed Internal Code (LIC) driver 3G at microcode level (MCL) J11206 or later must be installed on the IBM server.
Planning Considerations for Fibre Channel Topologies b. If required, click the Hardware tab. The Hardware View (Figure 19) displays. Verify that the status bar at the bottom left corner of the window displays a green circle, indicating director or switch status is operational. If a problem is indicated, go to MAP 0000: Start MAP in the product-specific Installation and Service Manual. c. Verify operation of non-cascaded FICON applications at each director or switch. 3.
Planning Considerations for Fibre Channel Topologies b. In the HAFM application, configure fabric binding. Refer to installation instructions in the HA-Fabric Manager User Guide. c. In the Element Manager application, configure switch binding. Refer to the installation instructions in the appropriate installation manual for your Director or Edge Switch. 5. Ensure FICON devices are logged in — Verify FICON devices are logged in to each director or switch as follows: a.
Planning Considerations for Fibre Channel Topologies Figure 52: Enterprise Fabric Mode Dialog Box b. Select the fabric to be configured from the Fabric Name drop-down list. The selected fabric’s status displays in the Enterprise Fabric Mode field. c. Click Activate to close the dialog box and enable Enterprise Fabric Mode for the selected fabric. 7. Verify FICON devices are still logged in — Maximize the Element Manager application.
Planning Considerations for Fibre Channel Topologies b. Ensure that the Enable Switch Binding check box is checked (enabled). c. Select the Restrict E_Ports radio button to restrict connections from specific fabric elements to E_Ports. WWNs can be added to the membership list to allow fabric element connection and removed from the list to prohibit fabric element connection. Devices are allowed to connect to any F_Port or FL_Port without restriction. d.
Physical Planning Considerations 4 This chapter describes the physical planning considerations for incorporating Hewlett-Packard (HP) Director 2/64s, Director 2/140s, Edge Switch 2/12s, Edge Switch 2/16s, Edge Switch 2/24s, and Edge Switch 2/32s into storage area networks (SANs) and Fibre Channel fabric topologies.
Physical Planning Considerations Port Connectivity and Fiber-Optic Cabling This section provides planning recommendations for director and switch port connectivity and fiber-optic cabling.
Physical Planning Considerations ■ Director 2/140 — The director is configured from a minimum of 16 universal port module (UPM) cards (64 ports total) to a maximum of 35 UPM cards (140 ports total). UPM cards provide four 2.125 Gbps port connections and can be configured with shortwave transceivers, longwave transceivers, and extended longwave transceivers or a combination of all three. ■ Edge Switch 2/12 — The switch provides up to 12 duplex SFP fiber-optic port transceivers (2.
Physical Planning Considerations Consider the following when determining the number and type of each transceiver to use: ■ Distance between a director or switch and the attached Fibre Channel device or between fabric elements communicating through an ISL. ■ Cost effectiveness. ■ Device restrictions or requirements with respect to existing fiber-optic cable (multimode or single-mode).
Physical Planning Considerations Extended-Distance Ports Through longwave laser transceivers and repeaters or dense wavelength division multiplexing (DWDM) equipment, Directors and Edge Switches support Fibre Channel data transmission distances of up to 100 km at 1 Gbps, or 50 km at 2 Gbps. The extended distance feature is enabled on a port-by-port basis by activating the 10-100 km check box for a specified port on the Element Manager application’s Configure Ports dialog box.
Physical Planning Considerations Cables and Connectors This section provides Fibre Channel cable and connector planning information as follows: ■ Cables for all directors and switches. ■ SFP transceivers for Director 2/64, Director 2/140, Edge Switch 2/12, Edge Switch 2/16, Switch 2/24, and Edge Switch 2/32. Cables Fiber-optic jumper cables are required to connect director and switch ports to servers, devices, distribution panels, or other elements in a multi-switch fabric.
Physical Planning Considerations Figure 54: SFP transceiver and LC duplex connector Routing Fiber-Optic Cables Follow a logical plan for routing fiber-optic cables to avoid confusing connections during installation and operation. Route cables from the access holes at the bottom or top of the equipment rack, and then to director and switch ports.
Physical Planning Considerations other units, while maintaining its original connections. To account for these possibilities, consider installing excess fiber-optic cable, especially in hard-to-reach places like underground trenches.
Physical Planning Considerations HAFM Appliance, LAN, and Remote Access Support Out-of-band (non-Fibre Channel) console access to directors and switches is provided to perform a variety of operations and management functions. These functions are performed from one or more of the following consoles: ■ Through the HAFM appliance attached to an Ethernet port on a director control processor (CTP) card or switch front panel.
Physical Planning Considerations HAFM Appliance Connectivity The HAFM appliance provides an auto-detecting 10/100 Base-T Ethernet interface that connects to a hub. Each director CTP card or switch front panel also provides an auto-detecting 10/100 Base-T Ethernet interface that connects to a hub. A 12-port hub can be ordered from HP and installed at the top front of the equipment rack. Although directors provide two Ethernet connections to a hub, only one connection is active at a time.
Physical Planning Considerations ■ Installing additional rack-mount products — When installing an additional director or switch, the length of Ethernet cable required to provide LAN connectivity is a function of rack position (top, bottom, or adjacent to the slide-out drawer). Ensure cable lengths provide sufficient cable inside the rack to route to the product’s Ethernet ports and to allow service clearance.
Physical Planning Considerations This single Ethernet connection is supported by HP, is Open View-Storage Node Manager (OV-SNM) compatible, and is the recommended configuration for a typical HP installation at a customer site. LAN security is provided by restricting password access and disabling the SNMP agent, Embedded Web Server interface, and command line interface (Telnet access) for each managed director or switch.
Physical Planning Considerations segment is optional and depends on customer requirements. This type of network configuration using both Ethernet connections is shown in Figure 56. Director 2/64s are used as an example. Although this dual Ethernet connection is supported by HP, it is not OV-SNM compatible, requires installation of an additional PCMCIA LAN adapter card (not supplied by HP), and is not the recommended configuration for a typical new HP installation at a customer site.
Physical Planning Considerations ■ A subset of the standard transmission control protocol/internet protocol (TCP/IP) MIB-II definition (RFC1213). ■ The director or switch-specific private enterprise MIB. The director or switch SNMP agent can be configured to send trap messages to up to six recipients. SNMP management is intended only for product monitoring; therefore, the default state of all MIB variables is read-only.
Physical Planning Considerations Inband Management Access (Optional) Inband management console access (through a Fibre Channel port) is provided by enabling user-specified features that allow Open Systems or FICON host control of a director or switch. The features are mutually exclusive; only one can be installed at a time. Features are enabled through a feature key encoded to work with the serial number of a unique director or switch.
Physical Planning Considerations The minimum OS/390 level for a director or switch without the control unit port (CUP) feature is version 2.6, plus service listed in PSP bucket upgrade 2032, device subset 2032OS390G5+. The minimum OS/390 level for a director or switch with the CUP feature is version 2.1, plus service listed in the preceding PSP bucket for that function. ■ A host-attached Hardware Management Console.
Physical Planning Considerations Security Provisions Security provisions are available to restrict unauthorized access to a director, switch, or attached Fibre Channel devices. Access to the director or switch (through the HAFM application, Element Manager application, and Web server interface) is restricted by implementing password protection.
Physical Planning Considerations Table 3: Types of User Rights User Right Operator Access Allowed Product Administrator The product administrator can make control and configuration changes through the Element Manager application. System Administrator The system administrator can make control and configuration changes, define users and passwords, and add or remove products through the HAFM application.
Physical Planning Considerations ■ Switch binding — Using the switch binding feature, an administrator allows only specified devices and fabric elements to connect to specified director or fabric switch ports. This provides security in environments that include a large number of devices by ensuring that only the intended set of devices attaches to a director or switch. This feature is managed through the Element Manager application.
Physical Planning Considerations Figure 57: Configure Allow/Prohibit Matrix - Active Dialog Box Figure 57 shows that port 1 (logical port address 05) is prohibited from communicating with port 6 (logical port address 0A), port 7 (logical port address 0B), and port 8 (logical port address 0C).
Physical Planning Considerations Figure 58: PDCM Array - Example Problem A PDCM array configured for Director A prohibits logical port address 05 from communicating with logical port addresses 0A, 0B, and 0C. No PDCM array is configured for Director B. The PDCM array configured for Director A prohibits the source server from transmitting or receiving data across ISL 2. However, internal route tables on both directors indicate a valid server-to-destination device path across ISL 1.
Physical Planning Considerations configured, an exit port on the director or switch, and the domain ID of the destination director or switch. Each participating director or switch must be configured as part of a desired path. The following rules apply when configuring a preferred path: ■ The switch domain ID must be set to Insistent. ■ Domain IDs range between 1 through 31. ■ Source and exit port numbers are limited to the range of ports available on the director or switch.
Physical Planning Considerations Figure 59: Preferred Path Configuration This procedure only specifies that data will enter and exit Director 1 through specific ports on the path to Director 3. The procedure must be repeated at the second director as follows: 1. Select the Preferred Path option from the Element Manager application’s Configure menu. The Configure Preferred Paths dialog box displays. 2. Click Add. The Add Preferred Path dialog box displays (top of Figure 59). 3.
Physical Planning Considerations 4. Click OK to save the path configuration and close the dialog box. Activating a preferred path can result in receipt of out-of-order frames (especially in FICON environments) if the path differs from the current path, if input and output (I/O) are active from the source port, and if congestion is present on the current path.
Physical Planning Considerations ■ Software-enforced zoning — For earlier versions of director or switch firmware (prior to version 6.0), the device configuration on a fabric element enforces zoning by limiting access to name server information in response to a device query. Only devices in the same zone as the requesting device are returned in the query response. This type of zoning is also called name server zoning or soft zoning.
Physical Planning Considerations Configuring Zones Zoning is configured through the HAFM application by authorizing or restricting access to name server information associated with device node ports (N_Ports) that attach to director or switch fabric ports (F_Ports). A device N_Port can belong to multiple zones.
Physical Planning Considerations — When a specific zone set is activated, that zone set replaces the active zone set. — If the active zone set is disabled, all devices attached to the fabric become members of the default zone. — All devices not included as members of the active zone set are included in the default zone. ■ Default zone — The default zone consists of all devices not configured as members of a zone in the active zone set.
Physical Planning Considerations If the configurations cannot merge, E_Ports that form the ISL for each fabric element become segmented. The ports cannot transmit data frames between attached switches (Class 2 or 3 traffic) but can transmit control frames (Class F traffic). Zoning configurations are compatible if there are no duplicate domain IDs, the active zone set name is the same for each fabric (or switch in the fabric), and zones with the same names in each fabric have identical members.
Physical Planning Considerations Server and Storage-Level Access Control To enhance the access barriers and network security provided by zoning through the director or switch, security measures for SANs should also be implemented at servers and storage devices. Server-level access control is called persistent binding. Persistent binding uses configuration information stored on the server and is implemented through the server’s HBA driver.
Physical Planning Considerations ■ Is typically proprietary and protects only a specific vendor’s storage devices. Storage-level access control may not be available for many legacy devices. Security Best Practices When implementing a enterprise data security policy, establish a set of best practice conventions using methods described in this section in the following order of precedence (most restrictive listed first): 1.
Physical Planning Considerations Devices in common zones can be prohibited from communicating through SANtegrity Binding or PDCM arrays, but hard zoning takes precedence over preferred path configurations, allowed connectivity through soft zoning, or device-level access control. 4. Preferred path — A preferred path provides soft control of fabric routing decisions on a switch-by-switch or port-by-port basis.
Physical Planning Considerations Logically work in sequence from the most restrictive method to the least restrictive method, ensuring the most restrictive routing or connectivity paths override all other paths.
Physical Planning Considerations Optional Features HP offers several operating features that are available for the switch as customer-specified options. Available features include: ■ Open Systems Management Server or FICON Management Server — Inband director or switch management is provided through purchase of the OSMS or FMS feature. Note: The Edge Switch 2/24 does not support out-of-band management through FMS.
Physical Planning Considerations Note: Director 2/64, Director 2/140, Edge Switch 2/16, and Edge Switch 2/32 do not provide CNT wide area network (WAN) support. ■ Element Manager application — This feature enables director or switch management through the Element Manager user interface. Directors and switches are delivered with the application enabled for a 31-day grace period. Before grace period expiration, the application must be reactivated through a PFE key.
Physical Planning Considerations FICON Management Server When the FMS feature key is enabled with the Element Manager application, host control and management of the director or switch is provided through a server attached to a product port. The server communicates with the product through a FICON channel.
Physical Planning Considerations Each port expansion kit includes eight SFP optical transceivers, upgrade instructions, and a feature key that enables the added port capacity through the Element Manager application. SANtegrity Binding SANtegrity Binding is a feature that enhances data security in large and complex SANs that have numerous fabrics and devices provided by multiple original equipment manufacturers (OEMs).
Physical Planning Considerations ■ Domain RSCNs — Domain RSCNs provide connectivity information to all HBAs and storage devices attached to a fabric. RSCNs are transmitted to all registered device N_Ports attached to a fabric if either a fabric-wide event or zoning configuration change occurs. ■ Insistent domain ID — When this parameter is enabled, the domain ID configured as the preferred domain ID for a director or switch becomes the active domain ID when the fabric initializes.
Physical Planning Considerations Server 1 Storage 1 Trunk TM Server 2 ISL 1 ISL 1 ISL 2 ISL 2 Director A TM Director B Storage 2 Server 3 Figure 61: Open Trunking configuration The figure illustrates two Director 2/64 directors connected by two ISLs. Three servers use the ISLs to communicate with two storage devices. Without trunking, servers 1 through 3 route Fibre Channel traffic to director B without regard to any data rates.
Physical Planning Considerations These Fibre Channel frames are then written to CD and included as part of the data collection procedure. This process constitutes a security breach if the frame data includes classified information. With the full-volatility feature installed and enabled, no frame data is stored, and the NV-RAM dump does not occur when the director or switch powers off or fails.
Physical Planning Considerations Note: The Edge Switch 2/12 is not supported by the HAFM appliance. In addition, the message Element Manager license key has not been installed Please follow up instructions to update permanent key is splashed across views, indicating the Element Manager PFE key must be installed. The Hardware View (Figure 63) for an Edge Switch 2/24 is shown as an example.
Configuration Planning Tasks 5 This chapter describes configuration planning tasks to be performed before installing the High Availability Fabric Manager (HAFM) server and one or more Director 2/64s, Director 2/140s, Edge Switch 2/12s, Edge Switch 2/16s, Edge Switch 2/24s, or Edge Switch 2/32s in a storage area network (SAN) configuration. Note: The Edge Switch 2/12 is not supported by the HAFM appliance. The following planning tasks are described in this chapter.
Configuration Planning Tasks 172 ■ Task 15: Plan a Multi-Switch Fabric (Optional), page 196 ■ Task 16: Plan Zone Sets for Multiple Products (Optional), page 197 SAN High Availability Planning Guide
Configuration Planning Tasks Task 1: Prepare a Site Plan For each director, switch, or equipment rack installed, design a site plan that provides efficient work flow, operator convenience and safety, and adequate service clearances for the equipment rack. A customer manager should review the site plan with a service representative and consider: ■ Location and relationship of the physical facilities, such as walls, doors, windows, partitions, furniture, and telephones.
Configuration Planning Tasks ■ Security necessary to protect the installation’s physical integrity, while maintaining accessibility to the director or switch. ■ Equipment rack front and rear service clearances, operator clearances, and maintenance access clearances. ■ Weight of an equipment rack. Either multiple persons or a lift must be available during installation to remove the rack from the packing crate. ■ Heat dissipation, temperature, and humidity requirements.
Configuration Planning Tasks Table 4: Physical Planning and Hardware Installation Tasks Activity Task Owner Due Date Comments Locate the physical facilities. Connect the facility alternating current (AC) power circuits. If more than one director or switch, consider separate power circuits for availability. Obtain an uninterruptable power supply (optional). Recommended. Obtain an outside-access phone line. Telephone for support personnel. Order the equipment rack with one or more HP products.
Configuration Planning Tasks Table 5 lists operational setup tasks and includes the task owner, due date, and comments. Table 5: Operational Setup Tasks Activity Obtain IP address and subnet mask. Task Owner Due Date Comments HAFM appliance (if installing on a LAN with non-HP devices). Directors and switches (if installing on a LAN with non-HP devices). Remote user workstation (optional). Simple network management protocol (SNMP) management stations (optional).
Configuration Planning Tasks Table 5: Operational Setup Tasks (Continued) Activity Determine SNMP access to directors and switches. Task Owner Due Date Comments Obtain SNMP trap recipient IP addresses. Determine SNMP information required (generic and productspecific). Determine if write permission is required for modifying SNMP variables. Determine if a multi-switch fabric is to be implemented. Determine if the zone management feature is to be used.
Configuration Planning Tasks Task 2: Plan Fibre Channel Cable Routing Plan for sufficient single-mode fiber-optic and multimode fiber-optic cabling to meet the connectivity requirements for all Fibre Channel servers and devices. If a multi-switch fabric is to be enabled, plan for sufficient fiber-optic cabling to meet interswitch link (ISL) connectivity requirements. Plan for at least one meter (39.
Configuration Planning Tasks Task 3: Consider Interoperability with Fabric Elements and End Devices HP conducts a substantial level of testing to ensure director and switch interoperability with fabric elements and end devices provided by multiple original equipment manufacturers (OEMs). New devices are tested and qualified on a continual basis. Contact your HP representative for the latest information about fabric element, server, host bus adapter (HBA), and device interoperability.
Configuration Planning Tasks Task 4: Plan Console Management Support Plan to implement one or more of the following methods to provide console management and support for directors and switches: ■ HAFM appliance — The rack-mounted HAFM appliance is used for product installation, initial software configuration, changing the configuration, and monitoring product operation.
Configuration Planning Tasks If director or switch management through an IBM host is planned, ensure that the FMS feature key is ordered with the Element Manager application. This feature key enables host control of the product from an IBM System/390 or zSeries 900 Parallel Enterprise server attached to a Fibre Channel port. ■ Web server interface — If Internet access to a director or switch Embedded Web Server interface is required, plan for access to an analog phone line.
Configuration Planning Tasks Task 5: Plan Ethernet Access Directors and the HAFM appliance can be ordered in an HP-supplied equipment rack in accordance with customer specifications; however, you may need to: ■ Connect equipment racks — Customer-supplied Ethernet hubs in multiple equipment racks can be connected to provide HAFM appliance access to up to 48 managed HP products. Racks can be placed at any distance up to the limit of the 10/100 megabit per second (Mbps) LAN segment.
Configuration Planning Tasks Task 6: Plan Network Addresses Depending on the configuration of the LAN on which directors, switches, and the HAFM appliance are installed, plan network addressing as follows: ■ If installing products and the HAFM appliance on a dedicated (private) LAN segment, there is no requirement to change any default network addresses. If multiple equipment racks are connected, ensure that all directors, switches, and servers have unique IP addresses.
Configuration Planning Tasks — Subnet mask is 255.0.0.0. — Gateway address is blank.
Configuration Planning Tasks Task 7: Plan SNMP Support (Optional) As an option, network administrators can use the HAFM application to configure an SNMP agent that runs on the HAFM appliance. This agent can be configured to send generic SNMP trap messages to up to 12 SNMP management workstations. Administrators can also use the Element Manager application to configure an SNMP agent that runs on each director or switch.
Configuration Planning Tasks Task 8: Plan E-Mail Notification (Optional) As an option, network administrators can configure director and switch e-mail support. The following support considerations are required if the e-mail notification feature is used: 186 ■ Determine if e-mail notification is to be configured and used for significant system events. ■ Determine which persons (up to five) require e-mail notification of significant director or switch events and record their e-mail addresses.
Configuration Planning Tasks Task 9: Establish Product and HAFM Appliance Security Measures Effective network security measures are recommended for directors, switches, and the HAFM appliance. Physical access to the network should be limited and monitored, and password control should be strictly enforced.
Configuration Planning Tasks Task 10: Plan Phone Connections Plan for one or more telephone connections near the HAFM appliance for service personnel use. While performing a diagnostic or repair action, a service representative or network administrator at the HAFM appliance may require voice technical support through a telephone connection.
Configuration Planning Tasks Task 11: Diagram the Planned Configuration Determine peripheral devices that will connect to each director or switch, and determine whether and where connectivity should be limited (zoning). These devices may include servers, storage control devices, and other fabric elements in a multi-switch fabric. Part of this task may have been performed when the configuration was determined. It might be helpful to draw the configuration diagram.
Configuration Planning Tasks Task 12: Assign Port Names and Nicknames During the planning process, consider assigning names to director and switch ports based upon devices connected to the ports. Though not required, port naming provides convenience and ease of use. Port naming also documents devices that connect through individual ports and identifies what is attached to each port. When it is necessary to change port connectivity, port names make it easier to identify the ports and attached end devices.
Configuration Planning Tasks Task 13: Complete the Planning Worksheet The planning worksheet included in this task is a four-page form that depicts port assignments for a director or switch. The worksheet lists 64 ports and provides fields to identify devices that connect to the ports.
Configuration Planning Tasks Product Planning Worksheet (Page 2 of 4) Director or Switch Name: ____________________________ IP Address: ____________________________ Unit Name: ____________________________ Attached Devices Port Type Port Name Location Model IP Address Zone IP Address Zone 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Product Planning Worksheet (Page 3 of 4) Director or Switch Name: ____________________________ IP Address: ____________________________ Unit Name: ________________
Configuration Planning Tasks Product Planning Worksheet (Page 3 of 4) 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 Product Planning Worksheet (Page 4 of 4) Director or Switch Name: ____________________________ IP Address: ____________________________ Unit Name: ____________________________ Attached Devices Port Type Port Name Location Model IP Address Zone 48 49 50 51 52 53 54 SAN High Availability Planning Guide 193
Configuration Planning Tasks Product Planning Worksheet (Page 4 of 4) 55 56 57 58 59 60 61 62 63 194 SAN High Availability Planning Guide
Configuration Planning Tasks Task 14: Plan AC Power Plan for facility power sources for each equipment rack. Directors and switches in the rack operate at 50 to 60 Hertz (Hz) and 100 to 240 volts alternating current (VAC) and require a minimum dedicated 5-ampere service. If two power sources are supplied (optional but recommended for high availability), the equipment rack contains two customer-specified external power cords.
Configuration Planning Tasks Task 15: Plan a Multi-Switch Fabric (Optional) If a multi-switch fabric topology is to be implemented, carefully plan the physical characteristics and performance objectives of the topology. Include the proposed number of fabric elements, characteristics of attached devices, cost, nondisruptive growth requirements, and service requirements.
Configuration Planning Tasks Task 16: Plan Zone Sets for Multiple Products (Optional) If name server zoning is to be implemented, carefully plan the characteristics and security objectives (differentiation of operating systems, data sets, user groups, devices, or processes) of zone members, zones, and zone sets. If a fabric topology is implemented, zoning is configured on a fabric-wide basis. Planning for zoned configurations must be carefully coordinated with planning a fabric topology.
Configuration Planning Tasks 198 SAN High Availability Planning Guide
Index A B backup HAFM data directory 49 NVRAM configuration 49 balancing data loads 83 bandwidth director 22 requirements 83 beaconing 37 best practices FICON cascading 126 security 160 binding SAN High Availability Planning Guide Index C Index addresses director gateway address 183 IP address 183 MAC address 183 subnet mask 183 HAFM appliance gateway address 184 IP address 183 MAC address 183 subnet mask 184 arbitrated loop switch, see FC-AL switch arbitrated loop typology characteristics 63 overview
Index core-to-edge fabric topology 95 D data access type 100 data collection 37 data transmission distance FCIP protocol 120 iFCP protocol 121 iSCSI protocol 122 WAN extension 119 default director network addresses 183 HAFM appliance network addresses 183 definition arbitrated loop 20 director 2/64 20 FC-AL switch 20 description fabric switch 28 software 50 design considerations fabric topology 109 device looplet 72 Tier 1 98 Tier 2 98 Tier 3 98 device fan-out ratio 103 device locality 102 device, private
Index performance requirements 99 redundant 106 ring 92 scalability 107 services 89 single 106 topologies 91 topology design considerations 109 implementation factors 82 WWN assignment 86 zoning configurations for joined fabrics 89 fabric binding 148, 166 fabric island topology 98 fabric switch description 28 performance 28 fabric-attached loop connectivity 75 fan-out ratio 103 FC-AL fabric attached-loop connectivity 75 FCIP protocol description 120 illustration 121 feature key CNT WAN support 169 descript
Index H HAFM appliance supported applications 43 HAFM application GUI 51 introduction 40 hard drive 44 Hardware View FRUs 55 Hardware view description 54 help, obtaining 17 heterogeneous fabric 81 high-availability director 22 fabric availability 105 fabric topology 80 high-integrity fabrics 124 hop count limit 81 HP authorized reseller 17 storage web site 17 technical support 17 hybrid topology 60 I I/O block size 100 I/O profile 100 I/O traffic requirements 99 iFCP protocol description 121 illustration
Index management HAFM application 40 out-of-band 40 SNMP agent 40 web server 40 management information bases director-specific MIB 38 Fabric Element MIB 38 Fibre Alliance MIB 37 management services application functions 50 maximum hop count 81 number of ISLs 81 memory HAFM appliance 44 mesh fabric topology 93 mode shared 63, 69 switched 64, 72 modem (external) 44 multi-cast support 34 multiswitch fabric support planning fabric multiswitch fabric support planning 80 multiswitch fabric topology 61 distance e
Index tuning 104 planning capacity 74 fabric-attached loop connectivity 75 Fibre Channel fabric topology 99 multiswitch fabric support 80 point-to-point connectivity 62 private arbitrated loop connectivity 69 planning considerations 59 planning tasks assign port names and nicknames 190 complete planning worksheet 191 consider interoperability with end devices 179 diagram planned configuration 189 establish security measures 187 plan AC power 195 plan console management support 180 plan e-mail notification
Index user restrictions 35 workstation restrictions 35 zoning 35 security provisions best practices 160 PDCM arrays 149 preferred path 151 zoning 154 server consolidation 77 service class support director 23 service rate 71 serviceability features 36 beaconing 37 data collection 37 diagnostic software 36 director-specific MIB 38 Fabric Element MIB 38 Fibre Alliance MIB 37 redundant FRUs 37 RS-232 maintenance port 37 services fabric 89 shared mode 63, 69 single fabric 106 SNMP introduction 40 trap messages
Index user workstation, planning support 180 utilization, loop 71 introduction 40 web server interface 56 web sites HP storage 17 WWN assignment 86 W Z overview 60 planning 62 U WAN extension description 119 FCIP protocol 120 iFCP protocol 121 iSCSI protocol 122 warning rack stability 16 symbols on equipment 14 web server 206 Zip drive 44 zone set naming conventions 197 zoning configurations for joined fabrics 89 description 154 naming conventions 197 planning 85 SAN High Availability Planning Gui
