SAN Design: March 29, 2001 3:18 pm Building and Scaling BROCADE© SAN Fabrics: Design and Best Practices Guide 53-0001575-01 BROCADE Technical Note Page: 1 of 31
BROCADE SAN Integration and Application Department Last Updated March 29, 2001 3:18 pm Design and Best Practices Guide This document contains BROCADE© recommendations and guidelines for configuring a Storage Area Network (SAN). The document includes several reference topologies and also provides pointers to products/solutions from BROCADE partners that can be used to implement the target configuration/solution.
SAN Design: March 29, 2001 3:18 pm 2.0 Fabric Topologies This section explores a variety of fabric topologies and provides some specific network examples for SAN fabrics. Topologies fall into the following general categories: • Meshed Topology-- a network of switches that has at least one link to each adjacent switch. Fully meshed designs will have a connection from each switch in the fabric to all other switches in the fabric. Other topologies are a specific instance of a mesh design.
SAN Design: March 29, 2001 3:18 pm Tier Architecture Designs • Typically three layers of switches, a host layer, core layer, and storage layer • Natural extension to star design (in some cases, Tier designs are Stars) • Core switches are used to provide connectivity between host and storage layer switches, includes redundant switching elements • Each layer can be scaled independently • Cores can be simple to more complex, easily replaced with higher port count switches • Can still use knowledge
SAN Design: March 29, 2001 3:18 pm useful when combined with a link extender that can allow from 30 to 120 kilometers distance between switch elements. There is a latency penalty for extended links that needs to considered where performance is a concern. Shorter links, lower latency -- with roughly 100 microseconds of delay per 10KM of distance for round trip traffic.
SAN Design: March 29, 2001 3:18 pm Half of the switches are connected to form one fabric, and the other half form an identical fabric, which is completely unconnected to the first fabric. There is no single point of failure in either fabric which could cause the fabric to segment. This can be used in combination with dual attach hosts and storage to keep a solution up even when one entire fabric fails. This is generally the best approach to take to SAN design for high availability. FIGURE 1.
SAN Design: March 29, 2001 3:18 pm FIGURE 2. Two Core Switch, Two ISL, Star Design. The above SAN is a single fabric, resilient design. To deploy a dual fabric, resilient SAN based upon this architecture, the following SAN would be built FIGURE 3. Dual Fabric Design, Hosts and Storage connection to Two Independent Fabrics: Hosts and Storage Dual connections No connection! Redundancy builds in the ability to allow for SAN management to take place on one SAN while the other SAN stays in operation.
SAN Design: March 29, 2001 3:18 pm Switch upgrades can take place on one SAN (firmware, hardware, both) and while this SAN is down the redundant SAN stays in operation • A switch failure in one SAN allow for failover to the redundant SAN while the failed switch is replaced.
SAN Design: March 29, 2001 3:18 pm 2.4 FIGURE 4.
SAN Design: March 29, 2001 3:18 pm 2.5 FIGURE 5. TWO SWITC H FA BRIC FOR MIRRORING AN D DIS ASTER TOLER ANCE Extended Fabric Example • Sample configuration showing only two hosts and two storage devices, larger configurations can be deployed H H SW D 10 KM • This example shows how data being used at the local site can be mirrored at a remote site via an extended fabric link. Primary system data is replicated at remote site where a backup failover system is located.
SAN Design: March 29, 2001 3:18 pm 2.6 FIGURE 6. HIGHLY AV AILA BLE SMALL FA BRIC CONFIGURA TION HIghly Available Small Fabric • In this design, any switch can fail and there will still be an alternative path to the host and storage devices. [Assumes hosts/storage have intelligence to fail over.] Any single switch could be powered off for servicing and replaced in the fabric without loosing device connectivity.
SAN Design: March 29, 2001 3:18 pm 2.7 SAN BUILD ING BLOCK - M ESHED FABR IC CON SISTIN G OF TWO STAR TOPOLOGIES CONNECTED BY A M ESH FIGURE 7. SAN Building Block showing expansion to 17 switch Mesh Fabric SW SW SW SW SW SW SW SW SW SW SW SW • SW An atomic building block of 8 switches (star) for larger SANs SW SW SW DESIGN GUIDANCE • Design can be expanded horizontally or vertically- horizontal expansion shown above using mesh design - Recommend use of FOS version 2.1.9g/2.2.
SAN Design: March 29, 2001 3:18 pm 2.8 STA R TOPOLOGY DESIGN In the traditional data networking world, star topology network has a core of networking equipment (hubs, switches, and routers) with other equipment radiating out from this. A star topology SAN is fundamentally the same: It contains one or more “core” switches, surrounded by one or more “edge” switches. FIGURE 8. Edge Core FIGURE 9.
SAN Design: March 29, 2001 3:18 pm FIGURE 10. Two Switch Core Star SAN Design Two, single-switch cores The strengths of this design are numerous. This provides lowest hop count design, uses the fewest switches for the given port count, has the best and easiest to analyze performance characteristics, and is the simplest to build, understand, and maintain. Support for more complex cores will be detailed in a future version of this document.
SAN Design: March 29, 2001 3:18 pm FIGURE 11. Star Topologies shown with 2x16 port switch at the core and with 4x16 port switch at the core Brocade has tested and validate star SAN designs today with our SilkWorm 2800 16 port switch. The two core switch topology shown in Figure 5 is a very good typical SAN design that can support many network storage applications. There is a maximum port count available of 224 ports. No more than two hops from any device to any device in the network.
SAN Design: March 29, 2001 3:18 pm FIGURE 12. Star Topologies Possible with 2 switch cores 2.
SAN Design: March 29, 2001 3:18 pm Three Tier designs in general allow for higher port count fabrics over Star designs (using the same switch building blocks) but can add additional hops to the design. A middle (core) set of switches can be used to provide connectivity between an upper and lower level of switches. These designs are generally used when data traffic flows between devices attached to the top tier and devices on the bottom tier.
SAN Design: March 29, 2001 3:18 pm FIGURE 14. MUlti Tier SAN Design across an Extended Link SITE 1 These links are via DWDM, not shown in diagram. SITE 2 DESIGN: 16 SWITCHES, TWO LOCATIONS; TOTA L PORTS= 212 USING 16 PORT SWITC HES Depicted in the next diagram is a 20 switch fabric, three tier, with 4 switches at the core. This fabric offers 2 hops from source to destinations, with multiple equal cost paths between devices allowing for device failover without a performance lost due to added latency.
SAN Design: March 29, 2001 3:18 pm FIGURE 15.
SAN Design: March 29, 2001 3:18 pm 2.9.1 SUMMA RY A variety of SAN designs topologies and port counts can be deployed today using Brocade switch. The Fabric Operating System provides for auto-discovery and configuration of the SAN as devices are added and new switches included in an existing SAN. The user should understand his environment, the components in the SAN, the relationship between hosts and storage, usage patterns and his needs for reliability and redundancy when building a SAN.
SAN Design: March 29, 2001 3:18 pm 3.0 3.1 Additional Considerations in Fabric Design and Implementation FABRIC BRING UP A large (greater than 8) switch fabric will require care and planning on initial power up and for expanding switch elements. Brocade has developed an extensive planning document to allow SAN administrators to successfully bring up, add to, delete from, and generally maintain a large switch fabric.
SAN Design: March 29, 2001 3:18 pm FIGURE 17. Valid and Invalid Configurations using Inter Switch Links Not Valid Config - 9 ISL’s Between Two Switches Note: connection of more than 8 ISLs leaving 7 or less open ports does not make sense for traffic routing in a 16 port switch. Valid Configuration Using 16 ISL’s 3.3 CABLING AND MED IA INTER FA CES Fibre channel supports several cable and optical media interface options. • The shortest supported optical cable length is 2 meters.
SAN Design: March 29, 2001 3:18 pm for wiring components that do not need to be a long distance from the switch. Copper connections tend to be less expensive. If c ost is not a driver, designing a solution with all optical media will provide for greater flexibility in future system upgrades and expansion and allow devices to be extended beyond the limits of one rack without having to replace any GBICs. 3.
SAN Design: March 29, 2001 3:18 pm The total size of a switch fabric is related to a number of factors that can affect overall performance. Some factors include: • Total nodes, entries in the name space. Name space table management as devices in the fabric change can effect overall fabric performance as a reconfiguration is needed to accommodate changes • Zoning entries, similar to names space, needs to be propagated through the fabric • Error handling.
SAN Design: March 29, 2001 3:18 pm of Fibre Channel is that data is transferred based on buffer credits assigned to ports and sending and receiving devices manage the credits so that there is never an overrun of data in the switch. Blocking is also called over-subscription -- where multiple initiator ports are limited by ISL links or paths in the fabric and, for example, 3 devices must share a single link between switches [a switch with 12 N-port nodes and 4 ISLs is 3:1 over-subscription].
SAN Design: March 29, 2001 3:18 pm fabric using IP over fibre channel protocol across switch links, thus reducing the need for a large switch management network. This configuration requires the primary switch with the Ethernet connection to be configured as the gateway switch for the Fibre Channel IP network. All other switches would have to have this switch configured as the gateway address.
SAN Design: March 29, 2001 3:18 pm width of 100 MBytes/sec. Thirty disks attached to a hub on a single physical loop will not perform as well as 5 disks attached to 6 switch QuickLoop ports. Other advantages are: • Loop tenancies can occur in parallel on different looplets. A loop tenancy occurs when a device takes control of the loop to perform I/O and I/O to any other loop device is restricted. In most loop implementations only one device is allowed control (tenancy) at a time.
SAN Design: March 29, 2001 3:18 pm 4.0 Glossary This glossary provides definitions for the fibre channel and switch terminology used in BROCADE guide. TABLE 1.
SAN Design: March 29, 2001 3:18 pm TABLE 1. Glossary of Fibre Channel and SAN Design Terminology Term Definition Frame Fibre channel structure used to transmit data. Consists of start-of-frame delimiter, header, any optional headers, data payload, cyclic redundancy check (CRC), and end-of-frame delimiter. There are two types: data frames and link control frames. Similar to the networking concept “packet”. FSPF Fabric Shortest Path First.
SAN Design: March 29, 2001 3:18 pm TABLE 1. Glossary of Fibre Channel and SAN Design Terminology Term Definition Topology As applies to fibre channel, the structure of the fibre channel network and the resulting possible communication paths. There are three fibre channel topologies: point-to-point, fabric, and arbitrated loop. Zone Set of hosts and devices attached to same fabric and having access permission, including RSCNs and user data, to each other.
SAN Design: March 29, 2001 3:18 pm Copyright IMPORTANT NOTICE This document is the property of BROCADE. It is intended solely as an aid for installing and configuring Storage Area Networks c onstructed with BROCADE switches. This document does not provide a warranty to any BROCADE software, equipment, or service, nor does it imply product availability. BROCADE is not responsible for the use of this document and does not guarantee the results of its use.