Dell EqualLogic Best Practices Series SAN Design Best Practices for the Dell PowerEdge M1000e Blade Enclosure and EqualLogic PS Series Storage (1GbE) A Dell Technical Whitepaper This document has been archived and will no longer be maintained or updated. For more information go to the Storage Solutions Technical Documents page on Dell TechCenter or contact support.
THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND. © 2012 Dell Inc. All rights reserved. Reproduction of this material in any manner whatsoever without the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell.
Table of Contents 1 Introduction ........................................................................................................................................................ 1 1.1 Audience ...................................................................................................................................................... 1 1.2 Terminology ..........................................................................................................................................
5.2.3 Results ............................................................................................................................................... 28 5.2.4 Recommendations .......................................................................................................................... 29 5.3 High availability ........................................................................................................................................ 29 5.3.1 TOR switch failure .........
Acknowledgements This white paper was produced by the PG Storage Infrastructure and Solutions team of Dell Inc. The team that created this white paper: Clay Cooper, Guy Westbrook, and Margaret Boeneke Feedback We encourage readers of this publication to provide feedback on the quality and usefulness of this information by sending an email to SISfeedback@Dell.com. SISfeedback@Dell.
1 Introduction With the Dell™ EqualLogic™ PS Series storage arrays, Dell provides a storage solution that delivers the benefits of consolidated networked storage in a self-managing iSCSI storage area network (SAN) that is affordable and easy to use, regardless of scale. By eliminating complex tasks and enabling fast and flexible storage provisioning, these solutions dramatically reduce the costs of storage acquisition and ongoing operations.
ISL – An inter-switch link that connects either the two blade IOM switches or the two TOR switches to each other. An ISL can be a stack or a LAG. Blade IOM switch only – A category of SAN design in which the network ports of both the hosts and the storage are connected to the M1000e blade IOM switches, which are isolated and dedicated to the SAN. No external TOR switches are required. The ISL can be a stack or a LAG, and no uplink is required.
SAN designs a like switch pair will need to be interconnected by an inter-switch link or ISL. Like the uplink, the ISL can be a stack or a LAG. The ISL is necessary to create a single layer 2 SAN fabric over which all PS Series array member network ports can communicate with each other. For a much more detailed description of each SAN design that was tested and evaluated see Section 4 – Tested SAN designs.
2 Overview of M1000e blade chassis solution The following section describes the M1000e blade chassis networking fabrics consisting of IO modules, a midplane, and the individual blade server network adapters. 2.1 Multiple fabrics Each M1000e can support up to three separate networking “fabrics” that interconnect ports on each blade server to a pair of blade IO modules within each chassis fabric through a passive chassis midplane.
Figure 2 Blade IO Modules and M1000e Chassis 2.2 Blade IO modules The following table lists the 1GbE blade IO module options (available at the time of this publication) and the number of ports available for EqualLogic SAN solutions.
3 Summary of SAN designs and recommendations The following section provides the high level conclusions reached after the course of comprehensive lab testing and analysis of various EqualLogic PS Series array SAN designs which incorporate M1000e blade server hosts on a 1GbE network. For complete results and recommendations see Section 5 - Detailed SAN design analysis and recommendations. For an illustration of each SAN design see Section 4 – Tested SAN designs.
Blade IOM and TOR with 4-way stack Multiple • • • Blade IOM and TOR with 3-way LAG Multiple • • A single switch stack to manage Uplinks required between tiers During stack reload, SAN is unavailable • Four switches to manage Uplinks required between tiers • The least amount of uplink bandwidth (32Gbps) • • Mid-range uplink bandwidth (40Gbps) • • Blade IOM and TOR with uplink stacks Multiple • • • • BP1031 Two switch stacks to manage Uplinks required between tiers Separate stacks allow f
Blade IOM and TOR with ISL stacks Multiple • • • Two switch stacks to manage Uplinks required between tiers During stack reloads, SAN is unavailable • Mid-range uplink bandwidth (40Gbps) • • Blade IOM switch failure reduces host ports by 50% Blade IOM or TOR switch failure reduces uplink bandwidth by 50% • • Two TOR switches accommodate twelve array members Beyond two blade chassis and a blade IOM ISL is required or TOR switches must be added to accommodate blade IOM switch uplinks 3.
3.4 Scalability For blade IOM switch only SAN designs, scaling the number of array members is only possible with the addition of M1000e blade chassis and scaling beyond three blade chassis is not recommended due to increased hop-count and latency over the ISL connection.
4 Tested SAN designs The following section describes each tested M1000e blade chassis SAN design in detail including diagrams and a table for comparison of important values such as bandwidth, maximum number of supported array members, and the host to storage port ratio. All information below assumes a single M1000e chassis and 16 half-height blade servers with two network ports each.
Figure 3 Blade IOM switch only with ISL stack 4.1.2 M6348 switch with ISL LAG This SAN design provides 20Gbps of ISL bandwidth between the two M6348 switches using the two integrated 10GbE SFP+ ports to create a LAG. Since there is only a single tier of switches, there is no uplink to external switches. The 16 external ports on each M6348 (32 ports total) can accommodate the connection of four 1GbE PS series array members, each of which require eight ports for the active and passive controllers combined.
The following diagram illustrates how the two PS6100XV array members directly connect to the two M6348 switches in Fabric B of the M1000e blade chassis and how the two M6348 switches are connected by a LAG using the integrated 10GbE SFP+ ports. Note how each array member controller connects to both M6348 switches for SAN redundancy.
4.2 TOR switch only These SAN designs include configurations where the blade server host ports are directly connected to TOR switches using 1GbE pass-through IOM in the M1000e blade chassis. The storage ports are also connected to the TOR switches, in this case a pair of PowerConnect 7048. One SAN design of this type was tested: 1.
Figure 5 TOR switch only with ISL LAG BP1031 SAN Design Best Practices for M1000e Blade Servers and EqualLogic PS Series Storage (1GbE) 14
4.3 Blade IOM switch with TOR switch These SAN designs include configurations in which the EqualLogic PS Series array member ports are connected to a tier of TOR switches while the server blade host ports are connected to a separate tier of blade IOM switches in the M1000e blade chassis. With the multiple switch tier designs it is a best practice to connect all array member ports to the TOR switches and not the blade IOM switches in the M1000e chassis.
Figure 6 Blade IOM switch with TOR switch and a four-way stack BP1031 SAN Design Best Practices for M1000e Blade Servers and EqualLogic PS Series Storage (1GbE) 16
4.3.2 M6348/PC7048 switches with three-way LAG This SAN design uses the 10GbE SFP+ ports provided by the PC7048 expansion module and the 10GbE SFP+ ports integrated into the M6348 to setup two separate uplink LAGs and one ISL LAG between the TOR PC7048 switches. It provides 40Gbps of uplink bandwidth between the storage tier of PC7048 switches and the host tier of M6348 switches, while providing 20Gbps of ISL bandwidth.
Figure 7 Blade IOM switch with TOR switch and a three-way LAG BP1031 SAN Design Best Practices for M1000e Blade Servers and EqualLogic PS Series Storage (1GbE) 18
4.3.3 M6348/PC7048 switches with uplinks stacked This SAN design uses the 16Gb stacking ports provided by the PC7048 expansion module and the stacking ports integrated into the M6348 to configure two uplink stacks between each pair of M6348 and PC7048 switches. Additionally, one ISL LAG connecting each uplink stack in a mesh pattern is created using the 10GbE SFP+ ports provided by the PC7048 expansion module and the 10GbE SFP+ ports integrated into the M6348.
Figure 8 Blade IOM switch with TOR switch and uplink stacks BP1031 SAN Design Best Practices for M1000e Blade Servers and EqualLogic PS Series Storage (1GbE) 20
4.3.4 M6348/PC7048 switches with ISL stacks This SAN design uses the 16Gb stacking ports provided by the PC7048 expansion module and the stacking ports integrated into the M6348 to setup two ISL stacks between each pair of like switches. Additionally, an uplink LAG connecting each switch tier in a mesh pattern is created using the 10GbE SFP+ ports provided by the PC7048 expansion module and the 10GbE SFP+ ports integrated into the M6348.
Figure 9 Blade IOM switch with TOR switch and ISL stacks BP1031 SAN Design Best Practices for M1000e Blade Servers and EqualLogic PS Series Storage (1GbE) 22
4.4 Summary table of tested SAN designs The following table assumes one fully populated M1000e blade chassis with 16 half-height blade servers each using two network ports (32 host ports total) and the maximum number of PS Series array members accommodated by the available ports of the array member switches -- either dual TOR PC7048 switches or dual M6348 switches in a single M1000e blade chassis IO fabric.
5 Detailed SAN design analysis and recommendations The following section examines each M1000e blade chassis and EqualLogic PS Series SAN design from the perspectives of administration, performance, high availability and scalability. In addition, SAN bandwidth, host to storage port ratios, and SAN performance and high availability test results are provided as a basis for SAN design recommendations. 5.
5.1.2 Hardware requirements The SAN design will determine the type and quantity of hardware and cabling required. Implementing a two tier switch SAN design will obviously require at least twice the number of switches as other more simple designs. The blade IOM switch only SAN design requires the fewest cables, with only the array member ports and a single ISL stack or LAG at the M1000e chassis to cable.
5.2 Performance The second criterion by which SAN designs will be evaluated is their performance relative to each other. Section 5.2 reports the performance results of each SAN design under two common IO workloads. 5.2.1 Test environment In order to determine the relative performance of each SAN design we used the performance tool Vdbench to capture throughput values at three distinct I/O workloads.
switches, multiple switch tier SAN design require an uplink stack or LAG between switch tiers. Each multiple switch tier design provides a different amount of uplink bandwidth between the host and storage switch tiers. Uplink bandwidth should be at least equal to the aggregate bandwidth of all active PS Series array member ports. For example, twelve array members with four active ports each would require 48Gbps of uplink bandwidth.
5.2.3 Results The following two figures show the relative aggregate Vdbench throughput of all four hosts within each SAN design at two different I/O workloads. Each throughput value is presented as a percentage of a baseline value. In each chart, the blade IOM switch only with ISL LAG design was chosen as the baseline value. All throughput values were achieved during a single one hour test run and are not an average of multiple test runs. 5.2.3.
Percentage of baseline throughput during Vdbench 256KB sequential read workload 120% 100% 80% 60% 40% 20% 0% % of baseline M6348 with ISL LAG M6348 with ISL stack PC7048 with ISL LAG 100% 102% 94% PC6348/PC PC6348/PC PC6348/PC 7048 with 7048 with 7048 with 3-way LAG 4-way stack ISL stacks 101% 100% 95% PC6348/PC 7048 with uplink stacks 100% Figure 11 Aggregate Vdbench throughput as a percentage of the baseline value in each SAN design during a 256KB sequential I/O, read workload 5.2.
remaining host ports. Also in these cases, the loss of 50% of the host ports reduced the Vdbench throughput by the same amount, as would be expected. 5.3.1 TOR switch failure The following table shows how each SAN design is affected by the loss of a TOR switch. Note that this failure is not applicable to the blade IOM switch only designs in which both host and storage ports are connected to blade IOM switches.
percentage the loss of bandwidth is not a factor. Of all multiple switch tier designs, the uplink stack (highlighted in green) retains by far the most uplink bandwidth. All applicable SAN designs retain enough ISL bandwidth to accommodate the expected ISL traffic of the remaining host ports. As discussed in Section 5.2.2 this is normally about 50% of the expected throughput between the host and storage ports.
array members as the blade IOM switch only designs, yielding an ideal 1:1 host/storage port ratio. The TOR switches in the multiple switch tier designs are entirely dedicated to storage and can accommodate up to 12 array members. With that many array members, it begins to make sense to add chassis to increase the host/storage port ratio.
switch only design will experience increased hop-counts and latency as the number of switches increases. Also, each storage pool should consist of array members that are all connected to the same pair of adjacent switches so that inter-array member traffic does not span more than one ISL. For both single tier switch SAN designs, an ISL stack or LAG will need to extend between the switches (whether blade IOM or TOR) as their number scales.
Figure 12 Incorporating an additional M1000e chassis into the blade IOM switch with TOR switch and uplink stacks SAN design BP1031 SAN Design Best Practices for M1000e Blade Servers and EqualLogic PS Series Storage (1GbE) 34
5.4.3 Recommendations For blade IOM switch only SAN designs, scaling the number of array members is only possible with the addition of M1000e blade chassis and scaling beyond three blade chassis is not recommended due to increased hop-count and latency over the ISL connection.
Appendix A Solution infrastructure detail The following table is a detailed inventory of the hardware and software configuration in the test environment. Table 9 A detailed inventory of the hardware and software configuration in the test environment Solution configuration - Hardware components: Description Blade Enclosure Storage host enclosure Dell PowerEdge M1000e chassis: CMC firmware: 4.
with uplink stacks Blade IOM switch and TOR switch with ISL stacks 1GbE External switches (2) Dell PowerConnect 7048 Firmware v4.2.1.
Appendix B Vdbench parameters Vdbench workloads were executed using the following parameters in the parameter file, where “N” is the number of iSCSI volumes under load. Common parameters: hd=default hd=one,system=localhost iSCSI volumes: sd=sd3,host=*,lun=\\.\PhysicalDrive3,size=1m,threads=5 sd=sd4,host=*,lun=\\.\PhysicalDrive4,size=1m,threads=5 sd=sd5,host=*,lun=\\.\PhysicalDrive5,size=1m,threads=5 sd=sd6,host=*,lun=\\.\PhysicalDrive6,size=1m,threads=5 8KB 67% read, random I/O workload: wd=wd1,sd=(sd3-s
Additional resources Support.dell.com is focused on meeting your needs with proven services and support. DellTechCenter.com is an IT Community where you can connect with Dell Customers and Dell employees for the purpose of sharing knowledge, best practices, and information about Dell products and your installations. Referenced or recommended Dell publications: • Dell EqualLogic Configuration Guide: http://en.community.dell.com/techcenter/storage/w/wiki/2639.equallogic-configurationguide.
THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND.
