Dell PowerEdge M1000e Blade Enclosure and EqualLogic PS Series 10 GbE SAN Design Best Practices A Dell EqualLogic Best Practices Technical White Paper Storage Infrastructure and Solutions Engineering Dell Product Group October 2012
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices © 2012 Dell Inc. All rights reserved. Reproduction of this material in any manner whatsoever without the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell. Dell, the DELL logo, and the DELL badge, PowerConnect™, EqualLogic™, PowerEdge™ and PowerVault™ are trademarks of Dell Inc. Intel® is a registered trademark of Intel Corporation in the U.S.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices Table of Contents 1 Introduction ............................................................................................................................................................................ 6 1.1 Audience ................................................................................................................................................................................ 6 1.
BP1042 5.2.3 Results........................................................................................................................................................................... 30 5.2.4 Recommendations ..................................................................................................................................................... 32 5.3 High availability ...................................................................................................................
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices Acknowledgements This white paper was produced by the PG Storage Infrastructure and Solutions team of Dell Inc. The team that created this white paper: Clay Cooper, Guy Westbrook, and Camille Daily Feedback We encourage readers of this publication to provide feedback on the quality and usefulness of this information by sending an email to SISfeedback@Dell.com. SISfeedback@Dell.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices 1 Introduction With the Dell™ EqualLogic™ PS Series storage arrays, Dell provides a storage solution that delivers the benefits of consolidated networked storage in a self-managing iSCSI storage area network (SAN) that is affordable and easy to use, regardless of scale.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices ISL – An inter-switch link that connects either the two blade IOM switches or the two TOR switches to each other. An ISL can be a stack or a LAG. Blade IOM switch only – A category of SAN design in which the network ports of both the hosts and the storage are connected to the M1000e blade IOM switches, which are isolated and dedicated to the SAN. No external TOR switches are required.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices can be a stack or a LAG. The ISL is necessary to create a single layer 2 SAN Fabric over which all PS Series array member network ports can communicate with each other. For a much more detailed description of each SAN design that was tested and evaluated see Section 4 titled, “Tested SAN designs”.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices 2 Overview of M1000e blade chassis solution The following section describes the M1000e blade chassis networking Fabrics consisting of IO modules, a midplane, and the individual blade server network adapters. 2.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices 2.2 Blade IO modules The following table lists the 10 GbE blade IO module options (available at the time of this publication) and the number of ports available for EqualLogic SAN solutions.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices 3 Summary of SAN designs and recommendations This section provides the high level conclusions reached after the course of comprehensive lab testing and analysis of various EqualLogic PS Series array SAN designs which incorporate M1000e blade server hosts on a 10 GbE network.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices Blade IOM and TOR with 3-way LAG Switch tier topology Multiple Ease of administration • Four switches to manage • Uplinks required between tiers Performance • Equivalent performance during small scale SAN testing High availability • Blade IOM or TOR switch failure reduces host ports by 50% • Blade IOM or TOR switch failure reduces uplink bandwidth by 50% Scalability • Highly scalable, allowing 16 ar
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices 3.3 High availability In both the TOR and blade IOM switch failure scenarios, all tested SAN designs suffer a temporary 50% reduction in the number of connected host ports and for the multiple switch tier three-way LAG SAN design, a 50% reduction in uplink bandwidth.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices 4 Tested SAN designs The following section describes each tested M1000e blade chassis SAN design in detail including diagrams and a table for comparison of important values such as bandwidth, maximum number of supported array members, and the host to storage port ratio. All information below assumes a single M1000e chassis and 16 half-height blade servers with two network ports each.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices Figure 3 Blade IOM switch only with ISL stack 15
BP1042 4.1.2 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices M8024-k switch with ISL LAG This SAN design provides 30 Gbps of ISL bandwidth between the two M8024-k switches using three integrated 10 GbE SFP+ ports on each switch to create a LAG. Since there is only a single tier of switches, there is no uplink to the TOR switches.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices Figure 4 Blade IOM switch only with ISL LAG 17
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices 4.2 TOR switch only These SAN designs include configurations where the blade server host ports are directly connected to TOR switches using 10 GbE pass-through IOM in the M1000e blade chassis. The storage ports are also connected to the TOR switches, in this case a pair of PowerConnect switches. Two SAN designs of this type were tested: • • 4.2.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices Figure 5 TOR switch only with ISL stack 19
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices 4.2.2 PC8024F switch with ISL LAG This SAN design provides 30 Gbps of ISL bandwidth between the two PC8024F switches using three 10 GbE SFP+ ports on each switch to create a LAG. Since there is only a single tier of switches, there is no uplink from the blade IOM pass-through modules. 16 ports on each switch are required for the connections of the 16 hosts with two network ports each.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices Figure 6 TOR switch only with ISL stack 21
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices 4.3 Blade IOM switch with TOR switch These SAN designs include configurations in which the EqualLogic PS Series array member ports are connected to a tier of TOR switches while the server blade host ports are connected to a separate tier of blade IOM switches in the M1000e blade chassis.
BP1042 4.3.1 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices M8024-k/PC8024F switches with three-way LAG This SAN design uses the 10 GbE SFP+ ports of the PC8024F and the M8024-k to setup two separate uplink LAGs and one ISL LAG between the TOR PC8024F switches. It provides 120 Gbps of uplink bandwidth between the storage tier of PC8024F switches and the host tier of M8024F switches, while providing 30 Gbps of ISL bandwidth.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices Figure 7 Blade IOM switch with TOR switch and a three-way LAG 24
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices 4.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices 5 Detailed SAN design analysis and recommendations The following section examines each M1000e blade chassis and EqualLogic PS Series SAN design from the perspectives of administration, performance, high availability, and scalability. In addition, SAN bandwidth, host to storage port ratios, and SAN performance and high availability test results are provided as a basis for SAN design recommendations. 5.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices The blade IOM switch only SAN design requires the fewest cables, with only the array member ports and a single ISL stack or LAG at the M1000e chassis to cable.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices 5.2 Performance The second criterion by which SAN designs will be evaluated is their performance relative to each other. This section reports the performance results of each SAN design under two common IO workloads. 5.2.1 Test environment In order to determine the relative performance of each SAN design we used the performance tool vdbench to capture throughput values at three distinct I/O workloads.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices 5.2.2 Bandwidth All SAN designs provide different amounts of ISL bandwidth between the two switches within each switch tier. While single switch tier designs have host and storage ports connected to the same switches, multiple switch tier SAN design require an uplink stack or LAG between switch tiers.
BP1042 5.2.3 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices Results The following three figures show the relative aggregate vdbench throughput of all four hosts within each SAN design at three different I/O workloads. Each throughput value is presented as a percentage of a baseline value. In each chart, the PC8024F with ISL LAG design was chosen as the baseline value. All throughput values were achieved during a single one hour test run.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices 256 KB sequential I/O, read workload The following figure shows the aggregate vdbench throughput of all four hosts within each SAN design at a 256 KB sequential I/O, read workload. All SAN designs yielded throughput results within 4% of the baseline value.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices 256 KB sequential I/O, write workload The following figure shows the aggregate vdbench throughput of all four hosts within each SAN design at a 256 KB sequential I/O, write workload. All SAN designs yielded throughput results within 1% of the baseline value. Figure 10 Aggregate Vdbench throughput as a percentage of the baseline value in each SAN design during a 256 KB sequential I/O, write workload 5.2.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices To test SAN design high availability, an ungraceful switch power down was executed while the SAN was under load. The test environment was the same as the environment that was used during performance testing, and the workload was 256 KB sequential I/O write using vdbench. In all cases, vdbench I/O continued without error and no iSCSI volume disruptions were observed.
BP1042 5.3.2 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices Blade IOM switch failure The following table shows how each SAN design is affected by the loss of a blade IOM switch. Note that this failure is not applicable to TOR switch only designs in which both host and storage ports are connected to the TOR switches.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices 5.4 Scalability The final criterion by which SAN designs will be evaluated is scalability. Note that the scalability data presented in this section is based primarily on available port count. Actual workload, host to array port ratios, and other factors may affect performance.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices 5.4.2 TOR switch only The following two tables show the scalability data for TOR switch only designs where both host and storage are connected to PC8024F switches. Because changing the number of total host ports actually changes number of switch ports available for array members, the 16 blade server and eight blade server scalability data is presented in separate tables.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices 5.4.3 Blade IOM and TOR switch The following table shows the scalability data for Blade IOM and TOR switch design where hosts are connected to the M1000e Blade IOM switches and storage is connected to the PC8024F switches.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices 5.4.4 Recommendations The following recommendations assume two M8024-k switches or two pass-through IO modules per blade chassis, two SAN ports per host and, if applicable, 24-port PC8024F TOR switches. The blade IOM switch only SAN design does not yield acceptable host/storage port ratios even with the maximum number of arrays and only eight full-height blade servers per blade chassis.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices Appendix A Solution infrastructure detail The following table is a detailed inventory of the hardware and software configuration in the test environment. Table 12 39 A detailed inventory of the hardware and software configuration in the test environment Solution configuration - Hardware components: Blade Enclosure Dell PowerEdge M1000e chassis: CMC firmware: 4.
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices Appendix B Vdbench parameters Vdbench workloads were executed using the following parameters in the parameter file. Common parameters: hd=default hd=one,system=localhost iSCSI volumes (random IO): sd=sd3,host=*,lun=\\.\PhysicalDrive3,size=256000m,threads=5 sd=sd4,host=*,lun=\\.\PhysicalDrive4,size=256000m,threads=5 sd=sd5,host=*,lun=\\.\PhysicalDrive5,size=256000m,threads=5 sd=sd6,host=*,lun=\\.\Physical
BP1042 Dell PowerEdge M1000e Blade Enclosure and Equallogic PS Series 10 GbE SAN Design Best Practices Additional resources Support.dell.com is focused on meeting your needs with proven services and support. DellTechCenter.com is an IT Community where you can connect with Dell Customers and Dell employees for the purpose of sharing knowledge, best practices, and information about Dell products and your installations.
This white paper is for informational purposes only. The content is provided as is, without express or implied warranties of any kind.
