Dell PowerEdge M1000e Blade Enclosure and Dell PS Series SAN Design Best Practices Using Dell S-Series and M-Series Networking Dell EMC Storage Engineering May 2017 Dell EMC Best Practices
Revisions Date Description February 2013 Initial release May 2017 Minor updates Acknowledgements This white paper was produced by the PG Storage Infrastructure and Solutions team of Dell Inc. The team that created this white paper: Clay Cooper, Guy Westbrook, and Camille Daily Updates provided by: Steven Lemons The information in this publication is provided “as is.” Dell Inc.
Table of Contents 1 2 Introduction ...................................................................................................................................................................4 1.1 Audience .............................................................................................................................................................4 1.2 Terminology .......................................................................................................................
5.1.4 Active System Manager ....................................................................................................................................25 5.1.5 Hardware requirements ....................................................................................................................................26 5.1.6 Using alternate switch vendors .........................................................................................................................26 5.1.7 Recommendations ...
1 Introduction Dell™ PS Series arrays provide a storage solution that delivers the benefits of consolidated networked storage in a self-managing iSCSI storage area network (SAN) that is affordable and easy to use, regardless of scale. By eliminating complex tasks and enabling fast and flexible storage provisioning, these solutions dramatically reduce the costs of storage acquisition and ongoing operation.
1.2 Terminology This section defines terms that are commonly used in this paper and the context in which they are used. Blade I/O module (IOM) switch – A switch that resides in an M1000e Fabric slot. Blade IOM switch only – A category of SAN design in which the network ports of both the hosts and the storage are connected to the M1000e blade IOM switches, which are isolated and dedicated to the SAN. No external ToR switches are required.
Switch tier – A pair or more of like switches connected by a switch interconnect which together create a redundant SAN Fabric. A switch tier might accommodate network connections from host ports, from storage ports, or from both. If all switches in a switch tier are reset simultaneously, for example the switch tier is stacked and the firmware is updated, then the SAN is temporarily offline. ToR switch – A top of rack switch, external to the M1000e blade chassis.
2 Concept Overview 2.1 PowerEdge M1000e blade chassis solution The following section describes the M1000e blade chassis networking Fabrics consisting of I/O modules, a midplane, and the individual blade server network adapters. 2.1.1 Multiple Fabrics Each M1000e can support up to three separate networking Fabrics that interconnect ports on each blade server to a pair of blade I/O modules within each chassis Fabric through a passive chassis midplane.
2.1.2 M-Series Blade I/O modules The following table lists the 1 GbE, 10 GbE & 40 GbE M-Series blade I/O module options and the number of ports available for PS Series SAN solution. 1 GbE, 10 GbE & 40 GbE M-Series Blade I/O Module Port options for PS Series storage 2.
If there are blade IOM switches and ToR switches in the SAN, these two switch tiers will need to be connected by an uplink, which can be a stack (if there is stack compatibility between the blade IOM and ToR switch types), a LAG, or a VLT LAG (if there is a VLT interconnect between the ToR switches). Uplink bandwidth should be at least equal to the aggregate bandwidth of all active PS Series array member ports.
An example PS Series SAN consisting of PS Series array members, an M1000e blade chassis with blade servers, and ToR and Blade IOM switches.
2.3 Data Center Bridging DCB standards are enhancements to IEEE 802.1 bridge specifications to support multiple protocols and applications in the data center. They support converged infrastructure implementations to carry applications and protocols on a single physical infrastructure. For more information on using DCB with PS Series SANs, see the following resources: Dell PS Series DCB Configuration Best Practice: http://en.community.dell.
Designed from the ground up as a converged infrastructure system, Active System integrates new unified infrastructure management, a new plug-and-play blade chassis I/O module – the PowerEdge M I/O Aggregator, modular servers and storage, and a converged LAN/SAN fabric. Key to Active System is Active System Manager, an intelligent and intuitive converged infrastructure manager that leverages templates to automate infrastructure on-boarding and re-configuration.
3 Summary of SAN designs and recommendations This section provides the high level conclusions reached after the course of comprehensive lab testing and analysis of various PS Series array SAN designs which incorporate M1000e blade server hosts on a 10 GbE network.
If the availability of the SAN is critical, a LAG or VLTi interconnect will be preferred over stacking. If a switch interconnect is stacked, then a switch stack reload (required for tasks such as switch firmware updates) will temporarily make the SAN unavailable. In this case, SAN downtime for firmware updates would have to be scheduled.
4 Tested SAN designs This section describes each tested M1000e blade chassis SAN design in detail including diagrams and a table for comparison of important values such as bandwidth, maximum number of supported array members, and the host to storage port ratio. The information below assumes a single M1000e chassis and 16 halfheight blade servers with two network ports each. There are three categories of SAN designs for M1000e blade chassis integration: 1.
M-Series MXL with LAG interconnect 16 Dell PowerEdge M1000e Blade Enclosure and Dell PS Series SAN Design Best Practices Using Dell S-Series and M-Series Networking | BP1039
4.2 ToR switch only These SAN designs include configurations where the storage ports and blade server host ports are directly connected to ToR switches. A 10 GbE pass-through IOM rather than a blade switch is used to connect the host ports to the ToR switches. For this SAN design, dual S-Series S4810P switches were used as the ToR switch.
S-Series S4810P with VLTi 18 Dell PowerEdge M1000e Blade Enclosure and Dell PS Series SAN Design Best Practices Using Dell S-Series and M-Series Networking | BP1039
4.3 Blade IOM switch with ToR switch These SAN designs include configurations in which the PS Series array member ports are connected to a tier of ToR switches while the server blade host ports are connected to a separate tier of blade IOM switches in the M1000e blade chassis. With the multiple switch tier designs, it is a best practice to connect all array member ports to the ToR switches and not the blade IOM switches in the M1000e chassis.
S-Series S4810P with VLTi / MXL with VLT LAG uplinks 20 Dell PowerEdge M1000e Blade Enclosure and Dell PS Series SAN Design Best Practices Using Dell S-Series and M-Series Networking | BP1039
4.3.2 S-Series S4810P with VLTi / M-Series I/O Aggregator with VLT LAG uplinks This SAN design uses two of the four integrated 40 GbE QSFP ports on each S-Series S4810P to create a VLTi between each ToR switch. Eight 10 GbE SFP+ ports will need to be set aside on each S4810P switch to provide a 160 Gbps uplink from the two M-Series I/O Aggregators (IOA).
Active System 800 -- Force10 S4810P with VLTi / IOA with VLT LAG uplinks 22 Dell PowerEdge M1000e Blade Enclosure and Dell PS Series SAN Design Best Practices Using Dell S-Series and M-Series Networking | BP1039
4.4 Summary table of tested SAN designs The following table assumes one fully populated M1000e blade chassis with 16 half-height blade servers each using two network ports (32 host ports total) and the maximum number of PS Series array members accommodated by the available ports of the array member switches -- either dual ToR S4810P switches or dual MXL switches in a single M1000e blade chassis I/O Fabric.
5 Detailed SAN design analysis and recommendations The following section examines each M1000e blade chassis and PS Series SAN design from the perspectives of administration, performance, high availability, and scalability. In addition, SAN bandwidth, host to storage port ratios, and SAN performance and high availability test results are provided as a basis for SAN design recommendations. 5.
that may occur prior to VLT being established. After VLT is established, RSTP may be used to prevent loops from forming with new links that are incorrectly connected and outside the VLT domain. When operating as VLT domain peers, the ToR switches appear as a single virtual switch from the point of view of any connected switch or server supporting LACP. This has many benefits including high availability, the use of all available uplink bandwidth, and fast convergence if either the link or the device fails.
Active System integration services provide a complete end-to-end deployment including switch configuration, DCB enablement, PS Series array member initialization, blade enclosure/server setup, and hypervisor installation. 5.1.5 Hardware requirements The SAN design will determine the type and quantity of hardware and cabling required. Implementing a multiple tier switch SAN design will obviously require at least twice the number of switches as other more simple designs.
5.2 Performance The second criterion by which SAN designs will be evaluated is their performance relative to each other. This section reports the performance results of each SAN design under three common I/O workloads. Note: The results provided in this paper are intended for the purpose of comparing specific configurations used in our lab environment. The results do not portray the maximum capabilities of any system, software, and/or storage. 5.2.
256KB transfer size, sequential I/O, 100% write Each vdbench workload was run for thirty minutes and the I/O rate was not capped (the vdbench “iorate” parameter was set to “max”). The throughput values used in the relative performance graphs are the sums of the values reported by each of the four hosts. Each host ran one instance of iperf server and one instance of iperf client.
5.2.3 Results The following three figures show the relative aggregate vdbench throughput of all four hosts within each SAN design at three different I/O workloads. Each throughput value is presented as a percentage of a baseline value. In each chart, the MXL with LAG interconnect design was chosen as the baseline value. All of the throughput values were achieved during a single thirty minute test run.
256 KB sequential I/O, read workload The following figure shows the aggregate vdbench throughput of all four hosts within each SAN design at a 256 KB sequential I/O, read workload. All SAN designs yielded throughput results within 5% of the baseline value.
256 KB sequential I/O, write workload The following figure shows the aggregate vdbench throughput of all four hosts within each SAN design at a 256 KB sequential I/O, write workload. All SAN designs yielded throughput results within 8% of the baseline value.
Previous generations of PS Series arrays did not have individual port failover and a single port, cable or switch failure could reduce the number of connected array member ports. To test SAN design high availability, an ungraceful switch power down was executed while the SAN was under load. The test environment was the same as the environment that was used during performance testing, and the workload was 256 KB sequential I/O write using vdbench. LAN traffic was generated with iperf.
5.3.2 Blade IOM switch failure The following table shows how each SAN design is affected by the loss of a blade IOM switch. Note that this failure is not applicable to ToR switch only designs in which both host and storage ports are connected to the ToR switches. In all applicable SAN designs, a blade IOM switch failure reduces the number of connected host ports by 50% and in the multiple switch tier SAN design the uplink bandwidth is also reduced by 50%.
5.4 Scalability The final criterion by which SAN designs will be evaluated is scalability. Note that the scalability data presented in this section is based primarily on available port count. Actual workload, host to array port ratios, and other factors may affect performance.
Two ToR S-Series S4810P and two M-Series MXL switches can also accommodate up to 16 array members with no expansion modules required. This SAN design has the highest number of available ports -- an additional sixty-four 10 GbE ports or, if 40 GbE QSFP to 10 GbE SFP+ breakout cables are used to uplink the MXL to the S4810P, forty-eight 10 GbE ports and four 40 GbE ports.
A Solution infrastructure detail The following table is a detailed inventory of the hardware and software configuration in the test environment. A detailed inventory of the hardware and software configuration in the test environment 36 Solution configuration - Hardware components: Description Blade Enclosure PowerEdge M1000e chassis: CMC firmware: 4.20 Storage host enclosure 10 GbE Blade Servers (4) PowerEdge M620 server: Windows Server 2008 R2 SP1 BIOS version: 1.4.9 iDRAC firmware: 1.23.
B Technical support and resources Dell.com/support is focused on meeting customer needs with proven services and support. Dell TechCenter is an online technical community where IT professionals have access to numerous resources for Dell EMC software, hardware and services. Dell.com/StorageResources on Dell TechCenter provide expertise that helps to ensure customer success on Dell EMC Storage platforms. B.
