Dell PS Series DCB Configuration Best Practices Dell Storage Engineering May 2017 Dell EMC Best Practices
Revisions Date Description February 2013 Initial publication February 2013 Changed PC81xx configuration steps to external SGC link May 2013 Added Force10 S4820T information May 2014 Updated links to new Switch Configuration Guides May 2017 Updated document to include consolidation of two other documents (BP1017 & BP1044) The information in this publication is provided “as is.” Dell Inc.
Table of Contents 1 2 Introduction ...................................................................................................................................................................5 1.1 Objective .............................................................................................................................................................5 1.2 Audience .........................................................................................................................
3.3.2 Single layer switching with rack servers ...........................................................................................................23 3.3.3 Dual layer switching ..........................................................................................................................................24 4 5 6 Host adapter configuration for DCB ...........................................................................................................................27 4.
1 Introduction Data Center Bridging (DCB) is a networking standard that first appeared on Fibre Channel over Ethernet (FCoE) SANs as a method to ensure lossless delivery of Fibre Channel (FC) traffic from Ethernet hosts to FC storage targets. Over recent years, this technology has been extended to deliver similar benefits for iSCSI storage solutions and continues to gain acceptance with storage networking device manufacturers and customers. Beginning with Dell™ PS Series Firmware release 5.
2 Data Center Bridging (DCB) Explained A key data center resource is the network infrastructure that interconnects various devices. These devices include server hardware systems that host enterprise applications and storage systems that host application data. Data Center Bridging (DCB) enables sharing the same network infrastructure between multiple traffic types such as server application traffic and storage data traffic.
2.2 Priority-based Flow Control (PFC): (IEEE 802.1Qbb) Independent traffic priority pausing and enablement of lossless packet buffers/queueing for iSCSI. DCB Terminology The following sub-sections explain the various DCB standards in more detail and why they are needed within a DCB enabled SAN. 2.2.1 Priority-based Flow Control Priority-based Flow control (PFC) is an evolution of the concept of Flow Control originally implemented in the MAC Pause feature of Ethernet (IEEE 802.3x).
2.2.2 Enhanced Transmission Selection Enhanced Transmission Selection (ETS) is a mechanism for guaranteeing a percentage of bandwidth to a traffic class (TC). A traffic class contains one or more Classes of Service from the VLAN Q-tag. Each traffic class is then assigned a percentage of bandwidth with a granularity of 1%. All traffic class bandwidths must add up to 100%; oversubscription is not allowed. The bandwidth percentage defined is a minimum guaranteed bandwidth for that traffic class.
Protocols identified by this method can be mapped to a priority value using the application priority feature. For example, iSCSI identified by TCP port number 3260 can be mapped to priority 4. An end device which is the source of a frame carrying iSCSI payload marks the frame with priority 4 in the VLAN tag. All devices in the network forward the frame with this priority value and process it with the same configured PFC lossless behavior and PG/TC bandwidth settings for this priority.
LLPD hierarchy supporting DCBx 2.3 DCB parameter configuration The key DCB technologies discussed in section 2.2 are applicable peer to peer on an Ethernet link between two ports. A link comprised of peer ports may be between a NIC/CNA and a switch, between two switches, or between a switch and a storage device. A successful converged network deployment based on DCB includes proper configuration and operation of DCB parameters on all peer device ports that carry converged traffic.
Step 2 – DCBX TLVs exchange between peer ports on the operational link Step 3 – CNA and storage port accept and confirm switch DCB parameters In step 2 depicted in Figure 6, peer ports communicate their local DCB configuration through DCBX and their willing mode. The server CNA and storage ports initially have a default local DCB configuration that they communicate. In step 3 depicted in Figure 7, they apply the switch advertised configuration locally and communicate the same parameters back to the switch.
2.4 Converged network example This section discusses a sample converged network implementation using DCB. The various DCB parameters (PFC, PG/TC, and application priority) are illustrated in the context of a deployment scenario. The scenario uses the hypothetical requirements given below: A virtualized server environment that has requirements to converge both LAN and SAN traffic on the same physical Ethernet infrastructure (Server adapters and Ethernet switches).
DCB parameters 2.5 The priority groups or traffic classes are the final step. The PG/TC table in the figure shows the priority group or traffic classes and the respective bandwidth percentage assignment. The Priority to PG/TC mapping table shows the priorities that are mapped to priority groups or traffic classes. This information is advertised as part of the DCBX ETS or PG TLV. It must be noted that multiple priorities can be mapped to the same PG or TC.
When configuring DCB on a new PS Series array member, ensure the switch is setup properly first. The PS Series CLI setup wizard or the PS Series Remote Setup Wizard will detect whether the switch is DCB capable and prompt for the required configuration information, including the VLAN ID, during setup. To configure DCB on an existing PS Series array member: 1. Disable DCB on the switch, if it is enabled. 2. Verify that all CNAs are enabled and configured in DCB willing mode. 3.
Next, a valid VLAN ID must be entered. The range of recommended values is 2-4096. However, verify that the VLAN ID choice does not conflict with any default or reserved IDs for your specific switch make and model. With firmware v7 and later, the VLAN ID will default to 2 when DCB is detected on the switch. Important: When DCB is enabled on the switch, it is necessary to configure a non-default VLAN on the array, switch, and all host ports that are part of the PS Series SAN.
1. In the PS Series Group Manager, click a member array and go to the Network tab. 2. Right-click on each 10 Gb interface (for example eth0) and click DCB Details to display the dialog box shown below. EqualLogic Group Manager NIC DCB Details 3. From these details, verify the following: a. Column 1: For the iSCSI Traffic Class Group (Group 1, in this example), only iSCSI is present (no other traffic classes should appear, such as FCoE). b.
0 will not have PFC enabled on the switch, which means that iSCSI traffic would be treated with the same priority as all other traffic or in some cases, with a lower priority. It is also possible that even when the iSCSI TLV is supported by the switch, the iSCSI priority may not have PFC enabled for that priority. Upon detection of either of these conditions, the array will issue a warning message that Ethernet flow control has been disabled on one or more interfaces.
3 Network configuration and deployment topologies This section discusses detailed DCB parameter configuration for network deployments with PS Series storage. It includes the configuration process for multi-layered switching environments and discusses sample topologies.
arbitration process is defined for one of them to propagate DCBX configuration internally and the other ports accept that configuration. b. Manual configuration: If this internal propagation method is not supported by your switch model, then peer and intermediate layer switches must be manually configured with the same DCB parameters as the source switches with willing mode off across all ports. 3. Configure and verify edge device DCBX parameters. a.
Note: The upstream port method can be used to configure other peer switches in the same top layer. A pair of source switches can be configured with DCBX parameters and other switches in the same layer can accept configuration from the source switches through the inter switch links (Willing mode on for ISLs in the same layer). All switches in this layer then pass on configuration to downstream links and edge devices like initiators and targets (willing mode off for these ports and downlinks).
For switches that support the DCB propagation mechanism: Switch inter-connectivity in the Layer 2 network must be designed so that there is no single point of failure that would cause a disjoint network. When designed properly, the failure of a single switch device, link, or port interface should not cause a disjoint network. The administrator must ensure that network redundancy design best practices are followed to avoid disjoint networks.
3.2.5 Configuration for switches that do not support DCB for PS Series The following switches do not fully support DCB requirements for PS Series, so DCB must be disabled on these SAN switches. Important: If DCB is disabled on a switch, then all DCB functionality will be disabled, including support for other protocols such as FCoE. Mixing of FCoE and iSCSI on the same converged fabric is not recommended and not supported by Dell. 3.2.
Single switching layer – Blade Servers 3.3.2 Single layer switching with rack servers This section demonstrates deployment configurations with rack servers and a single layer of top of rack (ToR) switches. The sample configuration for this discussion includes PowerEdge R620 servers attached to PS6110XV Series storage arrays through the PowerConnect 8132F ToR switches. The switches are interconnected through a LAG (Link Aggregation Group) and the configuration is illustrated in Figure 16.
Single switching layer – Rack severs In both the Blade server and Rack server configurations, the first step is to configure the switches with DCB parameters. 3.3.3 Dual layer switching A hierarchical switching configuration with two layers of switching, one for servers and another for storage, is covered in this section. An illustration of this configuration is given in Figure 17.
Dual Layer Switching In this scenario, the first step is to configure the external S4810 switches with DCB parameters. The Dell PowerEdge M I/O aggregator modules (FTOS version 9.11.0.P8) are by default configured with DCB enabled and all ports belonging to all VLANs. The external uplink ports (33 to 56) are auto configured to be in a single LACP LAG (LAG 128). They are also auto configured to be in the DCB auto-upstream mode. The auto-upstream ports are set with willing bit turned on.
accept the internally propagated DCB configuration from the auto- upstream port. These ports advertise the DCB configuration information to their server peers. The second step is to setup the server initiators and targets with correct iSCSI VLAN ID. DCB is enabled in willing mode by default. The last step is the verification of DCB parameters on the initiator, target, and corresponding switch ports.
4 Host adapter configuration for DCB 4.1 QLogic 57810 (previously Broadcom) configuration Configuring the QLogic 57810 adapter for iSCSI and DCB requires the use of the QLogic Control Suite (QCS) utility. For this paper, the following software versions were used: QCS version 17.0.14.0, QLogic driver version 7.12.32.0 and QLogic firmware version 7.12.19. QCS software and CNA drivers can be found at the Dell support site: https://support.dell.com. 4.
4.3 QLogic 57810 iSCSI VLAN configuration Figure 19 below illustrates the VLAN configuration for iSCSI traffic on the adapter port. Initiator VLAN ID configuration 4.4 QLogic 57810 DCB configuration verification Figure 20 below is a screen shot from the QCS software. This is applicable to the QLogic 57810 Dual-Port 10 GbE SFP+ adapter and the QLogic 57810 Dual-Port 10 GbE KR Blade Converged Mezzanine Card on Windows Server 2008 R2 SP1.
The local and remote DCBX settings are displayed in the DCBX Advanced section of the QCS, see Figure 21.
5 DCB testing The following workloads and tests were developed to provide empirical evidence of the benefits of DCB in two common scenarios: Dedicated iSCSI networks and converged traffic networks. In the dedicated iSCSI tests, traffic was only running from the iSCSI host to the target. No other traffic was being injected. This simulates environments where the customer is implementing DCB, but continuing to separate storage and LAN traffic through the use of dedicated, separate physical networks.
After comparison runs early in the testing process, it was determined that the sequential write and random read/write workloads did not provide sufficient throughput to cause a noticeable effect. As seen from the charts below, virtually no difference was seen from a DCB versus non-DCB standpoint.
5.2 Topologies Test topology In this topology, three PS6110 Series arrays were used to ensure that the throughput for the host ports could be maximized. All physical connections were made in a redundant manner, with ports connected across separate switches. These switches were then connected through a high speed link aggregation group (LAG) of 2 x 40 Gb ports, providing sufficient bandwidth for any traffic traversing the inter-switch link.
6 Results and analysis 6.1 DCB versus non-DCB in a dedicated SAN network The following charts show the difference in retransmitted iSCSI frames on the network given each flowcontrol method. Retransmitted network frames waste available network resources by transmitting data a second time resulting in lower throughput and I/O performance on the storage array.
Throughput comparison for iSCSI only traffic For I/Os per second, the result was consistent with a difference of approximately 15% improvement with PFC over no flowcontrol and a 5% improvement over MAC PAUSE. I/Os per second comparison for iSCSI only traffic 6.2 DCB versus non-DCB in a shared network Finally, the throughput for both DCB and non-DCB environments was compared for fully converged traffic.
In a shared network, controlling the amount of bandwidth given to different resources has been tough even in the best scenario. Traditional flowcontrol as implemented with MAC PAUSE provided a minimal method to stop some bursting traffic by pausing all traffic on a given link. With PFC it is now possible to stop specific traffic classes over a given link allowing a much more controlled environment for shared network traffic, in this case iSCSI traffic and other LAN data.
iSCSI throughput comparison for mixed traffic environment In Figure 30, we see the same benefit in IOPS performance that was noted in the previous throughput section. PFC is able to create a controlled environment providing consistent results.
A PS Series command line reference for DCB configuration verification From the CLI (GroupAdmin login), the following commands can be executed to obtain the same information that is presented in the Group Manager GUI: show grpparams (Display Global DCB and VLAN status): PG1> show grpparams __________________________ Group Information __________________________ Group-Ipaddress: 10.10.10.150 Name: PG1 Def-Snap-Reserve: 100% Group-Mgmt-Gateway: 192.168.140.
member select eth show (Display DCB ON/OFF): The example below is from a PS6110 array, so eth0 is the iSCSI port and eth1 is the management port which shows the DCB as off for the management port. PG1> Name eth0 eth1 member select P1 eth show ifType ifSpeed Mtu Ipaddress Status Errors DCB ethernetcsmacd ethernetcsmacd 10 Gbps 9000 10.10.10.141 up 0 on 100 Mbps 1500 192.168.140.
Class 1 2 39 50% 50% 4 0,1,2,3,5,6,7 Dell PS Series DCB Configuration Best Practices | BP1058
B Technical support and resources Dell.com/support is focused on meeting customer needs with proven services and support. Dell TechCenter is an online technical community where IT professionals have access to numerous resources for Dell EMC software, hardware and services. Dell.com/StorageResources on Dell TechCenter provide expertise that helps to ensure customer success on Dell EMC Storage platforms. B.
