VxFlex Network Deployment Guide using Dell EMC Networking 25GbE switches and OS9 A VxFlex Ready Node deployment guide using Dell EMC Networking S5000-ON switches Abstract The Dell EMC Networking S5048F-ON switches running OS9 provide the bandwidth and low latency support for a scalable storage architecture. This document details the deployment of the Dell EMC VxFlex Ready Node solution using these Dell EMC Networking 25GbE switches.
Revisions Date Description February 2018 v1.0 - initial release March 2019 v1.01 - minor revision: rebranded VxFlex, updated links to external resources, title changed from “ScaleIO 2.0 Deployment Guide with Dell Networking S5048F-ON 25GbE Switches” The information in this publication is provided “as is.” Dell Inc.
Table of contents Revisions.............................................................................................................................................................................2 Executive summary.............................................................................................................................................................5 1 2 3 4 Introduction .............................................................................................................
7.2 4 Configure Quality of Service using Differentiated Services (DiffServ) .............................................................36 A Troubleshooting SDS connectivity .............................................................................................................................38 B Routing VxFlex Virtual Machine traffic .......................................................................................................................39 C Validated hardware and components ..
Executive summary Dell EMC VxFlex is an industry leading software-defined storage (SDS) solution that enables customers to extend their existing virtual infrastructure into a high performing virtual SAN. VxFlex creates a virtual server SAN using industry-standard servers with direct attached storage (DAS). You can deploy VxFlex using as few as three hosts, and up to 1024 hosts. Each host can use storage media such as flash-based SSDs, NVMe SSDs, traditional spinning disks, or a mix.
1 Introduction VxFlex is a software-only solution that uses existing servers’ local disks and LAN to create a virtual SAN that has all the benefits of external storage – but at a fraction of cost and complexity. VxFlex uses the existing local storage devices and turns them into shared block storage. For many workloads, VxFlex storage is comparable to, or better than, external shared block storage.
In this environment, the VxFlex/ScaleIO Virtual Machine (SVM) hosts the MDM and SDS roles. Each VxFlex node, running ESXi, has a separate SVM. The following illustration shows the communication between these components: VxFlex component communication 1.2 Typographical conventions This document uses the following typographical conventions: 1.
2 Hardware overview This section briefly describes the hardware that is used to validate the deployment example in this guide. The appendix contains a complete listing of hardware and components. Steps in this document were validated using the specified Dell EMC Networking switches and OS9 but may be used for other Dell EMC Networking switch models that use the same networking operating system or later assuming the switch has the available port numbers, speeds, and types.
3 Networking In VxFlex, inter-node communication (for managing data locations, rebuilding, and rebalancing, and for application access to stored data) can be done on one IP network or spread across separate IP networks. Regardless of the model, VxFlex supports VLANs. Management is done in one of two ways: • • Via a separate network with access to the other VxFlex components On the same network Each VxFlex Ready Node R730xd has four 25GbE ports provided by two Mellanox Connect X-4 LX PCIe cards.
Spine switches Management Environment Network: 172.16.8.0/21 • • • • vCenter (atx01m01vc01.dell.local) NTP (ntp0.dell.local) DNS (172.16.11.4, 172.16.11.
Management Network Figure 6 shows the S3048-ON is connected to the S5048F-ON leaf switches through a port channel. The same S3048-ON switch provides iDRAC connectivity to the VxFlex Ready Nodes. All interfaces are configured to a dedicated subnet and VLAN, and that VLAN is tagged on the upstream port channel.
3.2 Network connectivity The figure below shows one VxFlex Ready Node connected to two S5048F-ON switches via two Mellanox ConnectX-4 LX PCIe cards installed in PCIe slots one and two. The leaf switches are Virtual Link Trunking (VLT) peers, and one port from each PCIe card connects to each leaf switch. The connections for the remaining VxFlex Ready Nodes (R730xd-2 through R730xd-4) are done in the same manner.
The figure below shows one VxFlex Ready Node’s iDRAC connected to the S3048-ON switch via the onboard iDRAC port. The S3048-Management switch is connected via a port channel to the S5048F-ON VLT pair. The connections for the remaining VxFlex Ready Nodes (R730xd-2 through R730xd-4) are done in the same manner.
In this guide, the S5048F-ON switches are configured as a VLT pair. Part of a successful VLT deployment is the use of a VLT backup link. The backup link monitors the connectivity between the VLT peer switches. The backup link sends configurable, periodic keepalive messages between the VLT peer switches. Figure 9 shows that the out-of-band (OOB) management interface (ma1/1) is configured as a point-to-point link to fulfill this requirement.
Table 1 and Table 2 show how to calculate the IP address requirements. To accommodate expansion each address pool is assigned a dedicated subnet.
The IP ranges and default gateway for each IP subnet are shown in Table 3. VMware vMotion, Data1, and Data2 do not require default gateways in this configuration. If routing is required for Data1 and Data2 see, Appendix B. IP address ranges IP Pool IP Subnet Default gateway IDRAC_MGMT_IP 172.16.30.0/24 172.16.30.253 ESXI_MGMT_IP 172.16.31.0/24 172.16.31.253 ESXI_VMOTION_IP 172.16.32.0/24 n/a SVM_MGMT_IP 172.16.33.0/24 172.16.33.253 Node_DATA1_IP & SVM_DATA1_IP 172.16.34.
4 Configure physical switches This section contains switch configuration details with explanations for one switch. The remaining switch uses a configuration very similar. Complete configuration files for both switches are provided as attachments. 4.1 Factory Default Settings The configuration commands in the sections below assume switches are at their factory default settings.
enable configure hostname S5048F-Leaf1 protocol lldp advertise management-tlv management-address system-description system-name advertise interface-port-desc interface ManagementEthernet 1/1 ip address 192.168.255.1/30 no shutdown policy-map-input TrustDSCPin trust diffserv Next, the VLT interfaces between S5048F-Leaf1 and S5048F-Leaf2 are configured. In this configuration, add interfaces hundredGigE 1/53-54 to static port channel 127 for the VLT interconnect.
switchport spanning-tree rstp edge-port mtu 9216 service-policy input TrustDSCPin no shutdown interface twentygigabitethernet 1/5 description To R730xd-3 switchport spanning-tree rstp edge-port mtu 9216 service-policy input TrustDSCPin no shutdown interface twentygigabitethernet 1/7 description To R730xd-4 switchport spanning-tree rstp edge-port mtu 9216 service-policy input TrustDSCPin no shutdown interface twentygigabitethernet 1/48 description uplink from S3048-Management port-channel-protocol LACP port-
interface Vlan 1631 description ESXI_MGMT_IP ip address 172.16.31.251 tagged twentyFiveGigE 1/1,1/3,1/5,1/7 vrrp-group 1631 description ESXI_MGMT_IP priority 254 virtual-address 172.16.31.254 no shutdown interface Vlan 1632 description ESXI_VMOTION_IP mtu 9216 tagged twentyFiveGigE 1/1,1/3,1/5,1/7 no shutdown interface Vlan 1633 ip address 172.16.33.251 description SVM_MGMT_IP tagged twentyFiveGigE 1/1,1/3,1/5,1/7 vrrp-group 1633 description SVM_MGMT_IP priority 254 virtual-address 172.16.33.
Note: IP configuration for VLAN IDs 1634 and 1635 is optional. A gateway is not required for the two VxFlex data networks. If multiple VxFlex data network subnets are required, the SVMs must be modified (see Appendix B). 4.3 S3048-ON management switch configuration The following section outlines the configuration commands issued to S3048-ON management switch. The switches start at their factory default settings per Section 5.1. Initial configuration involves setting the hostname and enabling LLDP.
interface GigabitEthernet 1/5 description R730xd-3_IDRAC switchport no shutdown interface GigabitEthernet 1/7 description R730xd-4_IDRAC switchport no shutdown VLAN ID 1630, used for iDRAC and switch management, is created and an IP address is assigned. The downstream interfaces, to the VxFlex Ready Nodes’ iDRAC interfaces, are configured in the next set of commands. The upstream port channel to the S5048-ON is tagged. interface vlan 1630 description IDRAC_MGMT_IP & SW_MGMT ip address 172.16.30.
4.4 Verify switch configuration The following sections show commands and output to verify switches are configured and connected correctly. Except where there are fundamental differences, the only output shown is the S5048F-Leaf1 switch. The output from remaining devices is similar. 4.4.1 show vlt brief The Inter-chassis link (ICL) Link Status, Heart Beat Status, and VLT Peer Status must all be up.
4.4.3 show vlt mismatch Show VLT mismatch lists VLANs configured on a single switch in the VLT domain. The output shows the two VLANs associated with Data 1 and Data 2 are listed. S5048F-Leaf-1#sh vlt mismatch Domain -----Parameters Local -------------- Peer ---- Vlan-config -----------Vlan-ID Local Mode Peer Mode -------------------- ----------- 4.4.4 1634 L3 -- 1635 -- L3 show vrrp brief The output from the show vrrp brief command should be similar to that shown below.
5 VMware virtual network design In this section, tables are provided that outline the virtual network design used in this deployment. Specific steps to create the distributed switches, VMkernels, and setting NIC teaming policies are not covered in this document. See vSphere Networking Guide for vSphere 6.5, ESXi 6.5, and vCenter Server 6.5. 5.1 ESXi management The default VMkernel, vmk0 is used for ESXi management and is migrated from the default standard switch to the VDS created in this section.
Port group settings VDS Port group name Teaming policy Teaming and Failover VLAN ID atx01-w01vds01 atx01-w01-vds01-management Route based on physical NIC load Active: Uplink 1 and Uplink 2 1631 atx01-w01vds01 atx01-w01-vds01-vmotion Route based on physical NIC load Active: Uplink 1 and Uplink 2 1632 atx01-w01vds01 atx01-w01-vds01-VxFlexmanagement Route based on physical NIC load Active: Uplink 1 and Uplink 2 1633 atx01-w01vds01 atx01-w01-vds01-VxFlex-data01 Route based on originating v
5.4 VMware vSphere VMkernel configuration The following table contains the configuration details for the VxFlex VDS with four VMkernel adapters assigned (see Section 6).
Figure 11 is taken from Home > Hosts and Clusters and shows a successfully deployed VxFlex HCI platform.
6 Deploying Dell EMC VxFlex Deploying VxFlex in this environment consists of the following topics: • • • Register the VxFlex plug-in Upload the VxFlex Open Virtual Appliance (OVA) template Deploy VxFlex This section does not contain step-by-step instructions for deploying VxFlex. For a detailed step-by-step guide, see the ScaleIO IP Fabric Best Practice and Deployment Guide. 6.
6.3 Deploy VxFlex This section describes how the deployment wizard is used in the deployment example provided in this guide. VxFlex deployment has four steps: • • • • SDC deployment and configuration VxFlex advanced configuration settings Deploy the VxFlex environment Install the VxFlex GUI (optional) Before an ESXi host can consume the virtual SAN, the SDC kernel driver must be installed on each ESXi host, regardless of the role that host is playing.
DNS Server 2 172.16.11.5 3. Using the following table, select the VxFlex wizard parameter settings for steps 5 through 7. VxFlex Wizard deployment settings Parameter Setting Protection domain name PD01 RAM read cache size per SDS 1,024 MB Storage Pools SSD01 Enable zero padding True SDS host selection atx01w01esx05, atx01w01esx06, atx01w01esx07, atx01w01esx08 Selected devices All empty device categorized into the appropriate storage pool.
VxFlex networking addressing ESXi name Management IP Default gateway Data 1 IP Data 2 IP atx01w01esx08 172.16.33.11/24 172.16.33.253 172.16.34.11/24 172.16.35.11/24 172.16.33.12/24 172.16.33.253 172.16.34.12/24 172.16.35.12/24 172.16.33.13/24 172.16.33.253 172.16.34.13/24 172.16.35.13/24 172.16.33.14/24 172.16.33.253 172.16.34.14/24 172.16.35.14/24 172.16.33.15/24 172.16.33.253 172.16.34.15/24 172.16.35.
6.4 VxFlex GUI The VxFlex graphical user interface (GUI) can be installed on a management workstation to provide an easy way to monitor and configure the VxFlex system. Once installed, the virtual IP assigned to Data1 (172.16.34.4) can be used to access the VxFlex GUI. The installation file is part of the VxFlex for Windows download. The VxFlex cluster created is shown below in Figure 12.
7 Best practices The post-installation information provided in this section consists of the following: • • Increase the Maximum Transmission Unit (MTU) for VMware vSphere, and Dell EMC VxFlex Configure Quality of Service using Differentiated Services (DiffServ) For additional performance tuning, including ESXi hosts and VxFlex VMs, see the VxFlex OS v2.x Performance Fine-Tuning Technical Notes Guide. 7.
To enable jumbo frames for the SVM, perform the following steps: 5. Run the ifconfig command to get the NIC information. The following is an example from an SVM deployed in this solution, ScaleIO-172.16.33.12: ScaleIO-172-16-33-12:~ # ifconfig eth0 Link encap:Ethernet HWaddr 00:50:56:B7:81:28 inet addr:172.16.33.12 Bcast:172.16.33.255 Mask:255.255.255.
ScaleIO-172-16-33-12:~ # service network restart Shutting down network interfaces: eth0 device: VMware VMXNET3 Ethernet Controller eth1 device: VMware VMXNET3 Ethernet Controller Eth2 device: VMware VMXNET3 Ethernet Controller Shutting down service network . . . . . . . . . Hint: you may set mandatory devices in /etc/sysconfig/network/config Setting up network interfaces: eth0 device: VMware VMXNET3 Ethernet Controller eth0 IP address: 172.16.33.
policy-map-input TrustDSCPin trust diffserv interface range tw1/1,tw1/3,tw1/5,tw1/7 description VxFlex nodes service-policy input TrustDSCPin DSCP values are inserted on a port-group basis. In the table below two port groups are enabled to filter traffic, atx01-w01-vds01-management, and atx01-w01-vds01-VxFlex-management. To create marking navigate to Home > Networking > port group > Edit > Traffic filtering and marking. Table 18 shows the values used.
A Troubleshooting SDS connectivity SDS connectivity problems affect VxFlex performance. VxFlex has a built-in tool to verify all SDS nodes in a given protection domain have connectivity. From the VxFlex Command Line Interface (SCLI), run the VxFlex internal network test to verify the network speed between all the SDS nodes in the Protection Domain. The command below tests all SDS nodes with a payload of 10 GB, using 8 parallel threads. ScaleIO-172-16-33-12:~ # scli --mdm_ip 172.16.33.
B Routing VxFlex Virtual Machine traffic In this section a possible solution to solve routing between SVMs in separate subnets is outlined. Each SVM contains three virtual NICs: • • • Eth0 for VxFlex management Eth1 for VxFlex Data01 Eth2 for VxFlex Data02 The SVM uses a single TCP/IP stack, and any unknown networks are limited to this single default gateway. If VxFlex Data01 or VxFlex Data02 needs to reach an SVM in another subnet, for instance in another rack in the data center, this traffic fails.
# #1 100 200 inr.ruhep eth1 eth2 6. For each interface enable PBR. Below is an example for eth1 and needs to be repeated for eth2. ScaleIO-172-16-33-12:~ ScaleIO-172-16-33-12:~ link table eth1 ScaleIO-172-16-33-12:~ eth1 ScaleIO-172-16-33-12:~ # ip route flush table eth1 # ip route add 172.16.34.0/24 dev eth1 proto kernel scope # ip route add default via 172.16.34.253 dev eth1 table # ip rule add from 172.16.34.0/24 lookup eth1 7.
Note: This solution is not persistent across reboots and needs to be automated. That effort is left to the reader.
C Validated hardware and components The following tables list the hardware and components used to configure and validate the example configurations in this guide. C.1 Dell EMC Networking switches Qty 1 Item S3048-ON - Management switch OS/Firmware version OS: DNOS 9.13.0.0 System CPLD: 9 Module CPLD: 7 2 S5048F-ON - Leaf switch OS: DNOS 9.12(1.0) CPLD: 1.0 C.2 42 Dell EMC PowerEdge 730xd servers Qty per server Item Firmware version 2 Intel(R) Xeon(R) CPU E5-2698 v4@2.2.
D Validated software The Software table lists the software components used to validate the example configurations in this guide. 43 Item Version Dell EMC VxFlex 2.0.-14000.228 VMware vSphere Power CLI 6.5.0 VxFlex vSphere Plug-in Installer 2.0.1.4 SVM OVA 2.0.1.4000.288.ova VMware ESXi 6.5 U1 - Dell EMC customized image build 6765664 VMware vCenter Server Appliance 6.
E Fabric Design Center The Dell EMC Fabric Design Center (FDC) is a cloud-based application that automates the planning, design and deployment of network fabrics that power Dell EMC compute, storage and hyper-converged infrastructure solutions, including VxFlex. The FDC is ideal for turnkey solutions and automation based on validated deployment guides like this one. FDC allows design customization and flexibility to go beyond validated deployment guides.
F Product manuals and technical guides F.1 Dell EMC Dell EMC Knowledge Library - An online technical community where IT professionals have access to numerous resources for Dell EMC software, hardware, and services. Dell EMC Networking Guides VxFlex Ready Nodes solution overview Manuals and documentation for Dell Networking S5048F-ON Manuals and documentation for Dell Networking S3048-ON Manuals and documentation for PowerEdge R730xd VxFlex OS: Documentation Library VxFlex Ready Node comparisons F.
G Support and feedback Contacting Technical Support Support Contact Information Web: http://support.dell.com/ Telephone: USA: 1-800-945-3355 Feedback for this document We encourage readers to provide feedback on the quality and usefulness of this publication by sending an email to Dell_Networking_Solutions@Dell.com.
