Dell EMC Networking OS10 Enterprise Edition Switch Configuration Guide for VxRail 4.5 Deploying S5148F-ON switches for a VxRail 4.5 cluster Abstract This document provides Dell EMC Networking OS10 Enterprise Edition switch configuration examples and topology options for a VxRail 4.5 cluster deployment using S5148F-ON.
Revisions Date Description August 2019 Removed Chapter 10 covering VxRail 4.7 with S5200 series switches. This information is now provided in Dell EMC SmartFabric OS10 Switch Configuration Guide for VxRail 4.7. Changed spanning tree protocol and bridge priority to use OS10 defaults. Added appendix for VxRail node discovery troubleshooting. March 2019 Changed switch node-facing interface settings from “flowcontrol transmit on” to “flowcontrol transmit off” as a best practice.
Table of contents Revisions.............................................................................................................................................................................2 1 2 Introduction ...................................................................................................................................................................5 1.1 Supported switches and operating systems ................................................................................
7.1.3 Configure interfaces .........................................................................................................................................23 7.1.4 Configure VLT...................................................................................................................................................24 7.2 Dual switch without VLT ...................................................................................................................................25 7.2.
1 Introduction VxRail sits at the forefront of a fundamental shift in IT infrastructure consumption – away from applicationspecific, “build-your-own” infrastructure and toward virtualized, general-purpose, engineered systems. Dell EMC and VMware have embraced this shift with the VxRail hyper-converged appliance. VxRail has a simple, scale-out architecture that leverages VMware vSphere and VMware vSAN to provide server virtualization and software-defined storage.
1.1 Supported switches and operating systems The examples provided in this deployment guide use VxRail 4.5 nodes connected to S5148F-ON switches running Dell Networking OS10EE. Dell EMC Networking supports the following switch and OS combinations for VxRail 4.0 and later: Supported Dell EMC PowerSwitches and operating systems 1.2 Typographical conventions The CLI and GUI examples in this document use the following conventions: 1.
2 Hardware overview This section briefly describes the hardware used to validate this deployment. Appendix A and Appendix B contain a complete listing of hardware and software validated for this guide. 2.1 Dell EMC PowerSwitch S5148F-ON The S5148F-ON is a 1-RU, multilayer switch with 48x25GbE, and 6x100GbE or 72x25GbE ports. This guide uses two S5148F-ONs as leaf switches. Dell EMC PowerSwitch S5148F-ON 2.
Note: The deployment examples in this guide use a VxRail cluster consisting of four P570 nodes for VxRail 4.5. 2.3.1 Dell EMC VxRail P Series node VxRail P Series nodes are ideal for CPU-intensive workloads such as databases. P Series nodes support up to 44 CPU cores, 1536GB memory, and 24TB hybrid or 46TB all-flash storage. 2.3.2 Dell EMC VxRail V Series node VxRail V Series nodes are graphics-ready for uses such as high-end 2D/3D visualization.
VxRail optimizes network traffic by splitting it across rNDC uplinks and by using Network I/O control (NIOC) shares for different traffic types. See Appendix C for more information. P, V, S, and E Series VxRail nodes also include a 1GbE BASE-T integrated Dell Remote Access Card (iDRAC) for OOB management. Note: P, V, and S Series nodes support additional network adapters in PCIe slots. See the Dell EMC VxRail Network Guide for more information.
3 Topology options VxRail may be deployed using a single or dual switch topology. Using a single switch provides the lowest initial cost but creates a single point of failure. A dual switch configuration helps ensure high availability by eliminating this single point of failure. A dual switch configuration may be used with or without VLT. Dell EMC recommends a dual switch configuration with VLT. The sections that follow explain the different options. 3.
• • • • Provides link-level resiliency Assures high availability Allows a single device to use a LAG across two upstream switches Provides Layer 2 multipathing Note: While VxRail nodes use active and standby network adapters instead of LAGs, other servers in the rack can connect to the VLT switch pair with an LACP LAG for active/active Layer 2 multipathing. For more information on VLT, see the OS10 Enterprise Edition User Guide Release 10.4.1.0. 3.1.
3.2 Single switch In a single switch topology, all VxRail nodes connect to a single switch. This topology is lower initial cost, but it is not recommended as the switch becomes a single point of failure. S5148F-ON 1 2 VxRail node 1 1 2 25GbE connections VxRail node 2 1 2 VxRail node n Rack 1 Single switch topology 12 Dell EMC Networking OS10 Enterprise Edition Switch Configuration Guide for VxRail 4.
4 VxRail in the data center 4.1 Leaf-spine network Dell EMC recommends using a leaf-spine network in the data center with leaf switches configured as VLT peers. The switches and VxRail nodes covered in this guide are shown in Rack 1 in Figure 10 and are incorporated into a data center’s leaf-spine network. Spine1 Spine2 Leaf-2A Leaf-2B L3 L2 L3 S5148F-Leaf1B S5148F-Leaf1A VLTi Leaf-nA Leaf-nB VLTi VLTi Additional VxRail clusters, servers, storage, etc.
4.2 OOB management network The out-of-band (OOB) management network is an isolated network for remote management of servers, switches, and storage devices via their dedicated hardware management ports. It is also commonly used to carry heartbeat messages sent between switches configured as VLT peers.
5 Configuration planning 5.1 VLANs and IP addresses VLANs and IP addresses used for VxRail node traffic must be planned before switch configuration and VxRail deployment can begin. VxRail node traffic is divided into four or more VLANs, as shown in Table 1.
5.2 VxRail network configuration table Information provided in the VxRail network configuration table is used during VxRail deployment. The values used for this deployment example are shown in the right column. The VLANs and IP addresses used are based on the information from Table 2. Note: For additional information on the VxRail network configuration table, see the Dell EMC VxRail Network Guide.
Row 35 36 37 38 39 40 41 42 43 Category vSAN VM Networks … (unlimited number) Solutions Logging 44 45 Description Values used Subnet mask VLAN ID Starting address for IP pool Ending address for IP pool Subnet mask VLAN ID VM Network name and VLAN ID 255.255.255.0 1612 172.16.13.1 172.16.13.40 255.255.255.0 1613 VM_Network_A, 1614 VM Network name and VLAN ID vRealize Log Insight™ hostname VM_Network_B, 1615 vxinsight vRealize Log Insight IP address Syslog server (instead of Log Insight) 172.
Hostname IP Address vxinsight.dell.local 172.16.11.103 An NTP server is not required but is recommended. If an NTP server is not provided, VxRail uses the time that is set on VxRail node 1. Note: For this deployment guide, the PowerEdge server used as the jump box also provides DNS and NTP services for the VxRail cluster. It is connected to both leaf switches for redundancy using Microsoft Windows NIC teaming (switch-independent load balancing).
6 Switch configuration prerequisites 6.1 Check switch OS version Run the show version command to check the OS version. Dell EMC recommends upgrading to the latest supported release available on Dell Digital Locker (account required). OS10# show version S5148F-ON# show version Dell EMC Networking OS10-Enterprise Copyright (c) 1999-2018 by Dell Inc. All Rights Reserved. OS Version: 10.4.1.1 Build Version: 10.4.1.1.
6.3 Factory default configuration The configuration commands in the sections that follow begin with S5148F-ON switches at their factory default settings. Dell EMC PowerSwitches running OS10EE can be reset to their default configuration as follows: OS10# delete startup-configuration Proceed to delete startup-configuration [confirm yes/no(default)]:y OS10# reload System configuration has been modified.
7 Configure switches This section covers switch configuration for S5148F-ON switches in the different topologies. Commands for all examples are provided as attachments to this guide. 7.1 Dual switch with VLT This example uses a four-node VxRail cluster connected to a pair of switches configured with VLT as shown below. Dell EMC recommends using this topology.
Note: Default spanning tree settings are used in this deployment. In OS10EE, RVPST+ is enabled by default, and RPVST+ VLAN priority numbers start at 32769. Modify the spanning tree settings if required for your environment. LLDP is enabled on each interface and globally by default. LLDP is useful for troubleshooting and validation. Dell EMC recommends leaving it enabled. General settings – dual switch with VLT 7.1.
7.1.3 S5148F-Leaf1A S5148F-Leaf1B interface vlan1615 description VM_Network_B no shutdown interface vlan1615 description VM_Network_B no shutdown Configure interfaces Configure the interfaces for the jump box or laptop computer connections to be used during VxRail deployment. This example uses interface 1/1/9 on each switch. While only one connection is required, two are used for redundancy. Add these interfaces as access ports on VLAN 1611 (the in-band management VLAN).
7.1.
7.2 S5148F-Leaf1A S5148F-Leaf1B vlt-domain 127 backup destination 100.67.172.37 discovery-interface ethernet1/1/531/1/54 peer-routing vlt-domain 127 backup destination 100.67.172.38 discovery-interface ethernet1/1/531/1/54 peer-routing end write memory end write memory Dual switch without VLT This example uses a four-node VxRail cluster connected to a switch pair without VLT as shown below.
Note: The commands in the tables below should be entered in the order shown. All commands are provided in the file attachments named S5148F-1A-no-vlt.txt and S5148F-1B-no-vlt.txt. 7.2.1 General settings Configure the hostname, OOB management IP address, and default gateway. Specify an NTP server accessible by the switch. In this example, this is an NTP server on the OOB management network, with a different IP address than the one used by the VxRail nodes on the in-band management network.
7.2.
7.2.
Configure switch interconnect - dual switch without VLT 7.
Note: The commands in the tables below should be entered in the order shown. All commands are provided in the file attachment named S5148F-single-switch.txt. 7.3.1 General settings Configure the hostname, OOB management IP address and default gateway. Specify an NTP server accessible by the switch. In this example, this is an NTP server on the OOB management network, with a different IP address than the one used by the VxRail nodes on the in-band management network.
S5148F-ON description VM_Network_A no shutdown interface vlan1615 description VM_Network_B no shutdown 7.3.3 Configure interfaces Configure the interfaces for the jump box or laptop computer connections to be used during VxRail deployment. This example uses interfaces 1/1/9 and 1/1/10. While only one connection is required, two are used for redundancy. Add these interfaces as access ports on VLAN 1611 (the in-band management VLAN).
S5148F-ON switchport trunk allowed vlan 1612-1615 spanning-tree port type edge flowcontrol receive on flowcontrol transmit off no shutdown interface ethernet1/1/18 description Node1_Port2 switchport mode trunk switchport access vlan 1611 switchport trunk allowed vlan 1612-1615 spanning-tree port type edge flowcontrol receive on flowcontrol transmit off no shutdown interface ethernet1/1/19 description Node2_Port1 switchport mode trunk switchport access vlan 1611 switchport trunk allowed vlan 1612-1615 spanni
S5148F-ON interface ethernet1/1/23 description Node4_Port1 switchport mode trunk switchport access vlan 1611 switchport trunk allowed vlan 1612-1615 spanning-tree port type edge flowcontrol receive on flowcontrol transmit off no shutdown interface ethernet1/1/24 description Node4_Port2 switchport mode trunk switchport access vlan 1611 switchport trunk allowed vlan 1612-1615 spanning-tree port type edge flowcontrol receive on flowcontrol transmit off no shutdown end write memory 33 Dell EMC Networking OS10
8 Switch validation After switches are configured and devices are connected, the switch CLI is used to validate the network configuration. This section provides a list of the most common commands and their output for the examples used in this guide. 8.1 General validation commands The commands and output shown below are for S5148F-Leaf1A in the dual switch with VLT topology. The output of its peer, S5148F-Leaf1B, is similar.
Note: The command output for the dual switch without VLT topology is shown below. In this example, Port channel 127 is up (U) and DYNAMIC indicates LACP is used. Ports 1/1/53 and 1/1/54 are port channel members and (P) indicates each is up and active.
Note: For the dual switch without VLT topology, the VLAN portion of the command output is as follows: NUM Status Description 1 Active 1/1/16,1/1/21-1/1/52 1611 Active InBand_Mgmt 1612 Active vMotion 1613 Active vSAN 1614 Active VM_Network_A 1615 Active VM_Network_B Q Ports A Eth1/1/1-1/1/8,1/1/10A T A T T T T T T T T Po127 Po127 Eth1/1/9,1/1/17-1/1/20 Eth1/1/17-1/1/20 Po127 Eth1/1/17-1/1/20 Po127 Eth1/1/17-1/1/20 Po127 Eth1/1/17-1/1/20 Po127 For the single switch topology, the VLAN portion
8.1.5 show vrrp brief In this deployment, VRRP is configured on the in-band management VLAN, 1611, to provide gateway redundancy for management traffic. The output from the show vrrp brief command shows the master and virtual IP addresses and whether the switch is in the master or backup role. The switch configured with the largest priority value, shown in the Priority column, becomes the master.
8.2.2 show vlt domain_id backup-link This command is used to verify VLT peers are communicating on the backup link over the OOB management network. The Destination is the management IP address of the peer. The Peer HeartBeat status must be Up. S5148F-Leaf1A#show vlt 127 backup-link VLT Backup Link -----------------------Destination : 100.67.172.37 Peer Heartbeat status : Up Heartbeat interval : 30 Heartbeat timeout : 90 8.2.
9 Deploy VxRail A laptop computer or jump box with a web browser for the VxRail user interface is required. It is either plugged into a leaf switch or able to logically reach the VxRail in-band management VLAN from elsewhere on the network. By default, the initial VxRail Manager IP address is 192.168.10.200/24. After initial configuration, the address changes to its new address on the in-band management network. The new VxRail Manager address used in this guide is 172.16.11.
In a web browser on the laptop computer or jump box, connect to https://192.168.10.200 and deploy VxRail using the planning data in Table 3. Note: Step-by-step VxRail deployment GUI screenshots are beyond the scope of this guide. 40 Dell EMC Networking OS10 Enterprise Edition Switch Configuration Guide for VxRail 4.
A Validated components for S5148F-ON using VxRail 4.5 The following tables include the hardware, software, and firmware used to configure and validate the examples in this guide. A.1 Dell EMC PowerSwitches Switches and OS versions A.2 Qty Item OS Version 2 Dell EMC PowerSwitch S5148F-ON leaf switches 10.4.1.1 1 Dell EMC PowerSwitch S3048-ON management switch 10.4.1.1 DAC Cables DAC cables used to connect nodes to ToR Qty Item Dell PN 8 DAC-SFP28-25G-3.
A.4 VxRail Appliance software The examples using S5148F-ON in this guide were developed using VxRail Appliance software 4.5.200. It consists of the component versions shown below. VxRail Appliance software component versions 42 Item Version VxRail Manager 4.5.200.7769314 VMware ESXi 6.5.0 Update 1 Patch 36, build 7388607 VMware vCenter Server Appliance 6.5 Update 1e, build 7515524 VMware vSAN 6.6.1 Patch 02 VMware vRealize Log Insight 4.
B VxRail network adapter traffic optimization VxRail optimizes network traffic by splitting traffic across rNDC uplinks and by using NIOC shares. Based on traffic type, uplinks are configured as active or standby, and NIOC shares are used to allocate network bandwidth. Uplink load balancing and NIOC shares are automatically configured by VxRail.
C VxRail node discovery troubleshooting If supported leaf switches other than S5148F-ON, S4248FB-ON, or S4248FBL-ON are used, and the OS version is 10.4.3.0 or later, also run the following command on each leaf switch to enable node discovery: (config)# no multicast snooping flood-restrict Note: The above command is not supported on S5148F-ON, S4248FB-ON, or S4248FBL-ON switches or for switches running an OS version prior to 10.4.3.0.
D Technical resources Dell EMC Networking Guides OS10 Enterprise Edition User Guide Release 10.4.1.0 OS10 Enterprise Edition User Guide Release 10.4.2.0 OS10 Enterprise Edition User Guide Release 10.4.3.
E Support and feedback Contacting Technical Support Support Contact Information Web: http://www.dell.com/support Telephone: USA: 1-800-945-3355 Feedback for this document We encourage readers to provide feedback on the quality and usefulness of this publication by sending an email to Dell_Networking_Solutions@Dell.com. 46 Dell EMC Networking OS10 Enterprise Edition Switch Configuration Guide for VxRail 4.
