Dell EqualLogic Best Practices Series Scaling and Best Practices for Implementing VMware vSphere Based Virtual Workload Environments with the Dell EqualLogic FS7500 A Dell Technical Whitepaper Storage Infrastructure and Solutions Engineering Dell Product Group July 2012
THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND. © 2012 Dell Inc. All rights reserved. Reproduction of this material in any manner whatsoever without the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell.
Table of contents 1 2 3 Introduction ....................................................................................................................................................... 2 1.1 Audience ..................................................................................................................................................... 2 1.2 Terms ............................................................................................................................................
5.3 Network..................................................................................................................................................... 41 5.3.1 SAN design ........................................................................................................................................ 41 5.3.2 LAN design ........................................................................................................................................
Acknowledgements This whitepaper was produced by the PG Storage Infrastructure and Solutions team based on testing conducted between August 2011 and February 2012 at the Dell Labs facility in Austin, Texas.
Executive summary Use of Network File System (NFS) based datastores for a VMware virtual environment is becoming increasingly popular due to the ease of implementation and management capabilities available with Network Attached Storage (NAS) solutions. Some of the challenges system administrators face while optimizing NAS based solutions for VMware includes the lack of native multipathing options within the hypervisor like those available for SAN based storage.
1 Introduction Today datacenter administrators in businesses of all sizes are facing an efficiency challenge. Due to the rapid growth of collaboration and e-business applications along with regulatory compliance requirements, server and storage needs are continually growing while IT budgets are either stagnant or shrinking.
By default, a group provides a single pool of storage. If there are multiple members, group space can be divide into different storage pools and then members assigned to each. Pools help organize storage according to application usage and provide a way to allocate resources, from a single-system management view. EqualLogic Volume Volumes are created to access storage space in a pool and are mounted on host operating systems as physical drives. Volume size and attributes can be modified on-demand.
2 Overview of Dell EqualLogic FS7500 Unified Storage solution The FS7500 is a high performance unified storage solution that integrates with new and existing EqualLogic PS Series storage arrays and enables customers to easily configure and manage iSCSI, CIFS, and NFS access in a single flexible storage pool using the EqualLogic Group Manager. The unique scale-out architecture of the solution lets customers expand storage capacity and/or system performance as needs change over time.
Figure 1 Typical high level architecture of virtualized environment with unified storage In SAN environments, also referred to as block based storage, VMware ESXi hosts use VMFS formatted block volumes for datastores. Typically ESXi servers will connect to the storage via a SAN which will be Ethernet based for iSCSI.
Dell EqualLogic PS Series arrays support these VAAI operations, and more information is available in this technical report - http://www.equallogic.com/WorkArea/DownloadAsset.aspx?id=10307. Another option for storing data in virtualized environment is NAS storage, also referred to as file based storage. In NAS environments, VMware ESXi servers will mount NFS exports as datastores via a LAN network.
3 NFS network and host configuration To develop the best practices for networking and ESX host configuration with the FS7500, we created a test configuration to simulate the I/O patterns of different application workloads. We ran three tests to ensure that the configuration was optimized to handle sequential write, sequential read, and random read/write workloads. The next sections cover the test results and best practices including the configuration for the tests. 3.
Figure 2 Hardware architecture for networking and host configuration studies BP1024 Scaling and Best Practices for Virtual Workload Environments with the FS7500 8
• A virtual switch was built on each ESXi host that was used to communicate with the NAS file systems exported by the FS7500 appliance. A VMkernel (vmknic) port and four uplink NICs were assigned to the vSwitch as shown in Figure 3. Figure 3 vSwitch on host 3.2 Test workloads Iometer workloads were run on 300 GB virtual hard drives assigned to the virtual machines. Depending on the type of test, up to eight virtual hard drives (vmdk) were assigned to each VM.
associated with the iSCSI software initiator and are load balanced using iSCSI MPIO stack within the host. For more information on configuring iSCSI MPIO on VMware with EqualLogic storage, please refer to the following technical reports: Configuring and Installing the EqualLogic Multipathing Extension Module for VMware vSphere 4.1 and PS Series SANs - http://www.equallogic.com/WorkArea/DownloadAsset.
Figure 4 vSwitch with MAC Based or Port based load balancing Using MAC or port-based vSwitch load balancing schemes does not provide the increased bandwidth or redundancy that can be achieved by using multiple physical NIC uplinks from the ESX host for NFS traffic. In order to balance the load on the physical uplink NICs on vSwitch using the single vmknic port, we have to rely on the IP hash-based load balancing policy available on the vSwitch.
Figure 5 vSwitch with IP Hashing load balancing To configure the ESXi host vSwitch for IP hash-based load balancing policy, use the following steps: Configure ESXi vSwitch and vmknic port: Build a virtual switch on each ESXi host to communicate with the NAS client network. Assign a vmknic port and two or more physical uplink NICs to it. 2.
Figure 6 Setting load balancing policy on the vSwitch 3. Configure corresponding switch ports to static LAG with the appropriate Hashing mode: Ensure that the participating physical uplink NICs on vSwitch are connected to the corresponding ports on the external physical switch configured with static link aggregation. For these ports on the switch, make a port-group to perform 802.3ad link aggregation in static mode. (See this procedure in Appendix B).
4. Configure NFS target to have multiple IP addresses: On the EqualLogic FS7500, configure the NAS service with multiple NAS service IP addresses (see Figure 7). In our configuration, we created eight NAS service IP addresses (maximum available with one FS7500). Load balancing techniques on the physical switch and the ESXi host are based on IP hashing. Therefore, more IPs are used in communication and more host, target NICs, and intermediate switch ports are used, which increases the overall throughput.
Figure 8 Multiple datastore mounts to same NFS export via multiple NAS services IP addresses We conducted sequential write I/O studies using Iometer from a single VM running on a single ESXi host. The vSwitch was studied with the three different load balancing policies. A vmknic was configured for mounting NFS exports as ESX datastores. For all of the load balancing algorithms, we mounted the same NFS export from the FS7500 using four different NAS service IP addresses as four different datastores.
Write throughput (MBps) 500 450 400 350 300 250 200 150 100 50 0 Port based load balancing MAC based load balancing Maximum available bandwidth from one ESX host (4 x 1GbE) utilized IP Hash based load balancing Figure 9 Write throughput with different load balancing techniques Since port-based and MAC-based load balancing maps traffic to a single NIC uplink, the measured throughput was close to the line rate of one NIC.
Figure 10 Single or multiple NAS file systems (FS) mapped to datastores We ran sequential read/write tests on the two configurations. In the first configuration, the NAS reserve on the FS7500 had one NAS file system with one NFS export. We mounted the NFS export on the ESXi host server as eight datastores. We used a separate NAS service virtual IP address for each of the eight mounts. The host vSwitch was configured with 4 x 1GbE uplinks.
500 450 Throughput (MBps) 400 350 300 250 Read 200 Write 150 100 50 0 1 8 Number of NAS file systems Figure 11 Throughput with single and multiple NAS file systems We did not find a significant difference in throughput behavior in either case as seen in Figure 11. We achieved close to line rate throughput in both cases. From a performance optimization standpoint, it does not make a difference to deploy a single or multiple NAS file systems within the FS7500.
450 400 Throughput MB/sec 350 300 250 200 150 100 50 0 1 2 3 4 Number of NICs used Figure 12 Increase in write throughput with increase in number of NICs and Virtual IP addresses With more NICs, we saw an increase in throughput measured. We tested configurations from one NIC up to eight NICs on the ESX host.
Figure 13 FS7500 NAS Service IP settings On the ESXi host, use alias connections to the same NFS export and mount multiple datastores each using different NAS service virtual IP addresses to the same export. In this configuration, multiple virtual IP addresses load balanced to separate NICs on the storage side and on the host side are utilized. On the host side, vSwitch load balancing will spread traffic across multiple host NICs.
The following figure shows logical connections (NFS mounts) between the ESXi hosts and the NFS exports on the FS7500.
Figure 15 Utilization of multiple IPs and datastores to same NFS file system It must be noted that the workload on ESXi hosts should be reasonably uniform over all the datastores to get optimal throughput and utilization of all host side and storage side NICs. This is because we distributed datastore mounts across different target virtual IP addresses which are mapped to different host NICs and target NICs based on load balancing schemes on the host, target, and network.
Figure 16 Read I/O throughput improvement with multiple hosts We simulated sequential read I/O from VMs using Iometer. In the first and second test, we used one and two hosts to connect to FS7500 respectively. Each host was configured with a vSwitch with four uplink NICs. We used a single NFS export from the FS7500 mounted as eight different datastores using eight different NAS service IP addresses. In the first case, all datastores were mounted on single host.
Read Throughput (MBps) 400 350 300 250 200 150 100 50 0 1 2 Number of hosts Figure 17 Increase in read throughput with increase in number of hosts When the number of hosts increased, we saw an increase in aggregate read throughput measured. The figure above shows that doubling the hosts almost doubled (194%) the read throughput. The FS7500 can deliver the optimal levels of read and write throughput based on appliance and disk configuration.
Write throughput (MBps) for 16 datastores 500 400 300 200 100 0 recommendations implemented recommendations not implemented Read throughput (MBps) for 16 datastores 250 200 150 100 50 0 recommendations implemented recommendations not implemented Figure 18 Read and write throughput with and without implementing VMware specific recommendations 3.
four machines are cloned, it is expected to take four times as long which is 240 minutes. However, by doing it over multiple IP addresses 38% of the time was saved. Similarly, for cloning eight machines, 31% of the expected time was saved. This is because of the additional throughput achieved by load balancing NFS traffic over multiple physical NICs when multiple NAS service IP addresses are in use.
4 Scalability of FS7500 A series of tests were conducted to study the scalability of the FS7500 solution with NFS exports hosted as ESXi datastores. The network and host configuration were based on the findings described in section 3. 4.1 Test workload 4.1.1 Workload modeling and application components A datacenter typically consolidates several diverse workloads onto a virtualization platform—a collection of physical servers accessing shared storage and network resources.
Table 3 Planned workload components Application # of VM’s Planned IOPS per VM Iometer I/O profile Typical VM resource allocation Windows Servers (for running Active Directory, Domain Controllers etc.
4.2 Test configuration 4.2.1 FS7500 controllers and EqualLogic PS Series arrays For the baseline test, three EqualLogic PS6100X arrays and one FS7500 system (two controllers) were used. The NAS reserve was divided into three NAS file systems; the first one to store the virtual hard drives (VMDKs) and OS, the second one to store the simulated application database data, and the third one to store the simulated application log file data.
Figure 20 Host to FS7500 connections for one FS7500 system In the next configuration, one FS7500 system (2 x controllers) and six PS6100X arrays were used. The number of datastores and IP address access policy was the same. Only the capacity of each datastore was expanded. The intent was to study the scalability of the solution when the PS Series arrays were doubled. In the third configuration, two FS7500 systems (4 x controllers) and six PS6100X arrays were used.
Figure 21 Host to FS7500 connections for 2 x FS7500 systems A logical view of the three scaling test configurations described above is illustrated in figure below. The FS7500 systems and the PS6100X arrays in each of the configurations are illustrated.
Figure 22 Three scaling test configurations BP1024 Scaling and Best Practices for Virtual Workload Environments with the FS7500 32
4.2.2 Host and Hypervisor Two PowerEdge R815s were used as ESXi hosts. Both of the hosts access the three datastores using the eight VIP addresses for the first and second configuration and 16 VIP addresses for third configuration. For the first configuration, one block consisting of 20 VMs was equally divided over these two hosts such that the I/O load on each host was equal. For the second and third configuration, two (40 VMs) and three (60 VMs) blocks respectively were divided between the two hosts.
Figure 24 Physical Network Connectivity BP1024 Scaling and Best Practices for Virtual Workload Environments with the FS7500 34
The physical connectivity details illustrated in Figure 24 are explained below: • • • • • • • • Six PS6100X series arrays (three PS Series arrays for first test and six PS Series arrays for the second and third tests) were used for scaling of FS7500. The arrays were connected in redundant fashion to two Dell PowerConnect 7048 stacked switches.
Table 4 I/O mix for FS7500 scaling studies Application # of VMs Per block 1 Planned IOPS per VM Actual IOPS per VM Test 1 Test 2A (2 blocks) *20 Test 2B (2.33 blocks) *24 Test 3 (3 blocks) *20 Windows 30 *20 Servers (for running Active Directory, Domain Controllers etc.
The load on all the VMs was proportionally increased to increase the response time for the One FS7500-six array configuration to 20 ms. The I/O mix achieved for this configuration is listed in Table 4 per single block in the column labeled Test 2B. Test 3: For the third test, three blocks of VMs were used to saturate the Two FS7500 – six array configuration. Response time was below 20ms. The final achieved I/O mix for this configuration is given in Table 4 per single block as Test 3. 4.3.
14000 12000 IOPS 10000 8000 6000 4000 2000 0 1 x FS7500 + 3 x 1 x FS7500 + 6 x PS6100X (1 VM Blocks) PS6100X (2 VM Blocks) 1 X FS7500 + 6 x PS6100X (2.33 VM Blocks) Figure 26 IOPS with scaling PS Series arrays On the One FS7500 - six array configuration with two VM blocks, response time was well below 20 ms while IOPS doubled in value compared to the One FS7500-three array configuration. There was headroom to increase the I/O workload based on our 20ms response time criteria.
20 I/O Response Time (ms) 18 16 14 12 10 8 6 4 2 0 1 x FS7500 + 3 x PS6100X (1 VM Blocks) 2 x FS7500 + 6 x PS6100X (2 VM Blocks) Figure 27 I/O Response time with scaling FS7500 and PS Series arrays 18000 16000 14000 IOPS 12000 10000 8000 6000 4000 2000 0 1 x FS7500 + 3 x PS6100X (1 VM Blocks) 2 x FS7500 + 6 x PS6100X (3 VM Blocks) Figure 28 IOPS with scaling FS7500 and PS Series arrays Doubling both the number of arrays and the number of FS7500 NAS appliances yielded approximately three times the
5 Best practices This section provides the best practices for the design and deployment of VMware vSphere virtualized infrastructure on EqualLogic FS7500. 5.1 Storage 5.1.1 • • • 5.1.2 • • • EqualLogic FS7500 recommendations Configure the maximum number of NAS service virtual IP addresses on the EqualLogic FS7500 so that the load can be distributed across all FS7500 client network NICs and controller resources to improve I/O performance and throughput.
5.3 Network You should use separate network infrastructure (network switches) for the client network (LAN) and the FS7500 SAN + Internal Communication network. 5.3.1 • • • • • • • SAN design Redundant inter-connected switches or switching fabrics are recommended for the SAN network infrastructure to ensure high-availability. Switch inter-connects (stack or LAG) should be properly sized with to provide the required bandwidth for the PS Series arrays.
Appendix A Solution hardware components Solution configuration - Hardware components: • ESXi hosts 2 x Dell PowerEdge R815 Servers: o ESXi 5.0 o BIOS Version: 1.5.2 o 4 x AMD Opteron(tm) Processors 6174, 2200 MHz. 12 cores o 96 GB RAM o 6 internal disk drives o NIC – Broadcom quad-port 1 Gb, Firmware: 6.0.1 BC 5.2.
Appendix B Switch configuration Use this script to turn on source and destination IP hashing on the switch: ********************************************************************************** enable configure interface port-channel 1 no shut mtu 9216 exit interface range gigabit 1/0/1-2 ESXi host** ** ports used to communicate with channel-group 1 mode ON ** default hashing is based on source and destination IP hash** mtu 9216 exit copy running-config startup-config exit BP1024 Scaling and Best Practices
Additional resources Referenced or recommended Dell publications: • • • • • • • Dell EqualLogic PS Series Network Performance Guidelines: http://www.equallogic.com/resourcecenter/assetview.aspx?id=5229 Dell EqualLogic PS Series Arrays: Advanced Storage Features in VMware vSphere: http://www.equallogic.com/WorkArea/DownloadAsset.aspx?id=10307 Configuring and Installing the EqualLogic Multipathing Extension Module for VMware vSphere 4.1 and PS Series SANs - http://www.equallogic.com/WorkArea/DownloadAsset.
THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND.
