Dell EqualLogic Best Practices Series Scalability & Deployment Best Practices for the Dell EqualLogic FS7500 NAS System as a File Share Solution A Dell Technical Whitepaper Storage Infrastructure and Solutions Engineering Dell Product Group February 2012
THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND. © 2011 Dell Inc. All rights reserved. Reproduction of this material in any manner whatsoever without the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell.
Table of Contents 1 Introduction ....................................................................................................................................................... 2 1.1 Audience ..................................................................................................................................................... 2 1.2 The rest of this paper ................................................................................................................................
Acknowledgements This whitepaper was produced by the PG Storage Infrastructure and Solutions team between March 2011 and August 2011 at the Dell Labs facility in Round Rock, Texas.
1 Introduction 1 The Dell™ EqualLogic™ FS7500 adds Network Attached Storage (NAS) capabilities to the EqualLogic 2 PS Series product line. Working with PS Series arrays, the FS7500 provides a high performance, scaleout, unified file and block storage solution for small to mid-size environments.
1.
2 Overview of the Dell EqualLogic FS7500 Unified Storage Solution The FS7500 is a high performance unified storage solution that integrates with new and existing EqualLogic PS Series storage arrays and enables customers to easily configure and manage iSCSI, CIFS, and NFS access in a single flexible storage pool using the EqualLogic Group Manager. The unique scale-out architecture of the solution lets customers expand storage capacity and/or system performance as needs change over time.
5 customer’s backup environment with support for Network Data Management Protocol (NDMP) backup. As with all Dell EqualLogic products, the FS7500’s NAS feature set, software licensing and future firmware enhancements are included in the base price of the system. Highly available file serving A dual active/active controller architecture and large onboard battery protected cache give the EqualLogic FS7500 outstanding performance.
• • • • • It uses a single EqualLogic storage pool. Only one storage pool in the EqualLogic storage group can be used for allocating the NAS reserve. It is formatted with Dell Fluid File system (FluidFS). It also stores NAS service metadata. The NAS reserve can be expanded later when more storage pool space is available. The NAS reserve is comprised of a variable number of automatically created EqualLogic storage volumes. The number of storage volumes created within the pool depends on the reserve size.
3 Test methodology The test methodology used to derive the results of this paper was comprised of two parts: 1. The creation of NFS exports or CIFS file shares on the Dell EqualLogic FS7500 Unified Storage Solution 2.
Table 1 File Size distribution Size Less than 4KB 4KB - 32KB 32KB - 256KB Percent 50% 22% 28% Note: The design of the data set as well as the distribution of NFS and CIFS file operations during each test was loosely based on the principles of the SPECsfs2008 file server benchmark. The test results presented in this paper should not be considered SPECsfs2008 benchmarks. 3.
Table 3 CIFS Simulation: File System Operation Distribution Operation Type READ_ANDX WRITE_ANDX 7 OTHER Distribution: READ Only Less than 4KB 4KB - 32KB 32KB - 256KB Distribution: WRITE Only Less than 4KB 4KB - 32KB 32KB - 256KB Percent 20% 9% 71% 60% 20% 20% 67% 18% 15% 3.5 Performance criteria The use case considered for simulations in this paper is end-user access of file shares.
4 Test hardware configurations This section outlines the major system hardware components involved in the simulations. These are illustrated in the figures below. Figure 3 illustrates one of the smallest configurations tested, which included two EqualLogic PS Series storage arrays and two EqualLogic FS7500 controllers.
Figure 3: Test setup using two EqualLogic FS7500 Controllers and two EqualLogic PS6500E arrays BP1015 Scalability & Deployment Best Practices for the Dell EqualLogic FS7500 NAS System 11
Figure 4: Test setup using four EqualLogic FS7500 Controllers and eight EqualLogic PS6000XV arrays BP1015 Scalability & Deployment Best Practices for the Dell EqualLogic FS7500 NAS System 12
4.1 Network configuration The connection paths from a single NAS node (FS7500 controller) to both the LAN and SAN are shown in Figure 5 below. The figure also details the connection paths for the EqualLogic PS Series storage array to the SAN.
4.1.1 NAS node NICs Four NICs are dedicated on each NAS node for each of the networks – SAN, client and internal. The connectivity of each of these NICS is shown in Figure 5 above. Refer to PS online help for more information on the FS7500 network configuration details and procedures. The FS7500 system LAN on motherboard (LOM) NICs are used for the client network and are shown connecting to the LAN switched stack in Figure 5. Client network NIC load balancing is set on each NAS node using a bonding mode.
• • The LAN was configured as single layer 2 network using the default VLAN. The default MAC based load-balancing mode (ALB) was used for distributing file system traffic across the FS7500 nodes. Flow control and Jumbo frames with MTU size = 9216 were enabled on the switch ports. SAN • • • • • Four Dell PowerConnect 6248 switches were configured for the SAN.
5 Test simulations A series of NFS and CIFS simulations were conducted to study the scalability and performance behavior of the NAS solution varying the number of NAS nodes, number of PS Series arrays, the PS Series array type and RAID type. This section describes the conduct of the tests and presents an analysis of the results obtained. These results provide scaling guidelines for designing FS7500 systems. 5.
Figure 6: PS6000XV RAID10 NFS scalability The results presented in Figure 6 show the following scaling characteristics: • Using a single FS7500 system, four arrays yielded 33% more operations per second than two arrays. In this comparison we increased the number of PS Series arrays from two to four PS6000XV arrays while keeping the number of FS7500 controllers constant at two.
comparison we doubled the number of the PS Series arrays from four to eight arrays and doubled the number of FS7500 controllers from two to four. From these results we can conclude the following: • • • Scaling the FS7500 systems along with the PS Series storage arrays can provide maximum system throughput and performance. Adding more FS7500 systems will produce more throughput for the same number of PS Series arrays.
The results presented in Figure 7 show the following scaling characteristics: • Using a single FS7500 system, four arrays yielded 38% more operations per second than two arrays. In this comparison we increased the number of the PS Series arrays from two to four PS6000XV arrays while keeping the number of FS7500 controllers constant at two. • Using four PS6000XV systems and two FS7500 systems we were able to support 93% more operations per second than when using a single FS7500 system.
Table 5 PS6000XV RAID 50 Scalability Test Configurations File System Count Total File System Size (TB) Total Data Set Size (TB) 4 8 1.36 16 8 16 1.83 16 2 x FS7500 + 4 x PS6000XV RAID50 8 16 2.5 32 2 x FS7500 + 8 x PS6000XV RAID50 16 32 3.
• • RAID 10 performed approximately 30% to 35% better than RAID 50 in the first three configurations. This difference was due to the extra I/O write operations necessary to maintain RAID50 parity blocks at the storage layer. In the largest configuration, the performance difference decreased to approximately 10% in favor of RAID 10. The lower difference in this case was caused by the FS7500 system reaching its maximum achievable throughput in the RAID10 configuration.
Figure 9 below shows the NFS achieved operations per second for the configurations using a % relative scale from the minimal system configuration. Total average client response time across all file operations remained less than 10ms during all tests. Figure 9: PS6500E RAID50 NFS Scalability The results presented in Figure 9 show the following scaling characteristics: • Using a single FS7500 system, four arrays yielded 38% more operations per second than two arrays.
• When using two FS7500 systems and eight PS6500E arrays we were able to achieve a 56% performance improvement over using one FS7500 and four PS6500E arrays. In this comparison we doubled the number of the PS Series arrays from four to eight arrays and doubled the number of FS7500 controllers from two to four. FS7500 system saturation capped throughput performance for the eight array configuration.
• • • • Using four PS6500E systems and two FS7500 systems we were able to support 69% more operations per second than when using a single FS7500 system. In this comparison the number of PS Series arrays (four PS6500E arrays) remained constant while the number of FS7500 controllers doubled from two to four. When using two FS7500 systems and four PS6500E arrays we were able to achieve more than twice the performance (122% increase) than when using one FS7500 and two PS6500E arrays.
Figure 11: NFS Block Size Variation As seen in Figure 11, the achieved operations per second with 32KB block size was 17% greater than that for 8KB block size. This difference is due to the processing overhead involved in fragmenting larger sized client file operation requests into smaller sized chunks of 8KB for transmission while the block size was set at 8KB. Also the response time was slightly higher with 8KB block size due to the same factor. 5.
Figure 12: Client network MTU Variation As seen in Figure 12, there was virtually no difference when varying client side MTU size. This is primarily due to the random nature of the file I/O workload simulated in the file share use case. 5.6 RAID Studies – RAID50, RAID5 and RAID 6 NFS and CIFS simulations were conducted with RAID 50, RAID 5 and RAID 6 on PS6500E. The configurations tested are shown below.
Figure 13 below shows the percent difference in NFS achieved operations per second for the configurations in a normalized scale. Total average client response time across all file operations remained less than10ms. Figure 13: PS6500E NFS RAID Studies The achieved operations per second with RAID 5 and RAID 6 were lower than for RAID 50. RAID 5 and RAID 6 incur penalties for parity calculations during write operations.
5.7 NFS data and meta-data mix The default mix of NFS operations in our test simulations was 28% data and 72% meta-data. Data operations are the actual read and write I/O operations to files. They incur higher latencies and can be deemed heavy I/O. Meta-data operations are operations on file and file system attributes. They incur lower latencies and can be deemed light I/O. Different application use cases may have differing needs of data and meta-data operations on the file system.
5.8 Sequential write I/O simulation Studies were conducted to understand the maximum sequential I/O throughput of the solution using I/O simulation on CIFS shares. The I/O simulation tool used was IOMeter. In order to ensure sequential nature of the client I/O, each NIC on the FS7500 client network served only one client machine and also only one I/O thread per client was simulated. A NAS file system was created for each client with a corresponding CIFS share exposed to the client.
Figure 16: CIFS IOMeter sequential write throughput With two nodes 92% of the available client network bandwidth was utilized, offering close to line rate utilization. With four nodes, about 54% of the bandwidth was utilized. The primary constraint with the four node configuration was the number of backend disk drives used in testing. The end-to-end total throughput was limited by the number and type of disks utilized.
6 Best practices In this section we provide best practice design and deployment recommendations based on the test results presented in this paper. 6.1 Network You should use separate network infrastructure (network switches) for the client network (LAN) and the SAN+FS7500 Internal Communication network. 6.1.1 SAN design 1. 2. 3. 4. 5. 6. 7. Redundant inter-connected switches or switching fabrics are recommended for the SAN network infrastructure to ensure high-availability.
4. Enable jumbo frames (MTU 9216) on the switch ports connecting to the FS7500 client network NIC ports to assist with sequential file I/O requests. If enabled, ensure jumbo frames is enabled across the network end-to-end from the file client to the FS7500 system. 5. Ensure that the file system client software on the file clients is configured to utilize the maximum block size for the file protocol. This will be 32KB for NFS v3 clients and 64KB for CIFS clients. 6.
Appendix A A.1 Test system components The table below provides details for the major hardware and software system components used in the test system configuration. Table 12 Test System Configuration Details Hardware Components Test Client Servers • Network • • • • • • Storage • • • Software Components Test Clients • • Monitoring BP1015 • 2 x Dell PowerEdge R815 Server: 4 x 12 Core AMD Opteron TM 6174 2.
A.2 IP and Subnet information Table 13 Test Network IP Configuration Entity Subnet 255.255.255.0 NAS Service IP 172.16.101.x NAS Node Client Network: Node 1 172.16.101.x Node 2 172.16.101.x Node 3 172.16.101.x Node 4 172.16.101.x NAS Node SAN: Node 1 172.16.103.x Node 2 172.16.103.x Node 3 172.16.103.x Node 4 172.16.103.x NAS Node 172.16.102.x Internal Network NAS Service 172.16.103.x Management IP EqualLogic 172.16.103.x Group IP Member IP: Member 1 172.16.103.x Member 2 172.16.103.x Member 3 172.16.
Related Publications The following Dell publications are referenced in this document or are recommended sources for additional information. The Dell EqualLogic FS7500 Unified Storage Solution http://www.equallogic.com/products/default.aspx?id=10465 EqualLogic PS Series Architecture http://www.equallogic.com/products/default.aspx?id=5823 Dell Fluid File system White Paper http://www.dell.com/downloads/global/products/pvaul/en/dell-scalable-file-system-whitepaper.
THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND.
