Dell HPC NFS Storage Solution – High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays A Dell Technical White Paper Xin Chen, Garima Kochhar and Mario Gallegos Dell HPC Engineering May 2013| Version 2.
Dell HPC NFS Storage Solution - High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays This document is for informational purposes only and may contain typographical errors and technical inaccuracies. The content is provided as is, without express or implied warranties of any kind. © 2013 Dell Inc. All rights reserved. Dell and its affiliates cannot be responsible for errors or omissions in typography or photography.
Dell HPC NFS Storage Solution - High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays Contents Executive summary (updated May 2013) .............................................................................. 5 1. Introduction ........................................................................................................... 6 2. Overview of NSS-HA solutions ..................................................................................... 6 2.1.
Dell HPC NFS Storage Solution - High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays Figures Figure 1. The infrastructure of the NSS-HA solution ............................................................. 7 Figure 2. PowerVault MD3260/MD3060e dense-enclosure storage array ..................................... 9 Figure 3. NSS4.5-HA test bed ...................................................................................... 12 Figure 4.
Dell HPC NFS Storage Solution - High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays Executive summary (updated May 2013) This solution guide describes the Dell NFS Storage Solution - High Availability configurations (NSS-HA) with Dell PowerVault MD3260 and MD3060e storage arrays. The PowerVault MD3260 and MD3060e are high-density storage enclosures that are able to provide 60 3.5” drives in 4U of rack space.
Dell HPC NFS Storage Solution - High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays 1. Introduction This solution guide provides information on the latest Dell NFS Storage Solution - High Availability configurations (NSS-HA) with Dell PowerVault MD3260 and MD3060e storage arrays.
Dell HPC NFS Storage Solution - High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays Figure 1. The infrastructure of the NSS-HA solution Note: The iDRAC 7 enterprise is not shown in the figure, and it is installed on each NFS server for Dell NSS-HA solutions. The term of Network Power Switches refers to APC PDU (Power Distribution Unit) in Dell NSS-HA solutions. 2.2.
Dell HPC NFS Storage Solution - High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays Table 1. NSS-HA Solutions(1), (2), (3) NSS2-HA Release (April 2011) (1) NSS3-HA Release (February 2012) (2) “Large capacity configuration” NSS4-HA Release (July 2012) (3) “PowerEdge 12G based solution” Storage Capacity 48TB and 96TB of raw storage space. 144TB and 288TB of raw storage space. Network Connectivity QDR InfiniBand or 10GbE connectivity.
Dell HPC NFS Storage Solution - High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays 3. Dell PowerVault MD3260 and MD3060e storage arrays As compared to previous versions of the NSS-HA solution, a major change in the current version is the introduction of the Dell PowerVault MD3260 and MD3060e storage arrays. These arrays replace the PowerVault MD3200 and MD1200 storage arrays used in previous NSS-HA solutions.
Dell HPC NFS Storage Solution - High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays o With the RAID-6 8+2 choice for the virtual disk configuration, six virtual disks can now be constructed for each PowerVault MD3260 or MD3060e storage array (60 / (8+2) = 6). An 180TB configuration consists of a single PowerVault MD3260 and the logical volume will contain six virtual disks.
Dell HPC NFS Storage Solution - High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays 4. Evaluation The architecture proposed in this white paper was evaluated in the Dell HPC lab. This section describes the test methodology and the test bed used for verification. It also contains details on the functionality tests. Performance tests and results follow in Section 5. 4.1.
Dell HPC NFS Storage Solution - High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays Figure 3. NSS4.5-HA test bed Public network (IB or 10GbE) Clients Clients Private network R620 R620 1 MD3260 + 1 MD3060e PDU PDU NSS4.5-HA 360TB configuration Table 3. Public network Private network Power Storage connections NSS4.5-HA hardware configuration Server configuration NFS server model Two Dell PowerEdge R620s. Processor Dual Intel Xeon E5-2680 @ 2.70 GHz.
Dell HPC NFS Storage Solution - High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays Storage configuration Storage Enclosure One Dell PowerVault MD3260 enclosure and one MD3060e enclosure for the 360TB solution. RAID controllers Duplex RAID controllers in the Dell MD3260. Hard Disk Drives 60 - 3TB 7200 rpm NL SAS drives per array. Other components Private Gigabit Ethernet switch Dell PowerConnect 5524.
Dell HPC NFS Storage Solution - High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays Firmware and Drivers PERC H710P firmware 21.0.2-0001 PERC H710P driver megaraid_sas 00.00.06.14-rh1 6Gbps SAS firmware 07.03.05.00 6Gbps SAS driver mpt2sas 12.101.00.00 Table 6. NSS4.5-HA client configuration Client / HPC Compute Cluster Clients 64 PowerEdge R410 compute nodes with Red Hat Enterprise Linux 6.2 x86-64. InfiniBand Mellanox ConnectX-2 QDR HCA.
Dell HPC NFS Storage Solution - High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays tolerant mechanisms in NSS-HA solutions, then presents the HA functionality tests with regards to different potential failures and faults. 4.3.1. Potential failures and fault tolerant mechanisms in NSS-HA There are many different types of failures and faults that can impact the functionality of NSS-HA. Table 7 lists the potential failures that are tolerated in NSS-HA solutions.
Dell HPC NFS Storage Solution - High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays Server failure. Heartbeat link failure. Public link failure. Private switch failure. Fence device failure. Single SAS link failure. Multiple SAS link failures. The NSS-HA behaviors are outlined below in response to these failures. Server failure — simulated by introducing a kernel panic.
Dell HPC NFS Storage Solution - High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays Multiple SAS link failures — simulated by disconnecting all SAS links between one Dell PowerEdge R620 server and the Dell PowerVault MD3260 storage. When all SAS links on the active server fail, the multipath daemon on the active server retries the path to the storage based on the parameters configured in the multipath.conf file. This is set to timeout after 150 seconds by default.
Dell HPC NFS Storage Solution - High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays 5.1. IPoIB sequential writes and reads Figure 4 shows the sequential write and read performance. The peak read performance is 4138MB/sec, and the peak write performance is 1171MB/sec. The design choice of RAID-6 8+2 was to optimize the service availability and disk layout with the new storage enclosures. Figure 4.
Dell HPC NFS Storage Solution - High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays Figure 5. IPoIB random write and read performance NFS IPoIB random I/O performance 12000 10000 IOPS 8000 6000 Write Read 4000 2000 0 1 2 4 8 16 32 48 64 Number of concurrent clients 5.3. IPoIB metadata operations Figure 6, Figure 7, and Figure 8 show the results of file create, stat, and remove operations, respectively.
Dell HPC NFS Storage Solution - High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays IPoIB file create performance Figure 6. Number of create () operations per second File create 50000 45000 40000 35000 30000 25000 20000 15000 10000 5000 0 File create 1 2 4 8 16 32 48 64 128 256 512 Number of concurrent clients Figure 7.
Dell HPC NFS Storage Solution - High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays IPoIB file remove performance Figure 8. Number of remove () operations per second File remove 45000 40000 35000 30000 25000 20000 File remove 15000 10000 5000 0 1 2 4 8 16 32 48 64 128 256 512 Number of concurrent clients 6. Conclusion This solution guide provides details of the latest Dell HPC NSS-HA Solution.
Dell HPC NFS Storage Solution - High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays Appendix A: Benchmarks and test tools The IOzone benchmark was used to measure sequential read and write throughput (MB/sec) as well as random read and write I/O operations per second (IOPS). The mdtest benchmark was used to test metadata operation performance. The checkstream utility was used to test for data correctness under failure and failover cases.
Dell HPC NFS Storage Solution - High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays IOzone Argument Description -s File size. -t Number of threads. +m Location of clients to run IOzone when in clustered mode. -w Does not unlink (delete) temporary file. -I Use O_DIRECT, bypass client cache. -O Give results in ops/sec.
Dell HPC NFS Storage Solution - High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays A.2. mdtest You can download mdtest from http://sourceforge.net/projects/mdtest/. Version 1.8.3 was used in these tests. It was compiled and installed on a NFS share that was accessible by compute nodes. mdtest is launched with mpirun. For these tests, OpenMPI version 1.4.3 was used. The following table describes the mdtest command-line arguments.
Dell HPC NFS Storage Solution - High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays Start the cluster service on the server. Mount NFS Share on clients. Metadata file and directory creation test: # mpirun -np 32 --nolocal --hostfile ./hosts /nfs/share/mdtest -d /nfs/share/filedir -i 6 -b 320 -z 1 -L -I 3000 -y -u -t -C Metadata file and directory stat test: # mpirun -np 32 --nolocal --hostfile .
Dell HPC NFS Storage Solution - High Availability (NSS-HA) Configuration with Dell PowerVault MD3260/MD3060e Storage Arrays checkstream[compute-00-10]: (5.43342 err/sec) checkstream[compute-00-10]: GiB) checkstream[compute-00-10]: err/sec) checkstream[compute-00-10]: checkstream[compute-00-10]: seconds (344598 KiB/sec) checkstream[compute-00-10]: checkstream[compute-00-10]: [valid data] 1488 valid extents in 273.
