Dell HPC NFS Storage Solution – High Availability (NSS5.5-HA) Configuration with Dell PowerVault MD3460 and MD3060e Storage Arrays A Dell Technical White Paper Xin Chen Dell HPC Engineering April 2014| Version 1.
Dell HPC NFS Storage Solution - High Availability (NSS5.5-HA) Configuration with Dell PowerVault MD3460 and MD3060e Storage Arrays This document is for informational purposes only and may contain typographical errors and technical inaccuracies. The content is provided as is, without express or implied warranties of any kind. © 2014 Dell Inc. All rights reserved. Dell and its affiliates cannot be responsible for errors or omissions in typography or photography.
Dell HPC NFS Storage Solution - High Availability (NSS5.5-HA) Configuration with Dell PowerVault MD3460 and MD3060e Storage Arrays Contents Executive summary ....................................................................................................... 5 1. Introduction ........................................................................................................... 6 2. Overview of NSS-HA solutions .....................................................................................
Dell HPC NFS Storage Solution - High Availability (NSS5.5-HA) Configuration with Dell PowerVault MD3460 and MD3060e Storage Arrays Figures Figure 1. The infrastructure of the NSS-HA solution ............................................................. 7 Figure 2. NSS5.5-HA test bed ...................................................................................... 11 Figure 3. IPoIB large sequential write and read performance ............................................... 18 Figure 4.
Dell HPC NFS Storage Solution - High Availability (NSS5.5-HA) Configuration with Dell PowerVault MD3460 and MD3060e Storage Arrays Executive summary This solution guide describes the Dell NFS Storage Solution - High Availability configurations (NSS-HA) with Dell PowerVault MD3460 and MD3060e storage arrays. The PowerVault MD3460 and MD3060e are the latest generation high-density storage enclosures that provide up to 60 3.5” drives in 4U of rack space and support 12GBbps SAS connectivity.
Dell HPC NFS Storage Solution - High Availability (NSS5.5-HA) Configuration with Dell PowerVault MD3460 and MD3060e Storage Arrays 1. Introduction This solution guide provides information on the latest Dell NFS Storage Solution - High Availability configurations (NSS-HA) with Dell PowerVault MD3460 and MD3060e storage arrays.
Dell HPC NFS Storage Solution - High Availability (NSS5.5-HA) Configuration with Dell PowerVault MD3460 and MD3060e Storage Arrays Figure 1. The infrastructure of the NSS-HA solution Note: The iDRAC 7 enterprise is not shown in the figure, and it is installed on each NFS server for Dell NSS-HA solutions. The term of Network Power Switches refers to APC PDU (Power Distribution Unit) in Dell NSS-HA solutions. 2.2.
Dell HPC NFS Storage Solution - High Availability (NSS5.5-HA) Configuration with Dell PowerVault MD3460 and MD3060e Storage Arrays Table 1. NSS-HA Solutions(1), (2), (3), (4), (5) NSS4.0-HA Release (July 2012) (3) “PowerEdge 12G based solution” NSS5.0-HA Release (September 2013) (5) “PowerVault MD3260 based solution” Storage Capacity 144TB and 288TB of raw storage space. 180TB to 360TB of raw storage space. Network Connectivity FDR InfiniBand or 10GbE Connectivity.
Dell HPC NFS Storage Solution - High Availability (NSS5.5-HA) Configuration with Dell PowerVault MD3460 and MD3060e Storage Arrays 3. Dell PowerVault MD3460 and MD3060e storage arrays As compared to previous versions of the NSS-HA solution, a major change in the current version is the introduction of the Dell PowerVault MD3460 storage array. This array replaces the PowerVault MD3260 storage array used in previous NSS-HA solutions.
Dell HPC NFS Storage Solution - High Availability (NSS5.5-HA) Configuration with Dell PowerVault MD3460 and MD3060e Storage Arrays 4. Evaluation The architecture proposed in this white paper was evaluated in the Dell HPC lab. This section describes the test methodology and the test bed used for verification. It also contains details on the functionality tests. Performance tests and results follow in Section 5. 4.1.
Dell HPC NFS Storage Solution - High Availability (NSS5.5-HA) Configuration with Dell PowerVault MD3460 and MD3060e Storage Arrays Figure 2. NSS5.5-HA test bed Public network (IB or 10GbE) Clients Clients Private network R620 R620 1 MD3460 + 1 MD3060e PDU PDU NSS5.5-HA 360TB configuration Table 3. Public network Private network Power Storage connections NSS5.5-HA hardware configuration Server configuration NFS server model Two Dell PowerEdge R620s. Processor Dual Intel Xeon E5-2695 v2 @ 2.
Dell HPC NFS Storage Solution - High Availability (NSS5.5-HA) Configuration with Dell PowerVault MD3460 and MD3060e Storage Arrays Systems Management iDRAC7 Enterprise. Power Supply Dual Power Supply Units. Storage configuration Storage Enclosure One Dell PowerVault MD3460 array and one MD3060e array for the 360TB solution. RAID controllers Duplex RAID controllers in the Dell MD3460. Hard Disk Drives 60 - 3TB 7200 rpm NL SAS drives per array.
Dell HPC NFS Storage Solution - High Availability (NSS5.5-HA) Configuration with Dell PowerVault MD3460 and MD3060e Storage Arrays Firmware and Drivers InfiniBand driver Mellanox OFED 2.1-1.0.0 10GbE Ethernet driver Bnx2x 1.78.17-0 PERC H710P firmware 21.2.0-0007 PERC H710P driver megaraid_sas 06.700.06.00-rh1 Table 6. NSS5.
Dell HPC NFS Storage Solution - High Availability (NSS5.5-HA) Configuration with Dell PowerVault MD3460 and MD3060e Storage Arrays 4.3. HA functionality The HA functionality of the solution was tested by simulating several component failures. The design of the tests and the test results are similar to previous versions of the solution since the general architecture of the solution has not changed in this release.
Dell HPC NFS Storage Solution - High Availability (NSS5.5-HA) Configuration with Dell PowerVault MD3460 and MD3060e Storage Arrays Failure type RAID controller failure on Dell PowerVault MD3260 storage array Mechanism to handle failure Dual controllers in the Dell PowerVault MD3260. The second controller handles all data requests. Performance may be degraded, but functionality is not impacted. 4.3.2. HA tests for NSS-HA Functionality was verified for an NFSv3-based solution.
Dell HPC NFS Storage Solution - High Availability (NSS5.5-HA) Configuration with Dell PowerVault MD3460 and MD3060e Storage Arrays Fence device failure — simulated by disconnecting the iDRAC7 Enterprise cable from a server. If the iDRAC on a server fails, the server is fenced using the network PDUs, which are defined as secondary fence devices in the cluster configuration files. For the above cases, it was observed that the HA service failover takes in the range of 30 to 60 seconds.
Dell HPC NFS Storage Solution - High Availability (NSS5.5-HA) Configuration with Dell PowerVault MD3460 and MD3060e Storage Arrays 5. NSS5.5-HA I/O Performance This section presents the results of the I/O performance tests for the current NSS-HA solution. All performance tests were conducted in a failure-free scenario to measure the maximum capability of the solution. The tests focused on two types of I/O patterns: large sequential reads and writes, and small random reads and writes.
Dell HPC NFS Storage Solution - High Availability (NSS5.5-HA) Configuration with Dell PowerVault MD3460 and MD3060e Storage Arrays IPoIB large sequential write and read performance Figure 3. NFS IPoIB large sequential I/O performance Throughput in MB/s 6000 5000 4000 3000 2000 1000 0 1 2 4 8 16 32 48 64 Number of concurrent clients Write Read 5.2. IPoIB random writes and reads Figure 4 shows the random write and read performance.
Dell HPC NFS Storage Solution - High Availability (NSS5.5-HA) Configuration with Dell PowerVault MD3460 and MD3060e Storage Arrays Figure 4. IPoIB random write and read performance NFS IPoIB random I/O performance 10000 9000 8000 IOPS 7000 6000 5000 4000 3000 2000 1000 0 1 2 4 8 16 32 Number of concurrent clients Write 48 64 Read 6. Conclusion This solution guide provides details of the latest Dell HPC NSS-HA Solution.
Dell HPC NFS Storage Solution - High Availability (NSS5.5-HA) Configuration with Dell PowerVault MD3460 and MD3060e Storage Arrays 5. Dell HPC NFS Storage Solution - High Availability Solution NSS5-HA configurations http://en.community.dell.com/techcenter/high-performancecomputing/b/hpc_storage_and_file_systems/archive/2013/10/28/dell-hpc-nfs-storage-solutionhigh-availability-solution-nss5-ha-configurations.aspx 6.
Dell HPC NFS Storage Solution - High Availability (NSS5.5-HA) Configuration with Dell PowerVault MD3460 and MD3060e Storage Arrays Appendix A: Benchmarks and test tools The IOzone benchmark was used to measure sequential read and write throughput (MB/sec) as well as random read and write I/O operations per second (IOPS). The mdtest benchmark was used to test metadata operation performance. The checkstream utility was used to test for data correctness under failure and failover cases.
Dell HPC NFS Storage Solution - High Availability (NSS5.5-HA) Configuration with Dell PowerVault MD3460 and MD3060e Storage Arrays IOzone Argument Description -s File size. -t Number of threads. +m Location of clients to run IOzone when in clustered mode. -w Does not unlink (delete) temporary file. -I Use O_DIRECT, bypass client cache. -O Give results in ops/sec.
Dell HPC NFS Storage Solution - High Availability (NSS5.5-HA) Configuration with Dell PowerVault MD3460 and MD3060e Storage Arrays A.2. mdtest You can download mdtest from http://sourceforge.net/projects/mdtest/. Version 1.8.3 was used in these tests. It was compiled and installed on a NFS share that was accessible by compute nodes. mdtest is launched with mpirun. For these tests, OpenMPI version 1.6.4 was used. The following table describes the mdtest command-line arguments.
Dell HPC NFS Storage Solution - High Availability (NSS5.5-HA) Configuration with Dell PowerVault MD3460 and MD3060e Storage Arrays Start the cluster service on the server. Mount NFS Share on clients. Metadata file and directory creation test: # mpirun -np 32 --nolocal --hostfile ./hosts /nfs/share/mdtest -d /nfs/share/filedir -i 6 -b 320 -z 1 -L -I 3000 -y -u -t -C Metadata file and directory stat test: # mpirun -np 32 --nolocal --hostfile .
Dell HPC NFS Storage Solution - High Availability (NSS5.5-HA) Configuration with Dell PowerVault MD3460 and MD3060e Storage Arrays checkstream[compute-00-10]: (5.43342 err/sec) checkstream[compute-00-10]: GiB) checkstream[compute-00-10]: err/sec) checkstream[compute-00-10]: checkstream[compute-00-10]: seconds (344598 KiB/sec) checkstream[compute-00-10]: checkstream[compute-00-10]: [valid data] 1488 valid extents in 273.
