DELL EMC ISILON F800 AND H600 WHOLE GENOME ANALYSIS PERFORMANCE ABSTRACT This white paper provides performance data for a BWA-GATK whole genome analysis pipeline run using Dell EMC Isilon F800 and H600 storage. It is intended for performance-minded administrators of large compute clusters that run genomics pipelines. The paper does not discuss the details of running a variant calling pipeline with BWA and GATK.
The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any software described in this publication requires an applicable software license. Copyright © 2018 Dell Inc. or its subsidiaries. All Rights Reserved.
TABLE OF CONTENTS ABSTRACT ................................................................................................................................1 EXECUTIVE SUMMARY ...........................................................................................................4 INTRODUCTION ........................................................................................................................4 DELL EMC ISILON .............................................................................
EXECUTIVE SUMMARY This Dell EMC technical white paper describes whole genome analysis performance results for Dell EMC Isilon F800 and H600 storage clusters (4 Isilon nodes per cluster). The data is intended to inform administrators on the suitability of Isilon storage clusters for high performance genomic analysis.
Kit (GATK) for the variant calling step. These are considered standard tools for aligning and variant calling in whole genome or exome sequencing data analysis. STORAGE CONFIGURATIONS Table 1 lists the configuration details of the three storage systems benchmarked. Default OneFS settings, SmartConnect and NFSv3 were used in all the Isilon tests. A development release of OneFS was used on the F800. Upgrading to the same OneFS version as used on the H600 would likely yield slightly better results.
DELL HPC INNOVATION LABS ZENITH COMPUTE CLUSTER Compute Clients Processor Memory Operating System Kernel System BIOS Profile Network 64 x PowerEdge C6320s CPU: Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz No. of cores = 18 per processor (36 per node) Processor Base Frequency: 2.3GHz AVX Base: 2.0GHz 128 GB @ 2400 MHz per node Red Hat Enterprise Linux Server release 7.2 (7.3 for H600 tests) 3.10.0-327.13.1.el7.x86_64 (3.10.0-514.el7.
NETWORK CONNECTIVITY The Zenith cluster and F800 storage system were connected via 8 x 40GbE links. Figure 1 shows the network topology used in the tests. The H600 was configured in the exact same way as the F800. Figure 2 shows the network configuration of the Dell HPC Lustre Solution. An OmniPath network was used for the Lustre tests. Figure 1. Network Diagram Of The F800 Benchmark Configuration Figure 2.
WHOLE GENOME SEQUENCE VARIANT ANALYSIS GATK version 3.5 and BWA version 0.7.2-r1039 were used to benchmark variant calling on the Lustre system, while GATK version 3.6 was used for runs using the F800 and H600. The whole genome workflow was obtained from the workshop, GATK Best Practices4, and its implementation is detailed here5 and here2. The publicly available human genome data set used for the tests was ERR091571.
To determine the maximum sample throughput possible while using an H600 for data input and pipeline output, an increasing number of genome samples were run on an increasing number of compute nodes with either 2 or 3 samples being run simultaneously on each node. Batches of 32-192 samples were run on 16-64 compute nodes while using the H600 NFS-mounted to the compute nodes.
Figure 5. BWA-GATK performance results comparison between F800 and H600. Plotting genomes/day throughput versus sample size for the F800 and H600 shows that performance on both platforms scales similarly up to 128 samples (Figure 5). Past that, H600 performance levels off and then deteriorates while F800 performance continues to improve. Future tests will utilize more than 64 compute nodes in an attempt to maximize pipeline throughput on the F800.
Figure 6. BWA-GATK performance results comparison between H600 and Lustre. The 80* labeled samples were run using a Dell HPC Lustre Storage Solution. Isilon storage provides an additional performance advantage when running large numbers of genomic analyses that consume large amounts of disk space. As can be seen in Figure 7, genomic analysis performance remains consistent whether the H600 is nearly empty (1% full) or almost completely full (91%).
BWA-GATK v3.6 with Isilon H600 64 Samples on 32 nodes 14 140 129 131 134 129 131 132 127 120 Apply Recalibration 10 100 8 80 6 60 Genomes/Day Running Time (hours) 12 4 Variant Recalibration GenotypeGVCFs HaplotypeCaller Base Recalibration Realign around InDel Generate Realigning Targets Mark/Remove Duplicates 40 2 Aligning & Sorting Number of Genomes per Day 0 20 1% 14% 19% 37% 55% 73% 91% Storage Usage: % full Figure 7. BWA-GATK performance results on H600 as storage usage increases.
