Reference Architecture Reference Architecture of Dell EMC Ready Solution for HPC Life Sciences Refresh with Cascade Lake CPUs and Dell EMC Ready Solution for HPC Storage Abstract The Dell EMC’s flexible HPC architecture for Life Sciences has been through a dramatic improvement with new Intel® Xeon® Scalable Processors.
Revisions Revisions Date Description March 2018 Initial release with Intel® Xeon® Scalable processors (code name Skylake) October 2019 Revised with 2nd Generation Intel Xeon Scalable processors (code name Cascade Lake) Acknowledgements This paper was produced by the following: Author: Kihoon Yoon The information in this publication is provided “as is.” Dell Inc.
Table of contents Table of contents Revisions.............................................................................................................................................................................2 Acknowledgements .............................................................................................................................................................2 Table of contents .................................................................................................
Executive summary Executive summary Since Dell EMC announced the Dell EMC Ready Solution for HPC Life Science in March 2018, the solution has matured, and can now process roughly 150 50x whole human genomes per day using 64x C6420s and Dell EMC Ready Solution for HPC Lustre Storage.
Introduction 1 Introduction Dell EMC has updated PowerEdge 14th Generation servers with 2nd Gen Intel Xeon Scalable processors, a boon to HPC applications. In addition to offering more cores, the solution offers Optane support, faster DRAM (DDR4-2933 in 1 DPC configuration), and more DRAM configurations (1TB, 2TB, and 4TB). Consumers are generally expecting more performance, better efficiency, and lower power from a newer processor.
Solution overview 2 Solution overview HPC in Life Sciences requires a flexible architecture to accommodate various system requirements. The Dell EMC Ready Solution for HPC Life Sciences was created to meet this need. It is a pre-integrated, tested, tuned, and leverages the most relevant of Dell EMC’s high-performance computing line of products and bestin-class partner products (2).
Solution overview In addition to compute, network, and storage options, there are several other components that perform different functions in the Dell EMC Ready Solution for HPC Life Sciences. These include CIFS gateway, fat node, acceleration node and other management components. Each of these components is described in detail in the subsequent section. The solutions are nearly identical for Intel OPA and IB EDR versions except for a few changes in the switching infrastructure and network adapter.
Solution overview - Dell EMC PowerEdge C4140 with up to four NVIDIA Tesla V100 GPUs > > > > > > > > - Dell EMC PowerEdge R940 > > > > > > > • CPU: 4x Intel Xeon Platinum 8280M at 2.7 GHz 28 cores Memory: 48x 32 GB at 2666 MHz Disk: 12x 1.92 TB SSD SATA mixed use 6 Gbps 512e 2.5 in hot plug S4610 drive in RAID 0 BIOS system profile: Performance optimized Logical processor: Disabled Virtualization technology: Disabled Operating system: RHEL 7.
Solution overview Figure 1 PowerEdge C4140 Config-M 2.3 Storage components The storage infrastructure consists of the following components and the details of the configurations can be found from the references: • • • 2.3.1 Dell EMC Ready Solution for HPC NFS Storage (NSS7.
Solution overview Figure 2 Dell EMC Ready Solution for HPC Lustre Storage, large base configuration including components overview and network configurations. • Dell EMC PowerEdge R640 - • CPU: 2x Intel Xeon Gold 5118 with 12 cores at 2.3 GHz Memory: 12x 8 GB at 2666 MHz 2x Dell EMC PowerEdge R740 for OSS nodes - CPU: 2x Intel Xeon™ Gold 6136 CPU with 12 cores at 3.
Solution overview - CPU: 2x Intel Xeon™ Gold 6136 CPU with 12 cores at 3.0 GHz Memory: 24x 16 GB at 2666 MHz MDS SAS controller: 2 x Dell 12 Gbps SAS HBAs MDS Storage Array: 1 x Dell EMC PowerVault ME4024 > Disks: 12 x 960 GB SAS SSD (active/passive configuration, no DNE) or 24 x 960 GB SAS SSD (active/Aactive Configuration, DNE) 2.4 Network components 2.4.
Solution overview Figure 3 Dell EMC Ready Solution for HPC Life Sciences with Intel® OPA fabric Figure 4 illustrates the details of a management network. It requires three Dell Networking S3048-ON switches. Two switches in Rack 1 (Figure 3) make up the internal management network for deploying, provisioning, managing, and monitoring the cluster. One of the switches in Rack 2 has 48 ports that are split into multiple untagged virtual LANs to accommodate multiple internal networks.
Solution overview Figure 4 Management network Figure 5 shows high-speed interconnect with Intel® OPA. The network topology is 2:1 blocking fat tree which requires three Dell Network H1048-OPF 48 port switches. Figure 5 Interconnection with Intel® OPA; 2:1 blocking fat tree topology 2.
Solution overview interface monitors every single node and reports detection of any software or hardware events. - BioBuilds > 14 BioBuilds is a well maintained, versioned, and continuously growing collection of open-source bio-informatics tools from L7Informatics (14). They are prebuilt and optimized for a variety of platforms and environments. BioBuilds solves most software challenges faced in the life sciences domain.
Performance evaluation and analysis 3 Performance evaluation and analysis 3.1 Variant calling analysis performance A typical variant calling pipeline consists of three major steps: 1) aligning sequence reads to a reference genome sequence; 2) identifying regions containing SNPs/InDels; and 3) performing preliminary downstream analysis. In the tested pipeline, BWA 0.7.2-r1039 is used for the alignment step, and Genome Analysis Tool Kit (GATK) is selected for the variant calling step.
Performance evaluation and analysis 3.1.1 Single sample test Three 2nd generation CPUs are picked to compare with four 1st generation CPUs, listed in Table 1. The purpose of this single sample test is to determine suitable CPUs for next generation sequencing (NGS) data analysis. All steps described in Figure 10 are tested on the Dell PowerEdge R640 with 50x whole human genome listed in Table 1 below.
Performance evaluation and analysis Figure 7 Runtime comparisons of steps in variant calling pipeline Total runtime, sum of all runtime from each step, provides better insight what we can choose for BWA-GATK pipeline as well as other pipelines including aligning and sorting as sub steps. 6248 is the best choice in terms of performance. However, it is important to note that the runtimes in Table 2 are drawn from single sample test (one job on one compute node).
Performance evaluation and analysis Figure 8 Performance comparisons between Lustre/MD3 and Lustre/ME4 3.2 De novo assembly performance For de novo assembly, 8168 and 8280M are compared with the same amount of system memory, 1.5 TB in R940. The main reason Cascade Lake 8280M was chosen is for its higher core counts and because it supports more memory, which is beneficial ad the data size for de novo assembly continues to grow larger over the time.
Technical support and resources 3.2.1 SOAPdenovo2 The maximum performance gain by upgrading from 8168 to 8280M is roughly 1% as shown in 92 cores of 8168 versus 108 cores of 8280M comparisons from Figure 9. For the test, one core per CPU was unused and allocated for the operating system and other housekeeping operations.
Technical support and resources Figure 10 Runtime and peak memory consumption plots for SPAdes with various number of cores 20 Reference Architecture of Dell EMC Ready Solution for HPC Life Sciences | Document 309
Technical support and resources 4 Conclusion Overall, 2nd Gen CPUs tested here perform slightly better over Skylake CPUs for genomics workloads such as variant Calling and de novo Assembly. In addition to performance advantages, 2nd Gen CPUs provide more choices compared to the predecessors in terms of lower TDP and higher core count for variant calling kinds of workloads. It is notable that 1 DPC configuration with DDR4 2933 MHz DIMMS does not improve performance for SOAPdenovo2.
Technical support and resources A Technical support and resources A.1 Related resources 1. Blueprint for High Performance Computing. Dell TechCenter. [Online] http://en.community.dell.com/techcenter/blueprints/blueprint_for_hpc/m/mediagallery/20443473. 2. Dell EMC PowerEdge Servers. [Online] https://www.dellemc.com/en-us/servers/index.htm. 3. PowerEdge C6420. [Online] https://www.dell.
Technical support and resources 15. European Nucleotide Archive: ERR091571. [Online] https://www.ebi.ac.uk/ena/data/view/ERR091571. 16. Genome Analysis Toolkit. [Online] https://software.broadinstitute.org/gatk/.
