Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers This Dell technical white paper analyses the various BIOS options available in Dell PowerEdge 12th generation servers and provides recommendations for High Performance Computing workloads. Garima Kochhar and Nishanth Dandapanthula High Performance Computing Engineering July 2012 | Version 1.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers This document is for informational purposes only and may contain typographical errors and technical inaccuracies. The content is provided as is, without express or implied warranties of any kind. © 2012 Dell Inc. All rights reserved. Dell and its affiliates cannot be responsible for errors or omissions in typography or photography. Dell, the Dell logo, and PowerEdge are trademarks of Dell Inc.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers Contents Executive summary ..................................................................................................... 5 1. Introduction ....................................................................................................... 6 2. Dell PowerEdge 12th generation servers and Intel Sandy Bridge-EP architecture ..................... 6 2.1. 3. Intel SandyBridge architecture ...................................
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers Figures Figure 1. Sandy Bridge-EP architecture for a PowerEdge R620 ................................................ 8 Figure 2. Local, remote and interleaved memory bandwidth ................................................ 13 Figure 3. Idle power usage across different System Profiles ................................................. 16 Figure 4. Impact of power-based BIOS options on idle power ..............................
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers Executive summary The latest Dell PowerEdge 12th generation servers provide several BIOS options that can be tuned for performance and energy efficiency. In this technical white paper, the cluster-level impact of different BIOS options is quantified and presented for different types of high performance computing (HPC) workloads.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers 1. Introduction Dell PowerEdge 12th generation servers1 include the Intel Xeon E5-2600 series processors based on the Intel microarchitecture codenamed Sandy Bridge. With the new processor and chipset technology, the new servers support PCI-Gen3 capable PCI slots, memory DIMM speeds up to 1600 MT/s, four memory channels per socket, and Intel QuickPath Interconnect (QPI) lanes running at 8.0GT/s.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers The following features are common to the servers: Support for processors from the Intel Xeon E5-2600 series. 4 memory channels per socket. The number of DIMM slots per server varies by product line. o 3 DIMMs per channel for the PowerEdge R and M product line. Total of 12 DIMMs per socket, 24 DIMMs per server. o 2 DIMMs per channel for the PowerEdge C product. Total of 8 DIMMs per socket, 16 DIMMs per server.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers Like Westmere-EP, Sandy Bridge is also a NUMA-based architecture. Figure 1 shows a block diagram of the Sandy Bridge-EP architecture. Each processor socket has an integrated memory controller. A core’s access to the memory attached to its local memory controller is faster and has higher bandwidth than access to the memory attached to the other, remote socket’s, memory controller. Figure 1.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers Table 2.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers Details of each of these settings are provided in [4]. A quick overview is provided here. Table 3. 1. 2. Performance3. Per Watt Optimized (DAPC) CPU Power Management System DBPM System Profile options Performance4. Performance Per Watt Optimized Optimized (OS) OS DBPM 5. Dense 6.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers CPU Power Management determines what controls the power management for the processor. System DBPM (Demand Based Power Management) is Dell’s Active Power Controller (DAPC) and is hardware-based control from the iDRAC. OS DBPM hands control to the Operating System. In the case of Linux, this is controlled by the “cpuspeed” service.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers The Performance Optimized System Profile focuses on pure performance. Turbo Boost is enabled; C States and C1E are disabled. The Dense Configuration Optimized profile is for systems that have high DIMM count configurations, where reliability is prioritized over power savings or performance considerations. Performance options like Turbo Boost are disabled, and memory-based options are prioritized.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers Figure 2. Local, remote and interleaved memory bandwidth Memory Bandwidth - Local, Remote and Interleaved Memory Bandwidth in GB/s 12 10 8 26% lower 43% lower 6 Local Interleaved Remote 4 2 0 1 thread * Data collected on a PowerEdge R620, Dual Intel Xeon E5-2680 @ 2.7GHz, 8*8GB 1600MT/s memory. Performance Optimized System Profile, Logical Processor Disabled.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers C States, C1E, Monitor/Mwait – Disabled. This is to turn off all power saving features on the processor. System Profile – Max Performance. This ensures there is no system level power saving. Logical Processor – Disabled. This is especially needed when Monitor/Mwait is Disabled A separate study has covered this topic in great depth. Refer to this link for details.10 4.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers Table 5. Benchmark and application details Application Domain Version Benchmark data set High Performance Linpack Floating point CPU intensive system benchmark Intel MKL v10.3.9.293 NA, all problems consumed 90% of total memory. Stream Memory Bandwidth microbenchmark v5.9 Array size 160000000 ANSYS Fluent Computational Fluid Dynamics application v14.0.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers 5.1. Idle power Very few clusters tend to be fully utilized, i.e., running load at all times on all the servers. Most clusters have some downtime when all or part of the cluster is not being used, and this is where idle power usage is relevant. If there is anticipated downtime for a cluster, when the load is expected to be light or close to zero, the cluster can be configured to save power while it is idle.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers The next graph presents the impact of some BIOS options on idle power consumption. C States, C1E and Monitor/Mwait are options that reduce power consumption when enabled as described in Section 3.1. Turbo Boost, when enabled, increases power consumption when the system is under load. In this test case, these four BIOS options that directly impact power consumption were evaluated to understand their impact on cluster idle power.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers used for the chart is the DAPC profile; all results are relative to DAPC. A bar higher than 1.00 indicates that the application’s performance was that much better when compared to its performance with the DAPC profile. From the graph, it is seen that MILC performance is significantly impacted by choice of profile for DAPC, OS, and Performance. The Dense System Profile significantly reduces performance for all applications by 8 to 35%.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers The Performance profile significantly improves MILC performance (21%), and energy efficiency is within 8% of DAPC. Section 7 concludes with BIOS recommendations that optimize for performance and for performance/watt based on the test results in this section and those that follow in subsequent sections. 5.3.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers energy efficiency of Turbo Boost disabled will be better than with Turbo Boost enabled. The text within the bar quantifies the decrease in performance (-Perf) and power savings (+Power saving) with Turbo disabled when compared to Turbo enabled. Results are charted for both the DAPC and the Performance CPU Power Management Profiles. Figure 6 shows that energy efficiencies are uniformly better with Turbo Boost disabled.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers Results are shown in Figure 7. The bar graph plots the performance of the application with Node Interleaving on relative to Node Interleaving off. A value higher than one indicates that the application performed that much better with Node Interleaving enabled. The text in the bar indicates the percentage of additional power consumed with Node Interleaving enabled.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers The bar graphs plot the performance with Logical Processor enabled when compared to Logical Processor disabled. The 16-server cluster has a total of 256 physical cores. With Logical Processor enabled, tests were performed using only 256 cores as well as all the logical 512 cores. The scheduling of cores was left to the Operating System.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers Figure 9. Impact of C States and C1E on remote memory access Remote Memory bandwidth (GB/s) Impact of C states/C1E on remote bandwidth 25 20 24% 15 10 5 0 8 Threads C states/C1E Enabled C states/C1E Disabled * Data collected on a single PowerEdge M620, Dual Intel Xeon E5-2680 @ 2.7GHz, 8*8GB 1600MT/s memory. Logical Processor and Node Interleaving Disabled. Performance CPU Power Management. Turbo Enabled.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers Table 6. 11th and 12th generation cluster test bed details Component 11th generation cluster10 12th generation cluster SERVER Dell PowerEdge R610 rack server (1) Dell PowerEdge R620 rack server (1) Dell PowerEdge M610 blade server (16) in a PowerEdge M1000e chassis Dell PowerEdge M620 blade server (16) in a PowerEdge M1000e chassis PROCESSORS Dual Intel Xeon X5570 @ 2.93 GHz, 95W Dual Intel Xeon E5-2680 @ 2.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers Figure 10. 11th vs. 12th generation cluster – idle power comparison 11th Generation vs 12th Generation - Idle power Idle power saving with 12th Generation 30% 25% 20% 15% 10% 5% 0% DAPC OSCtl Perf * Logical Processor and Node Interleaving Disabled. All other options at defaults. 4.00 3.50 11th Generation vs 12th Generation - Performance and Energy efficiency 3.00 2.92 3.00 3.50 2.50 2.50 2.50 2.00 2.00 1.62 1.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers shows the performance improvement with the new cluster relative to the previous generation. The secondary y-axis plots the energy efficiency of the new cluster relative to the earlier one. The results are impressive. HPL performance is 3.6 times (3.6x) better with the latest generation and 2.93 times more energy efficient. Fluent and WRF also demonstrate a significant improvement with the new servers.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers Table 7 summarizes the different options and the recommended settings for different configuration goals. Based on studies like this, servers ordered from Dell with the HPC SKU are preconfigured with the “Balanced configuration” HPC specific BIOS options. Additionally, this paper includes a performance and energy efficiency comparison to a previous generation cluster solution.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers 8. References 1. Dell PowerEdge 12th generation servers http://content.dell.com/us/en/enterprise/d/campaigns/poweredge-12th-generation-servers-ple 2. Dell PowerEdge 12th generation features http://i.dell.com/sites/content/shared-content/data-sheets/en/Documents/dell-poweredgem620-technical-guide.pdf 3. Top 500 HPC architectures over time http://i.top500.org/overtime 4. Dell PowerEdge 12th generation BIOS white paper http://en.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers Appendix A – Summary of findings This section summarizes results from this white paper. For readability, The Performance Per Watt Optimized (DAPC) System Profile is abbreviated to DAPC. The Performance Per Watt Optimized (OS) System Profile is abbreviated to OS Control. The Performance Optimized System Profile is abbreviated to Perf.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers Turbo Boost 6) Turbo Boost disabled reduces application performance by up to 15% compared to Turbo Boost enabled. But, at the same time, Turbo Boost disabled consumes 10% to 27% less power. This translates in to 3% to 20% better energy efficiency with Turbo Boost disabled.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers Appendix B – Dell Deployment Toolkit to modify BIOS options from the command line The Dell Deployment Toolkit (DTK) is a component of Dell’s OpenManage suite of products for systems management. StackIQ Rocks+ includes DTK utilities like syscfg, and this utility is automatically installed on cluster nodes as part of the Dell roll.
Optimal BIOS settings for HPC with Dell PowerEdge 12th generation servers On StackIQ Rocks+ Dell clusters, syscfg is installed in /opt/dell/toolkit/bin cd /opt/dell/toolkit/bin and run ./syscfg to see the version and options available. ./syscfg –h
