White Papers

ManualsBrandsDell ManualsConverged InfrastructureHigh Performance Computing Solution Resources

Page 1 of 13

Available at http://dell.to/XVCU0c

BIOS tuning for HPC on 13

Generation Haswell servers

Garima Kochhar, September 2014

This blog discusses the performance and energy efficiency implications of BIOS tuning options available

on the new Haswell-based servers for HPC workloads. Specifically we looked at memory snoop modes,

performance profiles and Intel’s Hyper-Threading technology and their impact on HPC applications. This

blog is part two of a three part series. Blog one provided some initial results on HPC applications and

performance comparisons on these new servers and previous generations. The third blog in this series

will compare performance and energy efficiency across different Haswell processor models.

We’re familiar with performance profiles including power management, Turbo Boost and C-states.

Hyper-Threading or Logical Processor is a known feature as well. The new servers introduce three

different memory snoop modes – Early Snoop, Home Snoop and Cluster On Die. Our interest was in

quantifying the performance and power consumed across these different BIOS options.

The “System Profile Settings” category in the BIOS combines several performance and power related

options into a “meta” option. Turbo Boost, C-states, C1E, CPU Power Management, Memory Frequency,

Memory Patrol Scrub, Memory Refresh Rate, Uncore Frequency are some of the sub-options that are

pre-set by this “meta” option. There are four pre-configured profiles, Performance Per Watt (DAPC),

Performance Per Watt (OS), Performance and Dense Configuration, that can be used. The DAPC and OS

profiles balance performance and energy efficiency options aiming for good performance while

controlling the power consumption. With DAPC, the Power Management is handled by the Dell iDRAC

and system level components. With the OS profile, the operating system controls the power

management. In Linux this would be the cpuspeed service and cpufreq governors. The Performance

profile optimizes for only performance – most power management options are turned off here. The

Dense Configuration profile is aimed at dense memory configurations, memory patrol scrub is more

frequent and the memory refresh rate is higher and Turbo Boost is disabled. Additionally if the four pre-

set profiles do not meet the requirement, there is a fifth option “Custom” that allows each of the sub-

options to be tuned individually. In this study we focus only on the DAPC and Performance profiles. Past

studies have shown us that DAPC and OS perform similarly, and Dense Configuration performs lower for

HPC workloads.

The Logical Processor feature is based on Intel® Hyper-Threading (HT) technology. HT enabled systems

appear to the operating system as having twice as many processor cores as they actually do by ascribing

two “logical” cores to each physical core. HT can improve performance by assigning threads to each

logical core; logical cores execute their threads by sharing the physical cores’ resources.

Snoop Mode is a new category under Memory Setting. Coherence between sockets is maintained by

way of “snooping” the other sockets. There are two mechanisms for maintaining coherence between

sockets. Snoop broadcast (Snoopy) modes where the sockets are snooped for every memory transaction

and directory support where some information is maintained in memory that gives guidance on whether

there is a need to snoop.

Summary of content (13 pages)

PAGE 1
BIOS tuning for HPC on 13th Generation Haswell servers Garima Kochhar, September 2014 This blog discusses the performance and energy efficiency implications of BIOS tuning options available on the new Haswell-based servers for HPC workloads. Specifically we looked at memory snoop modes, performance profiles and Intel’s Hyper-Threading technology and their impact on HPC applications. This blog is part two of a three part series.
PAGE 2
The Intel® Xeon® Processor E5-2600 v3 Product Family (Haswell) supports three snoop modes in dual socket systems - Early Snoop, Home Snoop and Cluster On Die. Two of these modes are snoop broadcast modes. In Early Snoop (ES) mode, the distributed cache ring stops can send a snoop probe or a request to another caching agent directly. Since the snoop is initiated by the distributed cache ring stops itself, this mode has lower latency.
PAGE 3
Configuration Table 1 below details the applications we used and Table 2 describes the test configuration on the new 13G server. Table 1 - Applications and benchmarks Application Stream HPL Domain Memory bandwidth Ansys Fluent LS-DYNA WRF MILC Version v5.9 From Intel MKL Benchmark Triad Problem size 90% of total memory Computational fluid dynamics Finite element analysis v15.0 truck_poly_14m v7_0_0_79069 car2car with endtime=0.02 Weather Research and Forecasting Quantum chromo dynamics v3.5.
PAGE 4
  WRF – Average time step computed over the last 719 intervals for Conus 2.5km MILC – Time as reported by the application. Power was measured by using a power meter attached to the server and recording the power draw during the tests. The average steady state power is used as the power metric for each benchmark. Energy efficiency (EE) computed as Performance per Watt (performance/power).
PAGE 5
Memory bandwidth - COD 140000 120000 Memory Bandwidth - MB/s 116198 100000 Full System Local socket - 14 threads 80000 Local NUMA node - 7 threads 60000 Remote to same socket 58082 Remote to other socket 40000 20000 29647 ↓ 47% 15591 13944 ↓ 11% 0 Figure 1 - Memory bandwidth with COD mode Figure 2 and Figure 3 compare the three different snoop modes on two processor models across the different applications. The system profile was set to DAPC, HT disabled. All other options were at BIOS defaults.
PAGE 6
Performance relative to HS.DAPC (higher is better) 1.06 1.02 1.04 1.00 1.02 0.98 1.00 0.96 0.98 0.94 0.96 Power consumption relative to HS.DAPC Power (lower is better, less power) Comparing snoop modes - HS, ES, COD E5-2697 v3, 2.6 GHz, 14c, 145W 0.92 0.94 HS.DAPC 1.00 0.97 1.01 1.00 1.00 1.01 1.00 0.99 1.04 1.00 1.00 1.04 1.00 0.99 1.04 1.00 0.98 1.06 1.00 0.98 1.05 Stream HPL Fluent LS DYNA WRF MILC 7.6.3 MILC 7.7.11 N=124320, NB=168 truck_poly_14m car2car, endtime=0.02 Conus 2.
PAGE 7
Performance relative to HS.DAPC (higher is better) 1.06 1.02 1.04 1.00 1.02 0.98 1.00 0.96 0.98 0.94 0.96 Power consumption relative to HS.DAPC Power (lower is better, less power) Comparing snoop modes - HS, ES, COD E5-2660 v3, 2.6 GHz, 10c, 105W HS.DAPC 0.92 0.94 1.00 1.01 1.02 1.00 1.00 1.01 1.00 1.01 1.03 1.00 1.01 1.03 1.00 1.00 1.04 1.00 1.01 1.07 COD.DAPC 1.00 1.01 1.02 0.92 0.90 Stream HPL Fluent LS DYNA WRF MILC 7.6.3 MILC 7.7.
PAGE 8
1.06 1.20 1.04 1.15 1.10 1.02 1.05 1.00 1.00 0.98 1.04 1.04 1.01 1.00 1.05 1.06 0.95 1.04 1.01 0.96 Power relative to HS.DAPC Power (lower is better, less power) Performance relative to HS.DAPC (higher is better) Comparing profiles - DAPC, Perf E5-2697 v3, 2.6 GHz, 14c, 145W 1.00 1.00 1.00 HS.DAPC 1.00 1.00 1.00 0.90 HS.Perf COD.DAPC 0.94 0.85 0.87 0.88 0.97 0.99 0.97 1.02 0.98 1.02 0.95 0.97 0.92 0.97 0.92 0.92 0.80 Stream HPL Fluent LS DYNA WRF MILC 7.6.
PAGE 9
1.06 1.20 1.04 1.15 1.10 1.02 1.05 1.00 1.00 0.98 1.03 1.02 1.07 1.04 0.95 1.03 0.96 1.01 1.02 1.00 1.00 1.00 1.00 1.00 1.00 Power relative to HS.DAPC Power (lower is better, less power) Performance relative to HS.DAPC (higher is better) Comparing profiles - DAPC, Perf E5-2660 v3, 2.6 GHz, 10c, 105W HS.DAPC HS.Perf 1.00 0.90 0.94 0.85 COD.DAPC COD.Perf HS.DAPC.Power 0.93 0.96 0.99 1.01 0.99 1.04 0.99 1.02 0.99 1.01 0.98 1.02 0.99 1.02 0.92 0.
PAGE 10
processor models is similar; both support 2133 MT/s memory. With the 14c model, we’ve added 40% more cores when compared to the 10c model. It’s likely that the memory bandwidth per core with HT enabled on the 14c processor is smaller than LS-DYNA’s requirement and that is why there is no performance improvement with HT enabled on the 14c processor. The power consumption with HT enabled was higher for all cases when compared to HT disabled.
PAGE 11
Comparing hyperthreading - on, off E5-2660 v3, 2.6 GHz, 10c, 105W 1.20 1.10 1.12 1.00 1.07 0.90 1.02 0.80 0.97 0.70 1.00 0.94 1.000.99 1.001.08 1.001.02 1.00 0.94 1.00 1.05 1.00 1.00 Stream HPL Fluent LS DYNA WRF MILC 7.6.3 MILC 7.7.11 N=124320, NB=168 truck_poly_14m car2car, endtime=0.02 Conus 2.5km Intel file Intel file 0.92 Power relative to HS.DAPC Power (lower is better, less power) Performance relative to HS.DAPC no HT (higher is better) 1.17 0.60 HS.DAPC HS. DAPC.HT HS.
PAGE 12
Power - Idle and Peak E5-2697 v3, 2.6 GHz, 14c, 145W Measured Power in Watts (lower is better, less power) 600 500 496 489 499 493 490 484 400 300 200 168 168 100 100 100 HS.DAPC ES.DAPC 103 99 0 COD.DAPC Idle HS.Perf COD.Perf HS.DAPC.HT Peak Figure 8 - Idle and Peak power - E5-2697 v3 Power - Idle and Peak E5-2660 v3, 2.6 GHz, 10c, 105W Measured Power in Watts (lower is better, less power) 600 500 400 374 362 300 391 368 391 367 200 157 100 97 97 HS.DAPC ES.
PAGE 13
In terms of System Profile, DAPC appears to be a good choice providing performance similar to Performance profile but with some energy efficiency benefits. Note that the profile DAPC enables Cstates and C1E, and will not be a good fit for latency sensitive workloads. It is recommended that Hyper-Threading be turned off for general-purpose HPC clusters. Depending on the applications used, the benefit of this feature should be tested and enabled as appropriate.