White Papers

ManualsBrandsDell ManualsConverged InfrastructureHigh Performance Computing Solution Resources

Page 1 of 9

Available at http://dell.to/1w8xlKI

New 13G servers – what’s new and how much better are they for HPC?

Garima Kochhar, September 2014

It’s been an exciting week –Intel Haswell processors for two-socket servers, DDR4 memory and new Dell

servers were just released. We’ve had a busy few months leading up to this announcement – our team

had access to early server units for the HPC lab and we spent time kicking the tires, running benchmarks,

and measuring performance. This blog describes our study and initial results and is part one of a three

part series. The next blog will discuss the performance implications of some BIOS tuning options

available on the new servers, and a third blog will compare performance and energy efficiency across

different Haswell processor models.

Focusing on HPC applications, we ran two benchmarks and four applications on our server. Our interest

was in seeing how the server performed and specifically how it compared to the previous generations.

The server in question is part of Dell’s PowerEdge 13

generation (13G) server line-up. These servers

support DDR4 memory at up to 2133 MT/s and Intel’s latest Xeon® E5-2600 v3 Product Family

processors (based on the architecture code-named Haswell). Haswell (HSW) is a net new micro-

architecture when compared to the previous generation - Sandy Bridge/Ivy Bridge. HSW processors use

a 22nm process technology, so there’s no process-shrink this time around. Note the “v3” in the Intel

product name – that is what distinguishes a processor as one based on Haswell micro-architecture.

You’ll recall that “E5-2600 v2” processors are based on the Ivy Bridge micro-architecture and plain E5-

2600 series with no explicit version are Sandy Bridge based processors. Haswell based processors

require a new server/new motherboard and DDR4 memory. The platform we used is a standard dual-

socket rack server with two Haswell-EP based processors. Each socket has four memory channels and

can support up to 3 DIMMs per channel (DPC). For our study we used 1 DPC for a total of eight DDR4

DIMMs in the server.

From an HPC point of view, one of the most interesting aspects is the Intel® AVX2 technology that allows

the processor to execute 16 FLOP per cycle. The processor supports 256 bit registers, allows three-

operand non-destructive operations (i.e. A = B+C vs. A = A+B), and a Fuse-Multiply-Add (FMA)

instruction (A = A*B+C). The processor has two FMA units each of which can execute 4 double precision

calculations per cycle. With two floating point operations per FMA instructions, HSW can execute 16

FLOP/cycle. This value is double of what was possible with Sandy Bridge/Ivy Bridge (SB/IVB)! There are

many more instructions introduced with HSW and Intel® AVX2 and these are described in detail in this

Intel programming reference or on other blogs.

Double the FLOP/cycle - does this mean that HSW will have 2x the theoretical performance of an

equivalent IVB processor? Close but not quite - read on. In past generations, we’ve looked at the rated

base frequency of a processor and the available Turbo bins/max Turbo frequency. For example, the

Intel® Xeon® E5-2680 v2 has a base frequency of 2.8 GHz and a maximum of 300 MHz of turbo available

when all cores are active. HSW processors will consume more power when running the new Intel® AVX2

instructions than when running non-AVX instructions. And so, starting with Haswell product family there

will be two rated base frequencies provided. The first is the traditional base frequency which is the

frequency one could expect to run non-AVX workloads. The second frequency is the base frequency for

workloads that are running AVX code, the “AVX base frequency”. For example, the HSW Xeon® E5-2697

Summary of content (9 pages)

PAGE 1
New 13G servers – what’s new and how much better are they for HPC? Garima Kochhar, September 2014 It’s been an exciting week –Intel Haswell processors for two-socket servers, DDR4 memory and new Dell servers were just released. We’ve had a busy few months leading up to this announcement – our team had access to early server units for the HPC lab and we spent time kicking the tires, running benchmarks, and measuring performance.
PAGE 2
v3 has a base frequency of 2.6 GHz and an AVX base of 2.2 GHz. Compare that with the Xeon® E5-2680 v2 IVB processor running at 2.6 GHz. For the 2.6 GHz IVB processor, we would calculate HPL theoretical maximum performance as (2.6 GHz * 8 FLOP/cycle * total number of cores). But for a HSW processor with the same rated base frequency of 2.6 GHz and an AVX base of 2.2 GHz, we now calculate HPL theoretical maximum using the AVX base as (2.
PAGE 3
Table 2 - Server configuration Components Server Processor Memory Hard drive RAID controller Operating System Kernel BIOS settings MPI Math Library Compilers Details PowerEdge R730xd – engineering sample unit/prototype 2 x Intel® Xeon® E5-2697 v3 – 2.6 GHz, 14c, 145W 128GB - 8 x 16GB 2133 MHz DDR4 RDIMMs 1 x 300GB SAS 6Gbps 10K rpm PERC H330 mini Red Hat Enterprise Linux 6.5 x86_64 2.6.32-431.el6.
PAGE 4
Memory Bandwidth Memory Bandwidth - GB/s 4.2 120 100 18% + over IVB 80 4.5 4.3 4.1 4.0 3.4 3.5 3.0 45% + over SB 2.5 60 2.0 66% + over WSM 1.5 40 1.0 20 0.5 0 Memory bandwidth per core - GB/s 140 0.0 13G - HSW 12G - IVB 2.6GHz, 14c, 2133MHz 2.7GHz, 12c, 1866MHz 12G - SB 11G - WSM 2.7GHz, 8c, 1333MHz 2.93GHz, 6c, 1333MHz Full system Per Core Figure 1 - Memory bandwidth Figure 2 shows the HPL performance generation-over-generation.
PAGE 5
HPL Performance 1000 800 99% 99% 920 98% 700 97% 600 96% 500 513 400 300 95% 95% 94% 93% 94% HPL Efficiency HPL Performance - GFLOPs 900 100% 93% 330 200 92% 100 132 0 91% 90% 13G - HSW 12G - IVB 12G - SB 11G - WSM 2.6GHz, 14c, 2133MHz 2.7GHz, 12c, 1866MHz 2.7GHz, 8c, 1333MHz 2.93GHz, 6c, 1333MHz HPL Perf HPL Efficiency Figure 2 - HPL performance Figure 3 shows Ansys Fluent performance on 13G HSW when compared to 12G IVB.
PAGE 6
Percentage improvement in perf. with 13G over 12G Ansys Fluent - 12G vs. 13G 60% 50% 40% 48% 47% 43% 42% 37% 30% 33% 20% 10% 6% 5% 1% -2% 2% -5% 0% -10% Full system Per Core Figure 3 - Ansys Fluent performance Figure 4 shows WRF performance generation-over-generation for the Conus 12km data set. The average time step computed over the last 149 intervals for Conus 12km is the metric used for performance. There is a 40% improvement from 12G-IVB and a 3.
PAGE 7
Improvement in performance over generations baseline is 1.0, higher is better WRF Conus 12km performance 3.5 3.0 3.2 2.5 2.3 2.0 1.8 1.5 1.0 1.0 0.5 0.0 13G - HSW 12G - MLK IVB 12G - SB E5-2697v3 @ 2.6 GHz, E5-2680v2 @ 2.8 GHz, E5-2680 @ 2.7 GHz, 14c, 2133 MHz 10c, 1866MHz 8c, 1600 MHz 11G - WSM X5560 @ 2.8GHz, 6c, 1333 MHz WRF-full-system Figure 4 - WRF Conus 12km performance MILC performance is shown in Figure 5.
PAGE 8
Improvement in performance over generations baseline is 1.0, higher is better MILC performance 1.6 1.4 1.2 1.34 1.21 1.0 1.00 0.8 0.6 0.4 0.2 0.0 13G - HSW 12G - MLK IVB 12G - SB E5-2697v3 @ 2.6 GHz, E5-2680v2 @ 2.8 GHz, E5-2680 @ 2.7 GHz, 14c, 2133 MHz 10c, 1866MHz 8c, 1600 MHz MILC-full-system Figure 5 - MILC performance LS DYNA with car2car and WRF with Conus 2.5km performance comparisons are shown in Figure 6.
PAGE 9
Relative performance (higher is better) LS-DYNA, WRF Conus 2.5km - performance 1.6 1.4 1.38 1.2 1.37 1.0 0.8 1.00 1.00 0.6 0.4 0.2 0.0 LS DYNA WRF car2car, endtime=0.02 Conus 2.5km 12G - IVB. E5-2680 v2 @ 2.8 GHz, 10c, 1866 MT/s 13G - HSW. E5-2697 v3 @ 2.6/2.2 GHz, 14c, 2133 MTs/ Figure 6 - LS DYNA, WRF 2.