White Papers

ManualsBrandsDell ManualsConverged InfrastructureHigh Performance Computing Solution Resources

Performance study of four Socket PowerEdge R940 Server with Intel Skylake

processors

Author: Somanath Moharana, Dell EMC HPC Innovation Lab, August 2017

This blog explores the performance of the four socket Dell EMC PowerEdge R940 server with

Intel Skylake processors. The latest Dell EMC 14th generation servers support the new Intel®

Xeon® Processor Scalable Family (processor architecture codenamed “Skylake”), and the

increased number of cores and higher memory speed benefit a wide variety of HPC applications.

The PowerEdge R940 is Dell EMC’s latest 4-socket, 3U rack server designed to run complex

workloads, which supports up to 6TB of DDR4 memory and up to 122 TB of storage. The

system features the Intel® Xeon® Scalable Processor Family, 48 DDR4 DIMMs, up to 13 PCI

Express® (PCIe) 3.0 enabled expansion slots and a choice of embedded NIC technologies. It is a

general-purpose platform capable of handling demanding workloads and applications, such as

data warehouses, ecommerce, databases, and high-performance computing (HPC). With the

increase in storage capacity the PowerEdge R940 makes it well-suited for data intensive

applications that require greater storage.

This blog also describes the impact of BIOS tuning options on HPL, STREAM and scientific

applications ANSYS Fluent and WRF and compares performance of the new PowerEdge R940

to the previous generation PowerEdge R930 platform. It also analyses the performance with Sub

NUMA Cluster (SNC) modes (SNC=Enabled and SNC=Disabled). SNC enabled will expose

eight NUMA nodes to the OS on a four socket PowerEdge R940. Each NUMA node can

communicate with seven other remote NUMA nodes, six in other three sockets and one within

same socket. NUMA domains on different sockets communicate over the UPI interconnect.

Please visit BIOS characteristics of Skylake processor-blog for more details on BIOS options.

Table 1 lists the server configuration and the application details used for this study.

Table 1: Details of Server and HPC Applications used for R940 analysis

Platform PowerEdge R930 PowerEdge R930 PowerEdge R940

Processor 4 x Intel Xeon E7-8890

v3@2.5GHz (18 cores)

45MB L3 cache 165W

Codename=Haswell-

4 x Intel Xeon E7-8890

v4@2.2GHz (24 cores)

60MB L3 cache 165W

Codename=Broadwell-

4 x Intel Xeon Platinum

8180@2.5GHz,

10.4GT/s (Cross-bar

connection)

Codename=Skylake

Memory 1024 GB = 64 x 16GB

DDR4 @1866MHz

1024 GB = 32 x 32GB

DDR4 @1866 MHz

384GB = (24 x 16GB)

DDR4@2666MT/s

CPU

Interconne

Intel QuickPath

Interconnect (QPI)

8GT/s

Intel QuickPath

Interconnect (QPI)

8GT/s

Intel Ultra Path

Interconnect (UPI)

10.4GT/s

BIOS Settings

Summary of content (8 pages)

PAGE 1
Performance study of four Socket PowerEdge R940 Server with Intel Skylake processors Author: Somanath Moharana, Dell EMC HPC Innovation Lab, August 2017 This blog explores the performance of the four socket Dell EMC PowerEdge R940 server with Intel Skylake processors.
PAGE 2
BIOS Processor Settings > Logical Processors Processor Settings > UPI Speed Processor Settings > Sub NUMA cluster System Profiles Operating System Intel Compiler Intel MKL LINPACK STREAM WRF ANSYS Fluent Version 1.0.9 Disabled Version 2.0.1 Disabled Version 1.0.
PAGE 3
Note: The software versions were different for the older generation processors and results are compared against what was best configuration at that time. Due to big architectural changes in servers and processors generation over generation, the changes in software versions is not a significant factor. The High Performance Linpack (HPL) Benchmark is a measure of a system's floating-point computing power.
PAGE 4
Figure 2: HPL performance with AVX2 and AVX512 instructions sets Figure 3: HPL Performance over multiple generations of processors Figure 2 compares the performance of HPL ran with AVX2 instructions sets and AVX512 instructions sets on PowerEdge R940 (where AVX=Advanced Vector Instructions). AVX-512 are 512-bit extensions to the 256-bit AVX SIMD instructions for x86 instruction set architecture. By Setting “MKL_ENABLE_INSTRUCTIONS=AVX2/AVX512” environment variable we can set the AVX instructions.
PAGE 5
Figure 4: STREAM Performance across BIOS profiles Figure 5: STREAM Performance over multiple generations of processors As per Figure 4, With “SNC = Enabled” we are getting up to 3% better bandwidth in comparison to “SNC = Disabled” across all bios profiles. Figure 5, shows the comparison of memory bandwidth of PowerEdge R930 server with Haswell-EX, Broadwell-EX processors and PowerEdge R940 server with Skylake processors.
PAGE 6
case of remote to other socket, the memory bandwidth shows ~57% decrease in comparison to local memory bandwidth. With “SNC = Disabled” remote memory bandwidth is 77% lower compared to local memory bandwidth. The Weather Research and Forecasting (WRF) Model is a mesoscale numerical weather prediction system designed for both atmospheric research and operational forecasting needs.
PAGE 7
larger dataset we can observe 1-2% performance variations across the system profiles as it utilizes larger number of processors more efficiently. Figure 10: Comparison over multiple generations of processors Figure 11: Comparison over multiple generations of processors Figure 10 and Figure 11 shows the performance comparison between PowerEdge R940 powered by Skylake processors and PowerEdge R930 powered by Broadwell-EX processors and HaswellEX processors.
PAGE 8
Figure 12: Ansys Fluent Performance across BIOS profiles We used five different datasets for our analysis which are truck_poly_14m, combuster_12m, exhaust_system_33m, ice_2m and aircraft_wing_14m. Fluent with ‘Solver Rating’ (Higher is better) as the performance metric. Figure 12 shows that all datasets performed better with “Performance SNC = Enabled” BIOS option than others. For all datasets, the “SNC = Enabled” performs 2% to 4% better than “SNC = Disabled”.