White Papers

ManualsBrandsDell ManualsConverged InfrastructureHigh Performance Computing Solution Resources

Ready Solutions Engineering Test Results

Skylake memory study

Authors: Joseph Stanfield, Garima Kochhar and Donald Russell, Bruce Wagner.

HPC Engineering and System Performance Analysis Teams, HPC Innovation Lab, January 2018

To function efficiently in an HPC environment, a cluster of compute nodes must work in tandem to compile complex data and achieve

desired results. The user expects each node to function at peak performance as an individual system, as well as a part of an intricate

group of nodes processing data in parallel. To enable efficient cluster performance, we first need good single system performance. With

that in mind, we evaluated the impact of different memory configurations on single node memory bandwidth performance using the

STREAM benchmark. The servers used here support the latest Intel Skylake processor (Intel Scalable Processor Family) and are from

the Dell EMC 14th generation (14G) server product line.

Less than 6 DIMMS per socket

The Skylake processor has a built-in memory controller similar to previous generation Xeons but now supports *six* memory channels

per socket. This is an increase from the four memory channels found in previous generation Xeon E5-2600 v3 and E5-2600 v4

processors. Different Dell EMC server models offer a different number of memory slots based on server density, but all servers offer at

least one memory module slot on each of the six memory channels per socket.

For applications that are sensitive to memory bandwidth and require predictable performance, configuring memory for the underlying

architecture is an important consideration. For optimal memory performance, all six memory channels of a CPU should be populated

with memory modules (DIMMs), and populated identically. This is a called a balanced memory configuration. In a balanced

configuration all DIMMs are accessed uniformly and the full complement of memory channels are available to the application. An

unbalanced memory configuration will lead to lower memory performance as some channels will be unused or used unequally. Even

worse, an unbalanced memory configuration can lead to unpredictable memory performance based on how the system fractures the

memory space into multiple regions and how Linux maps out these memory domains.

Figure 1. Relative memory bandwidth with different number of DIMMs on one socket.

PowerEdge C6420. Platinum 8176. 32 GB 2666 MT/s DIMMs.

Figure 1 shows the drop in performance when all six memory channels of a 14G server are not populated. Using all six memory

channels per socket is the best configuration, and will give the most predictable performance. This data was collected using the Intel

0.17

0.35

0.51

0.69

0.36

1.00

0.00

0.20

0.40

0.60

0.80

1.00

1.20

1 2 3 4 5 6

Relative Memory Bandwidth

[higher is better]

Number of DIMMs populated in the socket, one per memory channel

Relative Memory bandwidth of one socket with different DIMMs

Summary of content (6 pages)

PAGE 1
Ready Solutions Engineering Test Results Skylake memory study Authors: Joseph Stanfield, Garima Kochhar and Donald Russell, Bruce Wagner. HPC Engineering and System Performance Analysis Teams, HPC Innovation Lab, January 2018 To function efficiently in an HPC environment, a cluster of compute nodes must work in tandem to compile complex data and achieve desired results.
PAGE 2
Ready Specs 2 Xeon Platinum 8176 processor. While the exact memory performance of a system depends on the CPU model, the general trends and conclusions presented here apply across all CPU models. Balanced memory configurations Focusing now on the recommended configurations that use all 12 memory channels in a two socket 14G system, there are different memory module options that allow different total system memory capacities.
PAGE 3
Ready Specs 3  With two 8GB single ranked DIMMs giving two ranks on each channel, some of the memory bandwidth lost with 1DPC SR DIMMs can be recovered with the 2DPC configuration.  16GB dual ranked DIMMS perform better than 32GB DIMMs in 2DPC configurations too. We also measured the impact of this memory population when 2400 MT/s memory is used, and the conclusions were identical to those for 2666 MT/s memory. For brevity, the 2400 MT/s results are not presented here.
PAGE 4
Ready Specs 4 250 0.65 200 0.35 0.39 0.60 0.50 0.49 150 0.70 0.41 0.40 0.30 100 0.20 50 0.10 0 0.00 8176 6142 5120 4114 3106 Processor SKU 384 GB 12x32GB 512 GB 16x32GB 16 DIMM relative bandwidth when compared to 12 DIMMs. Stream Triad GB/s (higher is better) Impact of unbalanced 512 GB memory configuration in 14G Rel perf 16x32GB Figure 4. Impact of unbalanced memory configurations. PowerEdge C6420. Processor and DIMM configuration as noted. All 2666 MT/s memory.
PAGE 5
Ready Specs 5 So the question is, how bad are mixed memory configurations for HPC? To address this, we tested valid “near-balanced configurations” as described in Table 1, with the results displayed in Figure 5. Table 1.
PAGE 6
Ready Solutions Engineering Test Results  Unbalanced configurations are not recommended when optimizing for performance  Some near-balanced configurations are a reasonable alternative when the memory capacity requirements demand a compromise Previous memory configuration studies 1) Performance and Energy Efficiency of the 14th Generation Dell PowerEdge Servers – Memory bandwidth results across Intel Skylake Processor Family (Skylake) CPU models (14G servers) 2) http://en.community.dell.