White Papers

Dell HPC General Computing – Configuration and Supported Architecture with Dell PowerEdge 13th Generation Servers
10
Stream bandwidth for FC430 nodes with 8C and 12C with varying Snoop Modes and impact on
Infiniband bandwidth and latency.
System info:
Dual socket Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz (8core) and Intel Xeon ® CPU E5-2680 v3 @ 2.50
GHz (12 core), 8 x 16GB DDR4
Bios: 1.0.4, perfoptimized, HT-disabled, nodeinterleave=disabled, ES=early snoop, HS=Homesnoop
Stream (FC430 -8Core Triad GB/s)
ES
HS
Full
89
89
c0m0
45
45
c0m1
7
36
Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz (8core)
ES/HS
ES/HS
ES/HS
ib_write
core /memory
Bw (MB/s)
Bibw (MB/s)
Lat (us)
c0m0
5600/ 5800
10130/ 10522
0.8/ 0.8
c0m1
5800/ 5820
10555/ 10550
0.77/ 0.78
c1m0
5800/ 5800
11850/ 11840
0.66/ 0.66
c1m1
5800/ 5820
11850/ 11850
0.66/ 0.66
without numactl
5600/ 5800
10800/ 10800
0.76/ 0.76
Stream (FC430 -12c Triad GB/s)
ES
HS
COD
Full
115
116
116
c0m0
58
58
30
c0m1
20
41
14
Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz (12core)
ES/HS/COD
ES/HS/COD
ES/HS/COD
ib_write
core /memory
bw (MB/s)
bibw (MB/s)
lat (us)
c0m0
5945 /5940 /5963
11800 /11820 /11493
0.69 /0.67 /0.69
c0m1
5957/ 5955 / 5988
11800 /11850 / 11734
0.67 / 0.66 /0.65
c1m0
6037 /6037 /5978
11840/11750 /11674
0.57/ 0.58 / 0.56
c1m1
6039/ 6039 /6010
11860 /11850 /11837
0.57/ 0.57 /0.58
without numactl
6060 /6050 / 6011
10800 /11850 / 11850
0.69 /0.66 /0.65
As seen in the performance numbers above, the 8core processor latencies are slightly higher by 0.1-0.2 us
sec. This is because the memory bandwidth is impacted by the processor architecture which includes only a
memory controller for 8core and lower processors. Also note that when there is no pinning the bandwidth
and bi-directional bandwidth is slightly less by 100-200 MB/s for 8core servers. Additionally, early snoop (ES)
has a greater impact in remote bandwidth as it falls to almost 7GB/s in memory bandwidth. Home snoop
mode aims to provide a more consistent bw/bibw across numerous core/memory mappings for processors
8core and lower.