White Papers

ManualsBrandsDell ManualsConverged InfrastructureHigh Performance Computing Solution Resources

HPCG Performance study with

Intel KNL

By Ashish Kumar Singh. January 2017 (HPC Innovation Lab)

This blog presents an in-depth analysis of the High Performance Conjugate Gradient (HPCG)

benchmark on the Intel Xeon Phi processor, which is based on Intel Xeon Phi architecture codenamed

“Knights Landing”. The analysis has been performed on PowerEdge C6320p platform with the new

Intel Xeon Phi 7230 processor.

Introduction to HPCG and Intel Xeon Phi 7230 processor

The HPCG benchmark constructs a logically global, physically distributed sparse linear system using a

27-point stencil at each grid point in 3D domain such that the equation at the point (I, j, k) depend on

its values and 26 surrounding neighbors. The global domain computed by benchmark is (NRx * Nx) X

(NRy*Ny) X (NRz*Nz), where Nx, Ny and Nz are dimensions of local subgrids, assigned to each MPI

process and number of MPI ranks are NR = (NRx X NRy X NRz). These values can be defined in

hpcg.dat file or passed in the command line arguments.

The HPCG benchmark is based on conjugate gradient solver, where the pre-conditioner is a three-level

hierarchical multi-grid (MG) method with Gauss-Seidel. The algorithm starts with MG and

contains Symmetric Gauss-Seidel (SymGS) and Sparse Matrix-vector multiplication (SPMV) routines

for each level. Both SYMGS and SPMV require data from their neighbor as data is distributed across

nodes which is provided by their predecessor, the Exchange Halos routine. The residual should be

lower than 1

-6

which is locally computed by Dot Product (DDOT), while MPI_Allreduce follows the

DDOT and completes the global operation. WAXPBY only updates a vector with sum of two scaled

vectors. Scaled vector addition is a simple operation that calculates the output vector by scaling the

input vectors with a constant and performing an addition on the values of the same index. So, HPCG

has four computational blocks SPMV, SymGS, WAXPBY and DDOT, while two communication

blocks MPI_Allreduce and Halos Exchange.

Intel Xeon Phi Processor is a new generation of processors from the Intel Xeon Phi family. Previous

generations of Intel Xeon Phi were available as a coprocessor, in a PCI card form factor and required

an Intel Xeon processor. The Intel Xeon Phi 7230 contains 64 cores @ 1.3GHz of core frequency along

Summary of content (6 pages)

PAGE 1
HPCG Performance study with Intel KNL By Ashish Kumar Singh. January 2017 (HPC Innovation Lab) This blog presents an in-depth analysis of the High Performance Conjugate Gradient (HPCG) benchmark on the Intel Xeon Phi processor, which is based on Intel Xeon Phi architecture codenamed “Knights Landing”. The analysis has been performed on PowerEdge C6320p platform with the new Intel Xeon Phi 7230 processor.
PAGE 2
with the turbo speed of 1.5GHz and 32MB of L2 cache. It supports DDR4-2400MHz memory up to 384GB and instruction set of AVX512. Intel Xeon Phi processor also encloses 16GB of MCDRAM memory on socket with a sustained memory bandwidth of up to ~480GB/s measured by the Stream benchmark. Intel Xeon Phi 7230 delivers up to ~1.8TFLOPS of double precision HPL performance.
PAGE 3
execution time of t=30 seconds. The local domain dimension defines the global domain dimension by (NR*Nx) x (NR*Ny) x (NR*Nz), where Nx=Ny=Nz=160 and NR is the number of MPI processes. Figure 1: HPCG Performance comparison with multiple local dimension grid size As shown in figure 1, the local dimension grid size of 160^3 gives the best performance of 48.83GFLOPS. The problem size bigger than 128^3 allows for more parallelism and it fits well inside the MCDRAM while 192^3 does not.
PAGE 4
Figure 2 demonstrates HPCG performance with multiple execution times for grid size of 160^3 on a single Intel KNL server. As per the graph, HPCG performance doesn’t change even by changing the execution time. It means execution time does not appear to be a factor for HPCG performance. So, we may not need to spend hours or days of time to benchmark large clusters, which in result, will save both time and power. Although, the official execution time should be >=1800 seconds as reported in the output file.
PAGE 5
Figure 4: A slice of HPCG output file Performance Comparison Here is the HPCG multi-nodes performance comparison between Intel Xeon E5-2697 v4 @2.3GHz (Broadwell processor) and Intel KNL processor 7230 with Intel Omni-path interconnect.
PAGE 6
Figure 5: HPCG performance comparison between Intel Xeon Broadwell processor and Intel Xeon Phi processor Figure 5 shows HPCG performance comparison between dual Intel Broadwell 18 cores processors and one Intel Xeon phi 64 cores processor. Dots in figure 5 show the performance acceleration of KNL servers over Broadwell dual socket servers. For single KNL node, HPCG performs 2.23X better than Intel Broadwell node.