White Papers

ManualsBrandsDell ManualsConverged InfrastructureHigh Performance Computing Solution Resources

Need for Speed: Comparing FDR and EDR InfiniBand (Part 1)

By Olumide Olusanya and Munira Hussain

The goal of this blog is to evaluate the performance of Mellanox Technologies’ FDR (Fourteen Data Rate)

Infiniband and their latest EDR (Enhanced Data Rate) Infiniband with speeds of 56Gb/s and 100Gb/s

respectively. This is the first of our two series blog and we will be showing how these interconnects

perform on a cluster using industry-wide micro-level benchmarks and applications on HPC cluster

configuration. In this part, we will show latency, bandwidth and HPL results for FDR vs EDR and in part 2

we will share more results with other applications which include ANSYS Fluent, WRF, and NAS Parallel

Benchmarks. You should also keep in mind that while some applications would benefit from the higher

bandwidth in EDR, other applications which have low communication overhead would show little

performance improvement in comparison.

General Overview:

Mellanox EDR adapters are based on a new generation ASIC also known as ConnectX-4 while the FDR

adapters are based on ConnectX-3. The theoretical uni-directional bandwidth for EDR is 100 Gb/s versus

FDR which is 56Gb/s. Another difference is that EDR adapters are x16 adapters while FDR adapters are

available in x8 and x16. Both of these adapters operate at a bus width of 4X link. The messaging rate for

EDR can reach up to 150 million messages per second compared with FDR ConnectX-3 adapters which

deliver more than 90 million messages per second.

Table 1 below shows the difference between EDR and FDR and Table 2 describes the configuration of

the cluster used in the test while Table 3 lists the applications and benchmarks used for this test.

Table 1 - Difference between EDR and FDR

FDR

EDR

Chipset

ConnectX

Link

and x16

Gen3

16 Gen3

Theoretical BW

56 Gb/

100 Gb/

Messaging rate

MMS

150

MMS

Port

QSFP

Table 2 - Cluster configuration

Components

Details

Server

16 nodes x

PowerEdge

C6320 [ 4 chassis ]

Processor

Intel®Xeon®Intel Xeon

2660 v3 @2.6/2.2 GHz , 10 cores, 105W

Summary of content (5 pages)

PAGE 1
Need for Speed: Comparing FDR and EDR InfiniBand (Part 1) By Olumide Olusanya and Munira Hussain The goal of this blog is to evaluate the performance of Mellanox Technologies’ FDR (Fourteen Data Rate) Infiniband and their latest EDR (Enhanced Data Rate) Infiniband with speeds of 56Gb/s and 100Gb/s respectively. This is the first of our two series blog and we will be showing how these interconnects perform on a cluster using industry-wide micro-level benchmarks and applications on HPC cluster configuration.
PAGE 2
BIOS 1.1.3 Memory 128 GB – 8 x16 GB @ 2133MHz Operating System Red Hat Enterprise Linux Server release 6.6.z (Santiago) Kernel 2.6.32-504.16.2.el6.x86_64 MPI Intel® MPI 5.0.3.048 Drivers MLNX_OFED_LINUX-3.0-1.0.
PAGE 3
NAS Parallel Benchmarks Computational Fluid Dynamics 3.3.1 CG, MG, IS, FT Results OSU Micro-Benchmarks To find the latency and bandwidth, we used the tests from the OSU Micro-Benchmark suite. These tests use the MPI message passing performance to check the quality of a network fabric. Using the same system configuration for EDR and FDR fabrics, we got latency results as shown in Figure 1 below. OSU Latency Latency (us) 3.0 2.4 1.8 1.2 0.6 0.0 0.80 0.
PAGE 4
OSU Bandwidth 30.0 24.2 Bandwidth (GB/s) 25.0 20.0 12.4 15.0 10.0 10.8 5.0 6.3 0.0 Message Size (Bytes) FDR Unidirectional EDR Unidirectional FDR Bidirectional EDR Bidirectional Figure 2 - OSU Bandwidth (using MPI from Mellanox HPC-X Toolkit) OSU unidirectional bandwidth is a ping-pong type of communication test where the sender sends a fixed size of messages back-to-back to a receiver and then the receiver responds only after receiving all the messages.
PAGE 5
HPL Peformance (EDR vs FDR) Efficiency 92.0% 91.1% 91.0% 90.7% 90.6% 90.8% 90.8% 90.6% 90.7% 90.6% 90.0% 40 80 160 320 Number of cores EDR FDR Figure 3 - HPL Performance HPL benchmark is a compute-intensive application. It could spend more than 80% of its runtime on computation depending on how you tune it. During the bulk of its communication time, it sends messages of small sizes across the cluster which may not benefit from a higher speed network.