White Papers

ManualsBrandsDell ManualsConverged InfrastructureHigh Performance Computing Solution Resources

Dell - Internal Use - Confidential

Introducing 100GBps with Intel® Omni-Path Fabric in HPC

By Munira Hussain, Deepthi Cherlopalle

This blog introduces the Omni-Path Fabric from Intel® as a cluster network fabric used for intra-node

communication for application, management and storage communication in High Performance

Computing (HPC). It is part of the new technology referring to Intel® Scalable System framework based on

IP generated from the coalition of Qlogic, Truescale and Cray Aries. The goal of Omni-Path is to eventually

be able to meet the demands of the exascale data centers in performance and scalability.

Dell provides complete validated and supported solution offering which includes the Networking H-series

Fabric switches and Host Fabric Interface (HFI) adapters. The Omni-Path HFI is a PCI-E Gen3 x16 adapter

capable of 100 Gbps unidirectional per port. The card supports 4 lanes supporting 25Gbps per lane.

HPC Program Overview with Omni-Path:

The current solution program is based on Red Hat Linux 7.2 (kernel version 3.10.0-327.el7.x86_64). The

Intel Fabric Suite (IFS) drivers are integrated in the current software solution stack Bright Cluster Manager

7.2 which helps to deploy, provision, install and configure an Omni-Path cluster seamlessly.

The following Dell servers support Intel

Omni-Path Host Fabric Interface (HFI) cards

PowerEdge R430,PowerEdge R630, PowerEdge R730, PowerEdge R730XD, PowerEdge R930, PowerEdge

C4130, PowerEdge C6320

The management and monitoring of the Fabric is done using the Fabric Manager (FM) GUI available from

Intel®. The FM GUI provides in-depth analysis and graphical overview of the fabric health including

detailed breakdown of status of the ports, mapping as well as investigative report on the errors.

Figure 1: Fabric Manager GUI

Summary of content (4 pages)

PAGE 1
Introducing 100GBps with Intel® Omni-Path Fabric in HPC By Munira Hussain, Deepthi Cherlopalle This blog introduces the Omni-Path Fabric from Intel® as a cluster network fabric used for intra-node communication for application, management and storage communication in High Performance Computing (HPC). It is part of the new technology referring to Intel® Scalable System framework based on IP generated from the coalition of Qlogic, Truescale and Cray Aries.
PAGE 2
The IFS tools include various debugging and management tools such as opareports, opainfo, opaconfig, opacaptureall, opafabricinfoall, opapingall, opafastfabric, etc. These help to capture a snapshot of the Fabric and to troubleshoot. The Host based subnet manager service known as opafm is also available with IFS and is able to scale up to 1000’s of nodes. The Fabric relies on the PSM2 libraries to provide optimal performance.
PAGE 3
OSU Bandwidth 30 Bandwidth(GB/s) 25 21.2 20 22.5 23 23.9 24.5 12.1 12.3 12.3 12.3 24.3 24.3 16.9 14 15 8.7 10 10.1 6.4 4 5 3.3 0 7.1 5.1 10.2 11.8 12.0 12.3 12.3 7.8 Message Size(bytes) Bi-directional Uni-directional Figure 3: OSU Bandwidth – E5-2697 v4 Conclusion: Omni-Path Fabric provides a value add to the HPC solution.
PAGE 4
More details on how Omni-Path perform in the other domains is available here . This document provides Intel® Omni-Path Fabric technology key features and provides a reference to performance data conducted on various commercial and open source applications.