White Papers

8 Dell HPC Omni-Path Fabric: Supported Architecture and Application Study June 2016
Sending the data
 Programmed I/O (PIO): This supports the on-load model. The host can be used to send small
messages since these can be sent by the CPU faster than an RDMA setup time.
 Send DMA (SDMA): For larger messages the CPU sets up a RDMA send and then the 16 SDMA engines
in the HFI transfer the data to the receiving host without CPU intervention.
Receiving the data
 Eager receive: The data is delivered to the host memory and then copied to the application memory.
This protocol is faster for smaller messages and does not require any responses
 Expected Receive: Data flows directly from HFI to application memory without CPU intervention.
Each data transfer method is independent. For example, SDMA can be used from Sender’s side and Eager
receive method can be used on the other side. This can be used for medium size messages since it does
not require full RDMA setup. However, the packet size threshold for small, medium and large have default
values but are configurable.
All these features make OPA an ideal component for HPC workloads. In the upcoming sections, this white
paper details Dell’s support of Intel OPA and dives deeper into the software ecosystem. Finally, it discusses
the performance characterization of several HPC workloads at scale using OPA.