Specifications

OFED+ Host SW
True Scale Fabric OFED+ Host Software
RN 7.2.2.0.8 February 2014
16 Order Number: H31512002US
• The OpenSHMEM effort (see http://www.openshmem.org) is defining a
standardized API specification for SHMEM. Although it is premature to claim
compliance, Intel
®
SHMEM aims to be compliant with the OpenSHMEM 1.0
specification. Intel provides a SHMEM API that is compatible with the OpenSHMEM
1.0 specification, other than any omissions or bugs documented in these release
notes. If compliance with OpenSHMEM's passive progress statements are required,
Intel
®
SHMEM's passive progress mechanism must be enabled.
• When using Mellanox HCAs, any changes to Virtual Fabrics (vFabrics) in the Fabric
Manager, may require a reboot of the hosts with Mellanox HCAs. This limitation
relates to the Mellanox HCAs not properly responding to changes to the Fabric
Manager service level (SL). For some vFabric configuration changes, if the Fabric
Manager SL changes or is mapped to a different Virtual Lane (VL) than previously,
the Mellanox HCA can continue to use the previous VL. If that VL is presently
disabled by the Fabric Manager, future uses of applications which use the Fabric
Manager SL may hang or timeout because there are no VL Arbitration cycles for
that VL. As a result, anytime vFabric configuration is changed, it is recommended
to reboot all hosts with Mellanox HCAs so that the desired Quality of Service (QoS)
configuration changes fully take effect. Any hosts with Intel
®
HCAs will not need to
be rebooted.
Due to Mellanox HCAs not correctly handling changes to the Fabric Manager SL,
Intel recommends that all the hosts using Mellanox ConnectX or ConnectX-2 HCAs
be rebooted when used in a virtual fabric configuration.
• When Dispersive Routing is enabled, it allows packets sent using an MPI program
run over PSM to take any one of several routes through a fabric, thus often
increasing performance. The number of routes is determined by the value of 2 to
the power of the Lid Mask Control setting (LMC). Because LMC defaults to 0, the
default number of routes through the fabric is 2
0
or 1. LMC can be set as high as 3,
allowing a total number of 2
3
or 8 routes through the fabric. Providing these
additional routes can reduce fabric congestion, and thus improve performance.
Dispersive Routing is supported when the Fabric Manager is used in the fabric.
Dispersive Routing is not supported when using OpenSM.
• When running MVAPICH2, Intel recommends turning off RDMA fast path. To turn off
RDMA fast path, specify
MV2_USE_RDMA_FAST_PATH=0 in the mpirun_rsh
command line or set this option in the parameter file for mvapich2.
• The
ib_send_bw benchmark, when run in UC mode, is written such that it will
hang if even one packet is dropped.
1.14 Documentation
Table 9 lists the Release 7.2 related documentation. All related documentation is
available on the Intel
®
download site.
Documentation for Intel
®
Partners is available at the vendors web site.
Table 9. Related Documentation for this Release
Document Title Document Number Revision
Intel
®
Hardware Documents
Intel
®
True Scale Fabric Switches 12000 Series Hardware Installation
Guide
G91928 002US
Intel
®
True Scale Fabric Switches 12000 Series Users Guide G91930 002US
Intel
®
True Scale Fabric Switches 12000 Series CLI Reference Guide G91931 002US
Intel
®
True Scale Fabric Adapter Hardware Installation Guide G91929 002US
Intel
®
OFED+ Documents