HP Fabric Clustering System for InfiniBand™ Interconnect Performance on HP-UX 11iv2

ManualsBrandsHP ManualsComputer AccessoriesHP Integrity PCIe 2-port 4X DDR Infiniband Host Channel Adapter

HP Fabric Clustering System for InfiniBand™ Interconnect

Performance on HP-UX 11iv2

White paper

Executive Summary ...........................................................................................................................2

Performance Overview ...........................................................................................................................2

Document Structure ................................................................................................................................2

Methodology ....................................................................................................................................2

itperf ....................................................................................................................................................3

HP Integrity Servers ................................................................................................................................3

Performance Considerations....................................................................................................................3

Cluster Solution Performance – Point-to-Point Configuration.................................................................4

Latency .................................................................................................................................................4

Bandwidth.............................................................................................................................................6

HP Fabric Clustering System – Switch Fabric......................................................................................7

Latency .................................................................................................................................................8

Bandwidth.............................................................................................................................................9

Receive Bandwidth...............................................................................................................................10

Transmit Bandwidth..............................................................................................................................10

Non-Optimal Configurations............................................................................................................11

Summary ........................................................................................................................................11

References ......................................................................................................................................11

Summary of content (11 pages)

PAGE 1
HP Fabric Clustering System for InfiniBand™ Interconnect Performance on HP-UX 11iv2 White paper Executive Summary ...........................................................................................................................2 Performance Overview ...........................................................................................................................2 Document Structure ...........................................................................................................
PAGE 2
Executive Summary This document presents the performance characteristics and capabilities of the recently announced HP Fabric Clustering System for InfiniBand cluster interconnect product on HP-UX 11iv2 platforms.
PAGE 3
itperf itperf is a user-space program developed over the ICSC defined IT-API v1.0. itperf provides two types of tests and a brief description of each of these types is listed below: • Latency tests itperf supports two kinds of latency tests: Send/Receive programming model RDMA programming model The following gives a high-level flow of Send/Receive programming model: o Transmit/receive buffers share the same physical memory. o Posts the same buffer repeatedly for the whole test.
PAGE 4
Cluster Solution Performance – Point-to-Point Configuration A pair of rx2600/rx4640 Integrity servers are connected via a single HCA on each server. One port on each HCA is connected to the other using a 4X cable.
PAGE 5
Point-to-point Latency (Send/Receive programming model) on rx2600 (2CPUs/1.5 GHz/4 GB RAM) and rx4640 (4 CPUs/1.5GHz/2GB RAM) 55 50 45 Latency (usec) 40 35 Point-to-point Latency (usec) on rx2600 30 Point-to-point Latency (usec) on rx4640 25 20 15 10 5 0 1 10 100 1000 10000 100000 Message Length (in bytes) The latency values remain virtually unaffected when the results are obtained from two independent pairs of HCAs plugged into rx4640 servers.
PAGE 6
Bi-directional 1-way latencies on a rx2600 in point-to-point configuration are 5.6usec on each stream in the RDMA programming model for 8-byte messages, and 7.1usec on each stream in the Send/Receive programming model for 1-byte messages. The bi-directional latencies for short messages, up to 512 bytes, remain pretty close to the respective uni-directional latencies. On a rx2600 in point-to-point configuration, the bi-directional 1-way latencies for 512 byte messages are 7.
PAGE 7
Bandwidth Scaling: Point-to-Point configuration (rx4640/3CPUs/2GB RAM) 1600 100 1400 90 Bandwidth (MB/s) CPU Utilization (%) 80 1200 70 1000 60 800 50 600 40 30 400 Bandwidth (MB/s) Bi-directional Bandwidth (MB/s) CPU Util-Uni Directional CPU Util - Bi-Directional 20 200 10 0 0 1 2 Number of HCAs The HP-UX Fabric Clustering System interconnect solution scales without adversely affecting the service demand requirements on the server.
PAGE 8
IB 4X Cable console mgmt-eth AB291A PCI-X 133 PWR 1 dnabi ni f nI PWR 2 PCI-X 133 Management Card LAN 10/100 IB 4X Cable CONSOLE VGA GSP RESETS SOFT CONSOLE / REMOTE / UPS PCI-X 133 HARD SERIAL A PCI-X 133 TOC LAN 10/100 LAN Gb SERIAL B USB rx2600 PWR 2 PCI-X 133 PWR 1 dnabi ni f nI SCSI LVD/SE PCI-X 133 Management Card LAN 10/100 1 2 3 4 5 6 7 8 9 10 11 12 CONSOLE VGA GSP RESETS SOFT CONSOLE / REMOTE / UPS PCI-X 133 HARD SERIAL A PCI-X 133 SCSI LVD/SE IB 4X Cab
PAGE 9
Bandwidth Uni-directional bandwidth on a rx2600 node cluster provides an application bandwidth of 761MB/s at 4MB message size with 0% service demand. Bi-directional bandwidth for the same message size is 790MB/s with 0% service demand on the server. Bandwidth on rx2600 (2 CPUs/1.
PAGE 10
Receive Bandwidth Three application streams can saturate the receive bandwidth offered by the HP-UX HCA. Cumulative receive bandwidth on a rx4640 (3 CPUs/1.5 GHz/2 GB RAM) for a message size of 64KB is 924.5MB/s; limited only by the limitations of the PCI-X point of attachment. Receive Bandwidth on rx4640 (Msg. Size - 64 KB, Processor Speed: 1.5GHz, 3 CPUs.
PAGE 11
Transm it Bandw idth on rx4640 (Msg. Size - 64 KB, Processor Speed: 1.5GHz, 3 CPUs.) 760 Transmit Side Bandwidth (MB/s) 750 2 740 1.5 730 1 720 0.5 710 700 Transmit Side CPU Utilization (%) 2.5 Transmit Bandw idth Series2 0 1 2 3 Num ber of Parallel Stream s Non-Optimal Configurations HCAs when plugged in single rope PCI-X slots on a rx2600 (2 CPUs/1.5 GHz/4GB RAM) in a point-to-point configuration offer 1-way latency of 7.8usec, as against 7.