Expand Configuration and Management Manual (G06.24+)

Subsystem Description
Expand Configuration and Management Manual—523347-008
18-32
Multi-CPU Paths
Neighbor Nodes
For neighbor nodes, Expand line-handler pairs apply only to each source and
destination processor combination, not to entire systems. This method allows traffic
between neighbor nodes to be distributed over all the paths in the multi-CPU path.
Message order is preserved only between processor pairs instead of between entire
systems.
$NCP does not establish Expand line-handler pairs with a neighbor node. Instead,
when the first message is sent from a processor in the source node to each processor
in the destination node, the NRT lookup routine in the source processor selects an
Expand line-handler process and saves that process in the NRT in its processor;
subsequent messages sent from the same processor in the source node to the same
processor in the destination node are sent using the same Expand line-handler
process.
When selecting an Expand line-handler process, the NRT lookup routine selects a
process that spreads the load over all the paths in the multi-CPU path. Pairing
information is not broadcast to other processors, and pairings are not symmetrical;
messages between the same two processors in the reverse direction may use a
different Expand line-handler pair.
Load Balancing
The formation of Expand line-handler pairs can interfere with the requirement to
balance traffic over all paths in a multi-CPU path. In addition, traffic patterns can
change radically over time, causing imbalances to occur after the formation of Expand
line-handler pairs. (This is especially true for non-neighbor pairs because they tend to
be made when the multi-CPU path is first started and no traffic information is
available.) For these reasons, $NCP periodically runs a rebalancing algorithm that
reconsiders the pairings of Expand line-handler processes on each multi-CPU path. If
the load is unbalanced, $NCP changes some Expand line-handler pairs.
Multi-CPU path (superpath) rebalancing is run periodically to correct path selection as
traffic patterns change. It has three goals:
•
CPU Matching: Make sure all source/destination pairs are using a path with the
most CPU matches available (same local/remote CPU).
•
Load Factor Balancing: Try to make the load factors of all paths within 0.5 of each
other.
•
Pair Count Balancing: Spread those pairs whose traffic have no adverse impact on
load factors (LFs) over all paths.
Note. $NCP does not know which Expand line-handler processes have been paired; this
information is maintained separately in each processor in its MPT.