ft D ra QLogic HCA and QLogic OFED Software Install Guide IB0056101-00 G.02 P re l i mi n a r y QLogic OFED Version 1.
S QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4 Information furnished in this manual is believed to be accurate and reliable. However, QLogic Corporation assumes no responsibility for its use, nor for any infringements of patents or other rights of third parties which may result from its use. QLogic Corporation reserves the right to change product specifications at any time without notice.
S QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4 Replace EM64T with 64-bit Intel Xeon. “Supported Linux Distributions” on page 5-1 Added more information on VNIC interface. “OpenSM” on page 6-3 Split the Install section into three separate sections. “Software Installation” on page 5-1 “Configuring Drivers and Services” on page 6-1 “Installation Verification and Additional Settings” on page 7-1 “Software Installation” on page 3-2 Remove sections on configuring ipath_ether.
S QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4 Sections rearranged, renamed, and expanded due to multiple install methods.
S QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4 “Write Combining” on page B-1 Minor wording change to introduction section. “Software Installation” on page 5-1 Added new issue in Troubleshooting. “openmpi_gcc Fails to Install Because of Dependency on gfortran (RHEL 4)” on page A-4 Added new issue in Troubleshooting. “Outdated ipath_ether Configuration Setup Generates Error” on page A-5 Configuration file qlogic_vnic.cfg changed to qlgc_vnic.cfg.
S QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4 Changed title of Troubleshooting issue. Was “ipath_ether Configuration Setup Gener- “Outdated ipath_ether Configuration Setup Generates Error” on page A-5 Change main configuration file from /etc/sysconfig/infinipath to /etc/infiniband/openib.conf. The infinipath file is still to enable or disable the ipath_mtrr script. Table C-1 on page C-1 and “SRP” on page 6-4. Updated note about setting datagram mode.
S QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4 Adapter Settings section name changed. Also added new bullet point “Check the PCIe bus width” “Adapter and Other Settings” on page 7-1 Updated tables of RPMs. Moved all documentation RPMs together. New table for OpenSM-Devel RPMs. Updated OtherMPIs.
S QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4 D ra ft Notes Page viii Preliminary IB0056101-00 G.
Table of Contents 1 Introduction 2 Feature Overview ft Who Should Read this Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . How this Guide is Organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Interoperability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
S QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4 Hardware Installation for QLE7240, QLE7280, or QLE7140 with PCI Express Riser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dual Adapter Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Installation Steps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hardware Installation for QHT7140 with HTX Riser . . . . . . . . . . . . . .
A QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4 Uninstalling Software with Rocks or Platform OCS . . . . . . . . . . . . . . . Downgrading RPMs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Configuring Drivers and Services ra ft InfiniPath and OpenFabrics Driver Overview . . . . . . . . . . . . . . . . . . . . . . . . OpenFabrics Drivers and Services Configuration and Startup. . . . . . . . . . . Configuring the IPoIB Network Interface . .
S QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4 ibsrpdm Command Hangs When Two HCAs are Installed but Only Unit 1 is Connected to the Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-5 Outdated ipath_ether Configuration Setup Generates Error . . . . . A-5 B Write Combining Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Verify Write Combining is Working . . . . . . . . . . . . . . . .
A QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4 List of Figures Figure Page 4-1 QLogic QLE7280 with IBA7220 ASIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-7 4-2 QLogic QLE7140 Card with Riser, Top View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-7 4-3 QLogic QHT7040/QHT7140 Full and Low Profile Cards with Riser, Top View . . . . 4-8 4-4 PCIe Slot in a Typical Motherboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
S QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4 D ra ft Notes xiv Preliminary IB0056101-00 G.
1 Introduction This chapter describes the contents, intended audience, and organization of the QLogic HCA and QLogic OFED Software Install Guide. The QLogic HCA and QLogic OFED Software Install Guide contains instructions for installing the QLogic Host Channel Adapters (HCAs) and the QLogic InfiniPath and OpenFabrics software.
S Introduction Overview Section 3, Step-by-Step Installation Checklist, provides a high-level overview of the hardware and software installation procedures. Section 4, Hardware Installation, includes instructions for installing the QLogic QLE7140, QLE7240, QLE7280, QHT7040, and QHT7140 HCAs. Section 5, Software Installation, includes instructions for installing the QLogic InfiniPath and OpenFabrics software.
A Introduction Interoperability QLogic OFED OpenFabrics software is interoperable with other vendors’ InfiniBand Host Channel Adapters (HCAs) running compatible OpenFabrics releases. There are several options for subnet management in your cluster: Use the embedded Subnet Manager (SM) in one or more managed switches supplied by your InfiniBand switch vendor. Use a host-based Subnet Manager. QLogic provides one, HSM, as a part of the InfiniBand fabric Suite download.
S Introduction Documentation Table 1-1. Typographical Conventions (Continued) Convention Meaning concept Italic font is used for emphasis and concepts, as well as for documentation names/titles. user input Bold fixed-space font is used for literal items in commands or constructs that you type. Indicates a command line prompt. # Indicates a command line prompt as root. [] Brackets enclose optional elements of a command or program construct. ...
A Introduction Contact Information Contact Information Support Headquarters QLogic Corporation 4601 Dean Lakes Blvd Shakopee, MN 55379 USA QLogic Web Site www.qlogic.com Technical Support Web Site support.qlogic.com Technical Support Email support@qlogic.com Technical Training Email tech.training@qlogic.com D ra ft Additional contact information is available from the Contact Support area of the Technical Support Web Site. IB0056101-00 G.
S Introduction Contact Information D ra ft Notes 1-6 Preliminary IB0056101-00 G.
2 Feature Overview This section contains the features for this release, the supported QLogic adapter models, supported distributions and kernels, and a list of the software components. Features ft The QLogic OFED 1.4 software release contains the complete OFED 1.4, plus additional QLogic improvements, including an enhanced QLogic HCA driver. The InfiniPath 2.3 components (libraries, QLogic MPI/PSM, and utilities) are also included.
S Feature Overview Features MVAPICH and OpenMPI compiled with newer versions of each of four different compilers (GCC, PGI, Intel and PathScale) are available. The QLogic InfiniBand Fabric Suite (IFS) is available separately for purchase. It includes FastFabric, the QLogic Host Subnet Manager (HSM), and the Fabric Viewer, and the InfiniServ Host Software.The QLogic OFED 1.4 software is supported by IFS. Support for newer compiler versions (PathScale 3.x, PGI 7.x, PGI 8.x, Intel 10.x, Intel 11.
A Feature Overview Features This version of the QLogic OFED software provides support for all of the QLogic HCAs in Table 2-1. Table 2-1. QLogic Adapter Model Numbers QLogic Model Number Description Single port 10Gbps SDR 4X InfiniBand to HTX adapter. For systems with HTX expansion slots. QHT7140 a Single port 10Gbps SDR 4X InfiniBand to HTX adapter. For systems with HTX expansion slots. QLE7140 Single port 10Gbps SDR 4X InfiniBand to PCI Express x8 adapter.
S Feature Overview Software Components Software Components This release includes all of OFED 1.4 with enhancements (QLogic OFED 1.4), including a new version of the VNIC tools and driver, and support for the QHT7xxx and QLE7xxx adapters. The software includes the QLogic InfiniPath HCA driver, libraries, QLogic MPI, Subnet Management Agent, and associated utilities.
3 Step-by-Step Installation Checklist This section provides an overview of the hardware and software installation procedures. Detailed steps are found in Section 4 “Hardware Installation” and Section 5 “Software Installation”. Hardware Installation The following steps summarize the basic hardware installation procedure: Check that the adapter hardware is appropriate for your platform. See Table 4-1. 2.
S Step-by-Step Installation Checklist Software Installation Software Installation The following steps summarize the basic QLogic OFED 1.4 software installation and startup. These steps must be performed on each node in the cluster: Make sure that the HCA hardware installation has been completed according to the instructions in “Hardware Installation” on page 4-1. 2. Verify that the Linux kernel software is installed on each node in the cluster.
A Step-by-Step Installation Checklist Software Installation You can optimize your system and adapter for the best performance. See “Adapter and Other Settings” on page 7-1. Also see the Performance Settings and Management Tips section in the QLogic HCA and QLogic OFED Software Users Guide. 12. Perform the recommended health checks. See “Customer Acceptance Utility” on page 7-2. 13.
S Step-by-Step Installation Checklist Software Installation D ra ft Notes 3-4 Preliminary IB0056101-00 G.
4 Hardware Installation This section lists the requirements and provides instructions for installing the QLogic InfiniPath Interconnect adapters. Instructions are included for the QLogic DDR PCI Express adapters, the QLE7240 and QLE7280; the QLogic PCIe adapter and PCIe riser card, QLE7140; and the QHT7040 or QHT7140 adapter hardware and HTX riser card. These components are collectively referred to as the adapter and the riser card in the remainder of this document.
S Hardware Installation Hardware Installation Requirements Table 4-1. Adapter Models and Related Platforms (Continued) QLogic Model Number Platform QLE7140 PCI Express systems Standard PCI Express x8 or x16 slot QHT7040 Motherboards with HTX connectors HyperTransport HTX slot QHT7140 Motherboards with HTX connectors HyperTransport HTX slot Plugs Into Installation of the QLE7240, QLE7280, QLE7140, QHT7040, or QHT7140 in a 1U or 2U chassis requires the use of a riser card.
A Hardware Installation Hardware Installation Requirements Cabling and Switches The cable installation uses a standard InfiniBand (IB) 4X cable. Any InfiniBand cable that has been qualified by the vendor should work. For SDR, the longest passive copper IB cable that QLogic has currently qualified is 20 meters. For DDR-capable adapters and switches, the DDR-capable passive copper cables cannot be longer than 10 meters. Active cables can eliminate some of the cable length restrictions.
S Hardware Installation Hardware Installation Requirements Optical Fibre Option The QLogic adapter also supports connection to the switch by means of optical fibres through optical media converters such as the EMCORE™ QT2400. Not all switches support these types of convertors. For more information on the EMCORE convertor, see www.emcore.com. Intel® and Zarlink™ also offer optical cable solutions. See www.intel.com and www.zarlink.com for more information.
A Hardware Installation Safety with Electricity Safety with Electricity Observe these guidelines and safety precautions when working around computer hardware and electrical equipment: Locate the power source shutoff for the computer room or lab where you are working. This is where you will turn OFF the power in the event of an emergency or accident. Never assume that power has been disconnected for a circuit; always check first. Do not wear loose clothing.
S Hardware Installation Unpacking Information The package contents for the QLE7280 adapter are: QLogic QLE7280 Additional short bracket Quick Start Guide Standard PCIe risers can be used, typically supplied by your system or motherboard vendor. The package contents for the QLE7140 adapter are: QLogic QLE7140 Quick Start Guide Standard PCIe risers can be used, typically supplied by your system or motherboard vendor. The contents are illustrated in Figure 4-2.
A Hardware Installation Unpacking Information PCI Express Edge Connectors InfiniBand Connector Face Plate ft IBA7220 ASIC ra Figure 4-1. QLogic QLE7280 with IBA7220 ASIC D PCI Express Riser Card. Not supplied; shown for reference. PCI Express Edge Connectors InfiniBand Connector IBA6120 ASIC Face Plate Figure 4-2. QLogic QLE7140 Card with Riser, Top View IB0056101-00 G.
S Hardware Installation Unpacking Information HTX Riser Card HTX Edge Connectors InfiniBand Connector ra PathScale QHT7140 Low Profile Card Face Plate ft IBA6110 ASIC InfiniBand Connector Face Plate PathScale D QHT7040 Full Height Short Card Figure 4-3. QLogic QHT7040/QHT7140 Full and Low Profile Cards with Riser, Top View Unpacking the QLogic Adapter Follow these steps when unpacking the QLogic adapter: 4-8 1.
A Hardware Installation Hardware Installation Hardware Installation This section contains hardware installation instructions for the QLE7240, QLE7280, QLE7140, QHT7040, and QHT7140. Hardware Installation for QLE7240, QLE7280, or QLE7140 with PCI Express Riser Installation for the QLE7240, QLE7280, and QLE7140 is similar. The following instructions are for the QLE7140, but can be used for any of these three adapters. Most installations will be in 1U and 2U chassis, using a PCIe right angle riser card.
S Hardware Installation Hardware Installation Remove the cover screws and cover plate to expose the system’s motherboard. For specific instructions on how to do this, follow the hardware documentation that came with your system. 5. Locate the PCIe slot on your motherboard. Note that the PCIe slot has two separate sections, with the smaller slot opening located towards the front (see Figure 4-4). These two sections correspond to the shorter and longer connector edges of the adapter and riser. ft 4.
A Hardware Installation Hardware Installation 9. Connect the QLogic adapter and PCIe riser card together, forming the assembly that you will insert into your motherboard. First, visually line up the adapter slot connector edge with the edge connector of the PCIe riser card (see Figure 4-5). . PCIe Riser Card QLogic Adapter ft Face Plate LEDs ra InfiniBand connector Figure 4-5.
S Hardware Installation Hardware Installation Insert the riser assembly into the motherboard’s PCIe slot, ensuring good contact. The QLogic adapter should now be parallel to the motherboard and about one inch above it (see Figure 4-6). ft 13. Figure 4-6. Assembled PCIe HCA with Riser Secure the face plate to the chassis. The QLogic adapter has a screw hole on the side of the face plate that can be attached to the chassis with a retention screw.
A Hardware Installation Hardware Installation To install the QLogic adapter with an HTX riser card: The BIOS should be already be configured properly by the motherboard manufacturer. However, if any additional BIOS configuration is required, it will usually need to be done before installing the QLogic adapter. See “Configuring the BIOS” on page 4-4. 2. Shut down the power supply to the system into which you will install the QLogic adapter. 3.
S Hardware Installation Hardware Installation 7. Remove the QLogic QHT7140 from the anti-static bag. NOTE: Be careful not to touch any of the components on the printed circuit board during these steps. You can hold the adapter by its face plate or edges. Locate the face plate on the connector edge of the card. 9. Connect the QLogic adapter and HTX riser card together, forming the assembly that you will insert into your motherboard.
A Hardware Installation Hardware Installation Insert the HT riser assembly into the motherboard’s HTX slot, ensuring good contact. The QLogic adapter should now be parallel to the motherboard and about one inch above it, as shown in Figure 4-9. ft 13. Figure 4-9. Assembled QHT7140 with Riser Secure the face plate to the chassis. The QLogic adapter has a screw hole on the side of the face plate that can be attached to the chassis with a retention screw.
S Hardware Installation Hardware Installation Hardware Installation for the QHT7140 Without an HTX Riser Installing the QLogic QHT7140 without an HTX riser card requires a 3U or larger chassis. The card slot connectors on the QHT7140 fit into the HTX slot in a vertical installation. To install the QLogic adapter without the HTX riser card: The BIOS should already be configured properly by the motherboard manufacturer.
A Hardware Installation Switch Configuration and Monitoring Insert the card by pressing firmly and evenly on the top of the horizontal bracket and the top rear corner of the card simultaneously. The card should insert evenly into the slot. Be careful not to push, grab, or put pressure on any other part of the card, and avoid touching any of the components. See Figure 4-10. ft 8. Secure the face plate to the chassis.
S Hardware Installation Completing the Installation The QLE7240, QLE7280, QLE7140, QHT7040, and QHT7140 adapters are all cabled the same way. To install the InfiniBand cables: 1. Check that you have removed the protector plugs from the cable connector ends. 2. Different vendor cables might have different latch mechanisms. Determine if your cable has a spring-loaded latch mechanism. If your cable is spring-loaded, grasp the metal shell and pull on the plastic latch to release the cable.
5 Software Installation This section provides instructions for installing QLogic OFED 1.4, which includes QLogic InfiniPath and the OpenFabrics software. The software includes drivers, protocol libraries, QLogic’s implementation of the MPI message passing standard, associated utilities, and example programs, including benchmarks. A complete list of the provided software is in “Software Components” on page 2-4.
S Software Installation Cluster Setup Table 5-1. InfiniPath/OpenFabrics Supported Distributions and Kernels InfiniPath/OpenFabrics Supported Kernels Distribution 2.6.9-55 (U5), RHEL 4.6 2.6.9-67 (U6) RHEL 4.7 2.6.9-78 (U7) CentOS 4.5 2.6.9.55 CentOS 4.6 2.6.9-67 CentOS 4.7 2.6.9-78 Scientific Linux 4.5 2.6.9.55 Scientific Linux 4.6 2.6.9-67 Scientific Linux 4.7 2.6.9-78 Red Hat Enterprise Linux 5.1 (RHEL 5.1) 2.6.18-53, 2.6.18-92 RHEL 5.2 2.6.
A Software Installation Compiler Support Table 5-2. Distribution Identifiers (Continued) Distribution Identifier Used On rhel5 Red Hat Enterprise Linux 5.1 (RHEL5.1), RHEL5.2, CentOS 5.1-5.2, Scientific Linux 5.1-5.2, for x86_64 systems sles10 SLES 10 SP1-SP2 for x86_64 systems Compiler Support QLogic MPI supports use of a number of compilers. These include: GNU gcc 3.3.x, 3.4.x, 4.0, 4.1, 4.2.x, and 4.3.x compiler suites PathScale Compiler Suite 3.0, 3.1 and 3.2 PGI 5.2, 6.0. 7.1, 7.2-4, and 8.
S Software Installation Choose the Appropriate Download Files Some operating system packages are required for OpenFabrics; they are listed in Table 5-3. Table 5-3.
A Software Installation Choose the Appropriate Download Files The components of the QLogic OFED 1.4 release are available packaged as separate downloads: QLogicIB-Basic with the Text User Interface Installer QLogic OFED 1.4 RPM Set QLogic OFED 1.4 User-level RPM Set Rocks Rolls Platform OCS Kits The InfiniBand Fabric Suite is available for purchase as a CD/ISO image Check Table 5-5 for the package contents available for each type of download.
S Software Installation Choose the Appropriate Download Files Table 5-5. Available Packages for QLogic OFED 1.4 Release (Continued) Package QLogic OFED 1.4 RPM Set Description Includes: QLogic OFED 1.
A Software Installation Install QLogicIB-Basic with the Installer Tool Table 5-5. Available Packages for QLogic OFED 1.4 Release (Continued) Package QLogic OFED 1.4 Platform OCS Kits Installation and Documentation Description Includes: QLogic OFED 1.
S Software Installation Install QLogicIB-Basic with the Installer Tool 2. After unpacking the .tgz file, change directory to: cd . 3. Become root, then type: # ./INSTALL (Note that if you need 32-bit support on 64 bit OSs, invoke the installer with ./INSTALL --32bit) You will see a screen similar to this: QLogic Inc. InfiniBand 4.4.1.0.
A Software Installation Install QLogicIB-Basic with the Installer Tool 4. Type 1, which will display the screen for software installation. The next screen shows the packages to select for installation: QLogic Inc. IB Install (4.4.1.0.8 release) Menu Please Select Install Action (screen 1 of 3): 0) OFED IB Stack [ Install ][Available] 1.4.0.1.5 1) TrueScale HCA Libs [ Install ][Available] 2.3.0.0.4237 2) QLogic IB Tools [ Install ][Available] 4.4.0.0.
S Software Installation Install QLogicIB-Basic with the Installer Tool The next screen contains these choices: QLogic Inc. IB Install (4.4.1.0.8 release) Menu Please Select Install Action (screen 2 of 3): [ Install ][Available] 1.4.0.1.5 1) MVAPICH/PSM (PGI) [ Install ][Available] 1.4.0.1.5 2) MVAPICH/PSM (PSc) [ Install ][Available] 1.4.0.1.5 3) MVAPICH/PSM (Intel) [ Install ][Available] 1.4.0.1.5 4) OpenMPI/PSM (gcc) [ Install ][Available] 1.4.0.1.
A Software Installation Install QLogicIB-Basic with the Installer Tool Open SM (1) should only be installed on one node in the cluster where it will be used. If desired, Type 1 for Open SM. 7. Finally, type p to start the installation. The installer will uninstall older OFED RPMs, and then will ask for input for a series of operations. You can accept the defaults, by pressing .
S Software Installation Install QLogicIB-Basic with the Installer Tool .
A Software Installation About rpm Installation Table 5-6.
S Software Installation Using rpm to Install InfiniPath and OpenFabrics RPMs contain config files. Your current config files will not be overwritten when new RPMs are installed. New config files will contain the suffix .rpmnew and can be found in /etc/sysconfig and /etc/infiniband. Check the new files to see if there is anything you want to add to your standard config files Please note: For convenience, QLogic recommends that the same set of RPMs are installed on all nodes (with the exception of OpenSM).
A Software Installation Using rpm to Install InfiniPath and OpenFabrics Become root, then: # cd QLogicOFED-
S Software Installation Using rpm to Install InfiniPath and OpenFabrics Note that you need to install the InfiniPath-MPI/32bit/mpi-frontend-*.rpm even if you do not plan to use any other 32-bit RPMs. However, you can add the other 32bit subdirectories to the rpm command, if you need 32-bit support. Next, install all non-prefixed RPMs: # rpm -Uvh InfiniPath/*.rpm \ InfiniPath-Devel/infinipath-devel*.rpm \ OpenFabrics/*.rpm OpenFabrics-Devel/*.rpm \ OtherHCAs/*.rpm OtherHCAs-Devel/*.
A Software Installation Install QLogic OFED User-level Software with the rpm Command OpenFabrics/32bit/ OpenFabrics-Static/ OpenFabrics-Static/32bit/ OpenFabrics-Devel/ OpenFabrics-Devel/32bit/ OpenFabrics-Devel-Static/ OpenFabrics-Debuginfo/ OpenFabrics-Debuginfo/32bit/ OpenSM/ OpenSM/32bit/ OpenSM-Debuginfo/ OpenSM-Debuginfo/32bit/ OpenSM-Devel/ OpenSM-Devel/32bit/ ft OpenSM-Static/ OpenSM-Static/32bit/ OtherHCAs/ OtherHCAs-Debuginfo ra OtherHCAs-Debuginfo/32bit/ OtherHCAs-Devel/ OtherHCAs-Devel/3
S Software Installation Install QLogic OFED User-level Software with the rpm Command The tar command creates a directory based on the tar file name and places the RPMs and other files in this directory. 2. The RPMs need to be available on each node on which they will be used. You can copy the RPMs to a directory on each node that will need them. Become root, then cd to: # cd InfiniPath---x86_64 (Another way is to put the RPMs in a directory that is accessible (e.g.
A Software Installation Rebuilding or Reinstalling the kernel-ib Driver with rpm After a Kernel Upgrade The complete RPM directories for this download are organized as follows: InfiniPath_license.txt,LEGAL.txt (top level) Documentation/ InfiniPath/ InfiniPath-Devel/ InfiniPath-MPI/ OtherMPIs/ Rebuilding or Reinstalling the kernel-ib Driver with rpm After a Kernel Upgrade If you upgrade the kernel, then you must reboot and then rebuild or reinstall the InfiniPath kernel modules (drivers).
S Software Installation Install QLogic OFED Using Rocks NOTE: Using the override may not result in a buildable or working driver if your distribution/kernel combination is not similar enough to a tested and supported distribution/kernel pair. Install QLogic OFED Using Rocks Rocks is a distribution designed for managing clusters from the San Diego Supercomputer center (SDSC). Rocks is a way to manage the kickstart automated installation method created by Red Hat.
A Software Installation Install QLogic OFED Using Rocks Boot, Core, OS Roll DVD Note that you may also need updates; look for the latest files with the service-pack prefix. Make sure you downloaded the .iso images correctly; verify by checking the md5 checksum from the web site. b. Build the frontend node with the above .iso images (step 2) from the Rocks web site: a. Insert the Kernel/Boot Roll CD into your frontend machine. After the frontend boots off the CD, follow the instructions on the screen.
S Software Installation Install QLogic OFED Using a Platform OCS Kit If you have a burned a CD version of InfiniPath roll from the .iso image: # mount /mnt/cdrom # rocks-dist --install copyroll # umount /mnt/cdrom # cd /home/install # rocks-dist dist You can use this method if you download the .
A Software Installation Install Additional Software See also the current version of the FastFabric Users Guide, available with the software or from the QLogic download page. Install Additional Software This section contains information about additional third-party software installation. Installing Lustre This InfiniPath release supports Lustre cluster filesystem Version 1.6.5.1. Lustre is a fast, scalable Linux cluster file system that interoperates with InfiniBand.
S Software Installation Removing Software Packages Documentation is found in: /usr/share/man /usr/share/doc/infinipath /usr/share/doc/mpich-infinipath Note that license information only is found in usr/share/doc/infinipath. InfiniPath user documentation can be found on the QLogic web site on the software download page for your distribution. Configuration files are found in: /etc/sysconfig Init scripts are found in: /etc/init.
A Software Installation Removing Software Packages For both InfiniPath and OpenFabrics, QLogic recommends that you remove all the packages at the same time. 1. Use the script ofed_uninstall.sh to uninstall the OFED software. See the OFED Installation release notes that are part of the OFED documentation. 2.
S Software Installation Removing Software Packages D ra ft Notes 5-26 Preliminary IB0056101-00 G.
6 Configuring Drivers and Services This section provides instructions for configuring and using the drivers and services available with QLogic OFED 1.4. InfiniPath and OpenFabrics Driver Overview ft The InfiniPath ib_ipath module provides low level QLogic hardware support, and is the base driver for both MPI/PSM programs, and general OpenFabrics protocols such as IPoIB and SDP. The driver also supplies the Subnet Management Agent (SMA) component. IPoIB network interface.
S Configuring Drivers and Services OpenFabrics Drivers and Services Configuration and Startup Configuring the IPoIB Network Interface The following instructions show you how to manually configure your OpenFabrics IPoIB network interface. This example assumes that you are using sh or bash as your shell, all required InfiniPath and OpenFabrics RPMs are installed, and your startup scripts have been run (either manually or at system boot). For this example, the IPoIB network is 10.1.17.
A Configuring Drivers and Services OpenFabrics Drivers and Services Configuration and Startup The output of the ping command will be similar to the following, with a line for each host already configured and connected: WARNING: pinging broadcast address PING 10.1.17.255 (10.1.17.255) 517(84) bytes of data. 174 bytes from 10.1.17.3: icmp_seq=0 ttl=174 time=0.022 ms 64 bytes from 10.1.17.1: icmp_seq=0 ttl=64 time=0.070 ms (DUP!) 64 bytes from 10.1.17.7: icmp_seq=0 ttl=64 time=0.
S Configuring Drivers and Services OpenFabrics Drivers and Services Configuration and Startup You can stop opensmd again like this: # /etc/init.d/opensmd stop If you want to pass any arguments to the OpenSM program, modify the following file, and add the arguments to the OPTIONS variable: /etc/init.d/opensmd For example: # Use the UPDN algorithm instead of the Min Hop algorithm. OPTIONS="-R updn" For more information on OpenSM, see the OpenSM man pages, or look on the OpenFabrics web site.
A Configuring Drivers and Services OpenFabrics Drivers and Services Configuration and Startup In the output, look for lines similar to these: GUID: 0002c90200402c04 ID: LSI Storage Systems SRP Driver 200400a0b8114527 service entries: 1 service[ 0]: 200400a0b8114527 / SRP.T10:200400A0B8114527 GUID: 0002c90200402c0c ID: LSI Storage Systems SRP Driver 200500a0b8114527 service entries: 1 service[ 0]: 200500a0b8114527 / SRP.
S Configuring Drivers and Services OpenFabrics Drivers and Services Configuration and Startup 5. You can look for the newly created devices in the /proc/partitions file. The file will look similar to this example (the partition names may vary): # cat /proc/partitions major minor 6. #blocks name 8 64 142325760 sde 8 65 142319834 sde1 8 80 71162880 sdf 8 81 71159917 sdf1 8 96 20480 sdg 8 97 20479 sdg1 Create a mount point (as root) where you will mount the SRP device.
A Configuring Drivers and Services OpenFabrics Drivers and Services Configuration and Startup their IOCGUID, IOCSTRING, etc. Use the ib_qlgc_vnic_query tool to get this information. When ib_qlgc_vnic_query is executed without any options, it displays detailed information about all the EVIC/VEx IOCs present on the fabric. Run it as root.
S Configuring Drivers and Services OpenFabrics Drivers and Services Configuration and Startup HCA No = 0, HCA = mlx4_0, Port = 2, Port GUID = 0x0002c903000010fa, State = Active IO Unit Info: port LID: 0009 port GID: fe8000000000000000066a11de000070 change ID: 0003 max controllers: 0x02 ft controller[ 1] GUID: 00066a01de000070 vendor ID: 00066a device ID: 000030 IO class : 2000 ID: EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 1 service entries: 2 service[ 0]: 1000066a00000001 / InfiniNIC.InfiniConSys.
A Configuring Drivers and Services OpenFabrics Drivers and Services Configuration and Startup 2. Create the VNIC interfaces using the configuration file: /etc/infiniband/qlgc_vnic.cfg. Look at the qlgc_vnic.cfg.sample file to see how VNIC configuration files are written. It can be found with the OFED documentation, or in the qlgc_vnictools subdirectory of the QLogicIB_Basic download.
S Configuring Drivers and Services OpenFabrics Drivers and Services Configuration and Startup NOTE: Do not create EIOC names with similar character strings (e.g., eioc3 and eioc30). There is a limitation with certain Linux operating systems that cannot recognize the subtle differences. The result is that the user will be unable to ping across the network. b. Format 2: Defining an IOC using the IOCSTRING.
A Configuring Drivers and Services OpenFabrics Drivers and Services Configuration and Startup c. Format 3: Starting VNIC using DGID. Following is an example of a DGID and IOCGUID VNIC configuration.
S Configuring Drivers and Services OpenFabrics Drivers and Services Configuration and Startup 5. Start the QLogic VNIC driver and the QLogic VNIC interfaces. Once you have created a configuration file, you can start the VNIC driver and create the VNIC interfaces specified in the configuration file by running the following command (as root): # /etc/init.d/qlgc_vnic start You can stop the VNIC driver and bring down the VNIC interfaces by running the following command: # /etc/init.
A Configuring Drivers and Services OpenFabrics Drivers and Services Configuration and Startup If you want to restart the QLogic VNIC interfaces, run the following command: # /etc/init.d/qlgc_vnic restart You can get information about the QLogic VNIC interfaces by using the following script (as root): # ib_qlgc_vnic_info This information is collected from the /sys/class/infiniband_qlgc_vnic/interfaces/ directory, under which there is a separate directory corresponding to each VNIC interface.
S Configuring Drivers and Services Other Configuration: Changing the MTU Size 2. Check that you have a /etc/dat.conf file. It should be installed by the dapl- RPM. The file dat.conf contains a list of interface adapters supported by uDAPL service providers. In particular, it must contain mapping entries for OpenIB-cma for dapl 1.2.x, in a form similar to this (all on one line): OpenIB-cma u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "ib0 0" "" 3.
A Configuring Drivers and Services Managing the InfiniPath Driver ra ft NOTE: To use 4K MTU, set the switch to have the same 4K default. If you are using QLogic switches the following will apply: For the Externally Managed 9024, use 4.2.2.0.3 firmware (9024DDR4KMTU_firmware.emfw) for the 9024 EM. This has the 4K MTU default, for use on fabrics where 4K MTU is required. If 4K MTU support is not required, then the 4.2.2.0.2 DDR *.emfw file should be used for DDR externally-managed switches.
S Configuring Drivers and Services Managing the InfiniPath Driver The device files are: /dev/ipath /dev/ipath0, /dev/ipath1, ... The numbered device files allow access to a specific InfiniPath unit. See the ib_ipath man page for more details. Configure InfiniPath Driver State Use the following commands to check or configure the state. These methods will not reboot the system. To check the configuration state, use this command.
A Configuring Drivers and Services Further Information on Configuring and Loading Drivers The sequence of commands to restart the driver are as follows. # /etc/init.d/opensmd stop # /etc/init.d/openibd stop ... # /etc/init.d/openibd start # /etc/init.d/opensmd start The ... represents whatever activity you are engaged in after infinipath is stopped. An equivalent way to restart the driver this is to use same sequence as above, except use the restart command instead of start and stop: # /etc/init.
S Configuring Drivers and Services Further Information on Configuring and Loading Drivers D ra ft Notes 6-18 Preliminary IB0056101-00 G.
7 Installation Verification and Additional Settings This section provides instructions for verifying that the software has been properly installed and that the drivers are loaded, and the fabric is active and ready to use. Information on adapter performance tuning is also provided. LED Link and Data Indicators ft The LEDs function as link and data indicators once the InfiniPath software has been installed, the driver has been loaded, and the fabric is being actively managed by a subnet manager.
S Installation Verification and Additional Settings Customer Acceptance Utility Make sure that write combining is enabled. The x86 Page Attribute Table (PAT) mechanism that allocates write-combining (WC) mappings for the PIO buffers has been added and is now the default. If PAT is unavailable or PAT initialization fails for some reason, the code will generate a message in the log and fall back to the MTRR mechanism. See B Write Combining for more information. Check the PCIe bus width.
A Installation Verification and Additional Settings Customer Acceptance Utility 3. Gathers and analyzes system configuration from the nodes. 4. Gathers and analyzes RPMs installed on the nodes. Missing RPMs can be found this way. 5. Verifies QLogic hardware and software status and configuration. Includes tests for link speed, PIO bandwidth (incorrect MTRR settings), and MTU size. 6. Verifies the ability to mpirun jobs on the nodes. 7.
S Installation Verification and Additional Settings Customer Acceptance Utility In most cases of failure, the script suggests recommended actions. Please see the ipath_checkout man page for more information and updates. D ra ft Also refer to the Troubleshooting appendix in the QLogic HCA and QLogic OFED Software Users Guide. 7-4 Preliminary IB0056101-00 G.
A Installation Troubleshooting The following sections contain information about issues that may occur during installation. Some of this material is repeated in the Troubleshooting appendix of the QLogic HCA and QLogic OFED Software User Guide. Hardware Issues ft Many programs and files are available that gather information about the cluster, and can be helpful for debugging. See appendix D, Useful Programs and Files, in the QLogic HCA and QLogic OFED Software User Guide.
S Installation Troubleshooting BIOS Settings BIOS Settings This section covers issues related to BIOS settings. You can check and adjust BIOS settings using the BIOS Setup utility. For specific instructions, follow the hardware documentation that came with your system. Enable Advanced Configuration and Power Interface (ACPI) This setting must be enabled.
A Installation Troubleshooting Software Installation Issues If these RPMs have not been installed, you will see error messages like this when installing InfiniPath: Building and installing InfiniPath modules for 2.6.16_sles10 2.6.16.21-0.8-debug kernel *** ERROR: /lib/modules/2.6.16.21-0.8-debug/build/.config is missing. *** Is the kernel-source rpm for 2.6.16.21-0.8-debug installed? ================ ================ Building and installing InfiniPath modules for 2.6.9_U4 2.6.9-42.
S Installation Troubleshooting Software Installation Issues Resolving Conflicts Occasionally, conflicts may arise when trying to install "on top of" an existing set of files that may come from a different set of RPMs. For example, if you install the QLogic MPI RPMs after having previously installed Local Area Multicomputer (LAM)/MPI, there will be conflicts, since both installations have versions of some of the same programs and documentation.
A Installation Troubleshooting Configuration Issues In older distributions, such as RHEL4, the 32-bit glibc is contained in the libgcc RPM. The RPM name will be similar to this: libgcc-.EL4.i386.rpm In newer distributions, glibc is an RPM name. The 32-bit glibc is named similarly to: glibc-.i686.rpm OR glibc-.i386.rpm Check your distribution for the exact RPM name.
S Installation Troubleshooting Configuration Issues When ifconfig or ifup are run, the error will look similar to this (assuming ipath_ether was used for eth2): eth2: error fetching interface information: Device not found To prevent the error message, remove the following files (assuming ipath_ether was used for eth2): /etc/sysconfig/network-scripts/ifcfg-eth2 (for RHEL) /etc/sysconfig/network/ifcfg-eth-eth2 (for SLES) D ra ft QLogic recommends using the IP over InfiniBand protocol (IPoIB-CM), includ
B Write Combining Introduction Write combining improves write bandwidth to the QLogic chip by writing multiple words in a single bus transaction (typically 64 bytes). Write combining applies only to x86_64 systems. The x86 Page Attribute Table (PAT) mechanism that allocates write-combining (WC) mappings for the PIO buffers has been added and is now the default.
S Write Combining PAT and Write Combining . . If you do not see any of these messages on your console, but suspect this problem, check the /var/log/messages file. Some systems suppress driver load messages but still output them to the log file. Methods for enabling and disabling the two write combining mechanisms are given below. There are no conflicts between the two methods. PAT and Write Combining This is the default mechanism for allocating write-combining (WC) mappings for the PIO buffers.
A Write Combining MTRR Mapping and Write Combining For systems with very large amounts of memory (32GB or more), it may also be necessary to adjust the BIOS setting for the PCI hole granularity to 2GB. This setting allows the memory to be mapped with fewer MTRRs, so that there will be one or more unused MTRRs for the InfiniPath driver. Some BIOS’ do not have the MTRR mapping option. It may have a different name, depending on the chipset, vendor, BIOS, or other factors.
S Write Combining MTRR Mapping and Write Combining D ra ft Notes B-4 Preliminary IB0056101-00 G.
C Configuration Files Table C-1 contains descriptions of the configuration and configuration template files used by the InfiniPath and OpenFabrics software. Table C-1. Configuration Files Configuration File Name VirtualNIC configuration file. Create this file after running ib_qlgc_vnic_query to get the information you need. This file was named /etc/infiniband/qlogic_vnic.cfg or /etc/sysconfig/ics_inic.cfg in previous releases. See the sample file qlgc_vnic.cfg.
S Configuration Files Table C-1. Configuration Files (Continued) Configuration File Name /etc/sysconfig/network/ifcfg Description Network configuration file for network interfaces When used for VNIC configuration, is in the form eiocX, where X is the device number. There will be one interface configuration file for each interface defined in /etc/infiniband/qlgc_vnic.cfg. For SLES systems.
D Package Descriptions The following sections contain detailed descriptions of the packages for the InfiniPath and OpenFabrics software. In this release, software may be installed in several ways. With the QLogicIB-Basic download, which uses the Installer tool, the package groupings are a bit different than those in the RPM-based downloads. Package Names with the QLogicIB-Basic Download ft The QLogic IB-Basic download has been packaged differently than the RPM download.
S Package Descriptions InfiniPath RPM Version Numbers and Identifiers Although QLogic recommends installing all RPMs on all nodes, not all InfiniPath software is required on all nodes. See Table D-2, Table D-3, or Table D-4 for information on installation of software RPMs on specific types of nodes. InfiniPath RPM Version Numbers and Identifiers The InfiniPath RPMs that are shipped have the InfiniPath release number, build identifiers, and distribution contained in the RPM name.
A Package Descriptions Documentation RPMs Documentation RPMs Table D-1. Documentation/RPMs RPM Name infinipath-doc-xxx_yyy.noarch.rpm Front End Compute Development Optional Optional Optional Optional Optional Optional Optional Optional Optional InfiniPath man pages mpi-doc-xxx_yyy.noarch.rpm Man pages for MPI functions and other MPI documents ofed-docs-xxx.x86_64.rpm InfiniPath RPMs ft OpenFabrics documentation Table D-2.
S Package Descriptions OpenFabrics RPMs Table D-3. InfiniPath-Devel/RPMs RPM Name infinipath-devel-xxx_yyy.noarch.rpm Front End Compute Development Optional Optional Optional Optional Optional Required Development files for InfiniPath mpi-devel-xxx_yyy.noarch.rpm Source code for the MPI development environment, including headers and libs, MPI examples and benchmarks. Use to build the examples or rebuild the benchmarks. ft Table D-4. InfiniPath-MPI/RPMs RPM Name mpi-benchmark-xxx_yyy.x86_64.
A Package Descriptions OpenFabrics RPMs Table D-5. OpenFabrics/RPMs (Continued) RPM Name Optional for OpenFabrics dapl-xxx.x86_64.rpm uDAPL 2.0.15 support dapl-utils-xxx.x86_64.rpm uDAPL support Optional for OpenFabrics Optional for OpenFabrics ibsim-xxx.x86_64.rpm InfiniBand Fabric Simulator ibutils-xxx.x86_64.rpm ibutils provides InfiniBand (IB) network and path diagnostics. ibvexdmtools-xxx.x86_64.
S Package Descriptions OpenFabrics RPMs Table D-5. OpenFabrics/RPMs (Continued) RPM Name Comments Required for OpenFabrics libibverbs-xxx.x86_64.rpm Library that allows userspace processes to use InfiniBand verbs as described in the InfiniBand Architecture Specification. This library includes direct hardware access for fast path operations. For this library to be useful, a device-specific plug-in module must also be installed. libibverbs-utils-xxx.x86_64.
A Package Descriptions OpenFabrics RPMs Table D-5. OpenFabrics/RPMs (Continued) RPM Name Comments qlgc_vnic_daemon-xxx.x86_64.rpm Optional for OpenFabrics Used with VNIC ULP service qlvnictools-xxx.x86_64.rpm Startup script, sample config file, and utilities Optional for OpenFabrics Optional for OpenFabrics qperf-xxx.x86_64.rpm IB performance tests rds-tools-xxx.x86_64.rpm Supports RDS scsi-target-utils-xxx.x86_64.rpm ft Contains the daemon and tools to setup SCSI (SRP) targets sdpnetstat-xxx.
S Package Descriptions OpenFabrics RPMs Table D-6. OpenFabrics-Devel/RPMs (Continued) RPM Name Comments Optional for OpenFabric libibcm-devel-xxx.x86_64.rpm Development files for the libibcm library Optional for OpenFabrics libibcommon-devel-xxx.x86_64.rpm Development files for the libibcommon library Optional for OpenFabrics libibmad-devel-xxx.x86_64.rpm Development files for the libibmad library Optional for OpenFabrics libibumad-devel-xxx.x86_64.
A Package Descriptions Other HCAs Table D-8. OpenSM-Devel/RPM RPM Name opensm-devel-xxx.x86_64.rpm Development files for OpenSM Comments Optional for OpenSM Other HCAs The packages in Table D-9 should be installed only if other HCAs are used. Table D-9. Other HCAs/RPMs RPM Name libcxgb3-xxx.x86_64.rpm ft Support for the Chelsio 10GbE HCA libmlx4-xxx.x86_64.rpm Userspace driver for Mellanox ConnectX™ InfiniBand HCAs ® ra libmthca-xxx.x86_64.
S Package Descriptions Other MPIs Table D-10. Other HCAs-Devel/RPMs (Continued) RPM Name Comments Optional for OpenFabrics libmlx4-devel-xxx.x86_64.rpm Development files for Mellanox ConnectX™ InfiniBand HCAs ® Other MPIs The packages in Table D-11 should be installed if other MPI implementations are desired.RPMs that include yyy in the name are supplied by QLogic; others are from OFED. Table D-11. OtherMPIs/RPMs Comments ft RPM Name mpi-selector-xxx.x86_64.
A Package Descriptions Other MPIs Table D-11. OtherMPIs/RPMs (Continued) RPM Name Comments Optional openmpi_gcc-xxx.x86_64.rpm Open MPI compiled with GNU Optional openmpi_gcc_qlc-xxx.yyy.x86_64.rpm Open MPI compiled with GNU for PSM support openmpi_intel_qlc-xxx.yyy.x86_64.rpm a Optional Open MPI compiled with Intel for PSM support openmpi_pathscale_qlc-xxx.yyy.x86_64.rpm Optional Open MPI compiled with PathScale for PSM support openmpi_pgi_qlc-xxx.yyy.x86_64.
S Package Descriptions Other MPIs D ra ft Notes D-12 Preliminary IB0056101-00 G.
Index A Driver starting, stopping, and restarting 6-16 unloading manually 6-17 Drivers configuration and loading information 6-17 InfiniPath and OpenFabrics overview 6-1 IPoIB configuration 6-1, 6-2 rebuilding on an unsupported distribution/kernel pair 5-19 rebuilding or reinstalling after a kernel upgrade 5-19 ACPI 4-4, A-2 Adapter, see HCA C E ra BIOS configuring 4-4 settings A-2 settings to fix MTRR issues B-2 ft B D -c 7-3 Cables supported 4-3 Compiler support 5-3 Configuration files C-1 ib_ip
S QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4 ra I D ib_ipath module 6-1 0000:04:01.
A QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4 P Lockable memory error A-5 Lustre, installing 5-23 M N O D OFED software, uninstalling 5-24, 5-25 OFED SRP 6-4 OpenFabrics devel/RPMs D-7 configuration 6-1 downgrading RPMs 5-25 environment 5-3 interoperability with InfiniPath 1-3 RPMs D-4 RPMs, uninstalling 5-24 using rpm to install 5-14 OpenSM 6-3 OpenSM/RPMs D-8, D-9 openssh 5-3 openssh-server 5-3 Optical media converters 4-4 IB0056101-00 G.
S QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.
D ra ft
ft ra D Corporate Headquarters QLogic Corporation 26650 Aliso Viejo Parkway Aliso Viejo, CA 92656 949.389.6000 Europe Headquarters QLogic (UK) LTD. Quatro House Lyon Way, Frimley Camberley Surrey, GU16 7ER UK www.qlogic.com +44 (0) 1276 804 670 © 2005-2009 QLogic Corporation. Specifications are subject to change without notice. All rights reserved worldwide. QLogic and the QLogic logo are registered trademarks of QLogic Corporation.
