HP XC System Software Installation Guide Version 4.
© Copyright 2003, 2009 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license. The information contained herein is subject to change without notice.
Table of Contents About This Document.......................................................................................................15 Intended Audience................................................................................................................................15 How to Use This Document..................................................................................................................15 Naming Conventions Used in This Document........................................
2.4 Installing Additional RPMs from the HP XC DVD.........................................................................47 2.4.1 Enable HP XC to Run Serviceguard (Optional)......................................................................47 2.4.2 Install SVA (Optional).............................................................................................................48 2.4.3 Install Additional Linux RPMs (Optional)..............................................................................
.7.1.3.2 Step 2: Install and Configure SFS G3.0-0 Clients on the Head Node......................82 5.7.1.3.3 Step 3: Verify that File Systems Can Be Mounted...................................................83 5.7.1.3.4 Step 4: Create the /etc/sfstab.proto File........................................................84 5.7.1.3.5 Step 5: Create an /etc/sfstab File on the Head Node.......................................85 5.7.1.3.6 Step 6: Mount the hptc_cluster File System....................................
8.7 Creating a Baseline Report of the System Configuration..............................................................135 8.8 You Are Done................................................................................................................................135 9 Upgrading to HP XC System Software Version 4.0...............................................137 9.1 Software Upgrade Overview.........................................................................................................137 9.
12.4.1 Download the Maui Scheduler Kit......................................................................................167 12.4.2 Compile the Maui Scheduler from Its Source Distribution.................................................167 12.4.3 Update the Maui Scheduler Configuration File..................................................................167 12.4.4 Edit the SLURM Configuration File....................................................................................168 12.4.
D Configuring Interconnect Switch Monitoring Cards...............................................197 D.1 Configuring Quadrics Switch Controller Cards...........................................................................197 D.2 Configuring Myrinet Switch Monitoring Line Cards..................................................................200 D.3 Configuring InfiniBand Switch Controller Cards........................................................................201 D.3.
G.4 Configuring an Ethernet Connection...........................................................................................224 G.5 Modifying Node Role Assignments.............................................................................................225 G.6 Analyzing and Comparing Current Role Assignments to HP Recommendations......................226 G.7 Customizing Service and Client Configurations..........................................................................227 G.7.
List of Figures 14-1 10 Discovery Flowchart....................................................................................................................
List of Tables 1 2 1-1 1-2 1-3 2-1 2-2 2-3 2-4 2-5 2-6 2-7 2-8 3-1 4-1 4-2 5-1 5-2 5-3 5-4 6-1 6-2 6-3 6-4 9-1 9-2 9-3 9-4 9-5 9-6 9-7 9-8 9-9 12-1 12-2 14-1 14-2 A-1 D-1 D-2 D-3 D-4 D-5 F-1 G-1 G-2 G-3 G-4 Installation Types..........................................................................................................................15 Naming Conventions....................................................................................................................
H-1 I-1 12 Network Type Based on System Topology..................................................................................233 Default Installation Values for LSF and SLURM.........................................................................
List of Examples 1-1 3-1 3-2 4-1 4-2 5-1 5-2 5-3 G-1 G-2 Sample XC.lic File......................................................................................................................25 Default Content of the base_addr.ini File...............................................................................56 Default Content of the base_addrV2.ini File..........................................................................56 discover Command Output on a Small Non-Blade Configuration..............
About This Document This document describes how to install and configure HP XC System Software Version 4.0 on HP Cluster Platforms CP3000, CP3000BL, CP4000, CP4000BL, and CP6000. An HP XC system is integrated with several open source software components. Some open source software components are being used for underlying technology, and their deployment is transparent.
To avoid duplication of information, from time to time you are instructed to refer to information in the other documents in the HP XC System Software documentation set. To reduce the size of screen displays and command output in this document, a three- or four-node system was used to generate most of the sample command output shown in this document. Naming Conventions Used in This Document This document uses the naming conventions and sample IP addresses listed in Table 2.
Computer output Text displayed by the computer. Ctrl-x A key sequence. A sequence such asCtrl-c indicates that you must hold down the key labeled Ctrl while you press the key for the letter c. ENVIRONMENT VARIABLE The name of an environment variable, for example, PATH. [ERROR NAME] The name of an error, usually returned in the errno variable. Key The name of a keyboard key. Return and Enter both refer to the same key. Term The defined use of an important word or phrase.
HP XC System Software User's Guide Provides an overview of managing the HP XC user environment with modules, managing jobs with LSF, and describes how to build, run, debug, and troubleshoot serial and parallel applications on an HP XC system.
software components are generic, and the HP XC adjective is not added to any reference to a third-party or open source command or product name. For example, the SLURM srun command is simply referred to as the srun command. The location of each website or link to a particular topic listed in this section is subject to change without notice by the site provider. • http://www.platform.com Home page for Platform Computing Inc, the developer of the Load Sharing Facility (LSF).
• http://systemimager.org Home page for SystemImager®, which is the underlying technology that distributes the golden image to all nodes and distributes configuration changes throughout the system. • http://linuxvirtualserver.org Home page for the Linux Virtual Server (LVS), the load balancer running on the Linux operating system that distributes login requests on the HP XC system. • http://www.acresso.
• http://www-unix.mcs.anl.gov/mpi/ A comprehensive site containing general information, such as the specification and FAQs, and pointers to other resources, including tutorials, implementations, and other MPI-related sites. Compiler Websites • http://www.intel.com/software/products/compilers/index.htm Website for Intel® compilers. • http://support.intel.com/support/performancetools/ Website for general Intel software development information. • http://www.pgroup.
$ man discover $ man 8 discover If you are not sure about a command you need to use, enter the man command with the -k option to obtain a list of commands that are related to a keyword. For example: $ man -k keyword HP Encourages Your Comments HP encourages comments concerning this document. We are committed to providing documentation that meets your needs. Send any errors found, suggestions for improvement, or compliments to: docsfeedback@hp.
1 Preparing for a New Installation This chapter describes tasks that are required before you install HP XC System Software Version 4.0, which you must perform in the following order: 1. 2. 3. 4. 5. 6. 7. 8. 9.
1.3 Preparing Existing HP XC Systems This task applies to anyone who is installing HP XC System Software Version 4.0 on an HP XC system that is already installed with an older version of the HP XC System Software. Omit this task if you are installing HP XC System Software Version 4.0 on new hardware for the first time. Before using the procedures described in this document to install and configure HP XC System Software Version 4.
The Master Firmware Tables for this release of the HP XC System Software is available at the following website: http://www.docs.hp.com/en/linuxhpc.html 1.6 Arranging for IP Address Assignments and Host Names Make arrangements with your site's network administrator to assign IP addresses for the following system components. All IP addresses must be defined in the site's Domain Name System (DNS) configuration: • The external IP address of the HP XC system, if it is to be connected to an external network.
VENDOR Compaq USE_SERVER INCREMENT XC Compaq 3.1 28-feb-2008 1 23BBAFCE6662c \ NOTICE="Authorization = BM05WHITMORE19772031 - permanent - HP \ XC System Software - BASE License" INCREMENT XC-PROCESSORS Compaq 3.0 permanent 68 7BA7E0876F0F \ NOTICE="Date 30-Jan-2008 01:29:36 - License Number = \ LAGA4D1958DL - Qty 68 - 434066-B21 - HP XC System Software 1 \ Proc Flex License" INCREMENT lsf_xc Compaq 6.
1.9.1 What Is Improved Availability? A service availability infrastructure is built into the HP XC System Software, which enables an availability tool to fail over a subset of services to nodes that have been designated as a second server of the service. Improved availability protects against service failure if a node that is serving vital services becomes unresponsive or becomes unavailable.
Table 1-1 Improved Availability Summary (continued) Task Appropriate Task Details Are Provided Here 6. If you are using HP Serviceguard as the availability tool, decide whether to use a quorum “Deciding on the server or a lock LUN to achieve quorum. You must create the lock LUN before running Method to Achieve Quorum for HP the cluster_config utility. Serviceguard Clusters” (page 86) 7. Initiate the cluster_config utility to configure the HP XC system. You are prompted Section 6.
Because HP Serviceguard is the recommended availability tool, translator scripts and other supporting scripts are already provided for you in the HP XC Serviceguard RPM. You do not have to write scripts if you are using Serviceguard. Availability Tools from Other Vendors If you are using an availability tool other than Serviceguard, you are responsible for completing the following tasks: 1. 2. You must write translator scripts and any other required scripts.
set under Serviceguard, neither node n8 nor n7 can be a member of another availability set being managed by Serviceguard. However, nodes n8 or n7 or both can be members of another availability set if it is managed by an availability tool other than Serviceguard. • If you intend to place the /hptc_cluster file system on shared storage, you must create an availability set that contains the head node and the other node in the cluster that is connected to the shared storage 1.9.
IMPORTANT: Any service that is only present on the head node cannot be moved. Those services are delivered in the avail_node_management role. Thus, to achieve improved availability of those services, you must: • • Configure the head node into an availability set with one additional node. Assign the avail_node_management role to the other node.
Table 1-2 Role and Service Placement for Improved Availability (continued) Service Name Crash dump (kdump) Service is Delivered in This Role Special Considerations for Role Assignment avail_node_management The kdump service enables you to save and access stored crash dumps if the HP XC head node becomes unavailable. The kdump service is present on the head node by default and cannot be moved.
Table 1-2 Role and Service Placement for Improved Availability (continued) Service Name Service is Delivered in This Role Nagios master management_server Special Considerations for Role Assignment By default, the management_server role is installed on the head node. If you want improved availability for Nagios, the management_server role must be assigned to two nodes, the head node and one additional node.
Table 1-3 Availability Sets Worksheet Availability Set Configuration First Node Name Second Node Name Availability Tool to Manage This Availability Roles to Assign to Nodes in the Availability Set Set First node in the availability set: • _________________________ • _________________________ • _________________________ • _________________________ • _________________________ Second node in the availability set: • • • • • _________________________ _________________________ _________________________ ______
2 Installing Software on the Head Node This chapter contains an overview of the software installation process and describes software installation tasks, which you must perform in the following order: 1. “Gathering Information Required for the Installation” (page 39) 2. “Starting the Installation Process” (page 42) 3. “Installing Additional RPMs from the HP XC DVD” (page 47) 2.
Table 2-1 HP XC Software Stack (continued) Software Product Name Description HP XC System Software Version 4.0 HP XC System Software provides the installation, configuration, administration, and management tools to support HP XC systems on HP Cluster Platforms 3000, 3000BL, 4000, 4000BL and 6000.
Table 2-1 HP XC Software Stack (continued) Software Product Name Description syslog-ng The syslog-ng logging tool improves upon traditional syslog functionality. It supplies more flexibility to handle logs, adds better filters, and contains a better forwarding mechanism. SystemImager The SystemImager tool synchronizes the configuration of nodes across the system using image propagation. This facilitates ease of installation of the initial software and ease of upgrading software and configuration files.
Table 2-3 Default Disk Partition Layout on the Head Node File System Name Partition Size with /hptc_cluster File System Partition Size without /hptc_cluster File System Maximum Partition Size swap 6 GB1 6 GB1 Not applicable 1% of remaining disk space 150 MB /boot (AMD Opteron™ and Intel® Xeon™ 1% of remaining disk space systems) or /boot/efi (Itanium® systems) / (root) 29% of remaining disk space 39% of remaining disk space 10 GB /var 50% of remaining disk space 60% of remaining disk space
2.1.5.1 Determining the Size of /hptc_cluster When it is Located on an SFS Server If you plan to create the /hptc_cluster an SFS file share, use one of the following calculations to determine the size of the/hptc_cluster file system: • • If improved availability of services will be configured on the system, use this calculation: (8.8 MB * Number of nodes) + 1.75 GB for additional directories and files If improved availability of services will not be configured on the system, use this calculation: (5.
Table 2-5 Chip Architecture by Cluster Platform (continued) 4. 5. 6. Cluster Platform Model Chip Architecture Cluster Platform 4000BL (HP server blades) AMD Opteron Cluster Platform 6000 (CP6000) Intel Itanium 2 Ensure that you have in your possession the DVD distribution media that is appropriate for the cluster platform architecture. The HP XC System Software and the open source software source code DVDs are included in the architecture-specific media kits that were shipped to you.
Table 2-6 Information Required for the Kickstart Installation Session (continued) Item Description and User Action Where to create a partition The /hptc_cluster file system is the global, or clusterwide, file system on an for the /hptc_cluster file HP XC system. This file system is shared and mounted by all nodes and contains system configuration and log file information that is required for all nodes in the system.
Table 2-6 Information Required for the Kickstart Installation Session (continued) Item Description and User Action Time zone Select the time zone in which the system is located. The default is America/New York (Eastern Standard Time, which is Greenwich Mean Time minus 5 hours). Use the Tab key to move through the list of time zones, and use the spacebar to highlight the selection. Then, use the Tab key to move to OK, and press the space bar to select OK.
IMPORTANT: After this document was published, it is possible that specific head node hardware models require additional parameters to be included on the command line. Before booting the head node, look in HP XC System Software Release Notes at http:// www.docs.hp.com/en/linuxhpc.html to make sure no additional command-line options are required for your model of head node.
IMPORTANT: After the software load is complete, ensure that the DVD is ejected from the drive before continuing. On systems with a retractable DVD device, you must remove the installation DVD before the system reboots. This is especially important if the head node is an HP workstation, which never ejects the DVD. If you do not remove the DVD, a second installation process is initiated from the DVD when the system reboots. If a second installation process is started, halt the process and remove the DVD.
4. 5. Log in to the Onboard Administrator with the default user name Administrator and the default password shown on the tag affixed to the Onboard Administrator. Proceed to Section 2.3.2.2 (page 45). Method 2 Assign a temporary static IP address to the Onboard Administrator: 1. 2. Use a network cable to connect the laptop or PC NIC directly into the Onboard Administrator or connect it to the administration network ProCurve switch.
Internet Explorer If you are using Internet Explorer as your browser, do the following: a. b. c. d. e. Click the Integrated Remote Console link to open a virtual console window that provides access to the graphics console, virtual media, and power control in the same window. Click the Drives pull-down menu. Click Mount to mount the appropriate DVD drive Click the Power button and then click Momentary Press to turn on power to the server and start booting from the DVD. Proceed to step 6.
When the Linux RPMs are loaded, the HP XC System Software RPMs are loaded, and the following message appears and remains on the screen for about 5 minutes until the software load is complete: Installing HP XC System Software RPMs IMPORTANT: After the software load is complete, ensure that the DVD is ejected from the drive before continuing. On systems with a retractable DVD device, you must remove the installation DVD before the system reboots.
4. Use the results obtained from the previous step to install the Serviceguard RPM: # rpm -ivh sg-avail-{version}.noarch.rpm 5. Unmount the DVD: # cd # umount /dev/cdrom 2.4.2 Install SVA (Optional) The HP Scalable Visualization Array (SVA) is a scalable visualization solution that brings the power of parallel computing to bear on many demanding visualization challenges. SVA is integrated with HP XC and shares a single interconnect with the compute nodes and a storage system.
• If the head node is an HP workstation, returning to the head node X server happens automatically. You then press Ctrl-Alt-F2 to return to the alternate console and log out. Press Ctrl-Alt-F7 and log in as root again. 2.4.3 Install Additional Linux RPMs (Optional) Follow this procedure to install additional, optional Linux RPMs from the HP XC distribution DVD: 1.
3 Preparing to Configure the System This chapter provides an overview of the system configuration process and describes the tasks required to prepare for the system configuration, which you must perform in the following order: 1. “Preparing to Answer cluster_prep Questions” (page 54) 2. “Changing the Default IP Address Base (Optional)” (page 55) 3. “Running the cluster_prep Command to Prepare the System” (page 57) 4. “Installing Patches or RPM Updates” (page 59) 5.
• In an enclosure-based system with single density HP server blades, the discover command uses a sparse node numbering scheme. This means that internal node names are assigned based on the enclosure in which the node is located and the bay the node is plugged into. For example, if a node is plugged into bay 10 of enclosure 1, the node is numbered {node_prefix}10.
3.1.2.1 Head Node Naming The head node naming scheme differs if HP server blades and enclosures are present in the hardware configuration. • • When the hardware configuration does not contain (real) enclosures and HP server blades, the cluster_prep command determines and assigns the head node name by using the number that represents the maximum number of nodes allowed in the hardware configuration, which you supply.
Two system services that are based on syslog-ng log the UDP based information sent by each client during installation and the node configuration phase (which is called the nconfigure phase). The hptc-ire-serverlog service runs persistently on the imaging server. It is responsible for accepting UDP messages from the imaging client and from the hptc-ire-clientlog service running when a node is running through its nconfigure phase.
Table 3-1 Information Required by the cluster_prep Command (continued) Item Description and User Action IP address for the external network connection Provide the IP address for the head node's Ethernet connection to the external network. This is the address assigned by your site's network administrator. The IP address must be in your site's Domain Name Service (DNS) configuration. Enter the address in dotted decimal format n.n.n.
file, and the file you modify depends on whether or not HP server blades and enclosures are included in the hardware configuration. base_addr.ini file Modify this file if the hardware configuration does not contain HP server blades and enclosures. base_addrV2.ini file. Modify this file if the hardware configuration contains HP server blades and enclosures. If you modify the IP addresses in the base_addrV2.ini file, you must modify the base_addr.ini file with the same changes.
netMask The netmask that is used for all connections on the administration network. dyStart The start of the dynamic IP address range used during the discovery process. This parameter is present only in the base_addrV2.ini file. dyEnd The end of the dynamic IP address range used during the discovery process. This parameter is present only in the base_addrV2.ini file. 3. Follow these guidelines if you make changes to the base_addr.ini or base_addrV2.
IMPORTANT: Do not press Ctrl-c or Ctrl-d, enter the kill command, or use any other abnormal termination method to terminate the cluster_prep utility. If you terminate cluster_prep before it completes its processing, you must use the Kickstart installation procedure to reinstall the head node. The cluster_prep utility prepares the head node for the automated discovery process. This utility is run during the initial cluster configuration or any time your cluster is reconfigured.
3 The internal host name of the head node is based on the node naming prefix and the maximum number of nodes in the hardware configuration. Therefore, as shown in this example, the head node host name is set to n16. However, if the head node is an HP server blade, the head node is named based on the node naming prefix and its physical location in the hardware configuration. For example, if the head node is located in enclosure 1, bay 1, the head node is node 1. Be sure to include a period (.
Proceed to Chapter 4 (page 63) when you are finished installing patches. 3.5.1 Download and Install Patches Follow this procedure to download and install HP XC patches from the ITRC website: 1. Create a temporary patch download directory on the head node.
IMPORTANT: After the initial system installation and configuration is complete, you might have to download and install patches as part of regular system maintenance. If a new kernel is supplied in a patch, you must rebuild all kernel dependent modules. This task is described in the HP XC System Software Administration Guide. 3.6 Creating or Mounting the /hptc_cluster File System (Optional) This task is optional.
IMPORTANT: 2. You must set the disk format to 8e to indicate that this is an LVM volume. Create a physical volume and volume group. In the following example, VolGroup00 is the name of the volume group. You are free to use any name you want to use. # /usr/sbin/pvcreate /dev/sdb1 # /usr/sbin/vgcreate VolGroup00 /dev/sdb1 3. Build a logical volume on the volume group. The following example creates a 50 GB logical volume named hptc_cluster_vol on volume group VolGroup00.
4 Discovering Hardware Components The next step in the configuration process is to discover all system (hardware) components. In addition to configuring the system topology by discovering all hardware components, the discover command also configures the administration network and interconnect and populates the configuration and management database (CMDB). See Table 4-1 (page 63) to prepare the information you will be asked to supply during the discovery process.
Table 4-1 Information Required by the discover Command (continued) Item Description and User Action MAC address of the root administration switch Look on the back panel of the ProCurve Root Administration Switch near the 9-pin console port and write down the MAC address. Use the switch management serial console with a terminal connected if you cannot read the MAC address. See Section 14.1.2 (page 174), which describes how you can identify which switch is the Root Administration Switch.
1. 2. Begin this procedure as the root user on the head node. Start a script to capture command output into a file. This step is optional, but HP recommends doing so. # script your_filename 3. Change to the configuration directory: # cd /opt/hptc/config/sbin NOTE: Follow this procedure before you run the discover command if you want to locate the console port of a non-blade head node on the administration network and not on the external network: 1.
Table 4-2 discover Command Options (continued) Option Description --ic= The --ic={AdminNet|Ethernet|Infiniband|Myrinet|Quadrics} option specifies the type of interconnect to configure. If you do not specify this option, the discover utility configures the fastest interface it finds available in the hardware configuration. Your entries are case sensitive. When the interconnect is configured on the administration network, a single LAN is used.
pinging 172.21.0.13 no response from 172.21.0.13 pinging 172.21.0.15 no response from 172.21.0.15 pinging 172.21.0.14 no response from 172.21.0.14 Following console ports did not respond: 172.21.0.13 172.21.0.15 172.21.0.14 number of cps to check, 3 pinging 172.21.0.13 pinging 172.21.0.15 pinging 172.21.0.14 .done Opening /etc/hosts Opening /etc/hosts.new.XC Opening /etc/powerd.conf Building /etc/powerd.conf ...
Opening /etc/hosts.new.XC Head Node CP device type set to iLO Waiting for power daemon to restart... done 1 2 3 4 5 Enter the MAC address of the switch that is connected to the administration ports. Do not enter the MAC address of the switch connected to the console ports. Enter the password for the Root Administration Switch that you previously defined when you prepared the hardware. If you did not preset a password, press the Enter key.
Discovering hardware configurations with server blades and more than one c3000 or c7000 enclosure is a multiple-step process. First, because the hardware configuration contain at least one external ProCurve switch, the switch or switches are discovered, which activates the administration network. Next, you perform hardware preparation tasks such as setting up passwords and making BIOS settings. Lastly, the enclosures and nodes are discovered. Complete these tasks in the following order: 1. 2. 3.
Restarting dhcpd waiting for network component at 172.31.32.2 to become available. ........ switchName blc1nxcsw001708ad3740-24 switchIP 172.31.32.2 type 2626 switchName blc1nxcsw000000000000-0 switchIP 172.31.32.1 type 2824 5. If you captured the output of the discover command in a file (described in step 2), press Ctrl-d to end the script session now. This file is for your information only. Store this file in a directory where you keep administrative data. Proceed to Section 4.3.
IMPORTANT: When you discover nodes, you must include the --extended option on the command line if the hardware configuration contains double density blade servers such as the ProLiant BL2x220c. The --extended option is only valid when used in conjunction with the --nodesonly option. A ProLiant BL2x220c has two separate nodes for each server blade, which affects internal node numbering.
The discover command turns off the console management facility (CMF) daemon. If the CMF daemon is not running, a “FAILED” message is displayed. This message is expected, and you can ignore it. If the hardware configuration contains HP workstations and you installed SVA, turn on power to the render and display workstations. Then, when prompted, turn off power to the same nodes, leaving the head node power turned on.
Enter the MAC address of the OA for the enclosure in the format xx:xx:xx:xx:xx:xx : OA_MAC_address Enter the common user name for all console port management devices: your_user_name Please enter the password for your_user_name : Please re-enter password: Restarting dhcpd waiting for OA to obtain IP address. ...................................... Enclosure blc4n-enc092UX740006L found Restarting dhcpd Opening /etc/hosts Opening /etc/hosts.new.XC uploading database waiting for network component at 172.31.32.
IMPORTANT: HP does not support the future node expansion feature if the expansion nodes are double density server blades. In that situation you must rediscover the entire cluster. During the discovery process, the discover command creates placeholder database entries for these planned additional compute nodes and marks them as disabled to exclude them from command output and job resource planning.
# telnet cp-node_name login: admin password: admin 3. The interface you use to change the password differs depending upon the BMC firmware version. Thus, do one of the following: • For BMC Firmware Version 1.03: a. Press Esc and Shift-9 to enter the command-line mode. b. Use the C[hange Password] option to change the console port password. The factory default password is admin; change it to the password of your choice. This password must be the same on every node in your system.
For more information on how to perform this procedure, contact the HP XC Support Team at xc_support@hp.com. 4.8 You Are Done with Discovery Tasks You are done with system discovery tasks. Now, go to Chapter 5 (page 77) to set up the environment.
5 Setting Up the Environment Table 5-1 lists system environment setup tasks. Perform these tasks now before you run the cluster_config utility so that the information is propagated to the appropriate client nodes during the initial image synchronization. Some tasks are required and other tasks are conditionally optional, as noted in Table 5-1.
5.2 Configuring Interconnect Switch Monitoring Line Cards (Required) You must configure Quadrics switch controller cards, InfiniBand switch controller cards, and Myrinet monitoring line cards on the interconnect to diagnose and debug problems with the interconnect. These cards must be configured to pass the operation verification program (OVP), which is used to verify the proper operation of the system after it has been installed and configured.
9. Repeat steps 5 and 6 and make the same changes to the sendmail.cf file. 10. Save the changes to the file and exit the text editor. 11. Restart sendmail: # service sendmail restart To forward mail to users, the sendmail service requires users to create .forward files in their home directories to specify where mail is to be sent. If you intend to make additional, more advanced modifications to sendmail, HP recommends that you do not modify the .cf files directly. Rather, modify the .
Install the software now before the system is configured so that the software is transparently propagated to all nodes during the initial image synchronization. This section describes some of the HP software products that you can integrate into an HP XC system. Your computing environment might require some of these products depending upon how you plan to configure the HP XC environment. You must purchase and license these products from HP.
Use the following path to navigate to the Linux patch area on the ITRC: ITRC Home Page →Downloads→Patch database →find individual patches →Linux 3. 4. 5. 6. When you are at the Linux patch area, select hpxc as the vendor and version. On the drop down menu to the right, select either 4.0-xc6000 (for IA64 RPMs) or 4.0-xc4000_xc3000 (for x86_64 RPMs). Search for a keyword. Enter the string Lustre*SFS* in the text box. Click Search>>. Select the appropriate client package from the keyword search list.
1. Install the SFS G3.0-0 web-enabler-tool: # rpm -ivh hpls-lustre-server-webtool-SFS-G3.x86_64.rpm 2. Enter the following command on the SFS G3.0-0 server MGS node: # sfsg3_xcconfig –i -s [server_name] -w [DocumentRoot] See the /usr/opt/hpls_web/README file for more information about the sfsg3_xcconfig tool. 5.7.1.3.2 Step 2: Install and Configure SFS G3.0-0 Clients on the Head Node To install and configure the SFS G3.0-0 client on the head node: 1. 2. Unpack the SFS G3.
NOTE: The sfsconfig command uses the HTTP protocol to obtain configuration information from the SFS servers. If the head node does not have access to the SFS servers over a TCP/IP network, or if the servers are offline, the sfsconfig command cannot configure the head node correctly, and you will have to modify the configuration file manually. For instructions on how to do this manually, see the HP StorageWorks Scalable File Share System User Guide for Version 2.2: http://docs.hp.com/en/storage.html 9.
} + south2 { lnets o2ib0 o2ib0 172.22.0.201 role ost + ib { cards MT25208 } } } You can also use the /usr/sbin/sfsmount command on client nodes to obtain the NIDs configured on SFS G3.0-0 servers: # sfsmount -X http://MGS_host_name/hptc_cluster /hptc_cluster 2. Confirm that the file system is mounted: # mount 3. When you have verified that the /hptc_cluster file system has been mounted, unmount it: # sfsumount /hptc_cluster 4.
• • • • All nodes other than the head node must mount Lustre file systems in foreground mode; that is, they cannot use the bg option on any file system. This means that when the nodes have booted, all file systems are mounted on the nodes and jobs can run on the nodes. You must use the lnet: protocol in the mount directives in the /etc/sfstab file. You cannot use the HTTP protocol in the /etc/sfstab file. You must use the server=name mount option.
Deciding on the Method to Achieve Quorum for HP Serviceguard Clusters In a Serviceguard configuration, each availability set becomes its own two-node Serviceguard cluster, and each Serviceguard cluster requires some form of quorum. The quorum acts as a tie breaker in the Serviceguard cluster running on each availability set. If connectivity is lost between the nodes of the Serviceguard cluster, the node that can access the quorum continues to run the cluster and the other node is considered down.
5.8 Installing Third-Party Software Products (Optional) HP XC supports the use of several third-party software products. Use of these products is optional; the purchase and installation of these components is your decision depending on your requirements. HP recommends that you install additional software products now before the system is configured so that the software is propagated to all nodes during the initial image synchronization.
• The Portland Group, supplier of the C/C++ and Fortran compilers http://www.pgroup.com/ 5.9 Creating Space for Kernel Crash Dumps (Optional) This task is optional. If you want to configure the kdump facility to save and store kernel crash dumps, you must create the space to store them. The cluster_config utility will prompt you to enter the mount point for storing the crash dumps.
• Enter the following command to change the name of xw9400 workstations: # ./modify_node_type "hp workstation xw9300" \ "HP xw9400 Workstation" node_list 5.11 Enabling Software RAID-0 or RAID-1 on Client Nodes (Optional) This task is optional. Perform this task only if you want to configure software RAID-0 or RAID-1 on client nodes. You have the option to enable software RAID-0 (striping) or software RAID-1 (mirroring) on client nodes.
(GID) values). In that way, the cluster_config command uses these local accounts to properly configure Nagios and LSF with the right file permissions, access, and so on. 5.13 Overriding Default User and Group Account IDs (Optional) This task is optional. Perform this task only if the default user and group IDs are not suitable for your environment. Table 5-3 (page 90) lists the user and group account IDs that are configured by default on an HP XC system if they are not already in use.
5.15 Creating the HP Modular Cooling System Configuration File (Optional) This task is optional. Perform this task only if the hardware configuration includes a water-based HP Modular Cooling System (MCS) device. Follow this procedure to create and populate an MCS configuration file called mcs.ini. The discover command uses the information in this file to discover the MCS devices. 1. 2. Begin this procedure as the root user on the head node. Change to the following directory: # cd /opt/hptc/mcs/config 3.
status=offline [mcs3] name=mcs3 ipaddr=172.23.0.3 location=Cab CBB3 nodes=n[73-108] status=offline [mcs4] name=mcs4 ipaddr=172.23.0.4 location=Cab CBB4 nodes=n[109-144] status=offline [mcs5] name=mcs5 ipaddr=172.23.0.5 location=Cab CBB5 nodes=n[145-180] status=offline 5.16 Mounting Network File Systems (Optional) This task is optional. If you plan to mount NFS file systems, add the mount points to the /hptc_cluster/etc/fstab.proto file now so that the mount points are propagated to the golden image.
rm $i mv $NEWF $i fi fi done 5.
6 Configuring and Imaging the System This chapter describes system configuration and imaging tasks, which you must perform in the following order: 1. 2. 3. 4. 5. 6. 7. “Preparing to Answer cluster_config Questions” (page 95) “Running the cluster_config Utility to Configure the System” (page 100) “Configuring Availability Sets” (page 101) “Modifying and Assigning Node Roles” (page 103) “Responding to cluster_config Questions” (page 105) “Editing the /etc/dhcpd.
Table 6-1 Information Required by the cluster_config Utility (continued) Item Description and User Action Configure IB bonding Specify whether you want to configure InfiniBand (IB) bonding. With IB bonding, two network interfaces are configured to appear as one. The interface has one IP address. If one of the cards becomes unavailable, network traffic is automatically routed over the other card. IB bonding provides improved reliability (no single point of failure) and higher bandwidth.
Table 6-1 Information Required by the cluster_config Utility (continued) Item Description and User Action LVS configuration If you modified the default role assignments and assigned a login role to one or more nodes, you are prompted to enter an LVS alias. This is the host name that users will use to log in to the HP XC system.
Table 6-1 Information Required by the cluster_config Utility (continued) Item Description and User Action SVA and remote graphics software configuration If you installed SVA or optional remote graphics software,1 you are prompted to supply the following information: • Whether the visualization nodes (the workstations) have a KVM attached • The host names for display nodes, that is, the nodes that have monitors connected to them • Remote graphics software configuration information: — The host names of the
Table 6-1 Information Required by the cluster_config Utility (continued) Item Description and User Action • Configuration information for the LDAP admin user, the user who is granted LDAP administration privileges. In the following example, the common name (cn) of the user must be admin, the user ID (uid) is ldap and the organizational unit (ou) is Admin: Enter the LDAP admin user [cn=admin]: uid=ldap,ou=Admin • Password for the admin user.
Table 6-1 Information Required by the cluster_config Utility (continued) Item Description and User Action Kernel crash dump configuration You are prompted to configure support for kernel crash dumps, as follows: Do you want to configure kernel dump support (y/n)? []: kdump is the facility that enables support to save and store kernel crash dumps.
NOTE: The cluster_config utility processing time can take approximately ten minutes longer if it is run on a head node that is configured for improved availability with Serviceguard when the remaining nodes of the cluster are up and running. After the entire system has been imaged and booted, you might need to rerun the cluster_config procedure to modify the node configuration.
When the cluster_config utility prompts you to configure availability sets, you configure nodes n7 and n8 into an availability set and assign node roles accordingly. Using this example, you assign the external and management_server roles to both nodes. The following criteria must be met to configure improved availability of services: • • You have purchased, licensed, installed, and configured an availability tool (such as HP Serviceguard) .
a. If more than one availability tool has been installed, select the availability tool that will manage the services in the availability set. If only one tool is available, it is selected by default. avail> select serviceguard b. Create an availability set that will be managed by the selected tool. Specify both node names to associate into the availability set. Section 1.9.6 (page 29) described how to choose the nodes to associate in an availability set.
Q - Quits this utility leaving the cluster unmodified [L]ist Nodes, [M]odify Nodes, [C]heck, [H]elp, [P]roceed, [Q]uit: 2. Enter the letter l to list the default configuration; then enter the letter b or the letter a to view an abbreviated or full version of the output: [L]ist Nodes, [M]odify Nodes, [A]nalyze, [H]elp, [P]roceed, [Q]uit: l Output Type - a[B]breviated, [A]ll, [C]ancel: • The a[B]breviated output is similar to the following.
b. c. d. If you are configuring services for improved availability, read the information in Table 1-2 (page 31) to ensure that you assign the appropriate roles to the nodes in availability sets so that service failover can occur. If you plan to configure an external Ethernet connection on any node, you must assign the external role to that node. By default, the head node is assigned with the console_network and management_hub roles.
NOTE: Table 6-1 (page 95) describes each prompt and provides information to help you with your answers. 1. If an InfiniBand (IB) interconnect is in use, specify whether you want to configure IB bonding: Configuring system wide functions / policies / behaviors Executing C02hptc_cluster_sfs sconfigure Executing C02ibbond sconfigure Do you want to configure IB bonding (y/n)? [n]: IB bonding only applies if the hardware configuration has multiple IB connections.
HP Serviceguard Clusters” (page 86) or the Serviceguard documentation for more information. Executing C25syslimits sconfigure Executing C75mpiic sconfigure Configuring service specific functions Executing C03avail gconfigure You must configure a quorum method for the availability set with members n1 and n2. Valid choices are [q]uorum server or [l]ock LUN (q/l): [q] Enter the host name of the quorum server []: host_name OR Enter the path of the lock LUN [ ] full_path_to_lock_LUN 5.
Enter the number that corresponds to the availability method you want to use (1/2) []: If you enter 2 to configure improved availability of the hptc_cluster_fs service, provide the following information about the shared storage. You cannot proceed until the shared storage is in place. When prompted to supply the path to the system disk, enter the path to the local disk on the backup server where the OS will reside. You have elected to enable improved availability of the /hptc_cluster global file system.
Re-type new password: your_nagios_password Adding password for user nagiosadmin You can create or modify nagios web accounts using the command: # /usr/bin/htpasswd /opt/hptc/nagios/etc/htpasswd.users nagiosadmin Creating /opt/hptc/etc/sysconfig/managementData.ini ... 12.
The following configuration questions are specific to HP RGS: Do you want to use the RGS sender on the head node (y/n)? [n] You have installed RGS Sender 4.0.0. Will you be using RGS Receiver 3.0 to connect this cluster (y/n)? [n] 15. If you assigned the ldap_server role to a node, you are prompted to specify LDAP configuration information. Otherwise, skip this step. NOTE: The following sample configuration session sets up example.com as the domain name. Substitute example.com for your domain.
an LDAP database by opening another shell in another console. When you are done, press Enter to continue: Enter 16. Supply the name or IP address of the external NIS master server and the NIS domain name if you assigned the nis_server role to one or more nodes to configure them as NIS slave servers. If you did not assign a nis_server role to any node, you are not asked to supply this information.
NOTE: After cluster_config processing is complete, you have the option to modify default SLURM compute node and partition information, as described in Section 7.1.1 (page 123). 19. Define a SLURM user name and accept all default responses. NOTE: The output looks different if you assigned the resource_management role to one or more nodes because you will be prompted to assign the master and backup controller nodes. This command example does not show that prompt.
• If you intend to install another job management system, such as PBS Professional (see Chapter 11 (page 159)) or the Maui Scheduler (see Chapter 12 (page 165)) enter n. Proceed to step 25. If at a future time you want to install LSF, rerun the cluster_config utility, and answer y to this question. The remainder of this procedure does not describe how to install any other job management system other than LSF with SLURM or standard LSF. 21.
23. Provide responses to install and configure LSF. This requires you to supply information about the primary LSF administrator and administrator's password. The default user name for the primary LSF administrator is lsfadmin. If you accept the default user name and a NIS account exists with the same name, LSF is configured with the existing NIS account, and you are not be prompted to supply a password. Otherwise, accept all default answers.
to learn more about your new LSF cluster. ***Begin LSF-HPC Post-Processing*** No patches to be installed Created '/hptc_cluster/lsf/tmp'... Editing /opt/hptc/lsf/top/conf/lsf.cluster.hptclsf... Moving /opt/hptc/lsf/top/conf/lsf.cluster.hptclsf to /opt/hptc/lsf/top/conf/lsf.cluster.hptclsf.old.8549... Editing /opt/hptc/lsf/top/conf/lsf.conf... Moving /opt/hptc/lsf/top/conf/lsf.conf to /opt/hptc/lsf/top/conf/lsf.conf.old.8549... Editing /opt/hptc/lsf/top/conf/lsbatch/hptclsf/configdir/lsb.params...
info: info: info: info: info: info: info: info: info: info: info: info: info: info: info: info: info: info: info: info: info: info: info: info: info: info: info: info: info: info: info: info: info: info: info: info: Executing C10hptc_cluster_fs nconfigure Executing C10hptc_cluster_fs_client nconfigure Executing C12dbserver nconfigure Executing C20gmmon nconfigure Executing C20smartd nconfigure Executing C30syslogng_forward nconfigure Executing C33imageserver nconfigure Executing C35dhcp nconfigure Exe
info: info: info: info: info: info: info: info: info: info: info: 1 Executing C95hpwdt nrestart Executing C95kdump nrestart Executing C95lsf nrestart Executing C30syslogng_forward crestart Executing C35dhcp crestart Executing C50supermond crestart Executing C90munge crestart Executing C90slurm crestart Executing C95lsf crestart Executing C98db_ic crestart nconfig shut down You can safely ignore this warning because Use Your Own Kernel (UYOK) is not used by SystemImager in HP XC.
Multicast imaging uses the udpcast open source package, and the flamethrower functionality of SystemImager. A series of udp-sender daemons are run on the image server, and each client node runs a series of udp-receiver daemons during the imaging operation. The udp-sender daemons are managed by the startsys command.
Table 6-4 startsys Command-Line Options Based on Hardware Configuration Hardware Configuration startsys Command Line Fewer than 300 nodes For small-scale hardware configurations, nodes are imaged and rebooted in one operation. The nodes complete their per-node configuration phase, thus completing the installation. This option applies only for nodes that have previously been set up to network boot.
You must manually power on the following nodes: n1 Press enter after applying power to these nodes. continuing ........
12 nodes -> n[2-6,8-14] Fri Dec 05 09:06:48 2008 Processing completed for: 1 node -> n1 Fri Dec 05 09:07:03 2008 Processing completed for: 1 node -> n7 Fri Dec 05 09:07:18 2008 Processing completed for: 9 nodes -> n[4-5,8-14] *** Fri Dec 05 09:07:33 2008 Current statistics: Booted and available: 15 nodes -> n[1-15] Progress: Fri Dec 05 09:07:33 2008 Processing completed for: 3 nodes -> n[2-3,6] *** Fri Dec 05 09:07:33 2008 Current statistics: Booted and available: 15 nodes -> n[1-15] Progress: Fri Dec 05 09
7 Finalizing the Configuration This chapter describes postconfiguration tasks that you might have to complete depending upon the software and hardware configuration: • “Finalizing the Configuration of Compute Resources” (page 123) • “Enabling SSH for the InfiniBand Interconnect” (page 125) • “Creating a Lock LUN Device File” (page 125) • “Starting Availability Tools” (page 125) • “Configuring SNMP Trap Destination for Enclosures” (page 126) • “Configuring SNMP Trap Destination for Modular Cooling System Dev
a. b. Go to Appendix J (page 239) to determine the type of customizations that are available or required. For instance, if you installed and configured SVA, SVA requires certain SLURM customizations. Use the text editor of your choice to edit the SLURM configuration file: /hptc_cluster/slurm/etc/slurm.conf c. d. e. Use the information in Appendix J (page 239) to customize the SLURM configuration according to your requirements. If you make changes to the slurm.
Restart only the master candidate hosts? [y/n] n Do you really want to restart LIMs on all hosts? [y/n] y Restarting the LIM daemon is required because the licensing of LSF with SLURM occurs when the LIM daemon is started. This means that the LIM daemon will be licensed only for processors that are actually available at that time, which might be fewer than the total number of processors available after all nodes have been imaged and are up and running. 7.
Info: Starting transfer of services to ServiceGuard... prepForAvail: ========== Executing 'pdsh -S -w n7 '/opt/hptc/etc/nconfig.d/C50nat nxferto n7''... pdsh@n8: n7: RC=0 prepForAvail: ========== 'pdsh -S -w n7 '/opt/hptc/etc/nconfig.d/C50nat nxferto n7'' finished, exited with 0(0) prepForAvail: ========== Executing 'pdsh -S -w n8 '/opt/hptc/etc/nconfig.d/C50nat nxferto n8''... pdsh@n8: n8: RC=0 prepForAvail: ========== 'pdsh -S -w n8 '/opt/hptc/etc/nconfig.
This enables Nagios to generate alerts for all enclosure traps posted with a priority of WARNING or greater. 1. Configure the enclosures to send their SNMP traps to the management server IP alias: # manage_enclosure addtrap enc2n-enc09USE6281BJX: Added SNMP trap receiver destination IP_address 2. Verify that the SNMP trap destination has been set correctly on the enclosures: # manage_enclosure listtrap enc2n-enc09USE6281BJX: IP_address 7.
7.7 Generating an SVA Site Configuration File Perform this task only if you installed and configured SVA. Otherwise, omit this task. The SVA site configuration file identifies all nodes, characterizes the nodes by role, and defines all display surfaces by assuming each display node maps to a single display surface. Generating this file creates a set of display surfaces. Each display surface defaults to a single tile, which is defined by a display node and the display device physically connected it.
8 Verifying the System and Creating a Baseline Record of the Configuration Complete the tasks described in this chapter to verify the successful installation and configuration of the HP XC system components. With the exception of the tasks that are identified as optional, HP recommends that you perform all tasks in the following order: 1. “Verifying the LSF Configuration” (page 129) 2. “Verifying Availability Tools” (page 130) 3. “Running the OVP to Verify Software and Hardware Components” (page 131) 4.
# lsid Platform LSF 7 Update 3, LSF_build_date Copyright 1992-2005 Platform Computing Inc My cluster name is hptclsf My master name is n13 [root@n16 ~]# lshosts HOST_NAME type model n13 LINUX64 Itanium2 n16 LINUX64 Itanium2 n1 LINUX64 Itanium2 n2 LINUX64 Itanium2 n3 LINUX64 Itanium2 n4 LINUX64 Itanium2 n5 LINUX64 Itanium2 n6 LINUX64 Itanium2 n7 LINUX64 Itanium2 n8 LINUX64 Itanium2 n9 LINUX64 Itanium2 n10 LINUX64 Itanium2 n11 LINUX64 Itanium2 n12 LINUX64 Itanium2 cpuf ncpus maxmem maxswp server RESOURCES 16
Verifying HP Serviceguard Enter the following commands to verify the installation and configuration of Serviceguard: 1. List the nodes that are members of availability sets: # shownode config hostgroups hostgroups: headnode: n16 serviceguard:avail1: n14 n16 In this example, one availability set, avail1, has been configured. 2.
• Memory usage on all compute nodes but not the head node (by default). The OVP also runs the following benchmark tests. These tests compare values relative to each node and report results with values more than three standard deviations from the mean: • • • • LINPACK is a collection of Fortran subroutines that analyze and solve linear equations and linear least-squares problems. This test is CPU intensive and stresses the nodes, with limited data exchange.
6. When all OVP tests pass, proceed to Section 8.4 (page 133) (if SVA is installed) or Section 8.5 (page 133). 8.4 Running the SVA OVP Utility Run the SVA OVP utility only if you installed and configured SVA. The SVA OVP runs a series of Chromium demonstration applications on all defined display surfaces, which verifies the successful installation of SVA. Follow this procedure to start the SVA OVP: 1.
Host Monitor IP Assignment - DHCP Load Average LSF Failover Monitor Nagios Monitor NodeInfo PING Interconnect Resource Monitor Resource Status Root key synchronization Sensor Collection Monitor Slurm Monitor Slurm Status Supermon Metrics Monitor Switch Switch Data Collection Syslog Alert Monitor Syslog Alerts System Event Log System Event Log Monitor System Free Space Totals: 1-Ok 1-Ok 10-Ok 1-Ok 1-Ok 10-Ok 10-Ok 1-Ok 10-Ok 1-Ok 1-Ok 1-Ok 10-Ok 1-Ok 2-Ok 1-Ok 1-Ok 10-Ok 9-Ok 1-Ok 10-Ok 115-Ok 0-Warn 0-War
8.7 Creating a Baseline Report of the System Configuration The sys_check utility is a data collection tool that is used to diagnose system errors and problems. Use the sys_check utility now to create a baseline report of the system configuration (software and hardware). The sys_check utility collects configuration data only for the node on which it is run unless you set and export the SYS_CHECK_SYSWIDE variable, which collects configuration data for all nodes in the HP XC system.
9 Upgrading to HP XC System Software Version 4.0 This chapter describes how to upgrade an existing HP XC system to Version 4.0. You must complete these tasks in the following order: 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11.
NOTE: If you are not sure what version of the HP XC System Software is installed on the system, enter the following command: # cat /etc/hptc-release 9.1.2 Is Upgrading Appropriate for Your System Configuration? An upgrade path from previous HP XC System Software releases to Version 4.0 is provided, but HP recommends a new installation of Version 4.0.
Table 9-2 Upgrade Characteristics (continued) Characteristic Description Affect on nodes that are down If a node or nodes is in the DOWN state during the upgrade, you must reimage those nodes as soon as they are returned to operation. Affect on number of nodes You cannot add or remove nodes during the upgrade process. 9.1.4 Upgrade Commands Table 9-3 lists the commands and utilities that are run as part of a software upgrade process.
Table 9-4 Upgrade Readiness Criteria Component Requirement Version of HP XC System Software currently installed on the system Your current HP XC system must be running HP XC System Software Version 3.2 or Version 3.2.1. Firmware The firmware versions installed on all system components have been updated according to the HP XC Version 4.0 Master Firmware Tables: http://www.docs.hp.com/en/linuxhpc.
The InfiniBand firmware binary files are included in SFS and are automatically installed on all SFS nodes. However, the firmware on the HCA adapter must be upgraded manually, using these firmware binary files. If you need assistance with identifying and upgrading the firmware on the InfiniBand switches and HCA adapters, contact your local HP support center. 7. If the system is not configured with improved availability of services, determine whether you want to configure that feature.
b. Stop the processes that are using /hptc_cluster. Typically, the processes using /hptc_cluster correlate to the cmfd, nagios, lsf, slurm, and syslog-ng services. Use the following syntax to stop those services: # service service_name stop c. Run the fuser command again to make sure no processes are using /hptc_cluster: # fuser -vm /hptc_cluster d. 7. Proceed to step 7 when you have verified that no processes are using /hptc_cluster.
NOTE: If you run the preupgradesys command using SSH, do not close this session until you are done with any verification. # preupgradesys The preupgradesys utility performs all the necessary pre-processing steps needed to upgrade your cluster. This script must be run before you upgrade your system. Do you wish to continue? [y/n] y Copying /etc/sysconfig/iptables.proto to /etc/sysconfig/iptables.proto.bak Copying /var/log/postinstall.log to /var/log/postinstall.log.7483 Copying /etc/modprobe.
IMPORTANT: You must act fast, because the system does not pause very long at the boot prompt. Press any letter on the keyboard to pause at the boot prompt and give yourself more time to type in the boot command line. IMPORTANT: Some hardware models require additional parameters to be included on the command line.
NOTE: Ignore the following items when they appear on the screen: • The Kudzu New Hardware Installation Setup Menu • All errors seen during first reboot • The message Cannot get binary type, which you will see when the system reboots after the upgrade IMPORTANT: After the software load is complete, ensure that the DVD is ejected from the drive before continuing. On systems with a retractable DVD device, you must remove the DVD before the system reboots.
9.7 Installing Patches and Reinstalling Additional Software Follow this procedure to install patches and reinstall additional software on the upgraded system before the golden image is propagated to all client nodes: 1. Install all patches from the HP IT Resource Center (ITRC) website that might be available for HP XC Version 4.0. Section 3.5.1 (page 60) describes how to download HP XC patches from this website.
user customizations that were not merged into the new files in this release. This example uses the grep command to search for the affected files. # grep -E ".rpmsave|.rpmnew" /var/log/postinstall.log 2. 3. Use the method of your choice (the diff command, for example) to determine differences between the customized and current versions of configuration files. As software evolves, configuration file formats might change.
5. Verify that the HP XC head node is running the Version 4.0 kernel version that is listed in the HP XC System Software Release Notes: # uname -a 6. If SFS is used, verify that the HP XC head node can mount the SFS file system(s): # service sfs status # mount -t lustre 7. Remove the /boot/grub/grub.conf entry from /opt/hptc/systemimager/etc/ updgi_exclude_file file. Undo this change after the HP XC golden image is created. Proceed to Section 9.9 (page 148). 9.
NOTE: You will see a message "upgrading from v3.1 to 4.0" even though you are upgrading from another version. This message is expected and you can ignore it. IMPORTANT: Do not proceed to the next step in the upgrade process if the output from the upgradesys script indicates failures. If you cannot determine how to resolve these errors, contact your local HP support center. The state of the cluster is unstable at this point because the definition of all network interfaces has been removed.
Table 9-7 Upgrade Options for the cluster_config Utility --migrate Option --init Option Brings existing, recognized roles in the configuration and management database into alignment with the new roles introduced in this release. Initializes (resets) current node role assignments and configures the system with the default node role assignments in this release.
11. The cluster_config utility prompts you to supply system configuration information. When prompted, provide the answers listed in Table 9-8. Table 9-8 Responses to cluster_config Prompts During an Upgrade Prompt Answer Regenerate SSH keys? yes Re-create the qsnet database? (Seen only on systems that are configured with a QsNetII interconnect). yes Reconfigure SLURM? yes Create a new slurm.
# service powerd restart Proceed to Section 9.10 (page 152). 9.10 Imaging and Booting the System and Starting Compute Resources Follow this procedure to image and boot all nodes and start LSF. 1. Use the startsys command to turn on power to all nodes, image the nodes, and boot the nodes.
2. Make sure all nodes are up: # power --status 3. 4. If you have configured the kdump service, reboot all nodes except the head node to activate the kdump code. Set up the LSF environment by sourcing the LSF profile file: # . /opt/hptc/lsf/top/conf/profile.lsf 5. Complete this step if the system is configured with LSF with SLURM. Otherwise omit this step. a. If you assigned two or more nodes with the resource_management role and want to enable LSF failover, enter the following command.
Run the transfer_to_avail command to shut down all services and IP aliases associated with services that will be managed by an availability tool: # transfer_to_avail NOTE: To avoid duplicating command output here, transfer_to_avail command output is shown in Section 7.4 (page 125). After shutting down all services and IP aliases, the transfer_to_avail command starts each availability tool. Then, the availability tool starts up the services it manages. Proceed to Section 9.12 (page 154). 9.
10 Reinstalling HP XC System Software Version 4.0 This chapter describes how to reinstall HP XC System Software Version 4.0 on a system that is already running Version 4.0. Reinstalling an HP XC system with the same release might be necessary if you participated as a field test site of an advance development kit (ADK) or an early release candidate kit (RC).
# setnode --resync n[1-5] 4. Reboot nodes n1 through n5: # stopsys n[1-5] # startsys --image_and_boot n[1-5] The nodes automatically reboot when the reimaging is complete. 5. If SLURM is configured, reset the job state on nodes n1 through n5: # scontrol update NodeName=n[1-5] State=IDLE 10.
on the node or nodes by temporarily modifying the EFI environment to allow the node or nodes to network boot. This procedure assumes nodes 1 through 5 (n1–5) are to be reinstalled. 1. 2. Begin this procedure as the root user on the head node. Use the scontrol command to ensure that all jobs are drained from nodes n1 through n5: # scontrol update NodeName=n[1-5] State=DRAIN Reason="system shutdown" 3. Prepare all client nodes to network boot rather than boot from local disk: # setnode --resync --all 4.
11 Installing and Using PBS Professional This chapter describes the steps required to install and use PBS Professional, which you must complete in the following order: 1. 2. 3. 4. 5. 6. 7. 8. 9.
11.4 Performing Installation Actions Specific to HP XC Follow this installation procedure: 1. Install the PBS server node (front-end node) first, using the installation script provided by the software vendor, and specify the following values: a. Accept the default value offered for the PBS_HOME directory, which is /var/spool/ PBS. b. When prompted for the type of PBS installation, select: option 1 (Server, execution and commands). c. If available, enter the license key during the interactive installation.
11.5.1 Configure the OpenSSH scp Utility By default, PBS Professional uses the rcp utility to copy files between nodes. The default HP XC configuration disables rcp in favor of the more secure scp command provided by OpenSSH. To use PBS Professional on HP XC, configure HP XC to default to scp as follows: 1. 2. Using a text editor of your choice, open the /etc/pbs.conf file on the server node.
# # # # pdcp pdcp pdcp pdcp -rp -w "x[n-n]" /usr/pbs /usr -rp -w "x[n-n]" /var/spool/PBS /var/spool -p –w "x[n-n]" /etc/pbs.conf /etc -p –w "x[n-n]" /etc/init.d/pbs /etc/init.d Use the following as an example: • • • You have installed the PBS server on node n100. The first PBS execution node is node n49. You want to replicate the execution environment to nodes n1 through n48. In this case, the value of the node list expression is: "n[1-48]".
11.10 Running HP MPI Tasks The PBS Professional distribution contains a wrapper script named pbs_mpihp , which is used to run HP MPI jobs. The wrapper script uses information about the current PBS Professional allocation to construct a command line, and optionally, an appfile suitable for HP MPI. The wrapper also sets the MPI_REMSH environment variable to the PBS Professional pbs_tmrsh remote shell utility.
12 Installing the Maui Scheduler This chapter describes the steps required to install and configure the Maui Scheduler software tool to interoperate with SLURM on an HP XC system. Complete the steps in the following order: 1. 2. 3. 4. “Reviewing Readiness Criteria” (page 165) “Preparing for the Installation” (page 165) “Installing the Maui Scheduler” (page 166) “Verifying the Successful Installation of the Maui Scheduler” (page 168) 12.
• http://www.clusterresources.com/products/maui/ Ensure That LSF with SLURM Is Not Activated HP does not support the use of the Maui Scheduler with LSF with SLURM. These schedulers have not been integrated and will not work together on an HP XC system. Before you install the Maui Scheduler on an HP XC system, you must be sure that the HP XC version of LSF with SLURM is not activated on the system. If LSF with SLURM is activated, you must deactivate it before proceeding.
12.4.1 Download the Maui Scheduler Kit Follow this procedure to download the Maui Scheduler kit: 1. 2. Log in as the root user on the head node. Download the Maui Scheduler kit to a convenient directory on the system. The Maui Scheduler kit is available at: http://www.clusterresources.com/products/maui/ 12.4.
NODECFG[n14] PARTITION=PARTA NODECFG[n13] PARTITION=PARTA 6. Save the changes to the file and exit the text editor. 12.4.4 Edit the SLURM Configuration File Uncomment the following lines in the /hptc_cluster/slurm/etc/slurm.conf SLURM configuration file: SchedulerType=sched/wiki SchedulerAuth=42 SchedulerPort=7321 12.4.5 Configure the Maui Scheduler After you install and set up the Maui Scheduler to interoperate with SLURM on HP XC, you must perform the standard Maui Scheduler configuration steps.
Network: [NONE] Memory >= 1M Disk >= 1M Swap >= 0 Opsys: [NONE] Arch: [NONE] Features: [NONE] NodeCount: 1 Allocated Nodes: [n16:4][n15:2] IWD: [NONE] Executable: [NONE] Bypass: 0 StartCount: 1 PartitionMask: [lsf] Reservation '116' (00:00:00 -> 1:00:00 Duration: 1:00:00) PE: 6.00 StartPriority: 1 Table 12-2 lists several commands that provide diagnostic information about various aspects of resources, workload, and scheduling.
13 Adding Visualization Nodes to an Existing HP XC System This chapter describes how to install and configure visualization nodes into an existing HP XC system after the SVA nodes have been fully integrated into the hardware configuration. It addresses the following topics: • • “Prerequisites” (page 171) “Installation Scenarios” (page 171) 13.
13.2.2 New Visualization Nodes Do Not Exceed the Maximum Number of Nodes Supplied to the cluster_prep Command Follow this procedure if you added visualization nodes to the existing system and the number of new nodes does not exceed the maximum node number you set during the initial cluster_prep process. Because the number of new nodes does not exceed the previous maximum number of nodes, you do not need to run cluster_prep command again, but you do have to discover the new nodes. 1. 2.
14 Troubleshooting This chapter addresses the following topics: • “Troubleshooting the Discovery Process” (page 173) • “Troubleshooting the Cluster Configuration Process” (page 176) • “Troubleshooting LSF and Licensing” (page 180) • “Troubleshooting the Imaging Process” (page 178) • “Troubleshooting the OVP” (page 180) • “Troubleshooting SLURM” (page 183) • “Troubleshooting the Software Upgrade Procedure” (page 184) 14.
• • • “Not All Console Ports Are Discovered” (page 174) “Some Console Ports Have Not Obtained Their IP Addresses” (page 175) “Not All Nodes Are Discovered” (page 175) After performing the suggested corrective action, rerun the discover command. 14.1.1 Discovery Process Hangs While Discovering Console Ports This information applies only to HP XC systems with nodes that use Integrated Lights Out (iLO) as the console management device.
NOTE: If the --oldmp option was used on the discover command line, it is assumed that all Management Processors (MPs) have their IP addresses set statically, and therefore are not subject to this step in the discovery process. If some console ports are not configured to use DHCP, they will not be discovered. Therefore, the first item to verify is whether the nondiscovered console ports are configured to use DHCP.
. . . Switch Switch Switch Switch . . . 172.20.65.3 172.20.65.3 172.20.65.3 172.20.65.4 port port port port 4 5 6 1 ... ... ... ... Node Found Node Found NO Node Found Node Found In this case, a node is plugged into port 6 of the Branch Root switch at address 172.20.65.3. To resolve the discovery problem, examine this node to see what actions it is taking during power-on.
• Nodes that fail the configuration phase are put into single-user mode and marked as disabled in the database if an essential service failed. 14.2.1 lsadmin limrestart Command Fails “Finalizing the Configuration of Compute Resources” (page 123) describes LSF postconfiguration tasks. It is possible for the lsadmin limrestart command to fail if the LSF control node was assigned to the wrong node name.
14.3 Troubleshooting the Imaging Process This section describes hints to troubleshoot the imaging process. System imaging and node configuration information is stored in the following log files: • • • /hptc_cluster/adm/logs/imaging.log /var/log/systemimager/rsyncd /hptc_cluster/adm/logs/startsys.log Table 14-1 (page 178) lists problems you might encounter as the golden image is being propagated to client nodes and describes how to diagnose and resolve the problem.
Table 14-1 Diagnosing System Imaging Problems (continued) Symptom How To Diagnose Possible Solution The network boot times out. The system boots from local disk • Verify DHCP settings and status of daemon. and runs nconfigure. You can • Verify network status and connections. verify this by checking messages • Monitor the /var/log/dhcpd.log file for written to the imaging.log file. DHCPREQUEST messages from the client node MAC address. • Check boot order and BIOS settings.
Enter the following command on the affected node to fix the network boot problem: setnode --resync node_name 14.3.3 How To Monitor An Imaging Session To monitor an imaging operation, use the tail -f command in another terminal window to view the imaging log files. It is possible to actually view an installation through the remote serial console, but to do so, you must edit the /tftpboot/pxelinux.cfg/default file before the installation begins and add the correct serial console device to the APPEND line.
Verify perf_health: Testing memory_usage ... The headnode is excluded from the memory usage test. Number of nodes allocated for this test is 14 Job <2049> is submitted to default queue << Waiting for dispatch ...>> <> The following node has memory usage more than 25%: n3: memory usage is 34.38%, 12.
Virtual hostname is lsfhost.localdomain Comparing ncpus from Lsf lshosts to Slurm cpu count. The Lsf and Slurm cpu count are NOT in sync. The lshosts 'ncpus' value of 1560 differs from the cpu total of 2040 calculated from the sinfo output. Suggest running 'lshosts -w' manually and compare the ncpus value with the output from sinfo --- FAILED --Testing hosts_status ... Running 'bhosts -w'. Checking output from bhosts. Running 'controllsf show' to determine virtual hostname. Checking output from controllsf.
nodes ibblc64 and ibblc65 have an Exchange value of 2077.790000 14.5.2 OVP qsnet_database Test May Fail Due to Benign Errors Returned By the qsctrl Utility The following issue is specific to systems using the QsNetII interconnect. In this release, it is not possible for the qsnet2 utilities to detect that a node is missing. Therefore, the /usr/bin/qsctrl utility reports a warning message for all links in reset.
Use the following command to determine why SLURM marked this node as being down: # sinfo -R REASON NODELIST Low RealMemory [slurm@Mar 02 22:34] n15 The most common reason reported by the sinfo command is Not Responding, which means that something is wrong with the communication between the primary slurmctld daemon and the slurmd daemon on the affected node or nodes . In that situation, log in to the affected node or nodes and troubleshooting the slurmd daemon.
Table 14-2 Software Upgrade Log Files File Name Contents /opt/hptc/upgrade/rpm_qa_output.log List of RPMs installed on the system before the upgrade /var/log/preupgradesys/preupgradesys.log Results of the preupgradesys script • /var/log/upgradesys/upgradesys.
# service hptc-ire-serverlog status hptc-ire-serverlog is dead, but pid file exists 2. Determine if the /hptc_cluster/adm/logs/imaging.log file exists. If this file does not exist, use the following command to create it. If this file does not exist, the startsys utility fails. # touch /hptc_cluster/adm/logs/imaging.log 3. If hptc-ire-serverlog is not running, start the service: # service hptc-ire-serverlog start 14.7.
A Installation and Configuration Checklist Table A-1 provides a list of tasks performed during a new installation. Use this checklist to ensure you complete all installation and configuration tasks in the correct order. Perform all tasks on the head node unless otherwise noted. Table A-1 Installation and Configuration Checklist Task and Description Reference Preparing for the Installation 1. Read related documents, especially the HP XC System Software Release Notes.
Table A-1 Installation and Configuration Checklist (continued) Task and Description Reference 17. Create and mount the /hptc_cluster file system if it is not located on a partition on the system disk (optional). “Creating or Mounting the /hptc_cluster File System (Optional)” (page 61) Discovering Hardware Components 18. Run the discover command to discover network components. “Discovering Hardware Components” (page 63) Setting Up the Environment 19.
Table A-1 Installation and Configuration Checklist (continued) Task and Description Reference 28. Start availability tools if you configured services for improved availability “Starting Availability Tools” (page 125) (optional). 29. Configure SNMP trap destinations for enclosures (optional). “Configuring SNMP Trap Destination for Enclosures” (page 126) 30. Configure SNMP trap destinations for MCS devices (optional). “Configuring SNMP Trap Destination for Modular Cooling System Devices” (page 127) 31.
B Host Name and Password Guidelines This appendix contains guidelines for making informed decisions about information you are asked to supply during the installation and configuration process. It addresses the following topics: • “Host Name Guidelines” (page 191) • “Password Guidelines” (page 191) B.1 Host Name Guidelines Follow these guidelines when deciding on a host name: • Host names can contain from 2 to 63 alphanumeric uppercase or lowercase characters (a-z, A-Z, 0-9).
When choosing a password, do not use any of the following: • Single words found in any dictionary in any language. • Personal information about you or your family or significant others such as first and last names, addresses, birth dates, telephone numbers, names of pets, and so on. • Any combination of single words in the dictionary and personal information. • An obvious sequence of numbers or letters, such as 789 or xyz.
C Enabling telnet on iLO and iLO2 Devices The procedure described in this appendix applies only to HP XC systems with nodes that use Integrated Lights Out (iLO or iLO2) as the console management device. New nodes that are managed with iLO or iLO2 console management connections that have never been installed with HP XC software might have iLO interfaces that have not been configured properly for HP XC operation.
2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. Do one of the following: • If you cannot find an entry corresponding to the new node, check the network connections. Make repairs and rerun the discover command. • If you do find an entry corresponding to the new node, note the IP address on the line that begins with the string fixed-address, and proceed to step 3. Open a web browser on the head node.
fixed-address 172.21.0.2; # location "Level 2 Switch 172.20.65.4, Port 2"; } host cp-n3 { hardware ethernet 00:11:0a:30:b0:bc; option host-name "cp-n3"; fixed-address 172.21.0.3; # location "Level 2 Switch 172.20.65.4, Port 3"; } host cp-n4 { hardware ethernet 00:11:0a:2f:8d:fc; option host-name "cp-n4"; fixed-address 172.21.0.4; # location "Level 2 Switch 172.20.65.4, Port 4"; } 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17.
D Configuring Interconnect Switch Monitoring Cards You must configure the Quadrics switch controller cards, the InfiniBand switch controller cards, and the Myrinet monitoring line cards on the system interconnect to diagnose and debug problems with the system interconnect.
Table D-1 Quadrics Switch Controller Card Naming Conventions and IP Addresses for Reduced Bandwidth (continued) Number of Nodes Node-Level Switch Name 1025 to 2048 QR0N00 to QR0N31 (P) 172.20.66.1 to 172.20.66.32 QR0N00_S to QR0N31_S (S) 172.20.66.33 to Secondary not applicable 172.20.66.64 1 2 Node-Level IP Address Top-Level Switch Name Top-Level Switch IP Address QR0T00 to QR0T31 172.20.66.65 to 172.20.66.96 Secondary not applicable (P) represents the primary switch controller.
2. 3. 4. 5. 6. 7. 8. 9. Change network settings Run jtest Set module mode Firmware upgrade Quit Reboot Access Settings Self Test Enter 1-9 and press return: 3. Enter 2 to access the Change network settings menu option and set the switch to STATIC IP addresses: Quadrics Switch Control Select Protocol 1. BOOTP 2. STATIC 3.
D.2 Configuring Myrinet Switch Monitoring Line Cards You can use the Myrinet switch monitoring line card to run diagnostic tools and to check for events on each port of the line card. Table D-3 provides the switch names and associated IP addresses you need during the configuration procedure. The IP addresses for the switch monitoring line cards is based on the swBase address in the base_addr.ini file. The default address base for the switch monitoring line cards is 172.20.
. . host n3 { hardware ethernet 00:11:0a:ea:ea:41; option host-name "n3"; fixed-address 172.20.0.3; option xc-macaddress "00:11:0a:ea:ea:41"; # location "Level 2 Switch 172.20.65.3, Port 3"; } host MR0N00 { hardware ethernet your_MAC_address; option host-name "MR0N00"; fixed-address 172.20.66.1; } } } 5. 6. Save your changes and exit the text editor. Copy the contents of the /etc/dhcpd.conf file to another file name because the cluster_config utility rewrites the /etc/dhcpd.
Table D-4 InfiniBand Switch Controller Card Naming Conventions and IP Addresses (continued) Switch Order Switch Name IP Address Third switch IR0N02 172.20.66.3 Last switch IR0N0n 172.20.66.n Table D-5 InfiniBand Switch Controller Card Naming Conventions and IP Addresses for Hardware Configurations with HP Server Blades and Enclosures Switch Order Switch Name IP Address First switch IR0N00 172.31.33.1 Second switch IR0N01 172.31.33.2 Third switch IR0N02 172.31.33.
10. Set the IP address of the switch and the netmask using the data in Table D-4 or Table D-5 as a reference. Set the netmask to 255.255.0.0 as shown: ISR-9024(config-if-fast)# ip-address-fast set IP_address 255.255.0.0 11. Confirm the settings: ISR-9024(config-if-fast)# ip-address-fast show 12. Exit the interface fast mode: ISR-9024(config-if-fast)# exit 13. Access the route mode: ISR-9024(config)# route 14.
D.3.2 Configure Cisco InfiniBand Switches Cisco InfiniBand switches are supported as an alternative to Voltaire InfiniBand switches. Similar to the Voltaire switches, some basic configuration steps are required to configure network access to the switch controller card. Configuration instructions are located in the Cisco switch documentation and vary depending on the particular model of switch in your hardware configuration.
E Customizing Client Node Disks Use the information in this appendix to customize the disk partition layout on client node disk devices. It addresses the following topics: • “Overview of Client Node Disk Imaging” (page 205) • “Dynamically Configuring Client Node Disks” (page 205) • “Statically Configuring Client Node Disks ” (page 211) E.
issues encountered when the golden client node disk configuration differs from the client disk configuration. You also have the flexibility to configure client node disks on a per-image and per-node basis and to create an optional scratch partition. Partition sizes can be fixed or can be based on a percentage of total disk size. You can set the appropriate variables in the /opt/hptc/ systemimager/etc/make_partitions.sh file or in user-defined files with a .part extension.
E.2.2 Example 1: Modifying Partitions Using Fixed Sizes and Defining an Additional Partition This example applies fixed sizes to modify the default partition sizes on all compute nodes and creates an additional /scratch partition on each compute node. The user-defined .part files allow partition modifications to be done on a per-image or per-node basis. 1. Use the text editor of your choice to create the following file to define the partition format for compute nodes: /var/lib/systemimager/scripts/compute.
7. Do one of the following to install and image the client nodes: • If the client nodes were not previously installed with the HP XC System Software, see “Running the startsys Utility to Start the System and Propagate the Golden Image” (page 117) to continue the initial installation procedure.
8. Run the cluster_config utility, choosing the default answers, to create a new master autoinstallation script (/var/lib/systemimager/scripts/base_image.master.0) and generate an updated version of the golden image: # /opt/hptc/config/sbin/cluster_config 9. After the cluster_config utility completes its processing, the client nodes are ready to be installed.
# shownode servers lvs n[135-136] 9. Create a symbolic link from the node names of the login nodes to the newly created master autoinstallation script. Note that the node name is appended with a .sh extension: for i in n135 n136 do ln -sf login.master.0 $i.sh done 10.
NOTE: With software RAID, the Linux boot loader requires the /boot partition to be mirrored (RAID1). In addition, the swap partitions are not raided (with striping) because the operating system stripes them automatically. 6.
E.3.1 Enable Static Disk Configuration Dynamic disk configuration method is the default behavior for the HP XC imaging environment. Before you can use the static disk configuration method to customize the client disk configuration you must enable it. You cannot use a combination of the two methods simultaneously. Follow this procedure to enable static disk configuration: 1. Use the text editor of your choice to open the following file: /etc/systemimager/systemimager.conf 2.
# setnode --resync --all # stopsys # startsys --image_and_boot Wait until the stopsys command completes before invoking the startsys command. E.3.2.2 Example 2: Creating a New .conf File and Associated Master Autoinstallation Script If necessary, you can create your own master autoinstallation script with static disk configuration included from a customized .conf file by following the procedure shown here.
F Node Roles, Services, and the Default Configuration This appendix addresses the following topics: • “Default Node Role Assignments” (page 215) • “Special Considerations for Modifying Node Role Assignments” (page 215) • “Role Definitions” (page 216) F.1 Default Node Role Assignments Table F-1 lists the default role assignments. The default assignments are based on the number of total nodes in the system.
• • • • • • If you expect LSF to be accessible from outside the HP XC system, all nodes with the resource_management role must also be configured with the external role and have the appropriate hardware and wiring to directly access the external network. You must assign the disk_io role to any node that is exporting SAN storage. By default, no nodes are configured with the login role. Assigning a login role to a node enables the LVS director service.
• “Common Role” (page 218) • “Compute Role” (page 218) • “Console_network Role” (page 218) • “Disk_io Role” (page 219) • “External Role” (page 219) • “Ldap_server Role” (page 219) • “Login Role” (page 219) • “Management Hub Role” (page 219) • “Management Server Role” (page 220) • “NIS Server Role” (page 220) • “Node Management Role” (page 220) • “Resource Management Role” (page 221) • “Voltaire_hsm Role” (page 221) You can define multiple roles on any node.
F.3.3 Cisco_hsm Role This role defines where the optional Cisco InfiniBand Host Based Subnet Manager runs. The service associated with this role is called ib_sm. For more information about when this role is required, see Section F.2.1 (page 215). HP recommends that you assign this role to the head node and another service node. Assigning this role to another service node enables the Subnet Manager, and therefore the InfiniBand network, to continue to function even if the head node is down.
• Console access (cpaccess) F.3.7 Disk_io Role Nodes with the disk_io role provide access to storage and file systems mounted locally on the node. This role can be located on any node that provides local file system access to all nodes using NFS. Assign this role to any node that is exporting SAN storage. The configuration and management database for the NFS Server service supplied by this role is nfs_server.
• Supermon aggregator (supermond) • Syslogng (syslogng_forward) These services are used to support scaling of the cluster. Nodes with this role provide local storage for aggregation of system logs and performance information. Management hub services typically report up to the node with the management_server role. You can assign the management_hub role to several nodes, and HP recommends that you consider using one management hub for every 64 to 128 nodes.
The configuration and management database, in which all management configuration information is stored, runs on a node with this role. The power manager service manages the powering on and off of nodes in the system. The MPI interconnect settings are managed by nodes running this role. The image server service provides images from the SystemImager tool to all client nodes. The NTP server synchronizes the time on all nodes.
G Using the cluster_config Command-Line Menu This appendix describes how to use the configuration command-line menu that appears when you run the cluster_config utility. It addresses the following topics: • “Overview of the cluster_config Command-Line Menu” (page 223) • “Displaying Node Configuration Information” (page 223) • “Modifying a Node” (page 224) • “Analyzing and Comparing Current Role Assignments to HP Recommendations” (page 226) • “Customizing Service and Client Configurations” (page 227) G.
G.3 Modifying a Node From the command-line menu of the cluster_config utility, enter the letter m to modify node role assignments and Ethernet connections: [L]ist Nodes, [M]odify Nodes, [A]nalyze, [H]elp, [P]roceed, [Q]uit: m You are prompted to supply the node name of the node you want to modify. All operations you perform from this point are performed on this node until you specify a different node name.
• Enter the letter b to return to the previous menu. G.5 Modifying Node Role Assignments The cluster configuration menu enables you to assign roles to specific nodes. “Role Definitions” (page 216) provides definitions of all node roles and the services they provide. Follow this procedure to modify role assignments: 1. From the command-line menu of the cluster_config utility, enter the letter m : [L]ist Nodes, [M]odify Nodes, [A]nalyze, [H]elp, [P]roceed, [Q]uit: m 2.
G.6 Analyzing and Comparing Current Role Assignments to HP Recommendations From the command-line menu of the cluster_config utility, enter the letter a to analyze current node role assignments with those recommended by HP. [L]ist Nodes, [M]odify Nodes, [A]nalyze, [H]elp, [P]roceed, [Q]uit: a HP recommends using this option any time you run the cluster_config utility, even when you are configuring the system for the first time.
1 2 voltaire_hsm (optional) Note: n499 does not have external connection recommended by resource_management Role Rec: Role Recommended HN Req: Head Node Required HN Rec: Head Node Recommended Exc Rec: Exclusivity Recommended Ext Req: External Connection Required Ext Rec: External Connection Recommended Table G-3 provides an explanation of the analysis.
From the command-line menu of the cluster_config utility, enter the letter p to continue the system configuration process. [L]ist Nodes, [M]odify Nodes, [A]nalyze, [H]elp, [P]roceed, [Q]uit: p The following prompt appears: [S]ervices Config, [P]roceed, [Q]uit: Do one of the following: • Enter the letter s to perform customized services configuration on the nodes in the system. This option is intended for experienced HP XC administrators who want to customize service servers and clients.
from the list of cmf servers if you wanted that specific node not to serve that service, even though it was configured to do so by default. Example: Disable a Client Similarly, to disable node n1 as a client of the supermond service, enter the node attribute na_disable_client.supermond to the affected node. In this instance, it is not the server but a client of the service that is affected.
Table G-4 Service Configuration Command Descriptions (continued) Command Description and Sample Use [d]estroy attribute_name Destroys an existing attribute with a specific name; it is the opposite of the create function. All occurrences of this attribute on a node or nodes are removed as well. You will receive an error if you try to destroy an attribute that does not exist. Sample use: svcs> destroy na_disable_server.
Table G-4 Service Configuration Command Descriptions (continued) Command Description and Sample Use [h]elp Displays a help message. Sample use: svcs> help [b]ack Returns to the previous menu. Sample use: svcs> back G.
H Determining the Network Type The information in this appendix applies only to cluster platforms with a QsNetII interconnect. During the processing of the cluster_config utility, the swmlogger gconfig script prompts you to supply the network type of the system. The network type reflects the maximum number of ports the switch can support, and the network type is used to create the qsnet diagnostics database.
I LSF and SLURM Environment Variables This appendix lists the default values for LSF and SLURM environment variables that are set by the HP XC System Software installation process. For more information about setting LSF environment variables and parameters listed in Table I-1, see the Platform LSF Reference manual.
Table I-1 Default Installation Values for LSF and SLURM (continued) Environment Variable Default Value Description Where is this Value Stored? LSF_NON_PRIVILEGED_PORTS Y When LSF commands are run by the root user, this variable configures them to communicate using non-privileged ports (> 1024). lsf.conf file LSB_RLA_UPDATE 120 Controls how often LSF synchronizes with SLURM. This variable is not needed for standard LSF. lsf.conf file LSF_LICENSE none Defines the location of the LSF license.
Table I-1 Default Installation Values for LSF and SLURM (continued) Where is this Value Stored? Environment Variable Default Value Description One LSF partition RootOnly YES Specifies that only the root user or the SLURM administrator can create allocations for normal user jobs. slurm.conf file One LSF partition Shared FORCE Specifies that more than one job can run on the same node. LSF uses this facility to support preemption and scheduling of multiple serial jobs on the same node.
J Customizing the SLURM Configuration This appendix describes customizations you can make to the /hptc_cluster/slurm/etc/ slurm.conf SLURM configuration file. It addresses the following topics: • “Assigning Features” (page 239) • “Creating Additional SLURM Partitions” (page 239) • “Customizing SLURM for SVA” (page 239) J.1 Assigning Features Assigning features to nodes is common if the compute resources of the cluster are not consistent.
The following procedure provides an example of a cluster that has five nodes: node 5 is the head node, nodes 1 and 2 are visualization nodes, and nodes 3 and 4 are compute nodes. Using that example, you must modify the slurm.conf file to create two partitions: 1. Use the text editor of your choice to edit the /hptc_cluster/slurm/etc/slurm.conf file. 2.
K OVP Sample Output This appendix provides sample output from the OVP utility, which verifies successful installation and configuration of software and hardware components. # ovp --verbose LOG FILE NAME: ovp_n16_101608.log CMDLINE ARGS: --verbose TEST EXEC DIR: /hptc_cluster/ovp/ovp_n16_101608.tests XC CLUSTER VERIFICATION PROCEDURE Verify connectivity: Testing etc_hosts_integrity ... There are 15 IP addresses to ping. A total of 15 addresses were pinged. Test completed successfully.
Compaq: UP v9.2 Checking output from command. +++ PASSED +++ Verify SLURM: Testing spconfig ... Starting the command: /opt/hptc/bin/sinfo --exact --format="%m %N" Here is the output from the command: MEMORY NODELIST 3892 n[14-16] +++ PASSED +++ Testing daemon_responds ... Starting the command: /opt/hptc/bin/scontrol ping Here is the output from the command: Slurmctld(primary/backup) at n16/(NULL) are UP/DOWN Checking output from scontrol. +++ PASSED +++ Testing partition_state ...
Copyright 1992-2008 Platform Computing Corporation My cluster name is hptclsf My master name is lsfhost.localdomain Checking output from command. +++ PASSED +++ Testing hosts_static_resource_info ... Running 'lshosts -w'. Checking output from lshosts. Running 'controllsf show' to determine virtual hostname. Checking output from controllsf. Virtual hostname is lsfhost.localdomain Comparing ncpus from Lsf lshosts to Slurm cpu count. The Lsf and Slurm cpu count are in sync.
Checked 47 services. Checking hosts... Checked 5 hosts. Checking host groups... Checked 12 host groups. Checking service groups... Checked 6 service groups. Checking contacts... Checked 2 contacts. Checking contact groups... Checked 1 contact groups. Checking service escalations... Checked 0 service escalations. Checking service dependencies... Checked 36 service dependencies. Checking host escalations... Checked 0 host escalations. Checking host dependencies... Checked 0 host dependencies.
The nrg command "/opt/hptc/nagios/bin/nrg --mode summary" reported 3 services in the "Warning" state. The nrg command "/opt/hptc/nagios/bin/nrg --mode summary" reported 2 services in the "Critical" state. Suggest running "nrg --mode analyze" to get an analysis of why services may be failing. +++ PASSED +++ WARNING: Although this test passed, it wrote messages to standard error, which could indicate a problem. Please check the log file for details. Verify xring: Testing xring ...
The headnode is excluded from the memory test. Number of nodes allocated for this test is 2 Job [104] is submitted to queue [interactive]. [[Waiting for dispatch ...]] [[Starting on lsfhost.localdomain]] Detailed streams results for each node can be found in /hptc_cluster/ovp/ovp_n16_101608.tests/tests/100.perf_health/40.memory if the --keep flag was specified. Streams memory results summary (all values in mBytes/sec): min: 1010.420000 max: 1147.150300 median: 1029.866200 mean: 1062.478833 range: 136.
mean: 1712.393333 range: 1339.250000 variance: 578441.442633 std_dev: 760.553379 All nodes were in range for this test. +++ PASSED +++ Testing network_unidirectional ... Number of nodes allocated for this test is 3 Job [107] is submitted to queue [interactive]. [[Waiting for dispatch ...]] [[Starting on lsfhost.localdomain]] [0:n14:1] ping-pong 2839.26 usec/msg 1408.82 MB/sec [1:n15:2] ping-pong 5273.43 usec/msg 758.52 MB/sec [2:n16:0] ping-pong 5271.22 usec/msg 758.
Glossary A administration branch The half (branch) of the administration network that contains all of the general-purpose administration ports to the nodes of the HP XC system. administration network The private network within the HP XC system that is used for administrative operations. availability set An association of two individual nodes so that one node acts as the first server and the other node acts as the second server of a service. See also improved availability, availability tool.
operating system and its loader. Together, these provide a standard environment for booting an operating system and running preboot applications. enclosure The hardware and software infrastructure that houses HP BladeSystem servers. extensible firmware interface See EFI. external network node A node that is connected to a network external to the HP XC system. F fairshare An LSF job-scheduling policy that specifies how resources should be shared by competing users.
image server A node specifically designated to hold images that will be distributed to one or more client systems. In a standard HP XC installation, the head node acts as the image server and golden client. improved availability A service availability infrastructure that is built into the HP XC system software to enable an availability tool to fail over a subset of eligible services to nodes that have been designated as a second server of the service See also availability set, availability tool.
LVS Linux Virtual Server. Provides a centralized login capability for system users. LVS handles incoming login requests and directs them to a node with a login role. M Management Processor See MP. master host See LSF master host. MCS An optional integrated system that uses chilled water technology to triple the standard cooling capacity of a single rack. This system helps take the heat out of high-density deployments of servers and blades, enabling greater densities in data centers.
onboard administrator See OA. P parallel application An application that uses a distributed programming model and can run on multiple processors. An HP XC MPI application is a parallel application. That is, all interprocessor communication within an HP XC parallel application is performed through calls to the MPI message passing library. PXE Preboot Execution Environment.
an HP XC system, the use of SMP technology increases the number of CPUs (amount of computational power) available per unit of space. ssh Secure Shell. A shell program for logging in to and executing commands on a remote computer. It can provide secure encrypted communications between two untrusted hosts over an insecure network. standard LSF A workload manager for any kind of batch job.
Index A adduser command, 89 administration network activating, 68 testing, 131 using as interconnect network, 66 administrator password ProCurve switch, 64 Anaconda kickstart, 137 avail_node_management role, 217 availability role, 217 availability set choosing nodes as members, 29 configuring with cluster_config, 101 defined, 27 availability tool, 27 Serviceguard, 28 starting, 125 verifying operation, 130 B back up CMDB, 134 CMDB before cluster_config, 101 SFS server, 24, 140 base image removing, 115 base
D database cannot connect to MySQL server, 177 database administrator password, 54 database backup, 134 database server configuring, 108 default disk partition sizes, 37 file system layout, 37 node role assignments, 215 system configuration, 215 dense node numbering, 51 development environment tools, 87 disabled node inserted in database, 176 discover command, 63 BL2x220c server blade, 70 command line options, 65 discovering enclosures, 70 discovering nodes only, 70 double density serve blades, 70 - -extend
default role assignments, 215 golden client, 35 installation, 35 mirroring, 40 naming convention, 53 high availability (see improved availability) host name guidelines, 191 HowTo website, 17 HP documentation providing feedback for, 22 HP Integrity BMC/IPMI password, 79 HP MPI defined, 35 HP ProLiant BL2x220c discovering, 70 HP ProLiant DL140 discovering, 66 HP ProLiant DL145 discovering, 66 HP Remote Graphics Software (see RGS) HP Scalable File Share (see SFS) HP Scalable Visualization Array (see SVA) HP Se
Intel compiler, 26, 87 interconnect configure switch monitoring cards, 78, 197 diagnostic tests, 132 Myrinet switch controller card, 200 nfiniBand switch controller card, 201 on administration network, 66 Quadrics switch controller card, 197 internal node numbering , 51 IP address for Ethernet connection on head node, 55 in DNS configuration, 25 ProCurve switch, 174 IP address base base_addr.ini file, 55 base_addrV2.
Onboard Administrator, 72 MAC address of root administration switch, 64 managedb command, 134 management_hub role, 219 management_server role, 220 manpages, 21 Maui Scheduler, 165 defined, 165 documentation, 165 installation procedure, 166 readiness criteria, 165 maximum number of nodes, 54 MCS configuration file, 91 configuring snmptrapd service, 97, 109 enabling SNMP trap monitoring, 127 mcs.
P PALLAS, 132 parallel distributed shell (see pdsh) password BMC/IPMI on HP Integrity systems, 79 configuration and management database, 57 console port devices, 64 database administrator, 57 guidelines, 191 root, 42 patches, 59 PBS Professional adding nodes, 161 configuration tasks, 160 execution nodes, 161 HP MPI tasks, 163 installation planning, 159 installation process, 160 overview, 159 service daemons, 162 user setup, 162 pdsh defined, 35 performance maximizing on compute notes, 210 permission denied
eligible for improved availability, 31 making highly available, 26 SFS defined, 80 documentation, 80 documentation website, 81 installing, 80 mounting /hptc_cluster file system, 61 SFS client installation, 80 SFS server backing up, 24 si_rmimage command, 115 SIOCSIFFLAGS error, 186 SLURM assigning features, 239 configuration file, 123 configuring, 111 configuring failover, 33 creating additional partitions, 239 customizing, 239 customizing the slurm.
translator scipt, 28 troubleshooting, 173 using the nrg command, 133 workstation nodes, 63 changing database name, 88 U XC software version, 138 XC.
