Installation guide

ManualsBrandsRed Hat ManualsOtherAPPLICATION STACK 1.2 RELEASE

Release Notes

Scyld ClusterWare Release 4.9.0-490g0000

About This Release

Scyld ClusterWare Release 4.9.0-490g0000 (released April 22, 2011) is the latest update to Scyld ClusterWare 4.

Scyld ClusterWare 4.9.0 expects to execute in a Red Hat RHEL4 Update 9 or CentOS 4.9 base distribution environment, each

having been updated to the latest RHEL4/CentOS4 errata (https://rhn.redhat.com/errata/rhel4as-errata.html) as of the Scyld

ClusterWare 4.9.0 release date. Any compatibility issues between Scyld ClusterWare 4.9.0 and RHEL4 are documented on

the Penguin Computing Support Portal at http://www.penguincomputing.com/support.

Visit http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/index.html to view the Red Hat Enterprise Linux 4 4.9

Release Notes and 4.9 Technical Notes.

For the most up-to-date product documentation and other helpful information, visit the Penguin Computing Support Portal.

Important

Before continuing, make sure you are reading the most recent Release Notes, which can be found on the Penguin Computing

Support Portal at http://www.penguincomputing.com/ﬁles/scyld-docs/CW4/ReleaseNotes.pdf. The most recent version will

accurately reﬂect the current state of the Scyld ClusterWare yum repository of rpms that you are about to install. You may

consult the Installation Guide for its more generic and expansive details about the installation process. The Release Notes

document more speciﬁcally describes how to upgrade an earlier version of Scyld ClusterWare to Scyld ClusterWare 4.9.0

(see the Section called Upgrading Earlier Release of Scyld ClusterWare to Scyld ClusterWare 4.9.0), or how to install Scyld

ClusterWare 4.9.0 as a fresh install (see

the Section called Installing Scyld ClusterWare 4.9.0 on a Non-Scyld ClusterWare

System).

Important for clusters using Inﬁniband

Updating from RHEL4 Update 6 (or CentOS 4.6) to Update 7 will update the Open Fabric Inﬁniband (OFED) software

stack from version 1.2 to 1.3. Updating to Update 8 or beyond will update to OFED version 1.4. Scyld ClusterWare itself

is compatible with either OFED 1.2, 1.3, or 1.4, but OFED 1.3 and 1.4 may be incompatible with MPI stacks supplied

by certain ISV applications. Penguin Computing recommends that you do not upgrade Inﬁniband clusters beyond RHEL4

Update 6 without ﬁrst verifying that your applications that use Inﬁniband are compatible with OFED 1.4. See the Section

called OFED 1.2 vs. OFED 1.3 Issues in the Section called Known Issues And Workarounds for details.

Important for clusters using Panasas storage

If the cluster uses Panasas storage, then you must ensure that a Panasas kernel module is available that matches the

Scyld ClusterWare kernel you are about to install: 2.6.9-100.490g0000.ELsmp. Login to your Panasas account at

http://www.my.panasas.com, click on the Downloads tab, then click on DirectFLOW Client for Linux and then on Search

DirectFLOW Release, and do a Keyword search for 490g0000. If you ﬁnd a Panasas rpm matching the to-be-installed

2.6.9-100.490g0000.ELsmp kernel, then download that rpm and continue with the Scyld ClusterWare update or install.

Install the Panasas rpm after you ﬁnish installing the associated 2.6.9-100.490g0000.ELsmp kernel. If you do not ﬁnd an

appropriate Panasas rpm, then do not install this latest Scyld ClusterWare 4.9.0. The Panasas storage will not work with the

2.6.9-100.490g0000.ELsmp kernel without a matching Panasas kernel module.

Summary of content (24 pages)

PAGE 1
Release Notes Scyld ClusterWare Release 4.9.0-490g0000 About This Release Scyld ClusterWare Release 4.9.0-490g0000 (released April 22, 2011) is the latest update to Scyld ClusterWare 4. Scyld ClusterWare 4.9.0 expects to execute in a Red Hat RHEL4 Update 9 or CentOS 4.9 base distribution environment, each having been updated to the latest RHEL4/CentOS4 errata (https://rhn.redhat.com/errata/rhel4as-errata.html) as of the Scyld ClusterWare 4.9.0 release date.
PAGE 2
Release Notes Upgrading Earlier Release of Scyld ClusterWare to Scyld ClusterWare 4.9.0 When upgrading from an earlier Scyld ClusterWare 4 version to Scyld ClusterWare 4.9.0, you should perform the following steps: 1. Stop the Beowulf cluster: /sbin/service beowulf stop 2. Clean the yum cache to a known state: yum clean all 3. The latest Red Hat errata is RHEL4 Update 9. Penguin Computing recommends that Infiniband clusters do not upgrade beyond RHEL4 Update 6 (or CentOS 4.
PAGE 3
Release Notes If yum fails with a Transaction Check Error that complains that a base distribution rpm is newer than the Scyld ClusterWare rpm that is attempting to replace it, then you must manually install the downlevel Scyld ClusterWare rpm(s). For example, if the complaint is about the kernel rpms, then do: cd /var/cache/yum/ ls cw-*/packages/kernel-* and locate the newest Scyld ClusterWare 2.6.9-100.490g0000.ELsmp kernel, which should reside in either cw-core/packages/ or cw-updates/packages/.
PAGE 4
Release Notes Installing Scyld ClusterWare 4.9.0 on a Non-Scyld ClusterWare System When installing Scyld ClusterWare 4.9.0 on a system that does not yet contain Scyld ClusterWare, you should perform the following steps: 1. Clean the yum cache to a known state: yum clean all 2. The latest Red Hat errata is RHEL4 Update 9. Penguin Computing recommends that Infiniband clusters do not upgrade beyond RHEL4 Update 6 (or CentOS 4.6). See the Section called OFED 1.2 vs. OFED 1.
PAGE 5
Release Notes 7. If the cluster uses Panasas storage, then you should have already downloaded the Panasas rpm that matches the Scyld ClusterWare 4.9.0 kernel you have just installed. Now install the Panasas rpm using rpm -i. 8. Configure the network for Scyld ClusterWare: run /usr/sbin/beonetconf to specify the cluster interface, the maximum number of compute nodes, and the beginning IP address of the first compute node. See the Installation Guide for more details. 9.
PAGE 6
Release Notes Generally speaking, be careful when making changes to /etc/beowulf/config, as mistakes may leave your cluster in a non-working state. For example, in general do not manually change the existing keyword entries for interface, nodes, iprange, and nodeassign, as those are more accurately manipulated by the /usr/sbin/beonetconf or /usr/sbin/beosetup commands. The kernelimage and node entries are automatically managed by Beowulf services and should not be merged.
PAGE 7
Release Notes /usr/bin/htpasswd /etc/httpd/scyld-imf/htpasswd-users admin To use Scyld IMF, point your Web browser at the URL http://MasterNode/scyld-imf, where MasterNode is the name or IP address of the master node, whereupon you are prompted for a valid username (i.e., admin) and password (which was initialized as described above). See the Administrator’s Guide for more details.
PAGE 8
Release Notes Optionally adjust the size limit for locked memory OpenIB, MVAPICH, and MVAPICH2 require an override to the limit of how much memory can be locked. Scyld ClusterWare adds a memlock override entry to /etc/security/limits.conf during a Scyld ClusterWare upgrade (if the override entry does not already exist in that file), regardless of whether or not Infiniband is present in the cluster. The new override line, * - memlock unlimited raises the limit to unlimited.
PAGE 9
Release Notes options ip_conntrack hashsize=Nhash to /etc/modprobe.conf, and adding: net.ipv4.ip_conntrack_max=Nmax to /etc/sysctl.conf. Optionally reconfigure node names You may declare site-specific alternative node names for cluster nodes by adding entries to /etc/beowulf/config. The syntax for a node name entry is: nodename format-string [IPv4offset] [netgroup] For example, nodename node%N allows the user to refer to node 4 using the traditional .
PAGE 10
Release Notes Optionally increase the max number of processID values The kernel defaults to using a maximum of 32,768 processID values. Scyld ClusterWare automatically increases this default to 98,304 [= 3*32768], which likely is adequate for small- to medium-size clusters and which keeps pid values at a familiar 5-column width maximum.
PAGE 11
Release Notes Notable Feature Enhancements And Bug Fixes Beyond Scyld ClusterWare 4.3.1 New in Scyld ClusterWare 4.9.0 - Scyld Release 490g0000 1. The base kernel is upgraded to 2.6.9-100.490g0000.ELsmp. See https://rhn.redhat.com/errata/RHSA-2011-0263.html for details. 2. Fixes various BProc bpmaster bugs that either caused a silent death of the daemon or a segfault, most commonly associated with the death or rebooting of a compute node. 3.
PAGE 12
Release Notes 2. OpenMPI is upgraded to version 1.5.1. This release yet again restructures the locations of the compiler-specific libraries, executable binaries, manpages, and environment modules, but now each new release of OpenMPI can gracefully coexist with earlier releases, and existing applications that were built against an earlier version do not need to be immediately rebuilt against this new version. See the Section called Upgrading Earlier Release of Scyld ClusterWare to Scyld ClusterWare 4.9.
PAGE 13
Release Notes 3. The Scyld ClusterWare e1000e Ethernet driver is upgraded to version 1.2.17. This driver derives from source found at http://sourceforge.net/projects/e1000/files/. We recommend using this Scyld ClusterWare driver instead of the native RHEL4-U8 driver (version 0.3.3.3-k6). 4. Introduces beoweb, a web server that runs on a cluster’s master node to facilitate remote job submission and cluster monitoring.
PAGE 14
Release Notes New in Scyld ClusterWare 4.8.1 Update - Scyld Release 481g0007 1. The base kernel is upgraded to 2.6.9-89.0.29 (481g0004). See https://rhn.redhat.com/errata/RHSA-2010-0718.html for details. 2. The igb network driver has integrated an upstream fix to more frequently update /proc/net/dev statistics, which means that beostat and IMF more accurately report network usage for chipsets that use that driver. 3. beostatus no longer requires that TORQUE be installed. 4.
PAGE 15
Release Notes 6. The beostat service that supplies cluster performance statistics to Scyld IMF, beostatus, ganglia and other cluster status visualization utilities now understands bonded network devices. Previously, network statistics were double-reported: counting both the aggregated bonded pseudo-device and the individual devices that comprise the bonded device. 7. Eliminates a bogus recvstats syslog message of the form "Received stats from IP addr" that occasionally appeared as a compute node starts up.
PAGE 16
Release Notes 7. Fixes a bug where cluster startup would leave temporary files in /tmp/. These are now properly deleted. 8. The beoserv tftp server, which executes on the master node, now only listens on the private cluster interface. Previously it listened on all interfaces for tftp requests. Additionally, previously tftp requests only retrieved files that resided in the /var/beowulf/boot/ directory. Now it treats a requested filename as being a pathname relative to that base directory, i.e.
PAGE 17
Release Notes 3. Scyld ClusterWare includes the env-modules environment-modules package, which enables the dynamic modification of a user’s environment via modulefiles. Each modulefile contains the information needed to configure the shell for an application, allowing a user to easily switch between applications with a simple module switch command that resets environment variables like PATH and LD_LIBRARY_PATH.
PAGE 18
Release Notes Issues with TORQUE version 2.5.3 TORQUE version 2.5.3, which was introduced in Scyld Release 482g0000 and withdrawn in Scyld Release 482g0005, has seemingly introduced various problems running TORQUE jobs. If your cluster has experienced new TORQUE problems that you believe appeared after an upgrade to version 2.5.3, then we suggest reverting to an earlier version: rpm -qa | grep torque-2.5.3 | xargs rpm -e --nodeps yum install torque will install the preferred TORQUE version. OFED 1.2 vs.
PAGE 19
Release Notes libcxgb3-static libehca libibcm libibcm-devel libibcm-static libibcommon libibcommon-devel libibcommon-static libibmad libibmad-devel libibmad-static libibumad libibumad-devel libibumad-static libibverbs libibverbs-devel libibverbs-static libibverbs-utils libipathverbs libmlx4 libmthca libmthca-devel libmthca-static libnes librdmacm librdmacm-devel librdmacm-static librdmacm-utils libsdp ofed-docs openib openib-diags openib-mstflint openib-perftest openib-srptools openib-tvflash opensm opensm-
PAGE 20
Release Notes Caution using beosetup The /usr/sbin/beosetup utility is deprecated. At this time, we do not recommend using beosetup for observing or altering the cluster state while new compute nodes are booting. Caution using ethtool ethtool -G may be used to set the network interface’s ring buffer size. Performing this action on an interface that uses the forcedeth driver will cause that interface to stop working. Use ethtool -i interface-name to view the interface-name and driver pairing.
PAGE 21
Release Notes The modulefiles have pathnames /opt/scyld/modulefiles/openmpi/compiler/version, where version is a file that amends $PATH, $LD_LIBRARY_PATH, and $MANPATH with pathnames that point into the associated compiler-specific /opt/scyld/openmpi/version/compiler/ subdirectories.
PAGE 22
Release Notes If that reconfiguration is also not possible, you may need to increase the default Scyld ClusterWare timeout used by the node to a value safely greater than the STP delay: e.g., add rootfs_timeout=120 getfile_timeout=120 to the /etc/beowulf/config kernelcommandline entry to increase the timeouts to 120 seconds.
PAGE 23
Release Notes /etc/security/limits.conf /etc/sysconfig/syslog Additionally, Scyld ClusterWare uses /sbin/chkconfig to enable nfs. Running nscd service on master node may cause kickbackdaemon to misbehave The nscd (Name Service Cache Daemon) service executes by default on each compute node. However, if this service is also enabled on the master node, then it may cause the Scyld ClusterWare name service kickbackdaemon to misbehave.
PAGE 24
Release Notes RAM filesystem space on the node. It is likely that your cluster’s users and applications do not require all the international locale data that is present in the default file. With care, the cluster administrator may choose to rebuild locale-archive with a greatly reduced set of locales. For example, on a quiescent master node: cd /usr/lib mv locale locale.orig mkdir locale cp -a locale.