HP XC System Software Administration Guide Version 3.2

The environment in which the job is launched contains SLURM and LSF-HPC with SLURM
environment variables that describe the job's allocation. SLURM srun commands in the user's
job use the SLURM environment variables to distribute the tasks throughout the allocation.
The integration of LSF-HPC with SLURM has one drawback: the bsub command's -i option
for providing input to the user job is not supported. A workaround is to provide any file input
directly to the job. The SLURM srun command supports an --input option (also available in
its short form as the -i option) that provides input to all tasks.
16.2.1.1 Job Starter Scripts
LSF-HPC with SLURM dispatches all jobs locally. The default installation of LSF-HPC with
SLURM on the HP XC system provides a job starter script that is configured for use by all LSF
queues. This job starter script adjusts the LSB_HOSTS and LSB_MCPU_HOSTS environment
variables to the correct resource values in the allocation. Then, the job starter script uses the srun
command to launch the user task on the first node in the allocation.
If this job starter script is not configured for a queue, the user jobs begin execution locally on the
LSF execution host. In this case, it is recommended that the user job uses one or more srun
commands to make use of the resources allocated to the job. Work done on the LSF execution
host competes for core time with the LSF-HPC with SLURM daemons, and could affect the overall
performance of LSF-HPC with SLURM on the HP XC system.
The bqueues -l command displays the full queue configuration, including whether or not a
job starter script has been configured. See the Platform LSF documentation or bqueues(1) for more
information on the use of this command.
For example, consider an LSF-HPC with SLURM configuration in which node n20 is the LSF
execution host and nodes n[1-10] are in the SLURM lsf partition. The default normal queue
contains the job starter script, but the unscripted queue does not have the job starter script
configured.
Example 16-2 Comparison of Queues and the Configuration of the Job Starter Script
$ bqueues -l normal | grep JOB_STARTER
JOB_STARTER: /opt/hptc/lsf/bin/job_starter.sh
$ bqueues -l unscripted | grep JOB_STARTER
JOB_STARTER:
$ bsub -Is hostname
Job <66> is submitted to the default queue <normal>.
<<Waiting for dispatch...>>
<<Starting on lsfhost.localdomain>>
n10
$ bsub -Is -q unscripted hostname
Job <67> is submitted to the default queue <unscripted>.
<<Waiting for dispatch...>>
<<Starting on lsfhost.localdomain>>
n20
This release of the HP XC System Software provides an LSF queue JOB_STARTER script, which
is configured for all default queues during HP XC installation. This JOB_STARTER script performs
three tasks:
• It creates an accurate LSB_HOSTS environment variable.
• It creates an accurate LSB_MCPU_HOSTS environment variable.
• It uses a SLURM srun command to launch a user's interactive job on the first allocated
compute node.
192 Managing LSF