[Contents] [Prev] [Next] [End]
This chapter describes how to customize your batch jobs to take advantage of LSF and LSF Batch features.
When LSF Batch runs a batch job it sets several environment variables. Batch jobs can use the values of these variables to control how they execute. The environment variables set by LSF Batch are:
Each parallel programming package has different requirements for specifying and communicating with all the hosts used by a parallel job. LSF is not tailored to work with a specific parallel programming package. Instead, LSF provides a generic interface so that any parallel package can be supported by writing shell scripts or wrapper programs. Example shell scripts are provided for running PVM, P4, MPI, and POE programs as parallel batch jobs.
The hosts allocated for the parallel job are passed to the batch job in the LSB_HOSTS environment variable. Some applications can take this list of hosts directly as a command line parameter. For other applications you may need to process the host list. The following example shows a /bin/sh script that processes all the hosts in the host list, including identifying the host where the job script is executing.
#!/bin/sh # Process the list of host names in LSB_HOSTS for host in $LSB_HOSTS ; do handle_host $host done
LSF comes with a few scripts for running parallel jobs under LSF Batch, such as pvmjob, poejob, mpijob, p4job, etc. These scripts are installed In the LSF_BINDIR as defined in lsf.conf file. You can modify these scripts to support more parallel packages.
For simple parallel jobs you can use the lstools commands to start parts of the job on other hosts. Because the lstools commands handle signals transparently, LSF Batch can suspend and resume all components of your job without additional programming.
The simplest parallel job runs an identical copy of the executable on every host. The lsgrun command takes a list of host names and runs the specified task on each host. The lsgrun -p option specifies that the task should be run in parallel on each host. The example below submits a job that uses lsgrun to run myjob on all the selected batch hosts in parallel:
% bsub -n 10 'lsgrun -p -m "$LSB_HOSTS" myjob' Job <3856> is submitted to default queue <normal>.
For more complicated jobs, you can write a shell script that runs lsrun in the background to start each component.
For parallel jobs that have a variety of different components to run, you can use lsmake. Create a makefile that lists all the components of your batch job and then submit the lsmake command to LSF Batch. The following example shows a bsub command and Makefile for a simple parallel job.
% bsub -n 4 lsmake -f Parjob.makefile Job <3858> is submitted to default queue <normal>.
% cat Parjob.makefile # Makefile to run example parallel job using lsbatch and lsmake all: part1 part2 part3 part4 part1 part2 part3: myjob data.$@ part4: myjob2 data.part1 data.part2 data.part3
The batch job has four components. The first three components run the myjob command on the data.part1, data.part2 and data.part3 files. The fourth component runs the myjob2 command on all three data files. There are no dependencies between the components, so lsmake runs them in parallel.
PVM is a parallel programming system distributed by Oak Ridge National Laboratories. PVM programs are controlled by a file, the PVM hosts file, that contains host names and other information. The pvmjob shell script supplied with LSF can be used to run PVM programs as parallel LSF Batch jobs. The pvmjob script reads the LSF Batch environment variables, sets up the PVM hosts file and then runs the PVM job. If your PVM job needs special options in the hosts file, you can modify the pvmjob script.
For example, if the command line to run your PVM job is
% myjob data1 -o out1
the following command submits this job to LSF Batch to run on 10 hosts:
% bsub -n 10 pvmjob myjob data1 -o out1
Other parallel programming packages can be supported in the same way. The p4job shell script runs jobs that use the P4 parallel programming library. Other packages can be handled by creating similar scripts.
The Message Passing Interface (MPI) is a portable library that supports parallel programming. LSF supports MPICH, a joint implementation of MPI by Argonne National Laboratory and Mississippi State University. This version supports both TCP/IP and IBM's Message Passing Library (MPL) communication protocols.
LSF provides an mpijob shell script that you can use to submit MPI jobs to LSF Batch. The mpijob script writes the hosts allocated to the job by the LSF Batch system to a file and supplies the file as an option to MPICH's mpirun command. The syntax of the mpijob command is
mpijob option mpirun program [arguments]
where option is one of the following:
The following examples show how to use mpijob to submit MPI jobs to LSF Batch.
To submit a job requesting four hosts and using the default TCP/IP protocol, use
% bsub -n 4 mpijob mpirun myjob
Before you can submit a job to a particular pool of IBM SP-2 nodes, an LSF administrator must install the SP-2 ELIM. The SP-2 ELIM provides the pool number and lock status of each node.
To submit the same job to run on four nodes in pool 1 on an IBM SP-2 system using MPL, use
% bsub -n 4 -R "pool == 1" mpijob -mpl mpirun myjob
To submit the same job to run on four nodes in pool 1 that are not locked (dedicated to using the High Performance Switch) on an SP-2 system using MPL, use
% bsub -n 4 -q mpiq -R "pool == 1 && lock == 0" mpijob -mpl mpirun myjob
Before you can submit a job using the IBM SP-2 High Performance Switch in dedicated mode, an LSF administrator must set up a queue for automatic requeue on job failure. The job queue will automatically requeue a job that failed because an SP-2 node was locked after LSF Batch selected the node but before the job was dispatched.
The Parallel Operating Environment (POE) is an execution environment provided by IBM on SP-2 systems to hide the differences between serial and parallel execution.
LSF provides a poejob shell script that you can use to submit POE jobs to LSF Batch. The poejob script translates the hosts allocated to the job by the LSF Batch system into an appropriate POE host list and sets up environment variables necessary to run the job.
The poejob script does not set the MP_EUILIB and MP_EUIDEVICE environment variables, so you have to do this.
% setenv MP_EUILIB us
By default, MP_EUIDEVICE is css0. Or,
% setenv MP_EUILIB ip % setenv MP_EUIDEVICE en0
The following are examples of how to submit POE jobs.
To submit a job requesting four SP-2 nodes configured for the poeq queue, use
% bsub -n 4 -q poeq poejob myjob
By using LSF resource requirements, you can select appropriate nodes for your job.
To submit the same job requesting four SP-2 nodes from pool 2 configured for the poeq queue, use
% bsub -n 4 -R "pool == 2" -q poeq poejob myjob
To submit the same job requesting four SP-2 nodes from pool 2 with at least 20 megabytes of swap space, use
% bsub -n 4 -R "(pool == 2) && (swap > 20)" -q poeq poejob myjob
To submit the same job requesting four SP-2 nodes from pool 2 that are not locked (dedicated to using the High Performance Switch), use
% bsub -n 4 -R "(pool == 2) && (lock == 0)" -q poeq poejob myjob
The above examples use a script to run parallel jobs under LSF Batch. Alternatively, your LSF administrator could configure the script into your queue as a job starter. With a job starter configured at the queue, you can submit the above parallel jobs without having to type the script name. See 'Using A Job Starter' in the LSF Administrator's Guide for more information about a job starter.
To see if your queue already has a job starter defined, run the bqueues -l command.
Copyright © 1994-1997 Platform Computing Corporation.
All rights reserved.