[Contents] [Prev] [Next] [End]

Appendix D. Sample System Support

This section describes how LSF can be configured to support specific systems. Several systems are discussed: IRIX 6 processor sets, IBM SP-2, Cray Research NQS, and Atria Clearcase.

IRIX 6 Processor Sets

IRIX 6 allows the processors in a multiprocessor system to be divided into groups of processors call processor sets. IRIX 6 provides facilities to allow a user to define processor sets, and to run processes onto specific processor sets.

The pset(1M) command allows administrators to set up and manipulate processor sets and associate processes with sets. Once a process is associated with a processor set, the process and all its children will be scheduled only on the processors in that set. The definition of the processor set can be dynamically changed to increase or reduce the number of processors a process can be scheduled on.

LSF can be made to interface with processor sets by using pset(1M) in the queue-level pre- and post-execution commands (see 'Queue-Level Pre-/Post-Execution Commands'). This allows batch jobs to be assigned to certain processors.

Since the pset command must be run as root but, by default, queue-level pre- and post- execution commands are run as the user that submitted the command, you must define LSB_PRE_POST_EXEC_USER=root in /etc/lsf.sudoers. See 'The lsf.sudoers File' for details.

The following gives examples on how to handle different processor allocation situations for an 8-processor machine.

Time-Based Processor Allocation

During the day you want to ensure that batch jobs only use four processors with the remaining four dedicated for interactive users. During the night you want the batch jobs to be able to use all processors.

This can be accomplished by:

Step 1.
Create a 'batch' processor set and an 'interactive' processor set with half the processors in each set. This can be done by setting up an /etc/pssettab file as follows:
# <symbolic name>    <pset id>    <processor list>
batch       100        0,1,2,3
interactive 101        4,5,6,7
Run the pset command as root to tell the operating system to read the pssettab file:
% pset -i -v
Step 2.
All batch queues should specify the following PRE_EXEC parameter:
PRE_EXEC = pset -p $LS_JOBPID batch
The LS_JOBPID will be set to the process identifier of the batch job process. The pset command will tell IRIX to schedule that process and any of its children in the batch processor set.
Step 3.
Set up two cron jobs one of which runs every morning at 8 a.m. and the other every evening at 6 p.m. The cron job running at 8 a.m. should execute the command:
# Move processors 4,5,6,7 out of the batch processor set
pset -s batch \!4,5,6,7
The cron job running at 6 p.m. should execute the command:
#Move processors 4,5,6,7 into the batch processor set.
pset -s batch +4,5,6,7
Alternatively, you can use LSF JobScheduler to run this periodic job.

User-Based Processor Allocation

You want to give a particular user (Jane) exclusive access to one processor if she has jobs to run. Otherwise, users should be able to use all eight processors for batch jobs.

This can be accomplished by:

Step 1.
Create a processor set called 'excl' and another processor set 'batch' for normal batch jobs. The 'batch' processor set can initially contain all processors and the 'excl' set contains only processor 0.
Step 2.
Create a queue that will only accept jobs from Jane. The pre- and post-execution commands can be used to shuffle processors between processor sets such that Jane will get exclusive access to a processor:
Begin Queue
QUEUE_NAME    = exclusive
PRIORITY      = 43
USERS         = jane
# Move processor 0 from batch processor set to excl processor set
# Associate the job with the excl set.
PRE_EXEC      = pset -s excl +0; pset -s batch -0; pset -p $LS_JOBPID excl
#Move processor 0 back to the batch processor set
POST_EXEC     = pset -s excl -0; pset -s batch +0
DESCRIPTION   = Provides exclusive access to a processor for Jane's jobs
End Queue
Other queues should have their pre-execution command set to:
PRE_EXEC      = pset -p $LS_JOBPID batch

Other Situations

More complicated situations can be handled by using scripts in the pre- and post-execution commands which check for other conditions. For example in the above 'User-Based Processor Allocation' case, if you wanted to give Jane up to four processors to use (but not more), the pre-execution script could use pset to determine how many processors were already in the 'excl' set and move an additional processor from the 'batch' set into the 'excl' set until the 'excl' set has four processors.

IBM SP-2 Support

An IBM SP-2 system consists of multiple nodes running AIX. The system can be configured with a high-performance switch to allow high bandwidth, low latency communication between the nodes. The allocation of the switch to jobs as well as the division of nodes into pools is controlled by the SP-2 Resource Manager.

IBM's Parallel Operating Environment (POE) interfaces with the Resource Manager to allow users to run parallel jobs requiring dedicated access to the high performance switch.

The following are provided in LSF to support POE jobs running under LSF:

The 'poejob' script is installed as part of the standard installation procedure. The SP-2-specific ELIM can be found in the examples directory of the distribution.

The following steps should be followed to allow POE jobs to run under LSF:

Step 1.
Ensure that the SP-2 node names are the same as their host names. That is, jm_status should return the same names for the nodes that lsload returns.
Step 2.
Build and install the SP-2 ELIM. A README.sp2 file is provided in the examples directory with specific instructions.
Step 3.
Add the following to the lsf.shared file in the LSF_CONFDIR directory to pick up the indices reported by the ELIM:
Begin NewIndex
lock       60           Y        (IBM SP2 Node lock status)
pool       60           N        (IBM SP2 POWERparallel system pool)
End NewIndex

For all queues which accept POE jobs define a requeue exit value and a threshold for the lock index.

For example:

Begin Queue

The poejob script will exit with 133 if it is necessary to requeue the job. Note that other types of jobs should not be submitted to the same queue. Otherwise, they will get requeued if they happen to exit with 133. The scheduling threshold on the lock index prevents dispatching to nodes which are being used in exclusive mode by other jobs.

Note that it is only necessary to enable requeuing of POE jobs if some users submit jobs requiring exclusive access to the node.

Support for HP Exemplar Technical Servers

The HP Exemplar Technical Server is a high performance cache coherent Non Uniform Memory Access (ccNUMA) computer system. The Exemplar system supports the partitioning of the computing resources into subcomplexes which are collections of processors and memory from one or hypernodes in the system. The Subcomplex Manager (SCM) enables administrators to configure processor and memory resources into subcomplexes.

The following are provided in LSF to support the Exemplar:

LSF does not currently support dynamic load balancing between subcomplexes.

The following steps should be taken to setup an Exemplar system to run LSF.

  1. Build and install the Exemplar ELIM. Follow the instructions in the README file in the examples directory of the distribution.
  2. Add the definitions for the load indices for each subcomplex in the lsf.shared file.
  3. Add queue definitions for each subcomplex to the lsb.queues file.

Adding Load Indices Definitions

Edit the LSF_CONFDIR/lsf.shared configuration. Add the definitions for the load indices for each subcomplex. For example, if you have two subcomplexes, you need to configure 12 indices as follows:

Begin NewIndex
sc1Pme  60              N            (Subcomplex one private memory)
sc1Gme  60              N            (Subcomplex one global memory)
sc1cpu  60              N            (Subcomplex one number cpu)
sc1r5s  60              Y            (Subcomplex one five sec runQ)
sc1r30  60              Y            (Subcomplex one thirty sec runQ)
sc1r1m  60              Y            (Subcomplex one one minute runQ)
sc2Pme  60              N            (Subcomplex two private memory)
sc2Gme  60              N            (Subcomplex two global memory)
sc2cpu  60              N            (Subcomplex two number cpu)
sc2r5s  60              Y            (Subcomplex two five sec runQ)
sc2r30  60              Y            (Subcomplex two thirty sec runQ)
sc2r1m  60              Y            (Subcomplex two one minute runQ)
End NewIndex

The index names should be of the form scNxxx where N is the subcomplex number. The name of the subcomplexes defined on the system can be obtained by running the following command

% sysinfo -ls
System      load average:         4.30  4.28  4.09
largeGlbMem load average:         3.20  2.18  2.07

The subcomplex number corresponds to its position in the list. In the above example, System is subcomplex 1 and largeGlbMem is subcomplex 2. The names of the indices can be modified if appropriate changes are made to the supplied ELIM.

It is possible to change the name of the indices to include the subcomplex name instead of using a number. This requires changes to the supplied Exemplar ELIM.

The built-in load indices reported by the LIM on the Exemplar are implemented as follows:

Adding Queue Definitions

Edit the queue definitions in LSB_CONFDIR/cluster/configdir/lsb.queues to add queue definitions for each subcomplex. A Job Starter should be specified for each queue to control which subcomplex jobs from the queue will run on. For example:

Begin Queue
QUEUE_NAME   = long
JOB_STARTER  = mpa -sc largeGlbMem
DESCRIPTION = Long jobs on the largGlbMem subcomplex
End Queue

Begin Queue
QUEUE_NAME   = short
JOB_STARTER  = mpa -sc System
DESCRIPTION = Short jobs on the System subcomplex
End Queue

The JOB_STARTER parameter uses the mpa(1) command to start the job script file onto the subcomplex specified with the -sc option. LSF sets the LSB_JOBFILENAME environment variable, which specifies a shell script containing the user's commands.

You can use the load indices for each subcomplex to control the scheduling or suspension of jobs on that subcomplex. For example:

Begin Queue
QUEUE_NAME   = idle
JOB_STARTER  = mpa -sc System
sc1r1m       = 2.0/6.0
End Queue

would only start jobs on the System subcomplex if the 1-minute run queue was below 2.0 and suspend jobs if it goes above 6.0. Note that the load index specified in the scheduling constraints should correspond to the subcomplex specified in the JOB_STARTER parameter.

It is possible to make use of the queue level pre-/post-execution commands to move CPUs between subcomplexes on a per-job basis. For example, if an exclusive subcomplex has been set up, CPUs can be moved from the system subcomplex before job execution by the pre-exec command and moved back to the system subcomplex after job execution by the post-execution command.

Begin Queue
QUEUE_NAME   = exclusive
JOB_STARTER  = mpa -sc Exclusive
PRE_EXEC     = /usr/spp/moveCpuToEx
POST_EXEC    = /usr/spp/moveCpuToSys
End Queue
Step 1.
The Exemplar supports kernel level checkpointing using the chkpnt(1) and restart(1) commands. To enable checkpointing on Exemplar systems, copy erestart and echkpnt to the LSF_SERVDIR directory.

Users are not required to take any special actions for submitting jobs on a Exemplar system. If an Exemplar system is integrated into a larger cluster of machines, it is possible to set up queues that can dispatch to all machines. You need to specify a Job Starter script, which runs the job file through the mpa(1) for the Exemplar, and just executes the job file on non-Exemplar systems. Also scheduling constraints should be specified using the queue-level RES_REQ parameter to distinguish between Exemplar and non-Exemplar systems. For example:

RES_REQ= (type==Exemplar && sc1r1m < 2.0) || (type != Exemplar && r1m < 2.0)

Configuring NQS Interoperation

NQS (Network Queuing System) is a UNIX batch queuing facility that allows users to queue batch jobs to individual UNIX hosts from remote systems. This chapter describes how to configure and use LSF to submit and control batch jobs in NQS queues.

If you are not going to configure LSF to interoperate with NQS, you do not need to read this chapter.

While it is desirable to run LSF on all hosts for transparent resource sharing, this is not always possible. Some of the computing resources may be under separate administrative control, or LSF may not currently be available for some of the hosts.

An example of this is sites that use Cray supercomputers. The supercomputer is often not under the control of the workstation system administrators. Users on the workstation cluster still want to run jobs on the Cray supercomputer. LSF allows users to submit and control jobs on the Cray system using the same interface as they use for jobs on the local cluster.

LSF queues can be configured to forward jobs to remote NQS queues. Users can submit jobs, send signals to jobs, check the status of jobs, and delete jobs that are forwarded to the remote NQS. Although running on an NQS server outside the LSF cluster, jobs are still managed by LSF Batch in almost the same way as jobs running inside the LSF cluster.

Registering LSF with NQS

This section describes how to configure LSF and NQS so that jobs submitted to LSF can be run on NQS servers. You should already be familiar with the administration of the NQS system.


NQS uses a machine identification number (MID) to identify each NQS host in the network. The MID must be unique and must be the same in the NQS database of each host in the network. LSF uses the NQS protocol to talk with NQS daemons for routing, monitoring, signalling and deleting LSF Batch jobs that run on NQS hosts. Therefore, you must assign a MID to each of the LSF hosts that might become the master host.

To do this, perform the following steps:

Step 1.
Login to the NQS host as the NQS System Administrator or System Operator.
Step 2.
Run the nmapmgr command to create MIDs for each LSF host that can possibly become the master host. List all MIDs available. See the NQS nmapmgr(1) manual page for a description of this command.


NQS uses a mechanism similar to ruserok(3) to determine whether access is permitted. When a remote request from LSF is received, NQS looks in the /etc/hosts.equiv file. If the submitting host is found, requests are allowed as long as the user name is the same on both hosts. If the submitting host is not listed in the /etc/hosts.equiv file, NQS looks for a .rhosts file in the destination user's home directory. This file must contain the names of both the submitting host and the submitting user. Finally, if access still is not granted, NQS checks for a file called /etc/hosts.nqs. This file is similar to the .rhosts file, but it can provide mapping of remote usernames to local usernames. Cray NQS also looks for a .nqshosts file in the destination user's home directory. The .nqshosts file has the same format as the .rhosts file.

NQS treats the LSF cluster just as if it were a remote NQS server, except that jobs never flow to the LSF cluster from NQS hosts.

For LSF users to get permission to run jobs on NQS servers, you must make sure the above setup is done properly. Refer to your local NQS documentation for details on setting up the NQS side.


The lsb.nqsmaps file in the LSB_CONFDIR/cluster/configdir directory is for configuring inter-operation between LSF and NQS.


LSF must use the MIDs of NQS hosts when talking with NQS servers. The Hosts section of the LSB_CONFDIR/cluster/configdir/lsb.nqsmaps file contains the MIDs and operating system types of your NQS hosts.

Begin Hosts
cray001          1      UNICOS      #NQS host, must specify OS_TYPE
sun0101          2      SOLARIS     #NQS host
sgi006           3      IRIX        #NQS host
hostA            4      -           #LSF host; OS_TYPE is ignored
hostD            5      -           #LSF host
hostB            6      -           #LSF host
End Hosts

Note that the OS_TYPE column is required for NQS hosts only. For hosts in the LSF cluster, OS_TYPE is ignored; the type is specified by the TYPE field in the lsf.cluster.cluster file. The '-' entry is a placeholder.

User Name Mapping

LSF assumes that users have the same account names and user IDs on all LSF hosts. If the user accounts on the NQS hosts are not the same as on the LSF hosts, the LSF administrator must specify the NQS user names that correspond to LSF users.

The Users section of the lsb.nqsmaps file contains entries for LSF users and the corresponding account names on NQS hosts. The following example shows two users who have different accounts on the NQS server hosts.

Begin Users
user7          (user7l@cray001 luser7@sgi006)
user4          (suser4@cray001)
End Users

FROM_NAME is the user's login name in the LSF cluster, and TO_NAME is a list of the user's login names on the remote NQS hosts. If a user is not specified in the lsb.nqsmaps file, jobs are sent to the NQS hosts with the same user name.

Configuring Queues for NQS jobs

You must configure one or more LSF Batch queues to forward jobs to remote NQS hosts. A forward queue is an LSF Batch queue with the parameter NQS_QUEUES defined. The 'Adding a Batch Queue' describes how to add a queue to an LSF cluster. The following queue forwards jobs to the NQS queue named pipe on host cray001:

Begin Queue
QUEUE_NAME  = nqsUse
PRIORITY    = 30
NICE        = 15
CPULIMIT    = 15
NQS_QUEUES  = pipe@cray001
DESCRIPTION = Jobs submitted to this queue are forwarded to NQS_QUEUES
USERS       = all
End Queue

You can specify more than one NQS queue for the NQS_QUEUES parameter. LSF Batch tries to send the job to each queue in the order they are listed, until one of the queues accepts the job.

Since many features of LSF are not supported by NQS, the following queue configuration parameters are ignored for NQS forward queues: PJOB_LIMIT, POLICIES, RUN_WINDOW, DISPATCH_WINDOW, RUNLIMIT, HOSTS, and MIG. In addition, scheduling load threshold parameters are ignored because NQS does not provide load information about hosts.

Handling Cray NQS Incompatibilities

Cray NQS is incompatible with some of the public domain versions of NQS. Different versions of NQS on Cray may be incompatible with each other. If your NQS server host is a Cray, some additional steps may be needed in order for LSF to understand the NQS protocol correctly.

If the NQS version on a Cray is NQS 80.42 or NQS 71.3, then no extra setup is needed. For other versions of NQS on a Cray, you need to define NQS_REQUESTS_FLAGS and NQS_QUEUES_FLAGS in the lsb.params file.


If the version is NQS 1.1 on a Cray, the value of this flag is 251918848.

For other versions of NQS on a Cray, do the following to get the value for this flag. Run the NQS command:

% qstat -h CrayHost -a

on a workstation, where CrayHost is the host name of the Cray machine. Watch the messages logged by Cray NQS (you need access to the NQS log file on the Cray host):

03/02 12:31:59 I pre_server(): Packet type=<NPK_QSTAT(203)>.
03/02 12:31:59 I pre_server(): Packet contents are as follows:
03/02 12:31:59 I pre_server(): Npk_str[1] = <>.
03/02 12:31:59 I pre_server(): Npk_str[2] = <platform>.
03/02 12:31:59 I pre_server(): Npk_int[1] = <1392767360>.
03/02 12:31:59 I pre_server(): Npk_int[2] = <2147483647>.
03/02 12:31:59 I show_qstat_flags(): Flags=SHO_R_ALLUID SHO_R_SHORT

The value of Npk_int[1] in the above output is the value you need for the parameter NQS_REQUESTS_FLAGS.


To get the value for this flag, run the NQS command:

% qstat -h CrayHost -p -b -l

on a workstation, where CrayHost is the host name of the Cray machine. Watch the messages logged by Cray NQS (you need to have access to the Cray NQS log file):

03/02 12:32:57 I pre_server(): Packet type=<NPK_QSTAT(203)>.
03/02 12:32:57 I pre_server(): Packet contents are as follows:
03/02 12:32:57 I pre_server(): Npk_str[1] = <>.
03/02 12:32:57 I pre_server(): Npk_str[2] = <platform>.
03/02 12:32:57 I pre_server(): Npk_int[1] = <593494199>.
03/02 12:32:57 I pre_server(): Npk_int[2] = <2147483647>.
03/02 12:32:57 I show_qstat_flags(): Flags=SHO_H_ACCESS SHO_H_DEST 

The value of Npk_int[1] in the above output is the value you need for the parameter NQS_QUEUES_FLAGS.

If you are unable to get the required information after running the above NQS commands, make sure that your Cray NQS is configured properly to log these parameters. To do this, run:

% qmgr

and enter show all to get all information. The parameters related to the logging of the information you need are:

Debug level = 3
MESSAGE_Header = Short
   Accounting        OFF   CHeckpoint        OFF   COMmand_flow      OFF
   CONfig            OFF   DB_Misc           OFF   DB_Reads          OFF
   DB_Writes         OFF   Flow              OFF   NETWORK_Misc      ON
   NETWORK_Reads     ON    NETWORK_Writes    ON    OPer              OFF
   OUtput            OFF   PACKET_Contents   ON    PACKET_Flow       ON
   PROTOCOL_Contents ON    PROTOCOL_Flow     ON    RECovery          OFF
   REQuest           OFF   ROuting           OFF   Scheduling        OFF
   USER1             OFF   USER2             OFF   USER3             OFF
   USER4             OFF   USER5             OFF

Support for Atria ClearCase

Many sites use Atria's ClearCase environment for revision source control and development. A user uses the cleartool command to startup a ClearCase view. After the view is created, the user is presented with a file system containing the user's sources and binaries. The file system is not accessible outside the view. cleartool has an option to start up a view and run a command under the view.

LSF Job Starter can be used to set up a view then run the command (see 'Controlling LSF Batch Jobs' for further details). For example, if you set the Job Starter to a script 'clearstarter' similar to the following:

if [ x_$CLEARCASE_ROOT = x_ ]; then
    cd $LS_SUBCWD
    /usr/atria/bin/cleartool setview \
        -exec "cd $LS_SUBCWD;$*" \
        `basename $CLEARCASE_ROOT`

The user's job will run by LSF using the command line:

clearstarter myjob

which sets up a view the same as the user's on submission host, changes directory to the same as on submission host, then runs the job. Thus the remote job runs in the same view as on local host.

For interactive jobs, the user sets the environment variable LSF_JOB_STARTER to the ClearCase Job Starter. The RES on the remote host then will run user's job with the Job Starter. After the Job Starter is set, lsmake can run transparently in ClearCase view.

There are three steps to run an interactive job through the RES in a ClearCase view:

Step 1.
Write a ClearCase Job Starter script (see example above).
Step 2.
Set the LSF_JOB_STARTER environment variable. This can be done by each user or as part of the login process. For example:
% setenv LSF_JOB_STARTER clearstarter
Step 3.
Run the job. For example:
% lsmake -j 4 -V -f foo.mak

To run a batch job in ClearCase view, the csub command should be used instead of bsub. With csub, no Job Starter needs to be used(1).

csub checks whether the environment variable CLEARCASE_ROOT is set. If it is set, which means the job is submitted from a view, it wraps the user's job as following:

cleartool setview -exec "cd $LS_SUBCWD;job" `basename $CLEARCASE_ROOT`

and passes all options to bsub, except -i, -o, and -e. These three options will be translated to shell I/O redirection. For example, suppose CLEARCASE_ROOT=/view/myview and the user enters:

% csub -q myqueue -o myout -i myin myjob

csub will translate this into:

bsub -q myqueue cleartool setview -exec "cd $LS_SUBCWD; myjob < myin > myout \
2>&1" myview

An alternative way is to configure a queue level Job Starter (define JOB_STARTER in the file lsb.queues; see 'Controlling LSF Batch Jobs' for details), then use bsub to submit the job.

1. Look in the directory LSF_INDEP/misc/examples/clearcase for the files clearstarter and csub. LSF_INDEP is defined in the lsf.conf file.
[Contents] [Prev] [Next] [End]


Copyright © 1994-1997 Platform Computing Corporation.
All rights reserved.