Batch mode primer for dextrose

dextrose is the head node for a cluster of 34 compute nodes, all running the Linux operating system. You log into dextrose by using your favourite ssh client program to connect to

You can perform interactive work, such as editing or compiling files on the head node, dextrose, itself, but intensive computation should take place on compute nodes. The compute nodes are managed by a job queuing system called LSF (for "Load Sharing Facility"). This page is a bare bones introduction to submitting jobs to LSF on dextrose. See LSF's extensive documentation for full information.

Job submission

To oversimplify, you submit a job to the queue by prefixing the command line you wish to execute with the word bsub. For example,

bsub ls

asks the queuing system to run the ls command on the next compute node with resources to spare. Since the compute nodes mount the same home directory for your account as does the head node, there is not much point to such a command, but it does illustrate one thing about the batch queue: it is for non-interactive jobs.* If there are many jobs queued up, your job may not run immediately, and in any event the compute node that runs the job will not have access to your terminal. So if you ran the above command, the output from ls would not show up immediately on your screen. Instead, the batch system would email you* the output from ls after the command runs on some compute node.

You can modify how the batch system handles your job by inserting arguments to the bsub command between the word “bsub” and the command you are submitting to the queue.

For example, to have the standard output and standard error streams of your command redirected to files, rather than emailed to you, you could use

bsub -oo stdoutFile -eo stderrFile ls

to have the standard output stream written to a file named stdoutFile and the standard error stream written to stderrFile. The preceding command would overwrite any existing copies of stdoutFile and stderrFile. Use

bsub -o stdoutFile -e stderrFile ls

if you wish to append to existing files instead.

See the bsubmanual page (also readable by typing “man bsub” when you're logged into dextrose) for details on bsub's many options.

Here's a real-world example. To submit a job that will run Gaussian 09 on the input file ng1.gjf, save the output stream into ng1.out, and save the error stream into ng1.err, while assigning the job the name “NG job” (the job name shows up in some of the monitoring commands described below):

bsub -J "NG job" -oo ng1.stdout -eo ng1.stderr g09 ng1.gjf

Submitting MPI jobs

Some programs use a Message Passing Interface (MPI) library to spread their work over processors from multiple compute nodes. Starting such programs is complicated by the necessity to use some MPI launcher to start the program on each of the nodes it ends up using, and by the necessity to ask the queuing system to allocate multiple compute nodes.

This primer will only illustrate the idea with a single example, albeit a complicated one. The command:

bsub -n 24 -R "span[ptile=6]" -J "Big Vasp Job" -oo stdout -eo stderr \ /opt/platform_mpi/bin/mpirun -lsf -e MKL_NUM_THREADS=2 vasp5

asks the queuing system to allocate 24 processors, with up to 6 processors per node, to run a job called “Big Vasp Job”, saving the standard output and error streams to stdout and stderr, respectively. The queuing system will execute the command

/opt/platform_mpi/bin/mpirun -lsf -e MKL_NUM_THREADS=2 vasp5

on each assigned compute node.

The mpirun command is the launcher for Platform Computing's MPI library. mpirun's “-lsf” argument tells it to get the description of which hosts to communicate with from the queuing system, while “-e MLK_NUM_THREADS=2” specifies a variable setting that mpirun should make in the environnment of all the processes it starts to run the job. The actual command to be run is vasp5 (vasp5, in its turn, will get its job parameters from files with standard names in the directory where it starts, which is just how vasp5 happens to work).

The MKL_NUM_THREADS environment is a directive to the Math Kernel Library, which our build of vaspt5 is linked to. It essentially tells the library how many threads it should start up for parallelizing array operations. In the example invocations, we've combined “-R span[ptile=6]” and “-e MKL_NUM_THREADS=2” in the hopes the combination will produce 6x2=12 active threads on each of 4 compute nodes (each of our compute nodes have 12 processor cores).

See the “RESOURCE REQUIREMENT STRINGS” section of the output of “man lsfintro” for details about what can go after bsub's -R option.

Job monitoring

The command


lists your currently running jobs.

There are a number of options to the bjobs command to specify what it should tell you about currently running jobs. For example,

bjobs -u all -w

lists the jobs of other users as well as yours, and increases the width of output lines to avoid abbreviating the contents.

bjobs -l 3838

Lists extra detail about the job numbered 3838 (the number likely coming from the output of one of the other invocations of bjobs)

You can get some data about jobs that have terminated with bhist. For example,

bhist -l 3838

To check the state of the compute nodes, try running either




All of these commands take options that affect how they work. See “man bjobs”, “man bkill”, et cetera.

Aborting Jobs

To terminate a running job that you decide you do not need, run

bkill 1234

where 1234 stands for the jobid of the job you wish to terminate.


  1. Actually, mechanisms do exist for submitting interactive jobs to the batch queue, but that's an unusual special case which won't be discussed in this primer. See the complete LSF documentation, and in particular bsub's manual page if you need to be able to interact with your batch jobs while they are running.

  2. As this primer is being written, the destination to which the email should be directed is set by the contents of a file called .forward in your home directory on dextrose, so you can change the destination address by editing this file (or creating it if it does not exist). In the absence of a .forward file, email generated within the dextrose cluster is delivered on dextrose, where you can read it with the mail command, but you'd probably rather have it delivered to where you are accustomed to reading mail.