This is an old revision of the document!

The ''bsub'' Command: Submitting Jobs to the Cluster

bsub submits information on your job to the batch system:


LSF then matches the job’s requirements against the capabilities of available job slots. Once sufficient suitable job slots are found, the job is started. LSF considers jobs to be started in the order of their priority.

Available Queues

We have three generally available base queues, corresponding to broad application profiles:

interactive queue


This is the queue for SMP jobs, especially those requiring lots of memory. Serial jobs with very high memory requirements do also belong in this queue. Up to 64 cores and 256 GB are available on one host. Maximum runtime is 48 hours.

Both the “mpi” and “fat” queues are also available in manifestations, e.g. “mpi-long”, corresponding to special runtime requirements:



This queue is for jobs that require more than 256 GB RAM on single node. Nodes of fat+ queue have 512 GB, 1.5 and 2 TB RAM. Due to limited amount of such nodes, there are restrictions of using the queue, which you can find at the page for experienced Users

''bsub'' Syntax and Usage

bsub <bsub options> [mpirun.lsf] <path to program> <program parameters>

''bsub'' options for serial jobs

-q <queue>
Submission queue.

-W <hh:mm>
Maximum runtime. If this time is exceeded the job is killed.

-o <file>
Store the job output in “file” (otherwise it is sent by email). %J in the filename stands for the jobid.

''bsub'' options for parallel (SMP or MPI) jobs

-n <min>,<max>
The minimum and maximum process count. If “max” is left out, “min” is the exact number of job slots required.

-a <wrapper>
This option denotes a wrapper script required to run SMP or MPI jobs. The most important wrappers are openmp (for SMP jobs), intelmpi (for MPI jobs using the Intel MPI library), and openmpi (for MPI jobs using the OpenMPI library).

LSF’s substitute for mpirun. In MPI jobs mpirun.lsf needs to be put in front of the program path.

''bsub'': Specifying process distribution with ''-R''

-R span[hosts=1]
This puts all processes on one host. You always want to use this with SMP jobs.

-R span[ptile=<x>]
x denotes the exact number of job slots to be used on each host. If the total process number is not divisible by x, the residual processes will be put on one host.

-R span[ptile='!']
-R same[model]
With this special notation, x will become the maximum number of cores available on the node type used for the job. In other words, using '!' will acquire all job slots on all nodes the job runs on, provided the total number of job slots requested is divisible by x (otherwiese the residual will run on one shared host).

The GWDG Scientific Compute Cluster

This scheme shows the basic cluster setup at GWDG. The cluster is distributed across two facilities, with the “ehemalige Fernmeldezentrale” facility hosting the older resources and the shared /scratch file system and the “Faßberg” facility hosting the latest resources and the shared /scratch2 file system. The shared /scratch2 is usually the best choice for temporary data in your jobs, but it is only available at the Faßberg resources (selectable with -R scratch2). The scheme also shows the queues and resources by which nodes are selected using the -q and -R options of bsub.

''bsub'': Specifying node properties with ''-R''

-R scratch[2]

-R work
The node must have access to one of the shared /work directories.

-R “ncpus=<x>“
Choose only nodes with a job slot count of x. This is useful with span[ptile=<x>].

-R big
Choose the nodes with the maximum memory per core available in the queue. Currently only distinguishes gwdaxxx from gwdpxxx nodes.

-R latest
Always use the latest (and usually most powerful) nodes available in the queue. To get a list of current latest nodes run the command bhosts -R latest on one of the frontends. You can also check the Latest Nodes page for more information.

''bsub'': Using Job Scripts

A job script is a shell script with a special comment section: In each line beginning with #BSUB the following text is interpreted as a bsub option. Here is an example:

#BSUB -q mpi
#BSUB -W 00:10
#BSUB -o out.%J


Job scripts are submitted by the following command:

bsub < <script name>  

Exclusive jobs


To submit an exclusive job add -x to your bsub options. For example, to submit a single task job, which uses a complete fat node with 256 GB memory, you could use:

bsub -x -q fat -R big ./mytask

(-R big requests a 256 GB node, excluding the 128 GB nodes in the fat queue)


bsub -x -q mpi -n 8 -R span[ptile=4] -a intelmpi mpirun.lsf ./hybrid_job


Please note that fairshare evaluation and accounting is done based on the number of job slots allocated. So the first example would count as 64 slots for both fairshare and accounting.

Using exclusive jobs does not require reserving all of a node's slots explicitly (e.g., with span[ptile='!']) and subsequently using the MPI library's mpiexec or mpiexec.hydra to set the process number, as we explain in our introductory course. This makes submitting a hybrid job as exclusive job more straightforward.

However, there is a disadvantage: LSF will not reserve the additional job slots required to get a node exclusively. Therefore, when the cluster is very busy, an exclusive job needing a lot of nodes may wait significantly longer.

A Note On Job Memory Usage

LSF will try to fill up each node with processes up to its job slot limit. Therefore each process in your job must not use more memory than available per core! If your per core memory requirements are too high, you have to add more job slots in order to allow your job to use the memory of these slots as well. If your job's memory usage increases with the number of processes, you have to leave additional job slots empty, i.e., do not run processes on them.

Recipe: Reserving Memory for OpenMP

The following job script recipe demonstrates using empty job slots for reserving memory for OpenMP jobs:

#BSUB -q fat
#BSUB -W 00:10
#BSUB -o out.%J
#BSUB -n 64
#BSUB -R big
#BSUB -R "span[hosts=1]"


Disk Space Options




This space is the same as scratch described above except it is ONLY available on the nodes dfa, dge, dmp, dsu and dte and on the frontend gwdu103. You can use -R scratch2 to make sure to get a node with access to that space.


Recipe: Using ''/scratch''


#BSUB -q fat
#BSUB -n 64
#BSUB -R "span[hosts=1]"
#BSUB -R scratch
#BSUB -W 24:00
#BSUB -C 0
#BSUB -a openmp

export g09root="/usr/product/gaussian"
. $g09root/g09/bsd/g09.profile

mkdir -p /scratch/${USER}

g09 myjob.log

Using ''/scratch2''



#BSUB -R "scratch||scratch2"


Miscallaneous LSF Commands

While bsub is arguably the most important LSF command, you may also find the following commands useful:

Lists current jobs. Useful options are: -p, -l, -a, , <jobid>, -u all, -q <queue>, -m <host>.

Lists older jobs. Useful options are: -l, -n, <jobid>.

Status of cluster nodes. Useful options are: -l, <hostname>.

Status of batch queues. Useful options are: -l, <queue>.

Why do I have to wait? bhpart shows current user priorities. Useful options are: -r, <host partition>.

The Final Command. It has two use modes:

  1. bkill <jobid>: This kills a job with a specific jobid.
  2. bkill <selection options> 0: This kills all jobs fitting the selection options. Useful selection options are: -q <queue>, -m <host>.


Getting Help


Information sources

Using the GWDG Support Ticket System

Write an email to In the body:

  • State that your question is related to the batch system.
  • State your user id ($USER).

Scientific Computing
