The cluster uses Sun Grid Engine (SGE) to schedule jobs. This requires you to write a short shell script to specify how and where to run your jobs.

Detailed SGE user documentation can be found here either in HTML orPDF form.

Read me first

CPU and memory intensive jobs must not be run on the job submission hosts (chef.compbio.ucsf.edu and sous.compbio.ucsf.edu). They are only to be used to submit jobs to SGE, which will then schedule those jobs to be run on the cluster. The systems administrators reserve the right to kill without warning any simulations or similar programs running on the job submission hosts.

/!\ Note also that the network storage is for short term high performance use ONLY. It is not backed up and should not be used as a permanent data store. Move longer term data to your own systems and backups. /!\

See the Cluster_Topology page for a list of the disks and machines available to you in the cluster.

Please note that some of the cluster machines (ibm1 through ibm90) are more-or-less standard x86 PCs - i.e. they will run x86 code - while the other nodes will not:

Note that, in each of the above resources, the first character after the "=" is a lowercase letter "L", not the number one. If you do not select any SGE resource (see below) then your job will run on any node, and so will probably crash unless it is architecture independent, or you use a shell script to automatically select the correct binaries and libraries for each architecture.

The shared cluster machines run CentOS 6, so if you have to compile code for them on another Linux box, you should make sure that you either compile it statically, or that you make the necessary libraries available to it. Alternatively, log on to the Interactive_access nodes directly to compile. See the x86_node_config and Opteron_node_config pages for the libraries and compilers installed on each node.

Make sure that you also request the SGE resource(s) corresponding to the storage you want to use (see Cluster_Topology) as not all nodes have all disks mounted. Usually this means that you need to request the disk containing your home directory (use 'echo ~' to find out where this is) and local scratch space, if needed. (For example, if you know the job will need 4GB of space on /scratch, use '-l scratch=4G'.)

There are three queues on the shared cluster. They are sorted in the following sequence, with jobs preferentially going to the first available queue which matches the individual job's requirements:

For the greatest throughput, you should submit your jobs without specifying a queue. When doing this, however, it is very important that you specify a estimated runtime (-l h_rt=N) and that this runtime accurately reflects which queues your job can run in. If you don't specify a runtime and your job gets put into short.q, e.g., it will be killed when its runtime exceeds short.q's time limit, which is 30 minutes. Note that this runtime must also be longer than your expected job length, because your job will be killed if it overruns. If you are unsure how long your job will run, you should specify the maximum runtime, that is -l h_rt=336:0:0.

Most (but not all) nodes have a local /scratch disk which should be used ONLY as scratch space. Please do not store any permanent data there. The /scratch area will be periodically cleaned of files that are more than 15 days old. It is strongly recommended that you use /scratch for any storage during your calculations, as using your home directory for temporary files increases network traffic and slows down the whole cluster. You cannot see the /scratch directory from chef, so your job must copy back any data it needs on completion.

To create a separate directory within /scratch to use for your jobs, you might use the following shell trick (example given for bash):

export TMPDIR=/scratch
export MYTMP=`mktemp -d`
cd $MYTMP

This will use the utility mktemp to create a randomly-named directory within /scratch, store it in the TMPDIR environment variable, and change this to be the current working directory. Many other scripting languages (e.g. Perl, Python) have similar facilities for creating uniquely-named directories you can use to store working files.

Sample submit script

Before you can submit jobs to the nodes, you should prepare a script like the one below. Split your jobs into smaller tasks varying only in input parameters. You can then submit the jobs from chef using "qsub scriptname". (Note: do not include the #-- comments in your script - that won't work.)

#!/bin/bash                         #-- what is the language of this shell
#                                  #-- Any line that starts with #$ is an instruction to SGE
#$ -S /bin/bash                     #-- the shell for the job
#$ -o [dir]                        #-- output directory (fill in)
#$ -e [dir]                        #-- error directory (fill in)
#$ -cwd                            #-- tell the job that it should start in your working directory
#$ -r y                            #-- tell the system that if a job crashes, it should be restarted
#$ -j y                            #-- tell the system that the STDERR and STDOUT should be joined
#$ -l mem_free=1G                  #-- submits on nodes with enough free memory (required)
#$ -l arch=linux-x64               #-- SGE resources (CPU type)
#$ -l netapp=1G,scratch=1G         #-- SGE resources (home and scratch disks)
#$ -l h_rt=24:00:00                #-- runtime limit (see above; this requests 24 hours)
##$ -t 1-10                        #-- remove first '#' to specify the number of
                                   #-- tasks if desired (see Tips section)

# Anything under here can be a bash script

# If you used the -t option above, this same script will be run for each task,
# but with $SGE_TASK_ID set to a different value each time (1-10 in this case).
# The commands below are one way to select a different input (PDB codes in
# this example) for each task.  Note that the bash arrays are indexed from 0,
# while task IDs start at 1, so the first entry in the tasks array variable
# is simply a placeholder

#tasks=(0 1bac 2xyz 3ijk 4abc 5def 6ghi 7jkl 8mno 9pqr 1stu )
#input="${tasks[$SGE_TASK_ID]}"

date
hostname

qstat -j $JOB_ID                                  # This is useful for debugging and usage purposes,
                                                  # e.g. "did my job exceed its memory request?"

Some notes on job submissions

Tips

QB3cluster: Cluster_Usage (last edited 2016-10-21 20:31:58 by Joshua_Baker-LePain)