The saliweb.backend Python module

class saliweb.backend.Config(fh)

This class holds configuration information such as directory locations, etc. fh is either a filename or a file handle from which the configuration is read.

admin_email

The email address of the admin.

populate(config)

Populate data structures using the passed config, which is a configparser.ConfigParser object. This can be overridden in subclasses to read additional service-specific information from the configuration file.

send_admin_email(subject, body)

Send an email to the admin for this web service, with the given subject and body.

send_email(to, subject, body)

Send an email to the given user or list of users (to), with the given subject and body. body can either be the text itself, or an object from the Python email module (e.g. MIMEText, MIMEMultipart).

class saliweb.backend.Database(jobcls)

Management of the job database. Can be subclassed to add extra columns to the tables for service-specific metadata, or to use a different database engine. jobcls should be a subclass of Job, which will be used to instantiate new job objects.

add_field(field)

Add a new field (typically a MySQLField object) to each table in the database. Usually called in the constructor or immediately after creating the Database object.

set_track_hostname()

Add extra fields to support tracking the user’s hostname

class saliweb.backend.MySQLField(name, type, null=True, key=None, default=None, index=False)

Description of a single field in a MySQL database. Each field must have a unique name (e.g. ‘user’) and a given type (e.g. ‘VARCHAR(15)’). null specifies whether the field can be NULL (valid values are True, False, ‘YES’, ‘NO’). key if given specifies what kind of key it is (e.g. ‘PRIMARY’ or ‘PRI’). default specifies the default value of the field. If index is True, create an index on this field (the index gets the same name as the field, except with an ‘_index’ suffix).

get_schema()

Get the SQL schema needed to create a table containing this field.

class saliweb.backend.Runner

Base class for runners, which handle the actual running of a job, usually on an SGE cluster (see the WyntonSGERunner subclass). To create a subclass, you must implement both a _run method and a _check_completed class method, set the _runner_name attribute to a unique name for this class, and call Job.register_runner_class() passing this class.

class saliweb.backend.ClusterRunner(script, interpreter='/bin/sh')

Base class to run a set of commands on a compute cluster. Use a subclass specific to the cluster you want to use, such as WyntonSGERunner.

To use, pass a string script containing a set of commands to run, and use interpreter to specify the shell (e.g. /bin/sh, /bin/csh) or other interpreter (e.g. /usr/bin/python) that will run them. These commands will be automatically modified to update a job state file at job start and end, if your interpreter is /bin/sh, /bin/csh, /bin/bash or /bin/tcsh. If you want to use a different interpreter you will need to manually add code to your script to update a file called job-state in the working directory, to contain just the simple text “STARTED” (without the quotes) when the job starts and just “DONE” when it completes.

Once done, you can optionally call set_options() to set command line options (e.g. time limits, requested resources) and/or set_name() to set the job name.

set_name(name)

Set the job name (equivalent to SGE’s qsub -N option, or SLURM’s sbatch -J option). If the name is not a valid name (e.g. SGE job names cannot start with a digit) then it is mapped to one that is.

set_options(opts)

Set the options to use as a string, for example ‘-l mydisk=1G -p 0’. These will be specific to the queuing system (e.g. SGE, SLURM) you are using. Note that if you want to set the job name (SGE’s -N, SLURM’s -J option) it is better to use set_name().

class saliweb.backend.SGERunner(script, interpreter='/bin/sh')

Base class to run a set of commands on an SGE cluster. Use a subclass specific to the cluster you want to use, such as WyntonSGERunner.

See ClusterRunner for more information.

set_name(name)

Set the job name (equivalent to SGE’s qsub -N option, or SLURM’s sbatch -J option). If the name is not a valid name (e.g. SGE job names cannot start with a digit) then it is mapped to one that is.

set_sge_name(name)

Set the job name (equivalent to SGE’s qsub -N option, or SLURM’s sbatch -J option). If the name is not a valid name (e.g. SGE job names cannot start with a digit) then it is mapped to one that is.

set_sge_options(opts)

Set the options to use as a string, for example ‘-l mydisk=1G -p 0’. These will be specific to the queuing system (e.g. SGE, SLURM) you are using. Note that if you want to set the job name (SGE’s -N, SLURM’s -J option) it is better to use set_name().

class saliweb.backend.WyntonSGERunner(script, interpreter='/bin/sh')

Run commands on the Wynton SGE cluster.

See SGERunner for more details.

class saliweb.backend.SLURMRunner(script, interpreter='/bin/sh')

Base class to run a set of commands on a SLURM cluster (experimental).

See ClusterRunner for more information.

set_name(name)

Set the job name (equivalent to SGE’s qsub -N option, or SLURM’s sbatch -J option). If the name is not a valid name (e.g. SGE job names cannot start with a digit) then it is mapped to one that is.

class saliweb.backend.LocalRunner(cmd)

Run a program (given as a list of arguments or a single string) on the local machine.

The program must create a file called job-state in the working directory, to contain just the simple text “STARTED” (without the quotes) when the job starts and just “DONE” when it completes.

class saliweb.backend.SaliWebServiceRunner(url, args)

Run a job on another Sali lab web service. When the job completes, the resulting files are passed to the Job.postprocess() method, as a list of SaliWebServiceResult objects, where they can be downloaded if desired. This uses the web service’s REST interface (see Automated use for further details). For example, to submit to the fictional ModFoo service, use something like:

r = SaliWebServiceRunner('http://modbase.compbio.ucsf.edu/modfoo/job',
                         ['input_pdb=@input.pdb', 'job_name=testjob'])

class saliweb.backend.SaliWebServiceResult(url)

Represent a single file, resulting from a SaliWebServiceRunner job.

download(fh=None)

Download the result file. If fh is given, it should be a Python file-like object, to which the file is written. Otherwise, the file is written into the job directory with the same name.

get_filename()

Return the name of the file corresponding to this result.

class saliweb.backend.DoNothingRunner

Do nothing, i.e. have the job complete immediately.

class saliweb.backend.WebService(config, db)

Top-level class used by all web services. Pass in a Config (or subclass) object for the config argument, and a Database (or subclass) object for the db argument.

create_database_tables()

Create all tables in the database used to hold job state.

do_all_processing(daemonize=False, status_fh=None)

Process incoming jobs, completed jobs, and old jobs. This method will run forever, looping over the available jobs, until the web service is killed. If daemonize is True, this loop will be run as a daemon (subprocess), so that the main program can continue. If status_fh is specified, it is a file handle to which “OK” is printed on successful startup.

drop_database_tables()

Drop all tables in the database used to hold job state.

get_job_by_name(state, name)

Get the job with the given name in the given job state. Returns a Job object, or None if the job is not found.

get_running_pid()

Return the process ID of a currently running web service, by querying the state file. If no service is running, return None; if the last run of the service failed with an unrecoverable error, raise a StateFileError.

version = None

Version number of the service, or None.

class saliweb.backend.Job(db, metadata, state)

Class that encapsulates a single job in the system. Jobs are not created by the user directly, but by querying a WebService object.

logger

A standard Python logger object (i.e it provides methods such as debug, info, warning) that can be used by job methods such as run() (but not expire()). By default this logger logs to a file called ‘framework.log’ in the job directory; the file is created when the first log message is emitted. See also get_log_handler().

admin_fail(email)

Force a job into the FAILED state. This is intended to be used by the server administrator (via the failjob.py utility) to fail jobs which have erroneously completed.

archive()

Do any necessary processing when an old completed job reaches its archive time. Does nothing by default, but can be overridden by the user to compress files, etc. This method should not be called directly.

complete()

This method is called after a job completes. Does nothing by default, but can be overridden by the user. Note that it is rare to override this method - usually finalize() makes more sense. This method should not be called directly.

property config

Config object (read-only)

delete()

Delete the job directory and database row.

property directory

Current job working directory (read-only)

expire()

Do any necessary processing when an old completed job reaches its expire time; it is called just before the job directory is deleted. Does nothing by default, but can be overridden by the user to mail the admin, etc. This method should not be called directly.

finalize()

Do any necessary finalizing when the job completes successfully. Does nothing by default. This is like postprocess() except that it cannot schedule any more jobs (reschedule_run()), i.e. it is only called on the last cycle of a multi-job run. This method should not be called directly.

get_log_handler()

Create and return a standard Python log Handler object. By default it directs log messages to a file called ‘framework.log’ in the job directory. This can be overridden to send log output elsewhere, e.g. in an email. Do not call this method directly; instead use logger to access the logger object.

property name

Unique job name (read-only)

postprocess(results=None)

Do any necessary postprocessing when the job completes successfully. Does nothing by default. Note that a user-defined postprocess method can call reschedule_run() to request that the backend runs a new cluster job if necessary. results, if given, contains explicit results from the job runner. For example, the SaliWebServiceRunner returns a list of files generated by the web service (as SaliWebServiceResult objects). See also finalize(). This method should not be called directly.

preprocess()

Do any necessary preprocessing before the job is actually run. Does nothing by default. Note that a user-defined preprocess method can call skip_run() to skip running of the job on the cluster, if it is determined in preprocessing that a job run is not necessary. This method should not be called directly.

classmethod register_runner_class(runnercls)

Maintain a mapping from names to Runner classes. If you define a Runner subclass, you must call this method, passing that subclass.

rerun(data)

Run a rescheduled job (if postprocess() called reschedule_run() to run a new job). data is a Python object passed from the reschedule_run() method. Like run(), this should create and return a suitable Runner instance.

By default, this method simply discards data and calls the regular run() method. You can redefine this method if you want to do something different for rescheduled runs.

reschedule_run(data=None)

Tell the backend to schedule another job to be run on the cluster once postprocessing is complete (the job moves from the POSTPROCESSING state back to RUNNING).

It is only valid to call this method from the POSTPROCESSING state, usually from a user-defined postprocess() method.

The rescheduled job is run by calling the rerun() method, which is passed the data Python object (if any).

Note that because the rescheduled job itself will be postprocessed once finished, you must be careful not to create an infinite loop here. This could be done by using a file in the job directory or a custom field in the job database to prevent a job from being rescheduled more than a certain number of times, and/or to pass state to the postprocess() method (so that it knows it is postprocessing a rescheduled job rather than the first job).

resubmit()

Make a FAILED job eligible for running again.

run()

Run the job, e.g. on an SGE cluster. Must be implemented by the user for each web service; it should create and return a suitable Runner instance. For example, this could generate a simple script and pass it to an SGERunner instance. This method should never be called directly; it is automatically called by the backend when needed. To run a new job, call reschedule_run() instead.

send_job_completed_email()

Email the user (if requested) to let them know job results are available. (It does this by calling send_user_email().) Can be overridden to disable this behavior or to change the content of the email.

send_user_email(subject, body)

Email the owner of the job with the given subject and body. (If no email address was given by the user, this does nothing.)

property service_name

Web service name (read-only)

skip_run()

Tell the backend to skip the actual running of the job, so that when preprocessing has completed, it moves directly to the COMPLETED state, skipping RUNNING, POSTPROCESSING, and FINALIZING.

It is only valid to call this method from the PREPROCESSING state, usually from a user-defined preprocess() method.

property url

URL containing job results (read-only)

Exceptions

exception saliweb.backend.ConfigError

Exception raised if a configuration file is inconsistent.

exception saliweb.backend.InvalidStateError

Exception raised for invalid job states.

exception saliweb.backend.RunnerError

Exception raised if the runner (such as SGE) failed to run a job.

exception saliweb.backend.SanityError

Exception raised if a new job fails the sanity check, e.g. if the frontend added invalid or inconsistent information to the database.

exception saliweb.backend.StateFileError

Exception raised if a previous run is still running or crashed.