TABLE OF CONTENTS
2.7 Batch Queuing Support
DESCRIPTION
At present PWTK supports the SLURM, Load-Leveler, and the Platform-LSF batch queuing systems. There are two different ways of how PWTK can be used with respect to batch queuing:
Flexible Mode
This is a highly-recommended mode. In this mode the PWTK creates a small batch-queuing shell script with batch-queuing specific instructions and the PWTK script itself is then executed by PWTK within the so created batch queuing shell script. Here is an example: supposed we have a PWTK script job.pwtk. Without batch queuing, this script would be executed as:
pwtk job.pwtk
But to submit this script to batch-queuing system, we can either decorate the script with a specific batch-queuing command (say, SLURM in the current example) or write a new script that will import the job.pwtk script, i.e.:
SLURM {
import job.pwtk
}
Suppose we store this script into file slurm_job.pwtk. Now we can submit this script to SLURM batch queuing system simply by:
pwtk slurm_job.pwtk
The SLURM command if fully configurable. Its syntax is either:
SLURM script
or
SLURM profile script
or
SLURM profile --option1 value1 --option2 value2 ... script
where script is the PWTK script to be submitted to SLURM queue and profile is the name of the profile (or template) as defined in the user ~/.pwtk/slurm.tcl file and contains SLURM specific directives. If the profile is omitted the default profile is used, which is guaranteed to exist ( it is defined in file: config/slurm.tcl ).
Further SLURM directives can be specified via the --option1 value1 --option2 value2 ... options. Here is an example:
SLURM --nodes 4 --ntasks-per-node 16 --ntasks 64 {
import slurm_job.pwtk
}
For Load-Leveler and the Platform-LSF, the corresponding command analogous to SLURM is LL and LSF, respectively, and the usage is analogues to the above described SLURM command.
Rigid Mode (deprecated)
In the rigid-mode, the functionality is rather limited and only the basics of PWTK can be exploited. In this mode the PWTK first converts a PWTK script into a plain Bourne shell-script and then submits the shell script to a queue. But such a conversion works only for basics and it is also rather rigid.
N.B.: the rigid-mode has not been tested for years, hence the user is highly recommended to use instead the flexible-mode described above.
In this mode, the commands such as runPW and runPH should be substituted with ${queueSystem}_runPW and ${queueSystem}_runPH, where the $queueSystem stands for specific queing system, e.g., slurm, ll or lsf. Typically the last line in PWTK's batch queuing script will be ${queueSystem}_submit. This will actually submit the job to the queuing system.
As mentioned above, in the rigid-mode mode the PWTK constructs a plain shell script (Bourne shell), and submit it to batch queuing system. The PWTK's batch queuing shell scripts are composed of the following sections:
head (the batch-queuing specific instructions, set by ***_setHead command)
|
specs (optional user-specified piece of script, set by ***_setSpecs command)
|
body (part of script where calculations are specified, it is set indirectly via ***_run commands)
|
tail (optional user-specified piece of script, set by ***_setTail command)
As for the "head" part of the script, see also the tutorial.html, section "Configuring batch queuing system support".
Here you can see an example script for load-leveler: examples/ll_Si.pwtk
• COMMANDS related to SLURM (prefixed by slurm_):
setting various parts of batch queuing script
* slurm_setHead => ::pwtk::queue::setHead ; usage: slurm_setHead profile [option value] * slurm_setSpecs => ::pwtk::queue::setSpecs * slurm_setTail => ::pwtk::queue::setTail
retrieving various parts of batch queuing script
* slurm_get => ::pwtk::queue::get * slurm_getHead => ::pwtk::queue::getHead * slurm_getSpecs => ::pwtk::queue::getSpecs * slurm_getBody => ::pwtk::queue::getBody * slurm_getTail => ::pwtk::queue::getTail * slurm_getScript => ::pwtk::queue::getScript
printing batch queuing script to file or stdout
* slurm_fprint => ::pwtk::queue::fprint * slurm_print => ::pwtk::queue::print
running calculation via batch queuing system
* slurm_run => ::pwtk::queue::run * slurm_runPW => ::pwtk::queue::runPW * slurm_rerunPW => ::pwtk::queue::rerunPW * slurm_runPH => ::pwtk::queue::runPH * slurm_rerunPH => ::pwtk::queue::rerunPH * slurm_runNEB => ::pwtk::queue::runNEB * slurm_rerunNEB => ::pwtk::queue::rerunNEB * slurm_runPP => ::pwtk::queue::runPP * slurm_runPROJWFC => ::pwtk::queue::runPROJWFC * slurm_runDOS => ::pwtk::queue::runDOS * slurm_runBANDS => ::pwtk::queue::runBANDS * slurm_runMOPDOS => ::pwtk::queue::runMOPDOS
submitting the whole job to batch queuing system
* slurm_submit => ::pwtk::queue::submit
some extra stuff ...
* slurm_profile => ::pwtk::queue::profile * slurm_profileDefault => ::pwtk::queue::profileDefault
• COMMANDS related to Load-Leveler (prefixed by ll_):
setting various parts of batch queuing script
* ll_setHead => ::pwtk::queue::setHead ; usage: ll_setHead profile [option value] * ll_setSpecs => ::pwtk::queue::setSpecs * ll_setTail => ::pwtk::queue::setTail
retrieving various parts of batch queuing script
* ll_get => ::pwtk::queue::get * ll_getHead => ::pwtk::queue::getHead * ll_getSpecs => ::pwtk::queue::getSpecs * ll_getBody => ::pwtk::queue::getBody * ll_getTail => ::pwtk::queue::getTail * ll_getScript => ::pwtk::queue::getScript
printing batch queuing script to file or stdout
* ll_fprint => ::pwtk::queue::fprint * ll_print => ::pwtk::queue::print
running calculation via batch queuing system
* ll_run => ::pwtk::queue::run * ll_runPW => ::pwtk::queue::runPW * ll_rerunPW => ::pwtk::queue::rerunPW * ll_runPH => ::pwtk::queue::runPH * ll_rerunPH => ::pwtk::queue::rerunPH * ll_runNEB => ::pwtk::queue::runNEB * ll_rerunNEB => ::pwtk::queue::rerunNEB * ll_runPP => ::pwtk::queue::runPP * ll_runPROJWFC => ::pwtk::queue::runPROJWFC * ll_runDOS => ::pwtk::queue::runDOS * ll_runBANDS => ::pwtk::queue::runBANDS * ll_runMOPDOS => ::pwtk::queue::runMOPDOS
submitting the whole job to batch queuing system
* ll_submit => ::pwtk::queue::submit
some extra stuff ...
* ll_profile => ::pwtk::queue::profile * ll_profileDefault => ::pwtk::queue::profileDefault
• COMMANDS related to Platform LSF (prefixed by lsf_):
setting various parts of batch queuing script
* lsf_setHead => ::pwtk::queue::setHead ; usage: lsf_setHead profile [option value] * lsf_setSpecs => ::pwtk::queue::setSpecs * lsf_setTail => ::pwtk::queue::setTail
retrieving various parts of batch queuing script
* lsf_get => ::pwtk::queue::get * lsf_getHead => ::pwtk::queue::getHead * lsf_getSpecs => ::pwtk::queue::getSpecs * lsf_getBody => ::pwtk::queue::getBody * lsf_getTail => ::pwtk::queue::getTail * lsf_getScript => ::pwtk::queue::getScript
printing batch queuing script to file or stdout
* lsf_fprint => ::pwtk::queue::fprint * lsf_print => ::pwtk::queue::print
running calculation via batch queuing system
* lsf_run => ::pwtk::queue::run * lsf_runPW => ::pwtk::queue::runPW * lsf_rerunPW => ::pwtk::queue::rerunPW * lsf_runPH => ::pwtk::queue::runPH * lsf_rerunPH => ::pwtk::queue::rerunPH * lsf_runNEB => ::pwtk::queue::runNEB * lsf_rerunNEB => ::pwtk::queue::rerunNEB * lsf_runPP => ::pwtk::queue::runPP * lsf_runPROJWFC => ::pwtk::queue::runPROJWFC * lsf_runDOS => ::pwtk::queue::runDOS * lsf_runBANDS => ::pwtk::queue::runBANDS * lsf_runMOPDOS => ::pwtk::queue::runMOPDOS
submitting the whole job to batch queuing system
* lsf_submit => ::pwtk::queue::submit
some extra stuff ...
* lsf_profile => ::pwtk::queue::profile * lsf_profileDefault => ::pwtk::queue::profileDefault