queue.tcl

::pwtk::queue::QUEUE

::pwtk::queue::QUEUE

SYNOPSIS

proc ::pwtk::queue::QUEUE {queueSystem args} {

PURPOSE

This is a driver for the execution of PWTK script with the batch-queuing system. It creates a batch shell script and a pwtk script that is executed within batch shell script. Then it submits the batch shell script to the queue.

This is a generic command used by the implementation of specific queuing systems, such as SLURM, LL, LSF, which are created with "::pwtk::queue::init $queueSystem ..."

The batch shell script consists of four parts:

        profile   (always exists)
         |
        head      (optional)
         |
        pwtk      (always exists)
         |
        tail      (optional)

where:

profile -- contains the batch-queue directives (e.g. #SBATCH); it is set with ${queueSystem}_profile command
head -- contains optional shell commands that are executed prior to executing PWTK, such as "module load ..."; it is set with ${queueSystem}_head command
pwtk -- in this part, the PWTK script is executed; it is set internally by PWTK
tail -- contains optional shell commands that are executed after the PWTK script finishes; it is set with ${queueSystem}_tail command

USAGE

     ::pwtk::queue::QUEUE queueSystem ?profile? ?-option value ...? script
 or 
     ::pwtk::queue::QUEUE queueSystem ?profile? ?-option value ...? file

EXAMPLES

     ::pwtk::queue::QUEUE queueSystem file.pwtk
     ::pwtk::queue::QUEUE queueSystem { ...script code... }
     ::pwtk::queue::QUEUE queueSystem -option1 value1 -option2 -option3 ... file.pwtk
     ::pwtk::queue::QUEUE queueSystem -option1 value1 -option2 -option3 ... { ...script code... }

     ::pwtk::queue::QUEUE queueSystem profile file.pwtk
     ::pwtk::queue::QUEUE queueSystem profile { ...script code...}
     ::pwtk::queue::QUEUE queueSystem profile -option1 value1 -option2 -option3 ... file.pwtk
     ::pwtk::queue::QUEUE queueSystem profile -option1 value1 -option2 -option3 ... { ...script code... }

REMARK

The last argument of the ::pwtk::queue::QUEUE can be either a PWTK script-code or a PWTK file.

ARGUMENTS

queueSystem -- name of the batch queuing system (e.g. slurm, ll, lsf, ...)
args -- various arguments (see USAGE and EXAMPLES)

SOURCE

    variable direct
    variable queue
    variable counter
    variable pwtk_profile

    # direct mode (i.e. pwtk is executed directly from the created batch script)    
    set direct 1
    incr counter
    
    #
    # parse command-line arguments
    #

    set QUEUE   [string toupper $queueSystem]    
    set nargs   [llength $args]
    set lastarg [lindex $args end]
    set profileName default

    if { $nargs == 0 } {
        ::pwtk::error "no script provided to the $QUEUE command" 1
    } elseif { $nargs == 1 } {
        # called as: $QUEUE script
        # (no parsing of args needed)

        ::pwtk::queue::setProfile_ $queueSystem default

    } elseif { $nargs >= 2} {
        # called with profile and/or options
        # N.B. some options have values but others have not, hence do not make the even-odd check!

        if { [regexp ^- $args] } {
            # the profile was not specified, use default
            set args [concat default $args]            
            incr nargs
        }
        set profileName [lindex $args 0]
        ::pwtk::queue::setProfile_ $queueSystem {*}[lrange $args 0 $nargs-2]
    }

    # script or file ?
        
    if { [file exists $lastarg] } {
        # the last argument is a file
        set scriptHead [file tail [file rootname $lastarg]]
        set script     [::pwtk::readFile $lastarg]
    } else {
        # the last argument is a script code
        set scriptHead [file tail [file rootname [info script]]]
        set script     $lastarg
    }
    
    # PWTK profile script
    
    ::pwtk::ifnotempty pwtk_profile($queueSystem,$profileName) {
        set pwtk_profile_ $pwtk_profile($queueSystem,$profileName)\n
    }
    set script [::pwtk::varvalue pwtk_profile_]$script

    # filenames

    if { $scriptHead ne {} } { append scriptHead . }
    set pwtk_scriptName  [::pwtk::queue::scriptName_ ${scriptHead}$queueSystem.pwtk]
    set pwtk_logName     $pwtk_scriptName.log
    set queue_scriptName [::pwtk::queue::scriptName_ ${scriptHead}$queueSystem.sh]

    writeFile $pwtk_scriptName [::pwtk::propagate_dump]\n$script
    set queue($queueSystem,pwtk) "pwtk $pwtk_scriptName > $pwtk_logName 2>&1\n\n"

    # printout
    
    ::pwtk::printTitle $QUEUE "running PWTK script within the $QUEUE batch queuing system"    
    ::pwtk::print "batch PWTK script  :  $pwtk_scriptName"
    ::pwtk::print "batch PWTK log file:  $pwtk_logName"
    ::pwtk::print "batch profile name :  $profileName"
    ::pwtk::ifnotempty pwtk_profile_ {
        ::pwtk::print "batch PWTK script profile:\n"
        ::pwtk::print [::pwtk::trimIndent $pwtk_profile_]
    }
    ::pwtk::print "\n$QUEUE batch script:  $queue_scriptName
Content of the $QUEUE script:\n\n[::pwtk::skipEmptyLines [get $queueSystem]]\n"

    # submit to $QUEUE
    
    ::pwtk::queue::submit  $queueSystem  $queue_scriptName
    
    set direct 0
}

TABLE OF CONTENTS

::pwtk::queue::QUEUE