MSUB(1)               User Contributed Perl Documentation              MSUB(1)



Name
       msub

Synopsis
       msub  [-a datetime][-A account][-c interval][-C directive_prefix][-d
       path]
             [-e path][-E][-h][-I][-j join][-k keep][-K][-l resourcelist][-m
       mailoptions]
             [-M user_list][-N name][-o path][-p priority][-q destination][-r]
             [-S pathlist][-t jobarrays][-u userlist][-v variablelist][-V]
              [-W additionalattributes][-x][-z][script]

Overview
       msub allows users to submit jobs directly to Moab. When a job is
       submitted directly to a resource manager (such as TORQUE), it is
       constrained to run on only those nodes that the resource manager is
       directly monitoring. In many instances, a site may be controlling
       multiple resource managers. When a job is submitted to Moab rather than
       to a specific resource manager, it is not constrained as to what nodes
       it is executed on. msub can accept command line arguments (with the
       same syntax as qsub), job scripts (in either PBS or LoadLeveler
       syntax), or the SSS Job XML specification.

       Note: Moab must run as a root user in order for msub submissions to
       work. Workload submitted via msub when Moab is running as a non-root
       user fail immediately.

       Submitted jobs can then be viewed and controlled via the mjobctl
       command.

       Note: Flags specified in the following table are not necessairly
       supported by all resource managers.

Access
       When Moab is configured to run as root, any user may submit jobs via
       msub.

Flags
       -a (eligible date)
         Format: [[[[CC]YY]MM]DD]hhmm[.SS]

         Default: ---

         Description: Declares the time after which the job is eligible for
         execution.

         Example: msub -a 12041300 cmd.pbs Moab will not schedule the job
         until 1:00 pm on December 4, of the current year.

       -A (account)
         Format: <ACCOUNT NAME>

         Default: ---

         Description: Defines the account associated with the job.

         Example: msub -A research cmd.pbs
          Moab will associate this job with account research.

       -c (checkpoint interval)
         Format: [n|s|c|c=<minutes>]

           n
            No Checkpoint is to be performed.

           s
            Checkpointing is to be performed only when the server executing
           the job is shut down.

           c
            Checkpoint is to be performed at the default minimum time for the
           server executing the job.

           c=<minutes>
            Checkpoint is to be performed at an interval of minutes.

         Default: ---

         Description: Checkpoint of the job will occur at the specified
         interval.

         Example: msub -c c=12 cmd.pbs
          The job will be checkpointed every 12 minutes.

       -C (directive prefix)
         Format: '<PREFIX NAME>'

         Default: First known prefix (#PBS, #@, #BSUB, #!, #MOAB, #MSUB)

         Description: Specifies which directive prefix should be used from a
         job script.

         ? It is best to submit with single quotes. '#PBS'

         ? An empty prefix will cause Moab to not search for any prefix. -C ''

         ? Command line arguments have precedence over script arguments.

         ? Custom prefixes can be used with the -C flag. -C '#MYPREFIX'

         ? Custom directive prefixes must use PBS syntax.

         ? If the -C flag is not given, Moab will take the first default
           prefix found. Once a directive is found, others are ignored.

         Example: msub -C '#MYPREFIX' cmd.pbs
          #MYPREFIX -l walltime=5:00:00 (in cmd.pbs)

         Moab will use the #MYPREFIX directive specified in cmd.pbs, setting
         the wallclock limit to five hours.

       -d (execution directory)
         Format: <PATH>

         Default: Depends on the RM being used. If using TORQUE, the default
         is $HOME. If using SLURM, the default is the submission directory.

         Description: Specifies which directory the job should execute in.

         Example: msub -d /home/test/job12 cmd.pbs
          The job will begin execution in the /home/test/job12 directory.

       -e (error path)
         Format: [<hostname>:]<path>

         Default: $SUBMISSIONDIR/$JOBNAME.e$JOBID

         Description: Defines the path to be used for the standard error
         stream of the batch job.

         Example: msub -e test12/stderr.txt
          The STDERR stream of the job will be placed in the relative (to
         execution) directory specified.

       -E (environment variables)
         Format: ---

         Default: ---

         Description: Moab adds the following variables, if populated, to the
         job's environment:

         ? MOAB_ACCOUNT: Account name.

         ? MOAB_BATCH: Set if a batch job (non-interactive).

         ? MOAB_CLASS: Class name.

         ? MOAB_DEPEND: Job dependency string.

         ? MOAB_GROUP: Group name.

         ? MOAB_JOBID: Job ID. If submitted from the grid, grid jobid.

         ? MOAB_JOBNAME: Job name.

         ? MOAB_MACHINE: Name of the machine (ie. Destination RM) that the job
           is running on.

         ? MOAB_NODECOUNT: Number of nodes allocated to job.

         ? MOAB_NODELIST: Comma-separated list of nodes (listed singly with no
           ppn info).

         ? MOAB_PARTITION: Partition name the job is running in. If grid job,
           cluster scheduler's name.

         ? MOAB_PROCCOUNT: Number of processors allocated to job.

         ? MOAB_QOS: QOS name.

         ? MOAB_TASKMAP: Node list with procs per node listed.
           <nodename>.<procs>

         ? MOAB_USER: User name.

         In SLURM environments, not all variables will be populated since the
         variables are added at submission (such as NODELIST). With
         TORQUE/PBS, the variables are added just before the job is started.

         This feature only works with SLURM and TORQUE/PBS.

         Example: msub -E mySim.cmd
          The job mySim will be submitted with extra environment variables.

       -F (script flags)
         Format: "\"<STRING>\""

         Default: ---

         Description: Specifies the flags TORQUE will pass to the job script
         at execution time.

         Note: The -F flag is only compatible with TORQUE resource managers.

         Example: msub -F "\"arg1 arg2\"" -1 nodes=1,walltime-60 files/job.sh
          TORQUE will pass parameters arg1 and arg2 to the job.sh script when
         the job executes.

       -h (hold)
         Format: ---

         Default: ---

         Description: Specifies that a user hold be applied to the job at
         submission time.

         Example: msub -h cmd.ll
          The job will be submitted with a user hold on it.

       -I (interactive)
         Format: ---

         Default: ---

         Description: Declares the the job is to be run interactively. NOTE:
         qsub must exist on the same host as msub if the interactive job is
         destined for a TORQUE cluster, because the interactive msub request
         will be converted to a qsub -I request.

         Example: msub -I job117.sh
          The job will be submitted in interactive mode.

       -j (join)
         Format: [oe|n]

         Default: n (not merged)

         Description: Declares if the standard error stream of the job will be
         merged with the standard output stream of the job. If "oe" is
         specified, the error and output streams will be merged into the
         output stream.

         Example: msub -j oe cmd.sh
          STDOUT and STDERR will be merged into one file.

       -k (keep)
         Format: [e|o|eo|oe|n]

         Default: n (not retained)

         Description: Defines which (if either) of output and error streams
         will be retained on the execution host (overrides path for stream).

         Example: msub -k oe myjob.sh
          STDOUT and STDERR for the job will be retained on the execution
         host.

       -K (continue running)
         Format: ---

         Default: ---

         Description: Tells the client to continue running until the submitted
         job is completed. The client will query the status of the job every 5
         seconds. The time interval between queries can be specified or
         disabled via MSUBQUERYINTERVAL.

         Note: Use the -K option SPARINGLY (if at all) as it slows down the
         Moab scheduler with frequent queries. Running ten jobs with the -K
         option creates an additional fifty queries per minute for the
         scheduler.

         Example: msub -K newjob.sh
          3
          Job 3 completed*

         *Only shows up after job completion.

       -l (resource list)
         Format: <STRING> -l
         [BANDWIDTH|DDISK|DEADLINE|DEPEND|DMEM|EXCLUDENODES|FEATURE...|]
          Additional options can be references on the resource manager
         extensions page.

         Default: ---

         Description: Defines the resources that are required by the job and
         establishes a limit to the amount of resource that can be consumed.
         Resources native to the resource manager, scheduler resource manager
         extensions, or job flags may be specified. Note that resource lists
         are dependent on the resource manager in use.

         Example: msub -l
         nodes=32:ppn=2,pmem=1800mb,walltime=3600,VAR=testvar:myvalue cmd.sh
          > msub -l
         nodes=32:ppn=2,pmem=1800mb,walltime=3600,VAR=testvar:myvalue cmd.sh

          The job requires 32 nodes with 2 processors each, 1800 MB per task, a walltime of 3600 seconds, and a variable named testvar with a valur of myvalue.

         Note: If JOBNODEMATCHPOLICY is not set, Moab does not reserve the
         requested number of processors on the requested number of nodes. It
         reserves the total number of requested processors (nodes x ppn) on
         any number of nodes. Rather than setting nodes=<value>:ppn=<value>,
         set procs=<value>, replacing <value> with the total number of
         processors the job requires. Note that JOBNODEMATCHPOLICY is not set
         by default.

         Example: msub nodes-32:ppn-2 -l adres=!<resvid> This entry would tell
         Moab to only consider resources outside of the specified <reservation
         id>.

       -m (mail options)
         Format: <STRING> (either n or one or more of the characters a, b, and
         e)

         Default: ---

         Description: Defines the set of conditions (abort,begin,end) when the
         server will send a mail message about the job to the user.

         Example: msub -m be cmd.sh
          Mail notifications will be sent when the job begins and ends.

       -M (mail list)
         Format: <user>[@<host>][,<user>[@<host>],...]

         Default: $JOBOWNER

         Description: Specifies the list of users to whom mail is sent by the
         execution server. Overrides the EMAILADDRESS specified on the USERCFG
         credential.

         Example: msub -M jon@node01,bill@node01,jill@node02 cmd.sh
          Mail will be sent to the specified users if the job is aborted.

       -N (name)
         Format: <STRING>

         Default: STDIN or name of job script

         Description: Specifies the user-specified job name attribute.

         Example: msub -N chemjob3 cmd.sh
          Job will be associated with the name chemjob3.

       -o (output path)
         Format: [<hostname>:]<path> - %J and %I are acceptable variables. %J
         is the master array name and $I is the array member index in the
         array.

         Default: $SUBMISSIONDIR/$JOBNAME.o$JOBID

         Description: Defines the path to be used for the standard output
         stream of the batch job.

         Example: msub -o test12/stdout.txt
          The STDOUT stream of the job will be placed in the relative (to
         execution) directory specified.

          >msub -t 1-2 -o /home/jsmith/simulations/%J-%I.out ~/sim5.sh
          A job array is submitted and the name of the output files includes the master array index and the array member index.

       -p (priority)
         Format: <INTEGER> (between -1024 and 0)

         Default: 0

         Description: Defines the priority of the job. To enable priority
         range from -1024 to +1023, you must use the ENABLEPOSUSERPRIORITY
         parameter.

         Example: msub -p 25 cmd.sh
          The job will have a user priority of 25.

       -q (destination queue (class)
         Format: [<queue>][@<server>]

         Default: [<DEFAULT>]

         Description: Defines the destination of the job.

         Example: msub -q priority cmd.sh
          The job will be submitted to the priority queue.

       -r (rerunable)
         Format: [y|n]

         Default: n

         Description: Declares whether the job is rerunable.

         Example: msub -r n cmd.sh
          The job cannot be rerun.

       -S (shell)
         Format: <path>[@<host>][,<path>[@<host>],...]

         Default: $Shell

         Description: Declares the shell that interprets the job script.

         Example: msub -S /bin/bash
          The job script will be interpreted by the /bin/bash shell.

       -t (job arrays)
         Format: <name>.[<indexlist>]%<limit>

         Default: ---

         Description: Starts a job array with the jobs in the index list. The
         limit variable specifies how many jobs may run at a time. For more
         information, see Submitting Job Arrays.

         Note: Moab enforces an internal limit of 1000,000 sub-jobs that a
         single array job submission can specify.

         Example: > msub -t myarray.[1-1000]%4

       -u (user list)
         Format: <user>[@<host>[,<user>[@<host>],...]

         Default: UID of msub command

         Description: Defines the user name under which the job is to run on
         the execution system.

         Example: msub -u bill@node01 cmd.sh
          On node01 the job will run under Bill's UID, if permitted.

       -v (variable list)
         Format: <string>[,<string>,...]

         Default: ---

         Description: Expands the list the environment variables that are
         exported to the job (taken from the msub command environment).

         Example: msub -v DEBUG cmd.sh
          The DEBUG environment variable will be defined for the job.

       -V (all variables)
         Format: ---

         Default: ---

         Description: Declares that all environment variables in the msub
         environment are exported to the batch job.

         Example: msub -V cmd.sh
          All environment variables will be exported to the job.

       -W (additional attributes)
         Format: <STRING>

         Default: ---

         Description: Allows the for the specification of additional job
         attributes (see Resource Manager Extension).

         Example: msub -W x=GRES:matlab:1 cmd.sh
          The job requires one resource of "matlab".
          This flag can be used to set a filter for what namespaces will be
         passed from a job to a trigger using a comma-delimited list. This
         limits the triger's action to objects contained in certain workflows.
         For more information, see Filtering VC Triggers by Namespace.
          > msub -W x="trigns=vc1,vc2"
          The job passes namespaces vc1 and vc2 to triggers.

       -x
         Format: <SCRIPT> or <COMMAND>

         Default: ---

         Description: When running an interactive job, the -x flag makes it so
         that the corresponding script won't be parsed for PBS directives, but
         is instead a command that is launched once the interactive job has
         started. The job terminates at the completion of this command. This
         option works only when using TORQUE.

         Note: The -x option for msub differs from qsub in that qsub does not
         require the script name to come directly after the flag. The msub
         command requires a script or command immediately after the -x
         declaration.

         Example: msub -I -x ./script.pl

         > msub -I -x /tmp/command

       -z (silent mode)
         Format: ---

         Default: ---

         Description: The job's identifier will not be printed to stdout upon
         submission.

         Example: msub -z cmd.sh
          No job identifier will be printout the stdout upon successful
         submission.

   Job Script
       The msub command supports job scripts written in one of the following
       languages:

         PBS/TORQUE Job Submission Language

         LoadLeveler Job Submission Language
          Use the INSTANTSTAGE parameter as only a subset of the command file
         keywords are interpreted by Moab.

         SSS XML Job Object Specification

         LSF Job Submission Language
          enabled in Moab 4.2.4 and higher

/etc/msubrc
       Sites that wish to automatically add parameters to every job submission
       can populate the file '/etc/msubrc' with global parameters that every
       job submission will inherit.

       For example, if a site wished every job to request a particular generic
       resource they could use the following /etc/msubrc:
        -W x=GRES:matlab:2

Usage Notes
       msub is designed to be as flexible as possible, allowing users
       accustomed to PBS, LSF, or LoadLeveler syntax, to continue submitting
       jobs as they normally would. It is not recommended that different
       styles be mixed together in the same msub command.

       When only one resource manager is configured inside of Moab, all jobs
       are immediately staged to the only resource manager available. However,
       when multiple resource managers are configured Moab will determine
       which resource manager can run the job soonest. Once this has been
       determined, Moab will stage the job to the resource manager.

       It is possible to have Moab take a "best effort" approach at submission
       time using the forward flag. When this flag is specified, Moab will do
       a quick check and make an intelligent guess as to which resource
       manager can run the job soonest and then immediately stage the job.

       Moab can be configured to instantly stage a job to the underlying
       resource manager (like TORQUE/LOADLEVELER) through the parameter
       INSTANTSTAGE. When set inside moab.cfg, Moab will migrate the job
       instantly to an appropriate resource manager. Once migrated, Moab will
       destroy all knowledge of the job and refresh itself based on the
       information given to it from the underlying resource manager.

       In most instances Moab can determine what syntax style the job belongs
       to (PBS or LoadLeveler); if Moab is unable to make a guess, it will
       default the style to whatever resource manager was configured at
       compile time. If LoadLeveler and PBS were both compiled then
       LoadLeveler takes precedence.

       Moab can translate a subset of job attributes from one syntax to
       another. It is therefore possible to submit a PBS style job to a
       LoadLeveler resource manager, and vice versa, though not all job
       attributes will be translated.

Examples
   Example 1
       > msub -l nodes=3:ppn=2,walltime=1:00:00,pmem=100kb script2.pbs.cmd

        4364.orion

   Example 2
       This is the XML-formatted version of Example 1:

        <job>
          <InitialWorkingDirectory>/home/user/test/perlAPI</InitialWorkingDirectory>
          <Executable>/home/user/test/perlAPI/script2.pbs.cmd</Executable>
          <SubmitLanguage>PBS</SubmitLanguage>
          <Requested>
            <Feature>ppn2</Feature>
            <Processors>3</Processors>
            <WallclockDuration>3600</WallclockDuration>
          </Requested>
        </job>

See Also
       Moab Client Installation

       mjobctl

       checkjob

       mshow

       DEFAULTSUBMITLANGUAGE parameter

       MSUBQUERYINTERVAL parameter

       SUBMITFILTER parameter

       Applying the msub Submit Filter for job script sample



perl v5.10.1                      2012-09-28                           MSUB(1)