Simple Job Step Launch on CORAL systems

Not all users will need the detailed job step / task management options provided by CSM's jsrun. LLNL has developed a jsrun wrapper, lrun, that simplifies job step launch and provides a good task / thread placement and binding for most cases. lrun can also take jsrun options if more detailed controls or options are needed. Use lrun with:

% lrun -Tntasks_per_node | -pntasks [-Nnnodes] [--nolbind] [jsrun_options] app [app_args]

Note that no spaces are allowed within options for lrun. i.e., use '-p4' not '-p 4'. If -N is not given, all nodes in the allocation (from bsub) will be used. The --nolbind option will disable the built-in lrun mpibind task / thread affinity.

Job Step Launch under Slurm and Cluster System Management (CSM)

Both Slurm and IBM's CSM provide a way to launch tasks (aka, Linux processes) of a user's application in parallel across resources allocated to the job. For Slurm, the command is srun; for CSM, the command is jsrun. This page presents their similarities and their differences.

Similarities

A job submission to the batch system (Slurm's sbatch or LSF's bsub) results in a resource allocation when the job runs. Both srun and jsrun rely on the batch system to allocate the computing resources for the job. srun can do both resource allocation and job launch, but that is a difference discussed below). Both commands accept the file name of the user's application to launch as well as the number of tasks to launch. In response to an srun or jsrun invocation, both systems will select the resources (nodes, cores, memory, GPU's) on which to launch the tasks based on the arguments to the respective commands, and then spawn the requested number of tasks. A successful launch by either command is known as a job step.

Differences

The first difference is in terms. What Slurm refers to as "nodes", LSF refers to as "hosts". The discussion below will reflect that: srun launches tasks across nodes, jsrun launches tasks across hosts. In addition, Slurm defines the term CPU to generically refer to cores or hardware threads, depending on the node's configuration. Where Simultaneous Multithreading (SMT) is not available or disabled, "CPU" refers to a core. Where SMT is available and enabled, "CPU" refers to a hardware thread. There is no such distinction with jsrun; the processing element is the CPU.

The next difference is how the resources are selected. srun provides the (-N, --nodes) option to specify a number of nodes. The number of CPUs srun selects will by default be equal to the number of requested tasks. With those two essential specifications, srun will find the CPUs and nodes on which to launch the tasks. Other options to srun allow memory and GPUs to be independently specified.

jsrun's fundamental computing resource unit is known as the resource set. jsrun options allow the user to define the contents of a resource set (CPUs, memory, and GPUs) and the system will find and select the requested number of resource sets from the hosts allocated to the job. The default mapping is one task per resource set.

Both commands provide a means to bind a task (process) to the CPU(s), memory, and GPU(s) selected for it. srun offers the --cpu_bind, --mem_bind, and --acel-bind options to do this, using a number of optional patterns. jsrun always confines tasks to their selected resource sets and offers the --bind option to bind tasks to CPUs.

Similarly, both commands provide the ability to specify the task to resource mapping pattern: block, cyclic or plane.

As an aside, Livermore Computing offers the mpibind SPANK plugin on their Linux clusters in place of srun's native --cpu_bind option. This offers a default mapping and binding selection that considers the OpenMP thread count and makes optimal mapping/binding decisions by default for most cases.  The lrun command provides mpibind bindings on the IBM CORAL systems.

Typically, srun is invoked from within a batch script. The script is submitted to the Slurm scheduler using the sbatch command. Alternatively, users can invoke salloc. The salloc command will block until the job runs, and then return a prompt. From that prompt, users can invoke srun one or multiple times within the same resource allocation. This form of job submission is known as an interactive job.

Slurm allows the srun command to be invoked directly from the command line, i.e., outside of an sbatch or salloc job allocation. When that happens, srun will both request and receive a resource allocation and subsequently launch one job step on the allocated resources and return. The jsrun command does not offer this ability.

The following table presents the equivalent options for srun, jsrun, and lrun. This is a curated list for comparison purposes. For the complete list see the srun and jsrun man pages.

Environment variables are only propagated if they begin with JSM_, OMPI_, or OPAL_.