Job Limits

Each LC platform is a shared resource. Users are expected to adhere to the following usage policies to ensure that the resources can be effectively and productively used by everyone. You can view the policies on a system itself by running either:

joblimits
news job.lim.MACHINENAME

Web Version of Matrix Job Limits

Matrix is an NVIDIA GPU cluster. Each Matrix node has 4 NVIDIA H100 GPUs with Intel Sapphire Rapids processors with 56 cores per socket and 2 sockets per node, and 512 GB DDR5 memory.

Matrix jobs are scheduled through SLURM and jobs are scheduled per GPU. Matrix has three general use scheduling pools:

• pdebug - 8 GPUs (2 nodes)
• pbatch - 140 GPUs (35 nodes)
• pci - 80 GPUs (20 nodes overlapping pbatch)

Additional scheduling pools may exist for specific user groups.

Job Limits

Pools               Max nodes/job       Max runtime
---------------------------------------------------
pdebug                   1              1 hour
pbatch                  10              24 hours
pci                     10              24 hours
---------------------------------------------------   

Do NOT run computationally intensive work on the login nodes. There are a limited number of login nodes which are meant primarily for editing files and launching jobs. A majority of the time when a login node is laggy, it is because a user has started up a compile on that login node.

Pdebug is for compiling, visualization, and minor debugging.  It is not for production work. Do not use pdebug to run batch jobs.  Do not chain jobs to run one after the other. Individuals who misuse the pdebug queue in this or any similar manner may be denied access to running jobs in the pdebug queue.

Interactive access to a batch node is allowed while you have a batch job running on that node, and only for the purpose of monitoring your job. When logging into a batch node, be mindful of the impact your work has on the other jobs running on the node.

Pci is intended for CI/CD testing jobs originating from GitLab pipelines. It is not for production or debugging work. Do not use pci to run batch jobs. Individuals who misuse the pci queue in this or any similar manner may be denied access to running jobs in the pci queue. Technical limits are in place to restrict the total number of nodes that a given user may use across all jobs within pci. Large parameterized test suites may therefore submit many jobs without interrupting other users.  

Scheduling

Matrix is GPU scheduled. This means that the minimum allocation will include 1 GPU, 28 cores, and 128 GB of memory and all allocations will be multiples of 1 GPU, 28 cores, 128 GB of memory. Please use the Slurm -G, --gpus, or --gpus-per-task flags to specify how many resources you will need for your job. If you do not specify a GPU option, your allocation may not have the resources available that you expect. You can also allocate whole nodes with --exclusive.

Additional scheduling examples can be found below.

Matrix Scheduling Examples

Matrix is a GPU-scheduled machine where resources are allocated based on GPUs rather than entire nodes. Below are the key principles of how resources are allocated:

• Smallest Allocation: Includes 1 GPU and its 28 local CPU cores.
• Larger Allocations: Include multiple GPUs and all CPU cores local to those GPUs.
• Shared Nodes: Multiple users can share a node, each with dedicated GPUs, memory, and cores.
• Multi-Node Jobs: A job with multiple GPUs can span multiple nodes, even if using less than a full node.

Flags and Their Behavior

1.  -G (Number of GPUs)
   Specifies the number of GPUs needed for the job. Each GPU comes with its associated CPU cores and memory.
   • Examples:
     • salloc -G 1 : Allocates 1 GPU and 28 CPU cores.
     • salloc -G 2 : Allocates 2 GPUs and 56 CPU cores.
   • Notes:
     1. If the requested GPUs are available on the same node, the job will remain on one node.
     2. If the requested GPUs are not available on the same node, the job may span multiple nodes
         
2. -n (Number of Tasks)
   Specifies the number of tasks (processes) required for the job, with one CPU core allocated per task.
   • Examples:
     • salloc -n 1 : Allocates 1 GPU and 28 CPU cores (smallest allocation).
     • salloc -n 28: Allocates 1 GPU and 28 CPU cores (fully utilizes the cores associated with the GPU).
     • salloc -n 29 :Allocates 2 GPUs and 56 CPU cores (spans multiple GPUs when more than 28 tasks are requested).
        
3. -N (Number of Nodes)
   Specifies the number of nodes over which the allocation will be spread.
   • Examples:
     • salloc -N 1 : Allocates 1 GPU and 28 CPU cores (smallest possible allocation on one node).
     • salloc -N 2 : Allocates 2 GPUs and 56 CPU cores, distributed across two nodes.
   • Notes:
     • salloc -G 2 -N 1 : Allocates 2 GPUs on the same node.
     • salloc -G 2 -N 2 : Allocates 1 GPU on each of 2 nodes.
        
4. --exclusive  (Exclusive Node Access)
   Allocates the entire node for all nodes in the request. This ensures no other jobs share the node.
   • Examples:
     • salloc -G 1 --exclusive : Allocates 4 GPUs and 112 CPU cores on the same node (entire node).
     • salloc -G 5 --exclusive : Allocates 8 GPUs and 256 CPU cores, split across 2 nodes.

Additional Notes

• Multi-GPU Allocation:
  • salloc -G 2 allocates 2 GPUs and 56 cores on the same node if possible.
  • If the GPUs are not available on the same node, the job will be split across 2 nodes.
• Node-Specific Allocation:
  • salloc -G 2 -N 1 : Ensures 2 GPUs are allocated from the same node.
  • salloc -G 2 -N 2 : Allocates 1 GPU on each of 2 nodes.