Job Limits

Each LC platform is a shared resource. Users are expected to adhere to the following usage policies to ensure that the resources can be effectively and productively used by everyone. You can view the policies on a system itself by running either:

joblimits
news job.lim.MACHINENAME

Web version of Neo Job Limits

Hardware

Each Neo node is based on Intel Platinum 8480+ processor with 56 cores per socket, 2 sockets per node, and 1024 GB DDR5 memory, as well as 8 NVidia H200 GPUs.

Scheduling

Neo is GPU scheduled. This means that the minimum allocation will include 1 GPU, 14 cores, and 128 GB of memory and all allocations will be multiples of 1 GPU, 14 cores, 128 GB of memory. Please use the Slurm '-G', '--gpus', or '--gpus-per-task' flags to specify how many resources you will need for your job. If you do not specify a GPU option, your allocation may not have the resources available that you expect. You can also allocate whole nodes with '--exclusive'.

Neo scheduling is not technically enforced so users are expected to monitor their own behavior and keep themselves within the current limits while following these policies:

• Users will not compile on the login nodes during daytime hours
• A user can have a maximum of 3 gpu  with a runtime of up to 4 hours in queue during the day
  • Daytime is 0800-2000 Mondays-Fridays not including holidays
  • No production runs allowed, only development and debugging
  • Users won't run computationally intensive work on the login node

We are all family and expect developers to play nice. However if someone's job(s) have taken over the machine

• Call them or send them an email
• Email ramblings-help@llnl.gov with a screenshot so we can take care of the situation by killing work that violates policy.

This approach will be revisited later and additional limits will be set if necessary. If someone monopolizes the machine, developers can always shift to other resources.

Neo Scheduling Examples

Neo is a GPU-scheduled machine where resources are allocated based on GPUs rather than entire nodes. Below are the key principles of how resources are allocated:

• Smallest Allocation: Includes 1 GPU and its 14 local CPU cores.
• Larger Allocations: Include multiple GPUs and all CPU cores local to those GPUs.
• Shared Nodes: Multiple users can share a node, each with dedicated GPUs, memory, and cores.
• Multi-Node Jobs: A job with multiple GPUs can span multiple nodes, even if using less than a full node.

Flags and Their Behavior

1.  -G (Number of GPUs)
   Specifies the number of GPUs needed for the job. Each GPU comes with its associated CPU cores and memory.
   • Examples:
     • salloc -G 1 : Allocates 1 GPU and 14 CPU cores.
     • salloc -G 2 : Allocates 2 GPUs and 28 CPU cores.
   • Notes:
     1. If the requested GPUs are available on the same node, the job will remain on one node.
     2. If the requested GPUs are not available on the same node, the job may span multiple nodes
         
2. -n (Number of Tasks)
   Specifies the number of tasks (processes) required for the job, with one CPU core allocated per task.
   • Examples:
     • salloc -n 1 : Allocates 1 GPU and 14 CPU cores (smallest allocation).
     • salloc -n 14: Allocates 1 GPU and 14 CPU cores (fully utilizes the cores associated with the GPU).
     • salloc -n 15 :Allocates 2 GPUs and 28 CPU cores (spans multiple GPUs when more than 14 tasks are requested).
        
3. -N (Number of Nodes)
   Specifies the number of nodes over which the allocation will be spread.
   • Examples:
     • salloc -N 1 : Allocates 1 GPU and 14 CPU cores (smallest possible allocation on one node).
     • salloc -N 2 : Allocates 2 GPUs and 28 CPU cores, distributed across two nodes.
   • Notes:
     • salloc -G 2 -N 1 : Allocates 2 GPUs on the same node.
     • salloc -G 2 -N 2 : Allocates 1 GPU on each of 2 nodes.
        
4. --exclusive  (Exclusive Node Access)
   Allocates the entire node for all nodes in the request. This ensures no other jobs share the node.
   • Examples:
     • salloc -G 1 --exclusive : Allocates 8 GPUs and 112 CPU cores on the same node (entire node).
     • salloc -G 9 --exclusive : Allocates 16 GPUs and 256 CPU cores, split across 2 nodes.

Additional Notes

• Multi-GPU Allocation:
  • salloc -G 2 allocates 2 GPUs and 28 cores on the same node if possible.
  • If the GPUs are not available on the same node, the job will be split across 2 nodes.
• Node-Specific Allocation:
  • salloc -G 2 -N 1 : Ensures 2 GPUs are allocated from the same node.
  • salloc -G 2 -N 2 : Allocates 1 GPU on each of 2 nodes.