Once an allocation is granted, projects are assigned a "bank" or charge account (banks are called "accounts" in Slurm commands and documentation). Banks exist only on specific clusters, though the same bank name may appear on different clusters. 

Bank Information for Users

There are several tools for viewing bank information, including membership and default banks. LC recommends the use of the bankinfo utility, which works on both Flux and Slurm clusters.

$ bankinfo --help
usage: bankinfo [-h] [-v] [-p] [-n] [-t <bank> | -T <bank> | -r <bank> | -u <user>]
Display fairshare and priority information.
options:
 -h, --help  show this help message and exit
 -v          display detailed usage info
 -p          output "|" delimited columns for easy parsing
 -n          do not display headers
 -t <bank>   display all children of <bank>, including users
 -T <bank>   display all children of <bank>, except users
 -r <bank>   display all parents for <bank>
 -u <user>   display all banks for <user>

Viewing Banks and Usage for Users

Use the bankinfo command to view your current bank memberships and utilization statistics. For example:

$ bankinfo 
Default bank is lc
Name                     Shares  Norm Shares   Norm Usage   Level FS
ROOT                          1     0.000000     0.000000         --
 overhead                     1     1.000000     1.000000         --
  lc*                         1     1.000000     0.031002         --
    user                      1     1.000000     0.000000      0.507
ROOT                          1     0.000000     0.000000         --
 overhead                     1     1.000000     1.000000         --
  standby                     1     1.000000     0.006773         --
    user                      1     1.000000     0.000000      1.000

This output shows that "user" belongs to 2 banks, "lc" and "standby". The lc bank is the default bank for this system and it will be the bank for any jobs that do not specify a different one. The columns show:

  • Shares: The amount of a system time allocated to the bank or user.
  • Norm Shares: The normalized amount allocated to this bank or user. This amount is a proportion of the full system. See the image below for a visual description of how the calculation is made.
  • Norm Usage: This is the normalized amount that the current bank or user has used up.
    • If Norm Usage is higher than Norm Shares, the bank or user has used more than their allocated proportion.
    • If Norm Usage is lower than Norm Shares, the bank or user has used less than their allocated proportion.
  • Level FS (Fair share): This is an indication for how much priority any submitted jobs will be given.  The calculation for job priority is quite complex and additional details can be found below.
    • A fair share value below 1 indicates that jobs could be given lower priority.
    • A fair share value above 1 indicates that jobs could be given higher priority.
bank Shares diagram

Modify the Default Bank

You can change your own default bank, to any bank that you are already a member of. To do so you need to run the following command on a Slurm system (like oslic): 

sacctmgr update user <your-username> set defaultaccount=<new-default-bank> where cluster=<cluster-name>

By specifying <cluster-name> you change the default bank on any cluster in the current zone.  Flux scheduled clusters get their bank information from the Slurm database, so you can change your default bank on a Flux scheduled cluster by running the sacctmgr command on a Slurm cluster in the same network zone (e.g. on oslic to change your default bank on tuolumne).

Getting Added to A Bank

Bank modification requests should be sent by the bank owner (or other authorized person) to lc-support@llnl.gov.

Fairshare Scheduling

This is the primary mechanism used to determine job scheduling. It is based upon a dynamically calculated priority for your job that reflects your share allocation within a bank versus your actual usage.

  • Use more than your share, your priority/service degrades
  • Use less than your share, your priority/service improves
  • Your priority can become very low, but you never "run out of time" at LC.

Jobs with higher priorities often need to acquire their full set of nodes over time. While their nodes are being reserved, the nodes will appear to be idle. Scheduling is dynamic with job priorities and usage information being recalculated frequently.

Half-Life Decay

Without new usage, your current usage value decays to half its value in two weeks. Even though your allocation and/or job priority may be small your job will run if machine resources are sitting idle.

Backfill scheduling allows waiting jobs to use the reserved job slots of higher priority jobs, as long as they do not delay the start of the higher priority job.

The details of the Fair Share with Half-Life Decay algorithm are more complex than presented here. See the following documents for detailed information: https://slurm.schedmd.com/priority_multifactor.html, https://flux-framework.readthedocs.io/projects/flux-accounting/en/latest/components/fair-share.html, https://flux-framework.readthedocs.io/projects/flux-accounting/en/latest/components/job-usage-calculation.html.