Modules and Software Packaging

Livermore Computing uses LMod modules for packaging much of the software available on LC systems, including compilers, MPI, CUDA, debuggers, performance tools and math libraries. LMod is a hierarchical module system originally developed at TACC (Texas Advanced Computer Center). It can be used to easily manage your environment in complex many-compiler and many-MPI systems. For example, perhaps you want to use gcc 10.3.1-magic as your compiler, MVAPICH2 2.3.7 as your MPI implementation, and use HDF5 when building your code. By issuing a few LMod commands like the following:

% module load gcc/10.3.1-magic

% module load mvapich2/2.3.7

% module load hdf5

your PATH will be set to point at an mpicc command that used gcc 10.3.1-magic and mvapich2 2.3.7, along with having HDF5 in your LD_LIBRARY_PATH and LIBRARY_PATH. It's worth noting that LMod is helping setup your environment. It's not creating new files or making anything new accessible to you. It's also valid to use these packages by manually setting environment variables or using full paths.

Since LMod is a re-implementation and extension of the module system that is found on Cray and many other systems, many commands may be familiar to people. All commands begin by invoking module.

Basic LMod Commands

The table below summarizes the most common LMod module commands. Discussions on several common module topics follows. Full documentation for LMod modules can be found at: www.tacc.utexas.edu/research-development/tacc-projects/lmod. The direct link to the User Guide is: lmod.readthedocs.io/en/latest/010_user.html.

Module Command	Shortcut	Description
module avail	ml avail	List available modules
module list	ml	Show modules currently loaded
module load package module add package	ml load package ml add package	Load a selected module
module unload package module rm package	ml unload package ml rm package	Unload a previously loaded module
module swap package1 package2	ml swap package1 package2	Unload package1 and load package2
module purge	ml purge	Unload all loaded modules
module reset	ml reset	Reset loaded modules to system defaults
module display package module show package	n/a ml show package	Display the contents of a selected module
module spider	ml spider	List all modules (not just available ones)
module spider package	ml spider package	Display description of a selected module
module keyword key	ml keyword key	Search for available modules by keyword
module, module help	ml help	Display help, usage information for modules
module use path	ml use path	Add path to the MODULPATH search path

Setting Up the Modules Environment

The modules environment should be setup by default upon login, however, there are some known cases where a user may get a "module: command not found" error. One such instance is for csh/tcsh users who try to run scripts that do not explicitly "#!/usr/bin/csh" or "#!/usr/bin/tcsh" or for csh/tcsh users who run scripts that invoke "#!/bin/sh". To work around this, users are advised to always source the appropriate script in /usr/share/lmod/lmod/init/. For example:

#!/bin/sh
if test -e /usr/share/lmod/lmod/init/sh
then
  . /usr/share/lmod/lmod/init/sh
fi
module list

Loading Packages

New packages can be loaded into your environment with the module load command:

% which git
/usr/tce/bin/git
% module load git/2.45.0
% which git
/usr/tce/packages/git/git-2.45.0/bin/git

Unloading Packages

Packages can be unloaded with the module unload command:

% module unload git
% which git
/usr/tce/bin/git

Listing Packages

To see what modules are available now, use the module avail command:

% module avail
------------------ /usr/tce/modulefiles/Compiler/intel/16.0.3 ------------------
   mvapich2/2.2 (L)    openmpi/1.10.2
-------------------------- /usr/tce/modulefiles/Core ---------------------------
   StdEnv         (L)    inspector/2016.3          opt
   cmake/3.5.2           intel/16.0.2              stat/2.2.0
   dyninst/9.1.0         intel/16.0.3     (L,D)    totalview/2016.01.06
   gcc/4.8-redhat        launchmon/1.0.2           totalview/2016.04.08 (D)
   gcc/4.9.3      (D)    ld-auto-rpath             vtune/2016.3
   git/2.8.3      (L)    mkl/11.3.3
   graphlib/2.0.0        mrnet/5.0.1
-------------------- /usr/share/lmod/lmod/modulefiles/Core ---------------------
   lmod/6.0.15    settarg/6.0.15

  Where:
   L:  Module is loaded
   D:  Default Module

Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching
any of the "keys".

This shows us what modules are current loaded by marking them with an (L). Other modules are marked as defaults with a (D), and those are the versions of modules will be loaded if you don't specify a version number when loading that module (e.g., the command module load totalview will load version 2016.04.08).

Module Hierarchy

LMod can load modules in a hierarchical order, which means that loading module A may make module B become available. TOSS 3 uses this when loading compilers and MPIs. Loading a specific compiler makes all compatible MPIs appear as available modules, and loading an MPI makes compatible MPI packages appear. For example:

% module purge
% module avail

-------------------------- /usr/tce/modulefiles/Core ---------------------------
   StdEnv                inspector/2016.3        opt
   cmake/3.5.2           intel/16.0.2            stat/2.2.0
   dyninst/9.1.0         intel/16.0.3     (D)    totalview/2016.01.06
   gcc/4.8-redhat        launchmon/1.0.2         totalview/2016.04.08 (D)
   gcc/4.9.3      (D)    ld-auto-rpath           vtune/2016.3
   git/2.8.3             mkl/11.3.3
   graphlib/2.0.0        mrnet/5.0.1

% module load intel/16.0.3
% module avail

------------------ /usr/tce/modulefiles/Compiler/intel/16.0.3 ------------------
   impi/5.1.3    mvapich2/2.2    openmpi/1.10.2

-------------------------- /usr/tce/modulefiles/Core ---------------------------
   StdEnv                inspector/2016.3          opt
   cmake/3.5.2           intel/16.0.2              stat/2.2.0
   dyninst/9.1.0         intel/16.0.3     (L,D)    totalview/2016.01.06
   gcc/4.8-redhat        launchmon/1.0.2           totalview/2016.04.08 (D)
   gcc/4.9.3      (D)    ld-auto-rpath             vtune/2016.3
   git/2.8.3             mkl/11.3.3
   graphlib/2.0.0        mrnet/5.0.1

The initial module purge command simply empties the environment (module reset would have moved it back to the initial defaults). The first module avail doesn't list any MPIs available. After issuing a module load intel/16.0.3, the next module avail shows the three MPI implementations, impi/5.1.3, mvapich2/2.2 and openmpi/1.10.2, that are built by Intel 16.0.3. After loading one of those MPIs, more packages may become available.

List All Packages

You may want to see all packages, not just ones that are currently available. Use the module spider command to do so:

% module spider
----------------------------------------------------------------------------
The following is a list of the modules currently available:
----------------------------------------------------------------------------
  ParaView: ParaView/5.0
  StdEnv: StdEnv
  cmake: cmake/3.5, cmake/3.5.2
  cudatoolkit: cudatoolkit/7.5
  dyninst: dyninst/9.1.0
  fftw: fftw/2.1
  gcc: gcc/4.8-redhat, gcc/4.9.3
  git: git/2.8.3
  graphlib: graphlib/2.0.0
  inspector: inspector/2016.3
  intel: intel/15.0, intel/16.0, intel/16.0.2, intel/16.0.3
  intel-mpi: intel-mpi/5.1
  ...

You can also use spider to find out where in the hierarchy a specific module exists by passing it to module spider:

% module spider mvapich2/2.2
-----------------------------------------------------------------
  mvapich2: mvapich2/2.2
-----------------------------------------------------------------
    This module can only be loaded through the following modules:
      gcc/4.8-redhat
      gcc/4.9.3
      intel/14.0.3
      intel/15.0.6
      intel/16.0.2
      intel/16.0.3

Swapping Packages

With LMod you can swap out one level of the hierarchy with another while preserving which packages you have loaded with module swap:

% module load intel
% module load mvapich2
% which mpicc
/usr/tce/packages/mvapich2/mvapich2-2.2.7-intel-2022.1.0-magic/bin/mpicc
% module swap intel gcc

Due to MODULEPATH changes the following have been reloaded:
  1) mvapich2/2.3.7

% which mpicc
/usr/tce/packages/mvapich2/mvapich2-2.3.7-gcc-10.3.1-magic/bin/mpicc

In this example, the module swap intel gcc command removed the intel compiler from the environment and reloaded it with the gcc compiler in its place.

Module Families

LC has defined certain module families, most notably for compilers and MPI libraries. Only one package from each family may be loaded at once. Loading a second module from a given family will effectively unload the first module from that family. The example below shows what happens when gcc and impi are loaded on top of the default intel and mvapich environment:

% module list
Currently Loaded Modules:
  1) intel/16.0.3   2) mvapich2/2.2   3) StdEnv

% module load gcc
Lmod is automatically replacing "intel/16.0.3" with "gcc/4.9.3"

Due to MODULEPATH changes the following have been reloaded:
  1) mvapich2/2.2

% module list

Currently Loaded Modules:
  1) StdEnv   2) gcc/4.9.3   3) mvapich2/2.2

% module load impi
Lmod is automatically replacing "mvapich2/2.2" with "impi/5.1.3"

% module list

Currently Loaded Modules:
  1) StdEnv   2) gcc/4.9.3   3) impi/5.1.3

In this example, the module load intel command removed the Intel compiler from the module environment and replaced it with the gcc compiler in its place (note that icc/icpc/ifort commands will still be accessible from our /usr/tce/bin compiler wrappers). Similarly, the module load impi command removed mvapich from the module environment and replaced it with the Intel MPI. To disable the single package per family behavior, a user may set their LMOD_EXPERT environment variable. The most typical scenario when a user may want to enable expert mode is when requiring use of a specific GNU compiler and a specific Intel compiler simultaneously. For example:

% export LMOD_EXPERT=1

% module list

Currently Loaded Modules:
  1) intel/16.0.3   2) mvapich2/2.2   3) StdEnv

% module load gcc

% module list

Currently Loaded Modules:
  1) intel/16.0.3   2) StdEnv   3) gcc/4.9.3   4) mvapich2/2.2

Using the expert mode is is not generally recommended, however, as it may cause confusion as to which compiler (i.e., does mpicc use GNU or Intel?) is being used with the MPI wrappers.

Default Packages

When you first log in LMod is configured to load a default environment, which is currently set to include the intel compiler and MVAPICH2. You can change your default environment (or create other named environments) with the module save command:

% which mpicc
/usr/tce/packages/mvapich2/mvapich2-2.3.7-intel-2021.6.0-magic/bin/mpicc
% module swap intel gcc

Due to MODULEPATH changes the following have been reloaded:
  1) mvapich2/2.3.7

% module save default
Saved current collection of modules to: default
Then with your next login:

% which mpicc
/usr/tce/packages/mvapich2/mvapich2-2.3.7-gcc-10.3.1-magic/bin/mpicc

Using modules from a cron job

Under normal shell login, we source these files to initialize modules depending on your shell:

sh/bash/ksh users source /etc/profile.d/00-modulepath.sh
source /etc/profile.d/z00_lmod.sh
csh/tcsh users source /etc/profile.d/00-modulepath.csh
source /etc/profile.d/z00_lmod.csh

These files may not be automatically sourced in your cron job environment. To use modules from a cron job, write a script to manually source these files like in the following:

% cat cronmodules
#!/bin/bash
source /etc/profile.d/00-modulepath.sh
source /etc/profile.d/z00_lmod.sh
module load openmpi

% chmod +x cronmodules

% crontab -e
* * * * * /path/to/cronmodules

Known Module Issues

Sometimes it is necessary to reset the module cache. For example if you see an error like the following:

Lmod has detected the following error:  Unable to load module: intel
     /usr/tce/modulefiles/Core/intel/16.0.3.lua : Empty or non-existent file

While processing the following module(s):
    Module fullname  Module Filename
    ---------------  ---------------
    intel/16.0.3     /usr/tce/modulefiles/Core/intel/16.0.3.lua
    StdEnv           /usr/tce/modulefiles/Core/StdEnv.lua

To resolve this issue you can run rm –rf ~/.lmod.d/.cache.

Creating Your Own Module Files

Users and code teams can also create their own module files. Full documentation on how to create an LMod module is available at the LMod webpage. Below is a simple example using the LLNL-developed Stack Trace Analysis Tool (STAT). Various versions of STAT and its modules exist in the standard TCE environment:

[lee218@rzwiz2:~]$ module avail stat

-------------------------- /usr/tce/modulefiles/Core ---------------------------
   stat/2.2.0    stat/3.0.0    stat/4.0.0    stat/4.0.1 (D)

  Where:
   D:  Default Module

Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching
any of the "keys".

There is also a development/test version of STAT available in the /collab/usr/global/tools space. This can be exposed via a new module file:

[lee218@rzwiz2:~]$ cat /collab/usr/global/tools/stat/toss_3_x86_64_ib/modulefiles/stat/test.lua
help([[stat-debugger-test]])

whatis("Name: stat")
whatis("Version: test")
whatis("Category: Debugger")
whatis("URL: %{URL}")

prepend_path("PATH", "/collab/usr/global/tools/stat/toss_3_x86_64_ib/stat-test/bin")
prepend_path("MANPATH", "/collab/usr/global/tools/stat/toss_3_x86_64_ib/stat-test/share/man")
prepend_path("LD_LIBRARY_PATH", "/collab/usr/global/tools/stat/toss_3_x86_64_ib/stat-test/lib")

From the previous module avail command, we did not see the test version. However, a user may run module use to find the appropriate stat/test.lua module file:

[lee218@rzwiz2:~]$ module use /collab/usr/global/tools/stat/toss_3_x86_64_ib/modulefiles
[lee218@rzwiz2:~]$ module avail stat

---------- /collab/usr/global/tools/stat/toss_3_x86_64_ib/modulefiles ----------
   stat/test

-------------------------- /usr/tce/modulefiles/Core ---------------------------
   stat/2.2.0    stat/3.0.0    stat/4.0.0    stat/4.0.1 (D)

  Where:
   D:  Default Module

Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching
any of the "keys".
[lee218@rzwiz2:~]$ which stat-gui
/usr/tce/bin/stat-gui
[lee218@rzwiz2:~]$ module load stat/test
[lee218@rzwiz2:~]$ which stat-gui
/collab/usr/global/tools/stat/toss_3_x86_64_ib/stat-test/bin/stat-gui

We can now see the test version of STAT available via modules and can be loaded.

Users and code teams can create their own modulefiles directories and populate it with .lua module files. The module use command effectively adds the specified path to the front of a user's MODULEPATH environment variable, which instructs lmod where to look for module files. Some code teams may desire having their application added to the default MODULEPATH. This is typically done by submitting a request to the LC Hotline, who can generate a symlink from /usr/apps/modulefiles, which is in the default MODULEPATH, to the code team's module directory. Once the symlink is created, users can manage module files and versions in the target directory.