As described on the Compilers page, there are 2 compiler families being developed specifically for the El Capitan platforms: cce and rocmcc. While HPE is providing these compliers in the traditional Cray development environment (see Cray Modules Guide), LC is also packaging these compilers in a way that our users have become accustomed to.  

All LLNL / LC packaged compilers have a -magic postfix in the module version, in particular cce/VERSION-magic and rocmcc/VERSION-magic are specifically for the EAS3 systems.

These modules provide lots of LC magic, courtesy of John Gyllenhaal. Using these modules ensure that once your executable is built, it should run using the correct libraries, no matter which modules are loaded into the environment at runtime. They also make sure that the correct compilers/libraries are used during the entire build process, which is particularly useful for complex build systems which may spawn sub-shells during the build process. Using the -magic compilers ensures that executables can be run across similar LC platforms.

The content of these modules are available via the /usr/tce/packages/ prefix, in particular:

  • /usr/tce/packages/cce/cce-VERSION-magic/
  • /usr/tce/packages/rocmcc/rocmcc-VERSION-magic/
  • /usr/tce/packages/rocmcc/rocmcc-VERSION-cce-VERSION-magic/

Note for Fortan Users

Fortran users should note that the current versions of amdflang are not fully functional and it is recommend that Fortran codes instead use crayftn. See below for details on the rocmcc modules which automatically provide crayftn instead of amdflang.

amdflang is undergoing active development and an anticipated release date is early 2025.  

Versions and Naming Convention

LC-wrapped compilers are available via modules with a -magic at the end of the version number. TCE or tri-lab computing environment is the name of the development environment deployed by LC as part of the LLNL, LANL, and Sandia joint HPC coordination effort.

See the Compilers page for details on the cce and rocmcc compiler families. See the Announcements pages for information on the latests versions available for each platform.

cce/VERSION-magic

The LC-version of the cce compiler family is available via the cce/VERSION-magic modules. 

By default, a particular version of the ROCm libraries is linked by these compilers. For some versions of cce, there may multiple available ROCm versions. For example, cce version 15.0.0 can be used with four different rocm versions. The naming convention we have chosen is as follows:

  • cce/18.0.0-magic uses rocm/6.1.2
  • cce/18.0.0b-magic uses rocm/6.2.0

Users can determine the exact version of rocm linked by an LC compiler wrapper using grep rocm craycc, for example:

$ grep rocm /usr/tce/packages/cce/cce-18.0.0b-magic/bin/craycc
export COMPILER_ROCM_PATH=/opt/rocm-6.2.0

​​​​​Users can also see the paired modules with the module avail command:

$ ml avail 18.0.0b
 
----------------------- /usr/tce/modulefiles/toolchains/Core -------------------------------
   cce/18.0.0b-magic    rocmcc/6.2.0-cce-18.0.0b-magic

The deployment of new complier versions are always accompanied with a note to the machine status email list. Checkout the Announcements page for the latest deployments.

rocmcc/VERSION-magic

The LC-version of the rocmcc compiler family is available via the rocmcc/VERSION-magic module.

By default, the rocmcc and rocm libraries have the same version number. Thus, rocmcc version 5.4.1 includes rocm 5.4.1.

Note At this time, there are a number of known issues with the rocmcc compiler family:

  • The fortran compiler (amdflang) is still in the development stages.
  • Compiling the MPI library elicits a number of bugs.

If users are seeing performance issues, it is highly recommended that they use the rocmcc/VER-cce-VER-magic modules, see below. MPI libraries dependent on these packages are compiled with the cray compilers (cce family).

Fortran users: rocmcc/VERSION-cce-VERSION-magic

For Fortran users, LC has provided rocmcc-tce modules which provide the crayftn compiler instead of the amdflang compiler. These modules are denoted by their module version number, such as rocmcc/5.4.1-cce-15.0.0c-magic

Recommended Use of XPMEM and GTL Libraries

As of August 2024, we are recommending that users always link their application with -lxpmem and the GTL library. These recommended link modifications are done automatically with the -magic wrappers for cray-mpich/8.1.30 (and later), but can be turned off.

See additional details and documentation on the known issues page.

Example Compilation

The following examples describe the basic steps to compile with these compilers. See additional details in the Explicit build paths example page, including some guidance for constructing a basic Makefile.

cce/VERSION-magic Example

  1. ml +cce/18.0.0-magic

    Start by selecting LLNL-specific cray environment module. These modules will have -magic in the version number. This provides 3 compilers: craycc (C), crayCC (C++), and crayftn (fortran).
  2. (if needed) ml +cray-mpich

    Select an MPI module. The system will automatically ensure that the MPI version matches the loaded compiler. If you switch compilers, Lmod should reload any already-loaded MPI packages to the appropriate one.
  3. You can now use any of the following MPI compilers to build your project:

    mpic++, mpicc, mpicraycc, mpicrayCC, mpi-crayftn, mpicxx, mpif77, mpif90, mpifc, mpifort
    1. Use the -vvv flag with any -magic compiler to show details of what MPI wrapper is doing
    2. Use the -vvvv flag with any -magic compiler to view details on what the TCE compiler wrapper added before calling the vendor compiler.
    3. Both -vvv and -vvvv can be used at the same time.
  4. readelf -a a.out | grep PATH

    Verify that libraries have been properly linked using RPATH. If the readelf output contains any instances of RUNPATH something has gone horribly wrong.
  5. Since you've used the LLNL modules, you can now run your executable and it will continue to work, no matter which environment variables are loaded.

rocmcc/VERSION-magic Example

  1. ml +rocmcc/6.1.2-magic

    Start by selecting the LLNL-specific ROCMCC compiler family.
  2. (if needed) ml +cray-mpich

    Select an MPI module. The system will automatically ensure that the MPI matches the loaded compiler. If you switch compilers, Lmod should reload any already-loaded MPI packages to the appropriate one.