There is a wealth of documentation about using CUDA, OpenMP, and other programming models for NVIDIA GPUs. This page will focus more on the specifics of running on LC systems with NVIDIA GPUs. For information about running on systems with AMD GPUs, please refer to the El Capitan documentation at https://hpc.llnl.gov/documentation/user-guides/using-el-capitan-systems. For a list of LC systems with GPUs, please refer to https://hpc.llnl.gov/hardware/compute-platforms-gpus.
"Hello World" CUDA Example
Here is an example MPI hello world adapted from https://computer-graphics.se/hello-world-for-cuda.html:
// hello-world.cu // This is the REAL "hello world" for CUDA! // It takes the string "Hello ", prints it, then passes it to CUDA // with an array of offsets. Then the offsets are added in parallel // to produce the string "World!" // By Ingemar Ragnemalm 2010 // nvcc hello-world.cu
// nvcc -DUSEMPI -ccbin=mpic++ hello-world.cu #include <stdio.h> #include <unistd.h> #ifdef USEMPI #include "mpi.h" #endif const int N = 16; const int blocksize = 16; __global__ void hello(char *a, int *b) { a[threadIdx.x] += b[threadIdx.x]; } int main(int argc, char *argv[]) { char a[N] = "Hello \0\0\0\0\0\0"; int b[N] = {15, 10, 6, 0, -11, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}; char *ad; int *bd; const int csize = N*sizeof(char); const int isize = N*sizeof(int); #ifdef USEMPI int rank, size; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); printf("Rank %d/%d ", rank, size); #endif cudaMalloc( (void**)&ad, csize ); cudaMalloc( (void**)&bd, isize ); cudaMemcpy( ad, a, csize, cudaMemcpyHostToDevice ); cudaMemcpy( bd, b, isize, cudaMemcpyHostToDevice ); dim3 dimBlock( blocksize, 1 ); dim3 dimGrid( 1, 1 ); printf("%s", a); hello<<<dimGrid, dimBlock>>>(ad, bd); cudaMemcpy( a, ad, csize, cudaMemcpyDeviceToHost ); printf("%s\n", a); cudaFree( ad ); cudaFree( bd ); return EXIT_SUCCESS; }
To compile, first load the cuda module. You can run module avail cuda to see all the available cuda versions. You can then compile your code with the nvcc command.
$ module avail cuda -------------------- /usr/tce/modulefiles/toolchains/Core --------------------- cuda/10.1.168 cuda/11.3.0 cuda/11.7.0 cuda/12.9.1 cuda/10.2.89 cuda/11.4.1 cuda/11.8.0 cuda/13.1.1 (L,D) cuda/11.1.0 cuda/11.5.0 cuda/12.2.2 cuda/11.2.0 cuda/11.6.1 cuda/12.6.0 Where: L: Module is loaded D: Default Module $ module load cuda $ which nvcc /usr/tce/packages/cuda/cuda-13.1.1/bin/nvcc $ nvcc hello-world.cu
The login nodes do not have GPUs, so you will first need to allocate a compute node. In the example below, notice how the test code prints "Hello Hello" on the login node, meaning the hello kernel was not run on the GPU. After allocating a node with a GPU, the test code successfully prints "Hello World!".
$ ./a.out Hello Hello [lee218@rzvector1:cuda-hello-world]$ salloc -N 1 -G 1 -ppdebug salloc: Granted job allocation 17334 salloc: Waiting for resource configuration salloc: Nodes rzvector9 are ready for job bash-4.4$ ./a.out Hello World!
MPI + CUDA Example
If you have an MPI application that uses CUDA, you can use the nvcc command and specify mpic++ as via the -ccbin argument. This may require you to switch to the gcc compilers. In this example we also allocate using Slurm's --exclusive flag to get the full node, including all its GPUs.
$ module load gcc Lmod is automatically replacing "intel-classic/2021.6.0-magic" with "gcc/10.3.1-magic". Due to MODULEPATH changes, the following have been reloaded: 1) mvapich2/2.3.7 $ nvcc -ccbin=mpic++ hello-world.cu -DUSEMPI $ ./a.out Rank 0/1 Hello Hello $ salloc -N 1 --exclusive -ppdebug salloc: Granted job allocation 17335 salloc: Waiting for resource configuration salloc: Nodes rzvector10 are ready for job bash-4.4$ srun -n 4 a.out Rank 0/4 Hello World! Rank 2/4 Hello World! Rank 1/4 Hello World! Rank 3/4 Hello World!
mpibind
By default, Slurm uses our mpibind plugin (https://computing.llnl.gov/projects/mpibind) to bind processes to CPU cores and GPUs. You can see the distribution by adding --mpibind=verbose to your srun command line. If you need all processes to see all the GPUs, you will want to run with --mpibind=off on your srun command line.
$ srun -n 4 --mpibind=verbose python3 -c 'import torch ; print(torch.cuda.device_count())' mpibind: 4 GPUs on this node mpibind: task 0 nths 28 gpus 1 cpus 0-27 mpibind: task 1 nths 28 gpus 0 cpus 28-55 mpibind: task 2 nths 28 gpus 3 cpus 56-83 mpibind: task 3 nths 28 gpus 2 cpus 84-111 1 1 1 1 bash-4.4$ srun -n 4 --mpibind=off python3 -c 'import torch ; print(torch.cuda.device_count())' 4 4 4 4
Using the NVHPC Compilers
We have an NVHPC installation with an associated cuda-aware mvapich2 outside of the usual /usr/tce and module environment. The best way to access these is via their full paths:
$ /collab/usr/global/tools/nvidia/nvhpc/toss_4_x86_64_ib/nvhpc-26.3/Linux_x86_64/2026/compilers/bin/nvcc hello-world.cu $ /collab/usr/global/tools/mpi/toss_4_x86_64_ib/mvapich2-4.1-nvhpc-26.3/bin/mpicxx cuda_aware.cu
OpenMP Offload
The following is an example code that uses OpenMP offload and MPI:
/*
* Prints out one line for each rank:
* the MPI rank,
* host name of the node being run on,
* whether or not the GPU specified by CUDA_VISIBLE_DEVICES is usable,
* and what CPUs the MPI task is bound to.
*
* Rewritten by John Gyllenhaal at LLNL 6Oct2015
* in order to add CPU binding info based on omp_hello.c
* written by Edgar A. Leon
* Lawrence Livermore National Laboratory
*/
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h> // sysconf
#include <stdint.h>
#include <omp.h>
#include <string.h>
/* __USE_GNU is needed for CPU_ISSET definition */
#ifndef __USE_GNU
#define __USE_GNU 1
#endif
#include <sched.h> // sched_getaffinity
#include "mpi.h"
/* Print out info line for each MPI task */
int main(int argc, char *argv[])
{
int rank, np;
int rc;
char host[1024] = "unknown";
char gpuid[10];
const char *device=NULL;
char cpus[1024 * 6] = "none";
cpu_set_t cpumask;
int i, nc;
int runningOnGPU = 0;
rc = MPI_Init(&argc,&argv);
if (rc != MPI_SUCCESS) {
printf ("Error starting MPI program. Terminating.\n");
MPI_Abort(MPI_COMM_WORLD, rc);
}
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &np);
/* Which host are we running on */
gethostname (host, sizeof(host));
/* From env, get CUDA_VISIBLE_DEVICES which determines which
* GPU to use, defaults to 0 if not set
*/
device = getenv("CUDA_VISIBLE_DEVICES");
if (device == NULL)
device = "0";
strncpy (gpuid, device, sizeof(gpuid));
/* Get CPUs bound to. Based on Edgar Leon's omp_hello.c code */
/* Use static cpu_set_t mask, 1024 max size */
CPU_ZERO_S(sizeof(cpumask), &cpumask);
if (sched_getaffinity(0, sizeof(cpumask), &cpumask) == -1)
{
fprintf (stderr, "Error during sched_getaffinity\n");
exit (1);
}
/* Use hardcoded default max of 1024 cpus for cpumask
* in order to build up 'cpus' string of bound cpus
*/
nc =0;
for (i=0; i < 1024; i++)
{
if (CPU_ISSET_S(i, sizeof(cpumask), &cpumask))
{
nc += sprintf (cpus+nc, "%d ", i);
}
}
/* Determine if GPU is available but no longer use
* printfs on the GPU since it breaks the Cray Compiler
* in Nov 2015.
*/
#pragma omp target map(runningOnGPU)
{
if (omp_is_initial_device() == 0)
runningOnGPU = 1;
}
/* If still running on CPU, GPU must not be available */
if (!runningOnGPU)
printf("Rank %3i Host %-12s UNABLE to use GPU %s CPUs %s\n",
rank, host, gpuid, cpus);
else
printf("Rank %3i Host %-12s Able to use GPU %s CPUs %s\n",
rank, host, gpuid, cpus);
MPI_Finalize();
return 0;
}Here is an example compiling the mpihasgpu.c test and running it on an salloc --exclusive allocation:
$ /collab/usr/global/tools/mpi/toss_4_x86_64_ib/mvapich2-4.1-nvhpc-26.3/bin/mpicxx -fopenmp mpihasgpu.c -mp=gpu bash-4.4$ srun -n 4 a.out Rank 3 Host rzvector9 Able to use GPU 2 CPUs 84 Rank 0 Host rzvector9 Able to use GPU 1 CPUs 0 Rank 2 Host rzvector9 Able to use GPU 3 CPUs 56 Rank 1 Host rzvector9 Able to use GPU 0 CPUs 28