Containers: Using Containers in CI Pipelines

This is a simple example to build an Almalinux 8 image and inject LLNL CA certificate. After building, the container will be pushed to the Gitlab registry for this repository.

1. Build

Note that we are using the SLURM batch method to build the container in this example.

SLURM parameters

The build job runs in the batch queue using the Gitlab batch executor. The following segment of the CI yaml specifies which arguments should be passed to SLURM when allocating the job:

variables:
  LLNL_SLURM_SCHEDULER_PARAMETERS: "--nodes=1 -p pdebug --userns --exclusive"

The key SLURM parameters are:

--nodes=1: The build must be done with a single-node allocation
--userns: This indicates that SLURM should populate the user namespace mappings for your user
--exclusive: The build must be the only job running on the batch node. On CPU-shared systems, such as oslic and borax, this prevents other jobs from starting on the node.

Specifying the executor

We use tags to specify how and where the CI job should run. In this case, we're running on quartz using the batch executor. The shell executor is also available, but will run on a login node rather than allocating a compute node and will not use the SLURM parameters we set above.

build:
  stage: build
  tags:
    - quartz
    - batch

2. Container Registry

We are using the Gitlab registry to store the final image. The $CI_* variables are automatically populated to point to the registry for this repository.

podman push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"
podman logout $CI_REGISTRY

3. Working with the image

Once the image is stored in the registry, you'll be able to pull it and run containers on LC systems.

4. Pulling the image

First, you'll need to generate a Gitlab PAT (Personal Access Token) here: https://lc.llnl.gov/gitlab/-/profile/personal_access_tokens

The PAT you create should minimally include the read_registry scope. When prompted for your password when authenticating to the registry, the PAT should be used as your password.

5. Using the Image in Another CI Pipeline

Method 1: Pulling the Image in the CI Pipeline

To use the built container image in another CI pipeline, you can pull the image from GitLab's registry in your GitLab CI YAML file. Here's an example showing how to run a container from the pulled image:

variables:
  LLNL_SLURM_SCHEDULER_PARAMETERS: "--nodes=1 -p pdebug --userns --exclusive"

build:
  stage: build
  tags:
    - quartz
    - batch
  script:
    - /collab/usr/gapps/lcweg/containers/scripts/enable-podman.sh
    - echo "${CI_REGISTRY_PASSWORD}" | podman login -u="$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY
    - podman pull "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"
    - podman run -d --name my_deployed_app "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"
    - podman logout $CI_REGISTRY

Method 2: Referencing the Image in Another Dockerfile

Once the container image is pushed to the GitLab Container Registry, the image can be used as the base image in another project's Dockerfile. For exmaple, your Dockerfile might start out with:

FROM: czregistry.llnl.gov:5050/weg/examples/podman-ci-example:latest

Any necessary modifications to the image can be added in the Dockerfile. Then, the image can be rebuild using the build instructions detailed earlier, resulting in an updated version of the container image.

Podman

Configure podman for LC

Podman won't work out-of-the-box on LC systems and requires a few configuration tweaks. Since these settings may change over time, we have a script that automatically generates and installs an appropriate set of podman configuration files for you.

Run the podman configuration script:

/collab/usr/gapps/lcweg/containers/scripts/enable-podman.sh

Pulling the image with podman

podman login -u <your username> czregistry.llnl.gov:5050
podman pull czregistry.llnl.gov:5050/weg/examples/podman-ci-example:main

A note about subuid errors

You may see the following errors when using podman:

ERRO[0000] cannot find UID/GID for user green77: No subuid ranges found for user "green77" in /etc/subuid - check rootless mode in man pages. 
WARN[0000] using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids

When running a container, these messages are typically benign and may be ignored unless other failures occur.

Singularity

This will pull the image and convert it to a Singularity SIF file.

singularity remote login -u <your username> docker://czregistry.llnl.gov:5050
singularity build almalinux8.sif docker://czregistry.llnl.gov:5050/weg/examples/podman-ci-example:main