How to pull or build a container¶
1. Pulling from a registry with Singularity¶
Often we are interested in containers that live in a registry somewhere, such as Docker Hub, the Singularity Library, or Singularity Hub. In “pulling” a container, we download a pre-built image from a container registry.
Let us say we are interested in a Docker container like the one on Docker Hub at https://hub.docker.com/r/bitnami/python. We can now use the singularity pull command (more detail below) to grab this container with syntax that specifies both the user offering the container (here, bitnami) and the name of the container (here, python):
singularity pull docker://bitnami/python
By default, this creates a file with a .sif extension called python_latest.sif.
Similarily, we can pull the official image for the Julia programming language here with singularity pull via
singularity pull docker://julia
and this generates a file called julia_latest.sif. The .sif extension indicates a Singularity-specific version of a SquashFS image, a type of compressed file. Alternatively we could rename the output file or create a different file type by adding an input argument to our call to singularity pull:
singularity pull my_julia.img docker://julia
This generates the container image file my_julia.img.
Independent of whether you create a .sif or .img file, you can work with your output file using Singularity commands like singularity exec. (See later sections.) It no longer matters that this container originally came from Docker Hub.
2. Building from a registry with Singularity¶
Alternatively, we could use the command singularity build to interact with a registry. Because this command takes at least two input arguments, grabbing a container image file from a registry with singularity build looks more like our second example using singularity pull:
singularity build my_julia.img docker://julia
and similarly can generate an output file my_julia.img. Unlike singularity pull, singularity build can be used for a variety of tasks. In addition to grabbing a container image file from a container registry, you can use singularity build to convert between formats of a container image, as we'll see in the next section on building a container from a Dockerfile. (With singularity build, you can also make a container writeable or create a sandbox.)
3. Building from a Dockerfile with Podman¶
Our recommendation is that, if you want to build a container from scratch (e.g. using a Dockerfile), you first build that container using podman build initially and singularity build to change file formats. To do so, you'll want to
1. Build your container with podman.
2. Save the container to a docker `.tar` archive.
3. Re-build the container as a `.sif` file with `singularity build...`
In greater detail:
1. Building with podman¶
To build a container with podman build..., you'll first have to request an allocation with the --userns flag; this flag makes sure you have the necessary privileges to build a container successfully. For example, to ask slurm for a 1 node, 60 minute allocation, you would run
salloc -N 1 -t 60 --userns
Next, you'll want to configure your environment by running the enable-podman.sh script. On LC, it lives at /collab/usr/gapps/lcweg/containers/scripts/enable-podman.sh, so in a bash terminal you would run
. /collab/usr/gapps/lcweg/containers/scripts/enable-podman.sh
(Note that the default driver is vfs but that you can change the driver to overlayfs via . /collab/usr/gapps/lcweg/containers/scripts/enable-podman.sh overlay.)
Now build the container with podman build, specifying the Dockerfile with -f and the tag of the container created with -t. Using the Dockerfile Dockerfile.ubuntu and tag ubuntuimage, we'd run
podman build -f Dockerfile.ubuntu -t ubuntuimage
To try this, you can grab the contents of Dockerfile.ubuntu.
After podman build ..., run podman images to see your container image.
The whole build might look something like this:
janeh@pascal83:~$ salloc -N 1 -t 10 --userns
salloc: Pending job allocation 1535428
salloc: job 1535428 queued and waiting for resources
salloc: job 1535428 has been allocated resources
salloc: Granted job allocation 1535428
salloc: Waiting for resource configuration
salloc: Nodes pascal43 are ready for job
janeh@pascal43:~$ . /collab/usr/gapps/lcweg/containers/scripts/enable-podman.sh
janeh@pascal43:~$ podman build -f Dockerfile.ubuntu -t ubuntuimage
STEP 1: FROM ubuntu:18.04
Getting image source signatures
Copying blob 2f94e549220a done
Copying config 886eca19e6 done
Writing manifest to image destination
Storing signatures
STEP 2: COPY . /app
6ee326fc84d85e629b337aa35ec05fcde6706a859bed0ca3b922212b09499d51
STEP 3: CMD python /app/ubuntu-app.py
STEP 4: COMMIT ubuntuimage
7161b8fa255e24227b11a2b033fba8651ab513c3fb30e520b525220874434d23
7161b8fa255e24227b11a2b033fba8651ab513c3fb30e520b525220874434d23
janeh@pascal43:~$ podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/ubuntuimage latest 7161b8fa255e About a minute ago 4.22 GB
docker.io/library/ubuntu 18.04 886eca19e611 3 days ago 65.5 MB
2. Saving the container to a docker archive¶
By default the container files created on a compute node will be deleted when the allocation ends. After building and before the allocation ends, you need to explicitly save the container! For example, immediately after building the container in the above, if we terminate the allocation and start a new allocation, we see no remaining images with podman images:
janeh@pascal83:~$ salloc -N 1 -t 1 --userns
salloc: Pending job allocation 1535430
salloc: job 1535430 queued and waiting for resources
salloc: job 1535430 has been allocated resources
salloc: Granted job allocation 1535430
salloc: Waiting for resource configuration
salloc: Nodes pascal129 are ready for job
janeh@pascal129:~$ ./enable-podman.sh
janeh@pascal129:~$ podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
janeh@pascal129:~$
Instead, use the syntax podman save TAG_NAME > OUTPUT_FILENAME
to save the container after building. For a container with tag ubuntuimage, we might run
podman save ubuntuimage > ubuntuimage.tar
after building, as below:
janeh@pascal7:~$ podman build -f Dockerfile.ubuntu -t ubuntuimage
STEP 1: FROM ubuntu:18.04
Getting image source signatures
Copying blob 2f94e549220a done
Copying config 886eca19e6 [======================================] 1.4KiB / 1.4KiB
Writing manifest to image destination
Storing signatures
STEP 2: COPY . /app
88d6d0751fd16798dc127cd5e5ae463083f875280dd270849286fb8d8e009c9f
STEP 3: CMD python /app/ubuntu-app.py
STEP 4: COMMIT ubuntuimage
0ef15bc4f175622468ad741aa7ee3cdf3cb78f93ef693f5463fc864905aa24f1
0ef15bc4f175622468ad741aa7ee3cdf3cb78f93ef693f5463fc864905aa24f1
janeh@pascal7:~$ podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/ubuntuimage latest 0ef15bc4f175 10 seconds ago 4.22 GB
docker.io/library/ubuntu 18.04 886eca19e611 3 days ago 65.5 MB
janeh@pascal7:~$ podman save ubuntuimage > ubuntuimage.tar
This creates the file ubuntuimage.tar:
janeh@pascal7:~$ ls ubuntuimage.tar
ubuntuimage.tar
3. Re-building with singularity build...¶
Using the .tar created in the last step, you can build a singularity-compatible .sif file with singularity build using the syntax
singularity build OUTPUT_SIF_FILENAME docker-archive://DOCKER_TAR_FILENAME
For example, singularity build ubuntu.sif docker-archive://ubuntuimage.tar produces ubuntu.sif from ubuntuimage.tar:
janeh@pascal7:~$ singularity build ubuntu.sif docker-archive://ubuntuimage.tar
INFO: Starting build...
Getting image source signatures
Copying blob 40a154bd3352 done
Copying blob 9b11b519d681 done
Copying config 0ef15bc4f1 done
Writing manifest to image destination
Storing signatures
2022/01/10 14:56:00 info unpack layer: sha256:704fb613779082c6361b77dc933a7423f7b4fb6c5c4e2992397d4c1b456afbd8
2022/01/10 14:56:00 warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2022/01/10 14:56:00 warn xattr{/var/tmp/janeh/build-temp-768716639/rootfs/etc/gshadow} destination filesystem does not support xattrs, further warnings will be suppressed
2022/01/10 14:56:01 info unpack layer: sha256:82711e644b0c78c67fcc42949cb625e71d0bed1e6db18e63e870f44dce60e583
INFO: Creating SIF file...
INFO: Build complete: ubuntu.sif
We can now use singularity shell to test that we can run the container image ubuntu.sif in singularity and that it has the expected operating system:
janeh@pascal7:~$ singularity shell ubuntu.sif bash
Singularity> cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.6 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.6 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic