1. A note about performance

Containers can create an easy, friendly user experience, but are typically not optimized for specific systems or hardware. A cost of this is that containerized applications often will not achieve the same performance as would natively built applications, which might, for example, call libraries optimized for the specific hardware and underlying architecture.

This isn't to say that it's not possible for a containerized application to achieve the same (or better) performance as a native application. Adding system-specific hardware libraries or dependencies to a container increase performance at the cost of portability.

2. Limitations of --userns: Multi-node allocations

The --userns flag confers the additional privileges needed to work with a container to only one node in an allocation. If you use salloc --userns ... to request a multi-node allocation, you'll find that you are only able to build containers on the first node in that allocation (the node that you'll get a shell on, if using salloc, or the node that runs the batch script, if using sbatch or srun).

For example, allocating two nodes on pascal:

janeh@pascal83:~$ salloc -N 2 -t 10 --userns
salloc: Pending job allocation 1535493
salloc: job 1535493 queued and waiting for resources
salloc: job 1535493 has been allocated resources
salloc: Granted job allocation 1535493
salloc: Waiting for resource configuration
salloc: Nodes pascal[7-8] are ready for job

While podman build runs successfully on pascal8, it does not on pascal7:

janeh@pascal7:~$ podman build -f Dockerfile.ubuntu -t ubuntu_test
cannot clone: Invalid argument
user namespaces are not enabled in /proc/sys/user/max_user_namespaces
Error: could not get runtime: cannot re-exec process

3. Competing for --userns: Serially scheduled nodes

Avoid building containers on serially scheduled nodes, like those on oslic or boraxo. Only one user per node can use --userns to build containers successfully at a time. You may run into issues building containers if you are competing for resources on a shared node.

For example, I saw the following error messages while trying to build a container in an allocation on boraxo8 after starting another allocation on boraxo8 via salloc --userns under a second username:

janeh@boraxo3:~$ salloc -n1 -ppdebug -t 20 --userns
salloc: Granted job allocation 3387202
salloc: Waiting for resource configuration
salloc: Nodes boraxo8 are ready for job
janeh@boraxo8:~$ ./enable-podman.sh
janeh@boraxo8:~$ podman build -f Dockerfile.ubuntu -t ubuntuimage
ERRO[0000] cannot find mappings for user janeh: No subuid ranges found for user "janeh" in /etc/subuid
ERRO[0000] cannot find mappings for user janeh: No subuid ranges found for user "janeh" in /etc/subuid
STEP 1: FROM ubuntu:18.04 Error: error creating build container: error creating container: error creating read-write layer with ID "9e337546787915ffac51bbb2bd0e2e244b21baf26ca10a6ed33949eb54137e60": there might not be enough IDs available in the namespace (requested 0:65534 for /var/tmp/janeh/config/containers/storage/vfs/dir/9e337546787915ffac51bbb2bd0e2e244b21baf26ca10a6ed33949eb54137e60/etc/gshadow): lchown /var/tmp/janeh/config/containers/storage/vfs/dir/9e337546787915ffac51bbb2bd0e2e244b21baf26ca10a6ed33949eb54137e60/etc/gshadow: invalid argument

4. Memory issues with large container images

When working with sufficiently large container images, you can run into memory issues. For example, you could have a process terminated by an Out of Memory (OOM) killer, or you could see "FATAL" error messages as shown here:

janeh@pascal32:/p/lustre1/janeh$ singularity build oom-build-030122.img docker://ecpe4s/e4s-gpu
INFO: Starting build... Getting image source signatures (...)
INFO: Creating SIF file...
FATAL: While performing build: while creating squashfs: create command failed: exit status 1: Write failed because No space left on device
FATAL ERROR:Failed to write to output filesystem

FATAL ERROR:Failed to write to output filesystem

NOTE This example uses singularity, rather than podman, and that the possibility of memory issues when working with large container images exists for both. The threshold for encountering memory issues is lower when using podman and vfs together than it is for using either singularity or podman and overlayfs.


5. Incompatible architecture

When working with containerized applications, it's essential to ensure the container image's architecture aligns with the host system's architecture. Encountering the error:

standard_init_linux.go:211 exec user process caused exec format error

suggests that you're trying to run a container built for a different architecture than the host system's.

To resolve this, utilize a system that matches the architecture of the container image.