Containers: Troubleshooting

1. Podman in a bad state

Podman might sometimes run into state-related issues, especially when there have been changes to user configurations, such as when switching filesystem drivers.

To fix this error, first run

cat ~/.config/containers/storage.conf

From the values returned, save the value for graphroot. Then run

buildah unshare rm -rf $(graphroot_value)

Finally, run

podman system reset

NOTEthese steps should be ran on the same node that the issue is occurring on.

The entire process should look something like this

bash-4.4$ podman build -f Dockerfile.linux -t linuximage
ERRO[0000] User-selected graph driver "vfs" overwritten by graph driver "overlay" from database - delete libpod local files ("/tmp/mir2/config/containers/storage") to resolve.  May prevent use of images created by other tools 
ERRO[0000] User-selected graph driver "vfs" overwritten by graph driver "overlay" from database - delete libpod local files ("/tmp/mir2/config/containers/storage") to resolve.  May prevent use of images created by other tools 
Error: overlay: Unknown option vfs.ignore_chown_errors
bash-4.4$ cat ~/.config/containers/storage.conf
[storage]
  driver = "vfs"
  runroot = "/tmp/mir2/run-61136/containers"
  graphroot = "/tmp/mir2/config/containers/storage"
[storage.options.vfs]
  ignore_chown_errors = "true"
  mount_program = "/usr/bin/fuse-overlayfs"
bash-4.4$ buildah unshare rm -rf /tmp/mir2/config/containers/storage
bash-4.4$ podman system reset
WARNING! This will remove:
        - all containers
        - all pods
        - all images
        - all networks
        - all build cache
        - all machines
        - all volumes
Are you sure you want to continue? [y/N] y
 A "/g/g20/mir2/.config/containers/storage.conf" config file exists.
Remove this file if you did not modify the configuration.
bash-4.4$ podman build -f Dockerfile.linux -t linuximage
STEP 1/3: FROM alpine:latest
Resolved "alpine" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob 96526aa774ef done  
Copying config 8ca4688f4f done  
Writing manifest to image destination
Storing signatures
STEP 2/3: LABEL maintainer="your-email@example.com"
--> 591700879c5
STEP 3/3: CMD [ "sh" ]
COMMIT linuximage
--> b14896ac8d5
Successfully tagged localhost/linuximage:latest
b14896ac8d51fdbcc71a50ba6ea39bfb62bd2af22e367e2b783d0e2f035cc21c

2. seccomp issues

seccomp is a security layer that restricts the types and parameters of system calls that a container is able to make to the host operating system. Under normal circumstances, you shouldn't see any issues caused by this layer. However, in rare situations it's possible for the application in your container to behave strangely or fail to function correctly due to seccomp restrictions. The symptoms of this are unpredictable and differ for every application.

For example, the error below occurred from running an Ubuntu image on Lassen.

green77@izgw2:~$ podman run --rm -it ubuntu:22.04 bash
Trying to pull registry.access.redhat.com/ubuntu:22.04...
  name unknown: Repo not found
Trying to pull registry.redhat.io/ubuntu:22.04...
  unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication
Trying to pull docker.io/library/ubuntu:22.04...
Getting image source signatures
Copying blob aece8493d397 done  
Copying config e4c5895818 done  
Writing manifest to image destination
Storing signatures
root@1215f7b1743b:/# apt update
Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Get:2 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]
Err:1 http://security.ubuntu.com/ubuntu jammy-security InRelease   
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB]
Err:2 http://archive.ubuntu.com/ubuntu jammy InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
Err:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
Err:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
Reading package lists... Done
W: http://security.ubuntu.com/ubuntu/dists/jammy-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://security.ubuntu.com/ubuntu/dists/jammy-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://security.ubuntu.com/ubuntu jammy-security InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
E: The repository 'http://security.ubuntu.com/ubuntu jammy-security InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: http://archive.ubuntu.com/ubuntu/dists/jammy/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://archive.ubuntu.com/ubuntu/dists/jammy/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://archive.ubuntu.com/ubuntu jammy InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
E: The repository 'http://archive.ubuntu.com/ubuntu jammy InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: http://archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://archive.ubuntu.com/ubuntu jammy-updates InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
E: The repository 'http://archive.ubuntu.com/ubuntu jammy-updates InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: http://archive.ubuntu.com/ubuntu/dists/jammy-backports/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://archive.ubuntu.com/ubuntu/dists/jammy-backports/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://archive.ubuntu.com/ubuntu jammy-backports InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
E: The repository 'http://archive.ubuntu.com/ubuntu jammy-backports InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
E: Sub-process returned an error code

In the above case, seccomp restrictions are preventing apt from verifying GPG keys. The underlying cause was that the libseccomp on the host system didn't support the syscalls apt was making while performing this operation and returned an error code rather than passing it to the host operating system.

In order to work around this sort of issue, simply add the flag --security-opt=seccomp=unconfined to your podman run command. Note that this doesn't grant the application inside the container any more permissions than if it had been run natively outside the container. Rerunning the same command with the flag specified is show below.

green77@izgw2:~$ podman run --security-opt=seccomp=unconfined --rm -it ubuntu:22.04 bash
root@78d0bb238d97:/# apt update
Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Get:2 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]
Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB]
Get:5 http://archive.ubuntu.com/ubuntu jammy/restricted amd64 Packages [164 kB]
Get:6 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [266 kB]
Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages [1792 kB]     
Get:8 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [44.0 kB]
Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [17.5 MB]
Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1192 kB]
Get:11 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1008 kB]
Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1274 kB]     
Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1419 kB]   
Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [49.8 kB]   
Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1461 kB]        
Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1392 kB]
Get:17 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [78.3 kB]      
Get:18 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [32.6 kB]    
Fetched 28.3 MB in 3s (10.2 MB/s)                                                       
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
7 packages can be upgraded. Run 'apt list --upgradable' to see them.