Created attachment 1955740 [details] Complete log Description of problem: In CS9 compose CentOS-Stream-9-20230403.0, podman fails to run ubi8-minimal image (with root user), and also fails to run ubi8 image (as rootless user). The error is: "Error: copying system image from manifest list: Source image rejected: None of the signatures were accepted, reasons: No public keys imported..." Failing commands are: sudo podman run ubi8-minimal:latest cat /etc/redhat-release and, podman run ubi8:latest cat /etc/redhat-release Failing task is: TASK [run ubi8 image with root] ************************************************ FAILED - RETRYING: [192.168.100.50]: run ubi8 image with root (30 retries left). [...] FAILED - RETRYING: [192.168.100.50]: run ubi8 image with root (1 retries left). fatal: [192.168.100.50]: FAILED! => changed=true attempts: 30 cmd: - podman - run - ubi8-minimal:latest - cat - /etc/redhat-release delta: '0:00:01.457860' end: '2023-04-03 12:33:42.241795' msg: non-zero return code rc: 125 start: '2023-04-03 12:33:40.783935' stderr: |- Resolved "ubi8-minimal" as an alias (/etc/containers/registries.conf.d/001-rhel-shortnames.conf) Trying to pull registry.access.redhat.com/ubi8-minimal:latest... Error: copying system image from manifest list: Source image rejected: None of the signatures were accepted, reasons: No public keys imported; No public keys imported; No public keys imported; No public keys imported; No public keys imported; No public keys imported stderr_lines: <omitted> stdout: '' stdout_lines: <omitted> ...ignoring Version-Release number of selected component (if applicable): 2:4.4.1-3.el9 podman-4.4.1-3.el9.x86_64 How reproducible: 100% Steps to Reproduce: 1. Deploy CS9 VM in Openstack PSI 2. git clone https://github.com/virt-s1/rhel-edge.git 3. cd ~/rhel-edge 4. ./ostree.sh Actual results: Podman fails to run inside nested CS9 VM. Expected results: Podman not failing. Additional info:
@dornelas this looks like something is off in the CS9 make up. Is there someone you know we could ask to look at this?
Following lines show log-level debug of podman run: [admin@vm-1 ~]$ sudo podman run --log-level debug ubi8-minimal:latest cat /etc/os-release INFO[0000] podman filtering at log level debug inimal:latest cat /etc/os-release DEBU[0000] Called run.PersistentPreRunE(podman run --log-level debug ubi8-minimal:latest cat /etc/os-release) DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver overlay DEBU[0000] Using graph root /var/lib/containers/storage DEBU[0000] Using run root /run/containers/storage DEBU[0000] Using static dir /var/lib/containers/storage/libpod DEBU[0000] Using tmp dir /run/libpod DEBU[0000] Using volume path /var/lib/containers/storage/volumes DEBU[0000] Using transient store: false DEBU[0000] Set libpod namespace to "" DEBU[0000] [graphdriver] trying provided driver "overlay" DEBU[0000] overlay: imagestore=/usr/share/containers/storage DEBU[0000] Cached value indicated that overlay is supported DEBU[0000] Cached value indicated that overlay is supported DEBU[0000] Cached value indicated that metacopy is being used DEBU[0000] Cached value indicated that native-diff is not being used INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true DEBU[0000] Initializing event backend journald DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument DEBU[0000] Using OCI runtime "/usr/bin/crun" INFO[0000] Setting parallel job count to 7 DEBU[0000] Successfully loaded 1 networks DEBU[0000] Pulling image ubi8-minimal:latest (policy: missing) DEBU[0000] Looking up image "ubi8-minimal:latest" in local containers storage DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf" DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/001-rhel-shortnames.conf" DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/002-rhel-shortnames-overrides.conf" DEBU[0000] Trying "registry.access.redhat.com/ubi8-minimal:latest" ... DEBU[0000] Trying "localhost/ubi8-minimal:latest" ... DEBU[0000] Trying "registry.access.redhat.com/ubi8-minimal:latest" ... DEBU[0000] Trying "registry.redhat.io/ubi8-minimal:latest" ... DEBU[0000] Trying "docker.io/library/ubi8-minimal:latest" ... DEBU[0000] Trying "docker.io/library/ubi8-minimal:latest" ... DEBU[0000] Trying "ubi8-minimal:latest" ... DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Attempting to pull candidate registry.access.redhat.com/ubi8-minimal:latest for ubi8-minimal:latest DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/usr/share/containers/storage,overlay.mountopt=nodev,metacopy=on]registry.access.redhat.com/ubi8-minimal:latest" DEBU[0000] Resolved "ubi8-minimal" as an alias (/etc/containers/registries.conf.d/001-rhel-shortnames.conf) Resolved "ubi8-minimal" as an alias (/etc/containers/registries.conf.d/001-rhel-shortnames.conf) Trying to pull registry.access.redhat.com/ubi8-minimal:latest... DEBU[0000] Copying source image //registry.access.redhat.com/ubi8-minimal:latest to destination image [overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/usr/share/containers/storage,overlay.mountopt=nodev,metacopy=on]registry.access.redhat.com/ubi8-minimal:latest DEBU[0000] Using registries.d directory /etc/containers/registries.d DEBU[0000] Trying to access "registry.access.redhat.com/ubi8-minimal:latest" DEBU[0000] No credentials matching registry.access.redhat.com/ubi8-minimal found in /run/containers/0/auth.json DEBU[0000] No credentials matching registry.access.redhat.com/ubi8-minimal found in /root/.config/containers/auth.json DEBU[0000] No credentials matching registry.access.redhat.com/ubi8-minimal found in /root/.docker/config.json DEBU[0000] No credentials matching registry.access.redhat.com/ubi8-minimal found in /root/.dockercfg DEBU[0000] No credentials for registry.access.redhat.com/ubi8-minimal found DEBU[0000] Lookaside configuration: using "docker" namespace registry.access.redhat.com DEBU[0000] Using "sigstore" https://access.redhat.com/webassets/docker/content/sigstore DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.access.redhat.com DEBU[0000] Sigstore attachments: using "docker" namespace registry.access.redhat.com DEBU[0000] GET https://registry.access.redhat.com/v2/ DEBU[0005] Ping https://registry.access.redhat.com/v2/ status 200 DEBU[0005] GET https://registry.access.redhat.com/v2/ubi8-minimal/manifests/latest DEBU[0005] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json" DEBU[0005] Using blob info cache at /var/lib/containers/cache/blob-info-cache-v1.boltdb DEBU[0005] Source is a manifest list; copying (only) instance sha256:3e1adcc31c6073d010b8043b070bd089d7bf37ee2c397c110211a6273453433f for current system DEBU[0005] GET https://registry.access.redhat.com/v2/ubi8-minimal/manifests/sha256:3e1adcc31c6073d010b8043b070bd089d7bf37ee2c397c110211a6273453433f DEBU[0006] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json" DEBU[0006] IsRunningImageAllowed for image docker:registry.access.redhat.com/ubi8-minimal:latest DEBU[0006] Using transport "docker" specific policy section registry.access.redhat.com DEBU[0006] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8-minimal@sha256=3e1adcc31c6073d010b8043b070bd089d7bf37ee2c397c110211a6273453433f/signature-1 DEBU[0006] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8-minimal@sha256=3e1adcc31c6073d010b8043b070bd089d7bf37ee2c397c110211a6273453433f/signature-2 DEBU[0006] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8-minimal@sha256=3e1adcc31c6073d010b8043b070bd089d7bf37ee2c397c110211a6273453433f/signature-3 DEBU[0006] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8-minimal@sha256=3e1adcc31c6073d010b8043b070bd089d7bf37ee2c397c110211a6273453433f/signature-4 DEBU[0006] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8-minimal@sha256=3e1adcc31c6073d010b8043b070bd089d7bf37ee2c397c110211a6273453433f/signature-5 DEBU[0006] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8-minimal@sha256=3e1adcc31c6073d010b8043b070bd089d7bf37ee2c397c110211a6273453433f/signature-6 DEBU[0006] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8-minimal@sha256=3e1adcc31c6073d010b8043b070bd089d7bf37ee2c397c110211a6273453433f/signature-7 DEBU[0006] ... got status 404, as expected = end of signatures DEBU[0006] Not looking for sigstore attachments: disabled by configuration DEBU[0007] Requirement 0: denied, done DEBU[0007] Error pulling candidate registry.access.redhat.com/ubi8-minimal:latest: copying system image from manifest list: Source image rejected: None of the signatures were accepted, reasons: No public keys imported; No public keys imported; No public keys imported; No public keys imported; No public keys imported; No public keys imported Error: copying system image from manifest list: Source image rejected: None of the signatures were accepted, reasons: No public keys imported; No public keys imported; No public keys imported; No public keys imported; No public keys imported; No public keys imported DEBU[0007] Shutting down engines Rootless execution: [admin@vm-1 ~]$ podman run --log-level debug ubi8:latest cat /etc/os-release INFO[0000] podman filtering at log level debug DEBU[0000] Called run.PersistentPreRunE(podman run --log-level debug ubi8:latest cat /etc/os-release) DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /var/home/admin/.local/share/containers/storage/libpod/bolt_state.db DEBU[0000] Overriding graph root "/var/home/admin/.local/share/containers/storage" with "/home/admin/.local/share/containers/storage" from database DEBU[0000] Overriding static dir "/var/home/admin/.local/share/containers/storage/libpod" with "/home/admin/.local/share/containers/storage/libpod" from database DEBU[0000] Overriding volume path "/var/home/admin/.local/share/containers/storage/volumes" with "/home/admin/.local/share/containers/storage/volumes" from database DEBU[0000] systemd-logind: Unknown object '/'. DEBU[0000] Using graph driver overlay DEBU[0000] Using graph root /home/admin/.local/share/containers/storage DEBU[0000] Using run root /run/user/1000/containers DEBU[0000] Using static dir /home/admin/.local/share/containers/storage/libpod DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp DEBU[0000] Using volume path /home/admin/.local/share/containers/storage/volumes DEBU[0000] Using transient store: false DEBU[0000] Set libpod namespace to "" DEBU[0000] [graphdriver] trying provided driver "overlay" DEBU[0000] Cached value indicated that overlay is supported DEBU[0000] Cached value indicated that overlay is supported DEBU[0000] Cached value indicated that metacopy is not being used DEBU[0000] Cached value indicated that native-diff is usable DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false DEBU[0000] Initializing event backend journald DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument DEBU[0000] Using OCI runtime "/usr/bin/crun" INFO[0000] Setting parallel job count to 7 DEBU[0000] Successfully loaded 1 networks DEBU[0000] Pulling image ubi8:latest (policy: missing) DEBU[0000] Looking up image "ubi8:latest" in local containers storage DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf" DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/001-rhel-shortnames.conf" DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/002-rhel-shortnames-overrides.conf" DEBU[0000] Trying "registry.access.redhat.com/ubi8:latest" ... DEBU[0000] Trying "localhost/ubi8:latest" ... DEBU[0000] Trying "registry.access.redhat.com/ubi8:latest" ... DEBU[0000] Trying "registry.redhat.io/ubi8:latest" ... DEBU[0000] Trying "docker.io/library/ubi8:latest" ... DEBU[0000] Trying "docker.io/library/ubi8:latest" ... DEBU[0000] Trying "ubi8:latest" ... DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Attempting to pull candidate registry.access.redhat.com/ubi8:latest for ubi8:latest DEBU[0000] parsed reference into "[overlay@/home/admin/.local/share/containers/storage+/run/user/1000/containers]registry.access.redhat.com/ubi8:latest" DEBU[0000] Resolved "ubi8" as an alias (/etc/containers/registries.conf.d/001-rhel-shortnames.conf) Resolved "ubi8" as an alias (/etc/containers/registries.conf.d/001-rhel-shortnames.conf) Trying to pull registry.access.redhat.com/ubi8:latest... DEBU[0000] Copying source image //registry.access.redhat.com/ubi8:latest to destination image [overlay@/home/admin/.local/share/containers/storage+/run/user/1000/containers]registry.access.redhat.com/ubi8:latest DEBU[0000] Using registries.d directory /etc/containers/registries.d DEBU[0000] Trying to access "registry.access.redhat.com/ubi8:latest" DEBU[0000] No credentials matching registry.access.redhat.com/ubi8 found in /run/user/1000/containers/auth.json DEBU[0000] No credentials matching registry.access.redhat.com/ubi8 found in /var/home/admin/.config/containers/auth.json DEBU[0000] No credentials matching registry.access.redhat.com/ubi8 found in /home/admin/.docker/config.json DEBU[0000] No credentials matching registry.access.redhat.com/ubi8 found in /home/admin/.dockercfg DEBU[0000] No credentials for registry.access.redhat.com/ubi8 found DEBU[0000] Lookaside configuration: using "docker" namespace registry.access.redhat.com DEBU[0000] Using "sigstore" https://access.redhat.com/webassets/docker/content/sigstore DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.access.redhat.com DEBU[0000] Sigstore attachments: using "docker" namespace registry.access.redhat.com DEBU[0000] GET https://registry.access.redhat.com/v2/ DEBU[0000] Ping https://registry.access.redhat.com/v2/ status 200 DEBU[0000] GET https://registry.access.redhat.com/v2/ubi8/manifests/latest DEBU[0000] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json" DEBU[0000] Using blob info cache at /home/admin/.local/share/containers/cache/blob-info-cache-v1.boltdb DEBU[0000] Source is a manifest list; copying (only) instance sha256:4a6dbfbb845810dce5902ab80cb93ecb24c367460fff9d15438e0b3080e244b3 for current system DEBU[0000] GET https://registry.access.redhat.com/v2/ubi8/manifests/sha256:4a6dbfbb845810dce5902ab80cb93ecb24c367460fff9d15438e0b3080e244b3 DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json" DEBU[0001] IsRunningImageAllowed for image docker:registry.access.redhat.com/ubi8:latest DEBU[0001] Using transport "docker" specific policy section registry.access.redhat.com DEBU[0001] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8@sha256=4a6dbfbb845810dce5902ab80cb93ecb24c367460fff9d15438e0b3080e244b3/signature-1 DEBU[0001] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8@sha256=4a6dbfbb845810dce5902ab80cb93ecb24c367460fff9d15438e0b3080e244b3/signature-2 DEBU[0001] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8@sha256=4a6dbfbb845810dce5902ab80cb93ecb24c367460fff9d15438e0b3080e244b3/signature-3 DEBU[0001] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8@sha256=4a6dbfbb845810dce5902ab80cb93ecb24c367460fff9d15438e0b3080e244b3/signature-4 DEBU[0001] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8@sha256=4a6dbfbb845810dce5902ab80cb93ecb24c367460fff9d15438e0b3080e244b3/signature-5 DEBU[0001] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8@sha256=4a6dbfbb845810dce5902ab80cb93ecb24c367460fff9d15438e0b3080e244b3/signature-6 DEBU[0001] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8@sha256=4a6dbfbb845810dce5902ab80cb93ecb24c367460fff9d15438e0b3080e244b3/signature-7 DEBU[0001] ... got status 404, as expected = end of signatures DEBU[0001] Not looking for sigstore attachments: disabled by configuration DEBU[0002] Requirement 0: denied, done DEBU[0002] Error pulling candidate registry.access.redhat.com/ubi8:latest: copying system image from manifest list: Source image rejected: None of the signatures were accepted, reasons: No public keys imported; No public keys imported; No public keys imported; No public keys imported; No public keys imported; No public keys imported Error: copying system image from manifest list: Source image rejected: None of the signatures were accepted, reasons: No public keys imported; No public keys imported; No public keys imported; No public keys imported; No public keys imported; No public keys imported DEBU[0002] Shutting down engines
I can not reproduce this bug in composes CentOS-Stream-9-20230405.1 nor CentOS-Stream-9-20230410.0 (most recent composes) I will close this bug for that reason. In case of having this failure again, I will reopen the bug.
I'm seeing this problem on the current CentOS Stream 9 release (`dnf update` from yesterday)...is that to be expected? (When will the changes from 5 April be available as a "release"?)
@lsm5 Do you know the answer to Webb's question? I would have expected that would be in CentoS 9 by now, but maybe I'm off base.
(In reply to Webb Scales from comment #5) > I'm seeing this problem on the current CentOS Stream 9 release (`dnf update` > from yesterday)...is that to be expected? (When will the changes from 5 > April be available as a "release"?) Tried it just now on a fresh CentOS 9 Stream with podman 4.4.1-9 and it worked for both ubi8 and ubi8-minimal with both root and rootless. 4.4.1-9 seems to be the latest build on my env. Could you please let me know what version of the rpm you're using? (rpm -q podman)
@lsm5, I'm using Podman 4.4.1-9. It's the pull which is failing, so I'm not sure what you mean by "both root and rootless". I'm running Podman as a non-priv'd user. Below is the rpm output, with crypt thrown in, since there seems to be a signing problem. Thanks! ---- $ rpm -qa | grep -E -e crypt -e podman libxcrypt-4.4.18-3.el9.x86_64 libxcrypt-compat-4.4.18-3.el9.x86_64 libxcrypt-devel-4.4.18-3.el9.x86_64 cryptsetup-libs-2.6.0-2.el9.x86_64 python3.11-cryptography-37.0.2-5.el9.x86_64 libgcrypt-1.10.0-10.el9.x86_64 crypto-policies-20230505-1.gitf69bbc2.el9.noarch podman-4.4.1-9.el9.x86_64 crypto-policies-scripts-20230505-1.gitf69bbc2.el9.noarch python3-cryptography-36.0.1-4.el9.x86_64