Bug 2184462
| Summary: | Podman in CS9 error copying system image from manifest list reasons: No public keys imported | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Mario Cattamo <mcattamo> | ||||
| Component: | podman | Assignee: | Tom Sweeney <tsweeney> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | atomic-bugs <atomic-bugs> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | CentOS Stream | CC: | bbaude, bstinson, dornelas, dwalsh, jnovy, jwboyer, lsm5, mboddu, mheon, pthomas, tsweeney, umohnani, wscales | ||||
| Target Milestone: | rc | ||||||
| Target Release: | --- | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2023-04-11 08:51:10 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Mario Cattamo
2023-04-04 18:45:05 UTC
@dornelas this looks like something is off in the CS9 make up. Is there someone you know we could ask to look at this? Following lines show log-level debug of podman run:
[admin@vm-1 ~]$ sudo podman run --log-level debug ubi8-minimal:latest cat /etc/os-release
INFO[0000] podman filtering at log level debug inimal:latest cat /etc/os-release
DEBU[0000] Called run.PersistentPreRunE(podman run --log-level debug ubi8-minimal:latest cat /etc/os-release)
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: imagestore=/usr/share/containers/storage
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is being used
DEBU[0000] Cached value indicated that native-diff is not being used
INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true
DEBU[0000] Initializing event backend journald
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 7
DEBU[0000] Successfully loaded 1 networks
DEBU[0000] Pulling image ubi8-minimal:latest (policy: missing)
DEBU[0000] Looking up image "ubi8-minimal:latest" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] }
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/001-rhel-shortnames.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/002-rhel-shortnames-overrides.conf"
DEBU[0000] Trying "registry.access.redhat.com/ubi8-minimal:latest" ...
DEBU[0000] Trying "localhost/ubi8-minimal:latest" ...
DEBU[0000] Trying "registry.access.redhat.com/ubi8-minimal:latest" ...
DEBU[0000] Trying "registry.redhat.io/ubi8-minimal:latest" ...
DEBU[0000] Trying "docker.io/library/ubi8-minimal:latest" ...
DEBU[0000] Trying "docker.io/library/ubi8-minimal:latest" ...
DEBU[0000] Trying "ubi8-minimal:latest" ...
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] }
DEBU[0000] Attempting to pull candidate registry.access.redhat.com/ubi8-minimal:latest for ubi8-minimal:latest
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/usr/share/containers/storage,overlay.mountopt=nodev,metacopy=on]registry.access.redhat.com/ubi8-minimal:latest"
DEBU[0000] Resolved "ubi8-minimal" as an alias (/etc/containers/registries.conf.d/001-rhel-shortnames.conf)
Resolved "ubi8-minimal" as an alias (/etc/containers/registries.conf.d/001-rhel-shortnames.conf)
Trying to pull registry.access.redhat.com/ubi8-minimal:latest...
DEBU[0000] Copying source image //registry.access.redhat.com/ubi8-minimal:latest to destination image [overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/usr/share/containers/storage,overlay.mountopt=nodev,metacopy=on]registry.access.redhat.com/ubi8-minimal:latest
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Trying to access "registry.access.redhat.com/ubi8-minimal:latest"
DEBU[0000] No credentials matching registry.access.redhat.com/ubi8-minimal found in /run/containers/0/auth.json
DEBU[0000] No credentials matching registry.access.redhat.com/ubi8-minimal found in /root/.config/containers/auth.json
DEBU[0000] No credentials matching registry.access.redhat.com/ubi8-minimal found in /root/.docker/config.json
DEBU[0000] No credentials matching registry.access.redhat.com/ubi8-minimal found in /root/.dockercfg
DEBU[0000] No credentials for registry.access.redhat.com/ubi8-minimal found
DEBU[0000] Lookaside configuration: using "docker" namespace registry.access.redhat.com
DEBU[0000] Using "sigstore" https://access.redhat.com/webassets/docker/content/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.access.redhat.com
DEBU[0000] Sigstore attachments: using "docker" namespace registry.access.redhat.com
DEBU[0000] GET https://registry.access.redhat.com/v2/
DEBU[0005] Ping https://registry.access.redhat.com/v2/ status 200
DEBU[0005] GET https://registry.access.redhat.com/v2/ubi8-minimal/manifests/latest
DEBU[0005] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json"
DEBU[0005] Using blob info cache at /var/lib/containers/cache/blob-info-cache-v1.boltdb
DEBU[0005] Source is a manifest list; copying (only) instance sha256:3e1adcc31c6073d010b8043b070bd089d7bf37ee2c397c110211a6273453433f for current system
DEBU[0005] GET https://registry.access.redhat.com/v2/ubi8-minimal/manifests/sha256:3e1adcc31c6073d010b8043b070bd089d7bf37ee2c397c110211a6273453433f
DEBU[0006] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json"
DEBU[0006] IsRunningImageAllowed for image docker:registry.access.redhat.com/ubi8-minimal:latest
DEBU[0006] Using transport "docker" specific policy section registry.access.redhat.com
DEBU[0006] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8-minimal@sha256=3e1adcc31c6073d010b8043b070bd089d7bf37ee2c397c110211a6273453433f/signature-1
DEBU[0006] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8-minimal@sha256=3e1adcc31c6073d010b8043b070bd089d7bf37ee2c397c110211a6273453433f/signature-2
DEBU[0006] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8-minimal@sha256=3e1adcc31c6073d010b8043b070bd089d7bf37ee2c397c110211a6273453433f/signature-3
DEBU[0006] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8-minimal@sha256=3e1adcc31c6073d010b8043b070bd089d7bf37ee2c397c110211a6273453433f/signature-4
DEBU[0006] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8-minimal@sha256=3e1adcc31c6073d010b8043b070bd089d7bf37ee2c397c110211a6273453433f/signature-5
DEBU[0006] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8-minimal@sha256=3e1adcc31c6073d010b8043b070bd089d7bf37ee2c397c110211a6273453433f/signature-6
DEBU[0006] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8-minimal@sha256=3e1adcc31c6073d010b8043b070bd089d7bf37ee2c397c110211a6273453433f/signature-7
DEBU[0006] ... got status 404, as expected = end of signatures
DEBU[0006] Not looking for sigstore attachments: disabled by configuration
DEBU[0007] Requirement 0: denied, done
DEBU[0007] Error pulling candidate registry.access.redhat.com/ubi8-minimal:latest: copying system image from manifest list: Source image rejected: None of the signatures were accepted, reasons: No public keys imported; No public keys imported; No public keys imported; No public keys imported; No public keys imported; No public keys imported
Error: copying system image from manifest list: Source image rejected: None of the signatures were accepted, reasons: No public keys imported; No public keys imported; No public keys imported; No public keys imported; No public keys imported; No public keys imported
DEBU[0007] Shutting down engines
Rootless execution:
[admin@vm-1 ~]$ podman run --log-level debug ubi8:latest cat /etc/os-release
INFO[0000] podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(podman run --log-level debug ubi8:latest cat /etc/os-release)
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/home/admin/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Overriding graph root "/var/home/admin/.local/share/containers/storage" with "/home/admin/.local/share/containers/storage" from database
DEBU[0000] Overriding static dir "/var/home/admin/.local/share/containers/storage/libpod" with "/home/admin/.local/share/containers/storage/libpod" from database
DEBU[0000] Overriding volume path "/var/home/admin/.local/share/containers/storage/volumes" with "/home/admin/.local/share/containers/storage/volumes" from database
DEBU[0000] systemd-logind: Unknown object '/'.
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/admin/.local/share/containers/storage
DEBU[0000] Using run root /run/user/1000/containers
DEBU[0000] Using static dir /home/admin/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp
DEBU[0000] Using volume path /home/admin/.local/share/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend journald
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 7
DEBU[0000] Successfully loaded 1 networks
DEBU[0000] Pulling image ubi8:latest (policy: missing)
DEBU[0000] Looking up image "ubi8:latest" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] }
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/001-rhel-shortnames.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/002-rhel-shortnames-overrides.conf"
DEBU[0000] Trying "registry.access.redhat.com/ubi8:latest" ...
DEBU[0000] Trying "localhost/ubi8:latest" ...
DEBU[0000] Trying "registry.access.redhat.com/ubi8:latest" ...
DEBU[0000] Trying "registry.redhat.io/ubi8:latest" ...
DEBU[0000] Trying "docker.io/library/ubi8:latest" ...
DEBU[0000] Trying "docker.io/library/ubi8:latest" ...
DEBU[0000] Trying "ubi8:latest" ...
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] }
DEBU[0000] Attempting to pull candidate registry.access.redhat.com/ubi8:latest for ubi8:latest
DEBU[0000] parsed reference into "[overlay@/home/admin/.local/share/containers/storage+/run/user/1000/containers]registry.access.redhat.com/ubi8:latest"
DEBU[0000] Resolved "ubi8" as an alias (/etc/containers/registries.conf.d/001-rhel-shortnames.conf)
Resolved "ubi8" as an alias (/etc/containers/registries.conf.d/001-rhel-shortnames.conf)
Trying to pull registry.access.redhat.com/ubi8:latest...
DEBU[0000] Copying source image //registry.access.redhat.com/ubi8:latest to destination image [overlay@/home/admin/.local/share/containers/storage+/run/user/1000/containers]registry.access.redhat.com/ubi8:latest
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Trying to access "registry.access.redhat.com/ubi8:latest"
DEBU[0000] No credentials matching registry.access.redhat.com/ubi8 found in /run/user/1000/containers/auth.json
DEBU[0000] No credentials matching registry.access.redhat.com/ubi8 found in /var/home/admin/.config/containers/auth.json
DEBU[0000] No credentials matching registry.access.redhat.com/ubi8 found in /home/admin/.docker/config.json
DEBU[0000] No credentials matching registry.access.redhat.com/ubi8 found in /home/admin/.dockercfg
DEBU[0000] No credentials for registry.access.redhat.com/ubi8 found
DEBU[0000] Lookaside configuration: using "docker" namespace registry.access.redhat.com
DEBU[0000] Using "sigstore" https://access.redhat.com/webassets/docker/content/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.access.redhat.com
DEBU[0000] Sigstore attachments: using "docker" namespace registry.access.redhat.com
DEBU[0000] GET https://registry.access.redhat.com/v2/
DEBU[0000] Ping https://registry.access.redhat.com/v2/ status 200
DEBU[0000] GET https://registry.access.redhat.com/v2/ubi8/manifests/latest
DEBU[0000] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json"
DEBU[0000] Using blob info cache at /home/admin/.local/share/containers/cache/blob-info-cache-v1.boltdb
DEBU[0000] Source is a manifest list; copying (only) instance sha256:4a6dbfbb845810dce5902ab80cb93ecb24c367460fff9d15438e0b3080e244b3 for current system
DEBU[0000] GET https://registry.access.redhat.com/v2/ubi8/manifests/sha256:4a6dbfbb845810dce5902ab80cb93ecb24c367460fff9d15438e0b3080e244b3
DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json"
DEBU[0001] IsRunningImageAllowed for image docker:registry.access.redhat.com/ubi8:latest
DEBU[0001] Using transport "docker" specific policy section registry.access.redhat.com
DEBU[0001] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8@sha256=4a6dbfbb845810dce5902ab80cb93ecb24c367460fff9d15438e0b3080e244b3/signature-1
DEBU[0001] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8@sha256=4a6dbfbb845810dce5902ab80cb93ecb24c367460fff9d15438e0b3080e244b3/signature-2
DEBU[0001] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8@sha256=4a6dbfbb845810dce5902ab80cb93ecb24c367460fff9d15438e0b3080e244b3/signature-3
DEBU[0001] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8@sha256=4a6dbfbb845810dce5902ab80cb93ecb24c367460fff9d15438e0b3080e244b3/signature-4
DEBU[0001] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8@sha256=4a6dbfbb845810dce5902ab80cb93ecb24c367460fff9d15438e0b3080e244b3/signature-5
DEBU[0001] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8@sha256=4a6dbfbb845810dce5902ab80cb93ecb24c367460fff9d15438e0b3080e244b3/signature-6
DEBU[0001] GET https://access.redhat.com/webassets/docker/content/sigstore/ubi8@sha256=4a6dbfbb845810dce5902ab80cb93ecb24c367460fff9d15438e0b3080e244b3/signature-7
DEBU[0001] ... got status 404, as expected = end of signatures
DEBU[0001] Not looking for sigstore attachments: disabled by configuration
DEBU[0002] Requirement 0: denied, done
DEBU[0002] Error pulling candidate registry.access.redhat.com/ubi8:latest: copying system image from manifest list: Source image rejected: None of the signatures were accepted, reasons: No public keys imported; No public keys imported; No public keys imported; No public keys imported; No public keys imported; No public keys imported
Error: copying system image from manifest list: Source image rejected: None of the signatures were accepted, reasons: No public keys imported; No public keys imported; No public keys imported; No public keys imported; No public keys imported; No public keys imported
DEBU[0002] Shutting down engines
I can not reproduce this bug in composes CentOS-Stream-9-20230405.1 nor CentOS-Stream-9-20230410.0 (most recent composes) I will close this bug for that reason. In case of having this failure again, I will reopen the bug. I'm seeing this problem on the current CentOS Stream 9 release (`dnf update` from yesterday)...is that to be expected? (When will the changes from 5 April be available as a "release"?) @lsm5 Do you know the answer to Webb's question? I would have expected that would be in CentoS 9 by now, but maybe I'm off base. (In reply to Webb Scales from comment #5) > I'm seeing this problem on the current CentOS Stream 9 release (`dnf update` > from yesterday)...is that to be expected? (When will the changes from 5 > April be available as a "release"?) Tried it just now on a fresh CentOS 9 Stream with podman 4.4.1-9 and it worked for both ubi8 and ubi8-minimal with both root and rootless. 4.4.1-9 seems to be the latest build on my env. Could you please let me know what version of the rpm you're using? (rpm -q podman) @lsm5, I'm using Podman 4.4.1-9. It's the pull which is failing, so I'm not sure what you mean by "both root and rootless". I'm running Podman as a non-priv'd user. Below is the rpm output, with crypt thrown in, since there seems to be a signing problem. Thanks! ---- $ rpm -qa | grep -E -e crypt -e podman libxcrypt-4.4.18-3.el9.x86_64 libxcrypt-compat-4.4.18-3.el9.x86_64 libxcrypt-devel-4.4.18-3.el9.x86_64 cryptsetup-libs-2.6.0-2.el9.x86_64 python3.11-cryptography-37.0.2-5.el9.x86_64 libgcrypt-1.10.0-10.el9.x86_64 crypto-policies-20230505-1.gitf69bbc2.el9.noarch podman-4.4.1-9.el9.x86_64 crypto-policies-scripts-20230505-1.gitf69bbc2.el9.noarch python3-cryptography-36.0.1-4.el9.x86_64 |