Bug 2126100

Summary: Pulling Red Hat container images on CentOS 8 stream problem due to /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta
Product: Red Hat Enterprise Linux 8 Reporter: Lennart Schedin <lennart.schedin>
Component: containers-commonAssignee: Jindrich Novy <jnovy>
Status: CLOSED CURRENTRELEASE QA Contact: David Darrah/Red Hat QE <ddarrah>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: CentOS StreamCC: bstinson, carl.winback, dahorak, jwboyer
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-07-04 06:47:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Lennart Schedin 2022-09-12 12:18:36 UTC
How to reproduce:
1. Install a new CentOS 8 Stream VM. I used http://mirror.zetup.net/CentOS/8-stream/isos/x86_64/CentOS-Stream-8-x86_64-20220831-boot.iso as a base and selected “Server”

2. Install podman:
sudo dnf -y install podman

3. Pull the Red Hat Ubi8 image:
podman pull registry.access.redhat.com/ubi8:8.6-903.1661794351

Actual:
[root@localhost ~]# podman pull registry.access.redhat.com/ubi8:8.6-903.1661794351
Trying to pull registry.access.redhat.com/ubi8:8.6-903.1661794351...
Error: copying system image from manifest list: Source image rejected: None of the signatures were accepted, reasons: open /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta: no such file or directory; open /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta: no such file or directory; open /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta: no such file or directory; open /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta: no such file or directory; open /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta: no such file or directory; open /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta: no such file or directory

Expected:
The Red Hat Ubi8 image should be pulled and successfully stored in the local container image storage.

Environment:
[root@localhost ~]# cat /etc/redhat-release
CentOS Stream release 8
[root@localhost ~]# rpm -q centos-stream-release
centos-stream-release-8.6-1.el8.noarch
[root@localhost ~]# rpm -q containers-common
containers-common-1-40.module_el8.7.0+1196+721f4eb0.x86_64
[root@localhost ~]# rpm -q podman
podman-4.2.0-1.module_el8.7.0+1196+721f4eb0.x86_64

Analysis:
The root cause of the problem is somewhere related to the commit https://git.centos.org/rpms/containers-common/c/38a67aceb2ae0a4000ce2f29e8ebbccc2042215d where the file /etc/containers/policy.json was changed to also include the key /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta for the registry.access.redhat.com registry.

The sibling key /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release is included in containers-common RPM in CentOS 8 Stream, but on RHEL 8 both keys are distributed in the redhat-release RPM. The (half-)corresponding RPM on CentOS side (centos-stream-release) does not have the keys.

My guess is that CentOS 8 Stream has an inconsistency with respect to the keys and what is part of the policy.json file. Either this inconsistency needs to be solved, or making the podman pull command not crash when pulling images for where some key does not exist on disk.

(Ugly) workaround 1:
Create a dummy key file: 
[root@localhost ~]# touch /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta

(Ugly) workaround 2:
Edit the file /etc/containers/policy.json and remove /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta from it.

Comment 2 Lennart Schedin 2023-05-08 06:21:40 UTC
This ticket has been solved because the file /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release has been added to containers-common 1-46 for CentOS 8 stream:

[builder@7f7981bdc33c ~]$ cat  /etc/redhat-release
CentOS Stream release 8

[builder@7f7981bdc33c ~]$ rpm -q -l containers-common-1-49.module_el8.8.0+1254+78119b6e.x86_64 | grep RPM-GPG-KEY
/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta
/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release

Comment 3 Jindrich Novy 2023-07-04 06:47:48 UTC
Thank you for your comment Lennart.