2020026 – Unable to pull UBI 9 images from non-RHEL

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2020026 - Unable to pull UBI 9 images from non-RHEL

Summary: Unable to pull UBI 9 images from non-RHEL

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Red Hat Enterprise Linux 9
Classification:	Red Hat
Component:	doc-Release_Notes-9-en-US
Sub Component:
Version:	9.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	rc
Target Release:	9.1
Assignee:	Gabi Fialová
QA Contact:	RHEL DPM
Docs Contact:	Gabriela Nečasová
URL:
Whiteboard:
Depends On:
Blocks:	2020301 2094015
TreeView+	depends on / blocked

Reported:	2021-11-03 21:57 UTC by Shion Tanaka
Modified:	2022-11-14 11:35 UTC (History)
CC List:	15 users (show)
Fixed In Version:
Doc Type:	No Doc Update
Doc Text:
Clone Of:
Clones:	2020301 2094015 (view as bug list)
Environment:
Last Closed:	2022-06-06 13:28:38 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	RHELPLAN-101686	0	None	None	None	2021-11-03 21:58:28 UTC
Red Hat Knowledge Base (Solution)	6487081	0	None	None	None	2021-11-05 17:06:33 UTC

Description Shion Tanaka 2021-11-03 21:57:30 UTC

Description of problem:

Cannot pull UBI 9 images with Invalid GPG signature error.

```
$ podman pull registry.redhat.io/ubi9-beta/ubi
Trying to pull registry.redhat.io/ubi9-beta/ubi:latest...
Error: Source image rejected: None of the signatures were accepted, reasons: Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"938A80CAF21541EB", Status:gpgme.Error{err:0x9}, Timestamp:time.Time{wall:0x0, ext:63771440989, loc:(*time.Location)(0x5599955cd680)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x5599955cd680)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}; Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"938A80CAF21541EB", Status:gpgme.Error{err:0x9}, Timestamp:time.Time{wall:0x0, ext:63771440989, loc:(*time.Location)(0x5599955cd680)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x5599955cd680)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}; Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"938A80CAF21541EB", Status:gpgme.Error{err:0x9}, Timestamp:time.Time{wall:0x0, ext:63771440990, loc:(*time.Location)(0x5599955cd680)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x5599955cd680)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}; Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"938A80CAF21541EB", Status:gpgme.Error{err:0x9}, Timestamp:time.Time{wall:0x0, ext:63771440991, loc:(*time.Location)(0x5599955cd680)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x5599955cd680)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}; Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"938A80CAF21541EB", Status:gpgme.Error{err:0x9}, Timestamp:time.Time{wall:0x0, ext:63771440991, loc:(*time.Location)(0x5599955cd680)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x5599955cd680)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}; Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"938A80CAF21541EB", Status:gpgme.Error{err:0x9}, Timestamp:time.Time{wall:0x0, ext:63771440992, loc:(*time.Location)(0x5599955cd680)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x5599955cd680)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}
```

```
podman pull registry.access.redhat.com/ubi9-beta/ubi
```
Almost the same result.

Version-Release number of selected component (if applicable):

on CentOS Stream 8
```
$ cat /etc/redhat-release
CentOS Stream release 8
$ uname -a
Linux phenex 4.18.0-348.el8.x86_64 #1 SMP Tue Oct 19 15:14:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
$ podman --version
podman version 3.4.1-dev
```

on Fedora 35
```
$ cat /etc/redhat-release
Fedora release 35 (Thirty Five)
$ uname -a
Linux fedora35 5.14.14-300.fc35.x86_64 #1 SMP Wed Oct 20 16:14:50 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
$ podman --version
podman version 3.4.1
```

How reproducible:

All the time.

Steps to Reproduce:
1. podman pull registry.redhat.io/ubi9-beta/ubi
or
1. podman pull registry.access.redhat.com/ubi9-beta/ubi

Actual results:

Error above.

Expected results:


Additional info:

Comment 2 Shion Tanaka 2021-11-04 02:59:15 UTC

I got the same error in rhel9-beta/toolbox. It seems to be the same of other tags(rhel9-beta/*).
```
$ podman pull registry.access.redhat.com/rhel9-beta/toolbox
Trying to pull registry.access.redhat.com/rhel9-beta/toolbox:latest...
Error: Source image rejected: None of the signatures were accepted, reasons: Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"938A80CAF21541EB", Status:gpgme.Error{err:0x9}, Timestamp:time.Time{wall:0x0, ext:63771441427, loc:(*time.Location)(0x55d730c21680)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x55d730c21680)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}; Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"938A80CAF21541EB", Status:gpgme.Error{err:0x9}, Timestamp:time.Time{wall:0x0, ext:63771441427, loc:(*time.Location)(0x55d730c21680)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x55d730c21680)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}; Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"938A80CAF21541EB", Status:gpgme.Error{err:0x9}, Timestamp:time.Time{wall:0x0, ext:63771441428, loc:(*time.Location)(0x55d730c21680)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x55d730c21680)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}; Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"938A80CAF21541EB", Status:gpgme.Error{err:0x9}, Timestamp:time.Time{wall:0x0, ext:63771441428, loc:(*time.Location)(0x55d730c21680)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x55d730c21680)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}; Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"938A80CAF21541EB", Status:gpgme.Error{err:0x9}, Timestamp:time.Time{wall:0x0, ext:63771441429, loc:(*time.Location)(0x55d730c21680)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x55d730c21680)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}; Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"938A80CAF21541EB", Status:gpgme.Error{err:0x9}, Timestamp:time.Time{wall:0x0, ext:63771441430, loc:(*time.Location)(0x55d730c21680)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x55d730c21680)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}
```

Comment 4 Josh Boyer 2021-11-04 12:52:50 UTC

The ubi9 Beta images are signed with the Red Hat Beta GPG key, which can be found on https://access.redhat.com/security/team/key  New enough versions of podman, such as those found in Fedora CentOS Stream 8/9 or RHEL 9 Beta, default to validating signatures of images.  The current builds are configured to trust production keys for the Red Hat registries but do not trust the Beta keys by default.

We're looking into this issue, but a workaround is to ensure the Red Hat Beta key is on the local system and use podman image trust to set the Beta key as a trusted key for the ubi9-beta namespace.  Here is an example:

[jwboyer@zod ~]$ wget https://www.redhat.com/security/data/f21541eb.txt
[jwboyer@zod ~]$ sudo podman image trust set -f ./f21541eb.txt registry.access.redhat.com/ubi9-beta
[jwboyer@zod ~]$ podman pull registry.access.redhat.com/ubi9-beta/ubi
Trying to pull registry.access.redhat.com/ubi9-beta/ubi:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob c3aca5a03ade done  
Copying blob 76a1729eef6b done  
Copying config 28b0a4b69d done  
Writing manifest to image destination
Storing signatures
28b0a4b69d9b5dc8b55a9639dc372803a3145c65d2a0c36e80b52a208e18b2f6
[jwboyer@zod ~]$

Comment 5 Shion Tanaka 2021-11-04 13:33:09 UTC

@Josh Boyer
Thank you for your comments.
I can confirm that the workaround works.(Both ubi9-beta and rhel9-beta)

```
$ wget https://www.redhat.com/security/data/f21541eb.txt
$ sudo podman image trust set -f ./f21541eb.txt registry.access.redhat.com/ubi9-beta
$ sudo podman image trust set -f ./f21541eb.txt registry.access.redhat.com/rhel9-beta
$ podman image trust show
default                                accept
registry.access.redhat.com             signedBy                security  https://access.redhat.com/webassets/docker/content/sigstore
registry.access.redhat.com/rhel9-beta  signedBy                security  https://access.redhat.com/webassets/docker/content/sigstore
registry.access.redhat.com/ubi9-beta   signedBy                security  https://access.redhat.com/webassets/docker/content/sigstore
registry.redhat.io                     signedBy                security  https://registry.redhat.io/containers/sigstore
                                       insecureAcceptAnything
$ podman pull ubi9-beta/ubi
$ podman pull rhel9-beta/toolbox
$ podman images|grep -e ubi9 -e rhel9
registry.access.redhat.com/rhel9-beta/toolbox  latest       b4b8016fce23  3 weeks ago   591 MB
registry.access.redhat.com/ubi9-beta/ubi       latest       28b0a4b69d9b  3 weeks ago   229 MB
```

Comment 6 Josh Boyer 2021-11-04 13:52:03 UTC

One note: the file passed with -f should be an absolute path rather than a relative path.  Otherwise podman pull commands must be run from the directory that contains the key file.  

For systems that lack the Red Hat Beta key, it is best to copy the key to /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta and specify that path.

Comment 7 Shion Tanaka 2021-11-04 14:37:22 UTC

I moved the key and reassigned it to an absolute path. It's working perfectly.

```
$ sudo wget -O /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta https://www.redhat.com/security/data/f21541eb.txt
$ sudo podman image trust set -f /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta registry.access.redhat.com/ubi9-beta
$ sudo podman image trust set -f /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta registry.access.redhat.com/rhel9-beta
$ podman image trust show --raw |grep beta
            "registry.access.redhat.com/rhel9-beta": [
                    "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta"
            "registry.access.redhat.com/ubi9-beta": [
                    "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta"
```

Comment 8 Tom Sweeney 2021-11-04 14:47:28 UTC

Keeping this assigned to Jindrich for now, but have added Valentin to the cc list in case he has a thought.

Comment 14 Daniel Walsh 2021-11-05 12:48:21 UTC

The question is should we ship an updated version of Podman for RHEL8, Fedora and CoreOS Stream with the beta key preinstalled?

Comment 35 Lenka Špačková 2022-05-19 05:46:48 UTC

Published:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html-single/9.0_release_notes/index#BZ-2020026

Comment 42 Gabriela Nečasová 2022-08-08 08:00:13 UTC

Tom, please, is this issue fixed? 
Thank you in advance.

Note You need to log in before you can comment on or make changes to this bug.