Bug 2020026

Summary: Unable to pull UBI 9 images from non-RHEL
Product: Red Hat Enterprise Linux 9 Reporter: Shion Tanaka <shtanaka>
Component: doc-Release_Notes-9-en-USAssignee: Gabi Fialová <gfialova>
Status: CLOSED CURRENTRELEASE QA Contact: RHEL DPM <rhel-docs>
Severity: high Docs Contact: Gabriela Nečasová <gnecasov>
Priority: high    
Version: 9.0CC: agerstmayr, dmoppert, dornelas, dwalsh, gfialova, gnecasov, jnovy, jwboyer, lfriedma, mhofmann, mitr, ngompa13, rhel-docs, tsweeney, vrothber
Target Milestone: rcKeywords: Documentation, Reopened, Triaged
Target Release: 9.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 2020301 2094015 (view as bug list) Environment:
Last Closed: 2022-06-06 13:28:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2020301, 2094015    

Description Shion Tanaka 2021-11-03 21:57:30 UTC
Description of problem:

Cannot pull UBI 9 images with Invalid GPG signature error.

```
$ podman pull registry.redhat.io/ubi9-beta/ubi
Trying to pull registry.redhat.io/ubi9-beta/ubi:latest...
Error: Source image rejected: None of the signatures were accepted, reasons: Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"938A80CAF21541EB", Status:gpgme.Error{err:0x9}, Timestamp:time.Time{wall:0x0, ext:63771440989, loc:(*time.Location)(0x5599955cd680)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x5599955cd680)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}; Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"938A80CAF21541EB", Status:gpgme.Error{err:0x9}, Timestamp:time.Time{wall:0x0, ext:63771440989, loc:(*time.Location)(0x5599955cd680)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x5599955cd680)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}; Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"938A80CAF21541EB", Status:gpgme.Error{err:0x9}, Timestamp:time.Time{wall:0x0, ext:63771440990, loc:(*time.Location)(0x5599955cd680)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x5599955cd680)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}; Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"938A80CAF21541EB", Status:gpgme.Error{err:0x9}, Timestamp:time.Time{wall:0x0, ext:63771440991, loc:(*time.Location)(0x5599955cd680)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x5599955cd680)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}; Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"938A80CAF21541EB", Status:gpgme.Error{err:0x9}, Timestamp:time.Time{wall:0x0, ext:63771440991, loc:(*time.Location)(0x5599955cd680)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x5599955cd680)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}; Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"938A80CAF21541EB", Status:gpgme.Error{err:0x9}, Timestamp:time.Time{wall:0x0, ext:63771440992, loc:(*time.Location)(0x5599955cd680)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x5599955cd680)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}
```

```
podman pull registry.access.redhat.com/ubi9-beta/ubi
```
Almost the same result.

Version-Release number of selected component (if applicable):

on CentOS Stream 8
```
$ cat /etc/redhat-release
CentOS Stream release 8
$ uname -a
Linux phenex 4.18.0-348.el8.x86_64 #1 SMP Tue Oct 19 15:14:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
$ podman --version
podman version 3.4.1-dev
```

on Fedora 35
```
$ cat /etc/redhat-release
Fedora release 35 (Thirty Five)
$ uname -a
Linux fedora35 5.14.14-300.fc35.x86_64 #1 SMP Wed Oct 20 16:14:50 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
$ podman --version
podman version 3.4.1
```

How reproducible:

All the time.

Steps to Reproduce:
1. podman pull registry.redhat.io/ubi9-beta/ubi
or
1. podman pull registry.access.redhat.com/ubi9-beta/ubi

Actual results:

Error above.

Expected results:


Additional info:

Comment 2 Shion Tanaka 2021-11-04 02:59:15 UTC
I got the same error in rhel9-beta/toolbox. It seems to be the same of other tags(rhel9-beta/*).
```
$ podman pull registry.access.redhat.com/rhel9-beta/toolbox
Trying to pull registry.access.redhat.com/rhel9-beta/toolbox:latest...
Error: Source image rejected: None of the signatures were accepted, reasons: Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"938A80CAF21541EB", Status:gpgme.Error{err:0x9}, Timestamp:time.Time{wall:0x0, ext:63771441427, loc:(*time.Location)(0x55d730c21680)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x55d730c21680)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}; Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"938A80CAF21541EB", Status:gpgme.Error{err:0x9}, Timestamp:time.Time{wall:0x0, ext:63771441427, loc:(*time.Location)(0x55d730c21680)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x55d730c21680)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}; Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"938A80CAF21541EB", Status:gpgme.Error{err:0x9}, Timestamp:time.Time{wall:0x0, ext:63771441428, loc:(*time.Location)(0x55d730c21680)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x55d730c21680)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}; Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"938A80CAF21541EB", Status:gpgme.Error{err:0x9}, Timestamp:time.Time{wall:0x0, ext:63771441428, loc:(*time.Location)(0x55d730c21680)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x55d730c21680)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}; Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"938A80CAF21541EB", Status:gpgme.Error{err:0x9}, Timestamp:time.Time{wall:0x0, ext:63771441429, loc:(*time.Location)(0x55d730c21680)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x55d730c21680)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}; Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"938A80CAF21541EB", Status:gpgme.Error{err:0x9}, Timestamp:time.Time{wall:0x0, ext:63771441430, loc:(*time.Location)(0x55d730c21680)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x55d730c21680)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}
```

Comment 4 Josh Boyer 2021-11-04 12:52:50 UTC
The ubi9 Beta images are signed with the Red Hat Beta GPG key, which can be found on https://access.redhat.com/security/team/key  New enough versions of podman, such as those found in Fedora CentOS Stream 8/9 or RHEL 9 Beta, default to validating signatures of images.  The current builds are configured to trust production keys for the Red Hat registries but do not trust the Beta keys by default.

We're looking into this issue, but a workaround is to ensure the Red Hat Beta key is on the local system and use podman image trust to set the Beta key as a trusted key for the ubi9-beta namespace.  Here is an example:

[jwboyer@zod ~]$ wget https://www.redhat.com/security/data/f21541eb.txt
[jwboyer@zod ~]$ sudo podman image trust set -f ./f21541eb.txt registry.access.redhat.com/ubi9-beta
[jwboyer@zod ~]$ podman pull registry.access.redhat.com/ubi9-beta/ubi
Trying to pull registry.access.redhat.com/ubi9-beta/ubi:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob c3aca5a03ade done  
Copying blob 76a1729eef6b done  
Copying config 28b0a4b69d done  
Writing manifest to image destination
Storing signatures
28b0a4b69d9b5dc8b55a9639dc372803a3145c65d2a0c36e80b52a208e18b2f6
[jwboyer@zod ~]$

Comment 5 Shion Tanaka 2021-11-04 13:33:09 UTC
@Josh Boyer
Thank you for your comments.
I can confirm that the workaround works.(Both ubi9-beta and rhel9-beta)

```
$ wget https://www.redhat.com/security/data/f21541eb.txt
$ sudo podman image trust set -f ./f21541eb.txt registry.access.redhat.com/ubi9-beta
$ sudo podman image trust set -f ./f21541eb.txt registry.access.redhat.com/rhel9-beta
$ podman image trust show
default                                accept
registry.access.redhat.com             signedBy                security  https://access.redhat.com/webassets/docker/content/sigstore
registry.access.redhat.com/rhel9-beta  signedBy                security  https://access.redhat.com/webassets/docker/content/sigstore
registry.access.redhat.com/ubi9-beta   signedBy                security  https://access.redhat.com/webassets/docker/content/sigstore
registry.redhat.io                     signedBy                security  https://registry.redhat.io/containers/sigstore
                                       insecureAcceptAnything
$ podman pull ubi9-beta/ubi
$ podman pull rhel9-beta/toolbox
$ podman images|grep -e ubi9 -e rhel9
registry.access.redhat.com/rhel9-beta/toolbox  latest       b4b8016fce23  3 weeks ago   591 MB
registry.access.redhat.com/ubi9-beta/ubi       latest       28b0a4b69d9b  3 weeks ago   229 MB
```

Comment 6 Josh Boyer 2021-11-04 13:52:03 UTC
One note: the file passed with -f should be an absolute path rather than a relative path.  Otherwise podman pull commands must be run from the directory that contains the key file.  

For systems that lack the Red Hat Beta key, it is best to copy the key to /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta and specify that path.

Comment 7 Shion Tanaka 2021-11-04 14:37:22 UTC
I moved the key and reassigned it to an absolute path. It's working perfectly.

```
$ sudo wget -O /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta https://www.redhat.com/security/data/f21541eb.txt
$ sudo podman image trust set -f /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta registry.access.redhat.com/ubi9-beta
$ sudo podman image trust set -f /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta registry.access.redhat.com/rhel9-beta
$ podman image trust show --raw |grep beta
            "registry.access.redhat.com/rhel9-beta": [
                    "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta"
            "registry.access.redhat.com/ubi9-beta": [
                    "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta"
```

Comment 8 Tom Sweeney 2021-11-04 14:47:28 UTC
Keeping this assigned to Jindrich for now, but have added Valentin to the cc list in case he has a thought.

Comment 14 Daniel Walsh 2021-11-05 12:48:21 UTC
The question is should we ship an updated version of Podman for RHEL8, Fedora and CoreOS Stream with the beta key preinstalled?

Comment 42 Gabriela Nečasová 2022-08-08 08:00:13 UTC
Tom, please, is this issue fixed? 
Thank you in advance.