Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1889427

Summary: An update to 4.5.15 via UI fails: signature not available in the expected location
Product: OpenShift Container Platform Reporter: Christian LaPolt <christian.lapolt>
Component: ReleaseAssignee: Luke Meyer <lmeyer>
Status: CLOSED CURRENTRELEASE QA Contact: Wei Sun <wsun>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.5CC: aos-bugs, jokerman, talessio, wking, wzheng
Target Milestone: ---   
Target Release: 4.5.z   
Hardware: s390x   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-20 21:13:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Christian LaPolt 2020-10-19 16:15:15 UTC
Description of problem:
An update to 4.5.15 via UI fails with message
"Unable to apply 4.5.15: the image may not be safe to use"

Version-Release number of selected component (if applicable):

4.5.15
How reproducible:
Very

Steps to Reproduce:
1.Load a cluster with 4.4.27 or 4.5.14 (it fails with any version )
2.VIA UI select 4.5 candidate channel, select 4.5.15, perform update
3.The update will immediately fail and oc get clusterversion indicates
"Unable to apply 4.5.15: the image may not be safe to use"

Actual results:
As it is in the candidate channel would expect it to work

Expected results:
Fails

Additional info:

Comment 2 W. Trevor King 2020-10-19 17:52:28 UTC
> ... the image may not be safe to use ...

This means your cluster-version operator is unable to find a trusted signature for your release image.  'oc get -o yaml clusterversion version' may have more details, and will certainly include the digest of the image the CVO is trying to verify.  You don't mention your architecture, but assuming you're on amd64:

$ curl -s https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/4.5.15/release.txt | grep Pull
Pull From: quay.io/openshift-release-dev/ocp-release@sha256:1df294ebe5b84f0eeceaa85b2162862c390143f5e84cda5acc22cc4529273c4c

And while we have a signature for that:

$ curl -s https://mirror.openshift.com/pub/openshift-v4/signatures/openshift-release-dev/ocp-release-nightly/sha256=1df294ebe5b84f0eeceaa85b2162862c390143f5e84cda5acc22cc4529273c4c/signature-1 | gpg --decrypt
{"critical": {"image": {"docker-manifest-digest": "sha256:1df294ebe5b84f0eeceaa85b2162862c390143f5e84cda5acc22cc4529273c4c"}, "type": "atomic container signature", "identity": {"docker-reference": "quay.io/openshift-release-dev/ocp-release:4.5.15-x86_64"}}, "optional": {"creator": "Red Hat OpenShift Signing Authority 0.0.1"}}gpg: Signature made Wed 14 Oct 2020 02:58:42 AM PDT using RSA key ID FD431D51
gpg: Good signature from "Red Hat, Inc. (release key 2) <security>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 567E 347A D004 4ADE 55BA  8A5F 199E 2F91 FD43 1D51

It's not currently in the place that the CVO is pointing at [1].

$ curl -I https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release@sha256=1df294ebe5b84f0eeceaa85b2162862c390143f5e84cda5acc22cc4529273c4c/signature-1
HTTP/1.1 404 Not Found
Date: Mon, 19 Oct 2020 17:52:22 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips
Content-Type: text/html; charset=iso-8859-1

I'll talk to ART about getting that fixed.

[1]: https://github.com/openshift/cluster-update-keys/blob/4cc42fb333592c974070c7112577806549d3f039/stores/store-openshift-official-release-mirror

Comment 4 W. Trevor King 2020-10-19 17:59:48 UTC
The signature is available at the expected GCS location [1]:

$ curl -s https://storage.googleapis.com/openshift-release/official/signatures/openshift/release/sha256=1df294ebe5b84f0eeceaa85b2162862c390143f5e84cda5acc22cc4529273c4c/signature-1 | gpg --verify
gpg: Signature made Wed 14 Oct 2020 02:58:42 AM PDT using RSA key ID FD431D51
gpg: Good signature from "Red Hat, Inc. (release key 2) <security>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 567E 347A D004 4ADE 55BA  8A5F 199E 2F91 FD43 1D51

[1]: https://github.com/openshift/cluster-update-keys/blob/4cc42fb333592c974070c7112577806549d3f039/stores/store-openshift-official-release

Comment 5 Christian LaPolt 2020-10-19 20:09:08 UTC
This is on s390x

Comment 7 Christian LaPolt 2020-10-20 13:44:07 UTC
This is working now via GUI install.

Working towards 4.5.15: 77% complete

Comment 8 W. Trevor King 2020-10-20 19:38:44 UTC
> Ah, actually it should be https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/sha256=1df294ebe5b84f0eeceaa85b2162862c390143f5e84cda5acc22cc4529273c4c/signature-1 , which also 404s

Grr.  Reading [1] again, it should be:

$ curl -s https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/sha256=1df294ebe5b84f0eeceaa85b2162862c390143f5e84cda5acc22cc4529273c4c/signature-1 | gpg --decrypt
{"critical": {"image": {"docker-manifest-digest": "sha256:1df294ebe5b84f0eeceaa85b2162862c390143f5e84cda5acc22cc4529273c4c"}, "type": "atomic container signature", "identity": {"docker-reference": "quay.io/openshift-release-dev/ocp-release:4.5.15-x86_64"}}, "optional": {"creator": "Red Hat OpenShift Signing Authority 0.0.1"}}gpg: Signature made Wed 14 Oct 2020 02:58:42 AM PDT using RSA key ID FD431D51
gpg: Good signature from "Red Hat, Inc. (release key 2) <security>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 567E 347A D004 4ADE 55BA  8A5F 199E 2F91 FD43 1D51

But yeah, as Thiago pointed out the issue here was the lack of signatures for s390x and ppc64le, since fixed.

[1]: https://github.com/openshift/cluster-update-keys/blob/4cc42fb333592c974070c7112577806549d3f039/stores/store-openshift-official-release-mirror

Comment 9 Luke Meyer 2020-10-20 21:13:21 UTC
Closing as it seems solved - reopen if that's incorrect.

Comment 10 Christian LaPolt 2020-10-21 00:30:52 UTC
I agree.  Cool to close.