Bug 1889427 - An update to 4.5.15 via UI fails: signature not available in the expected location
Summary: An update to 4.5.15 via UI fails: signature not available in the expected loc...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Release
Version: 4.5
Hardware: s390x
OS: Linux
unspecified
medium
Target Milestone: ---
: 4.5.z
Assignee: Luke Meyer
QA Contact: Wei Sun
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-10-19 16:15 UTC by Christian LaPolt
Modified: 2020-10-21 00:30 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-20 21:13:21 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Christian LaPolt 2020-10-19 16:15:15 UTC
Description of problem:
An update to 4.5.15 via UI fails with message
"Unable to apply 4.5.15: the image may not be safe to use"

Version-Release number of selected component (if applicable):

4.5.15
How reproducible:
Very

Steps to Reproduce:
1.Load a cluster with 4.4.27 or 4.5.14 (it fails with any version )
2.VIA UI select 4.5 candidate channel, select 4.5.15, perform update
3.The update will immediately fail and oc get clusterversion indicates
"Unable to apply 4.5.15: the image may not be safe to use"

Actual results:
As it is in the candidate channel would expect it to work

Expected results:
Fails

Additional info:

Comment 2 W. Trevor King 2020-10-19 17:52:28 UTC
> ... the image may not be safe to use ...

This means your cluster-version operator is unable to find a trusted signature for your release image.  'oc get -o yaml clusterversion version' may have more details, and will certainly include the digest of the image the CVO is trying to verify.  You don't mention your architecture, but assuming you're on amd64:

$ curl -s https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/4.5.15/release.txt | grep Pull
Pull From: quay.io/openshift-release-dev/ocp-release@sha256:1df294ebe5b84f0eeceaa85b2162862c390143f5e84cda5acc22cc4529273c4c

And while we have a signature for that:

$ curl -s https://mirror.openshift.com/pub/openshift-v4/signatures/openshift-release-dev/ocp-release-nightly/sha256=1df294ebe5b84f0eeceaa85b2162862c390143f5e84cda5acc22cc4529273c4c/signature-1 | gpg --decrypt
{"critical": {"image": {"docker-manifest-digest": "sha256:1df294ebe5b84f0eeceaa85b2162862c390143f5e84cda5acc22cc4529273c4c"}, "type": "atomic container signature", "identity": {"docker-reference": "quay.io/openshift-release-dev/ocp-release:4.5.15-x86_64"}}, "optional": {"creator": "Red Hat OpenShift Signing Authority 0.0.1"}}gpg: Signature made Wed 14 Oct 2020 02:58:42 AM PDT using RSA key ID FD431D51
gpg: Good signature from "Red Hat, Inc. (release key 2) <security@redhat.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 567E 347A D004 4ADE 55BA  8A5F 199E 2F91 FD43 1D51

It's not currently in the place that the CVO is pointing at [1].

$ curl -I https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release@sha256=1df294ebe5b84f0eeceaa85b2162862c390143f5e84cda5acc22cc4529273c4c/signature-1
HTTP/1.1 404 Not Found
Date: Mon, 19 Oct 2020 17:52:22 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips
Content-Type: text/html; charset=iso-8859-1

I'll talk to ART about getting that fixed.

[1]: https://github.com/openshift/cluster-update-keys/blob/4cc42fb333592c974070c7112577806549d3f039/stores/store-openshift-official-release-mirror

Comment 4 W. Trevor King 2020-10-19 17:59:48 UTC
The signature is available at the expected GCS location [1]:

$ curl -s https://storage.googleapis.com/openshift-release/official/signatures/openshift/release/sha256=1df294ebe5b84f0eeceaa85b2162862c390143f5e84cda5acc22cc4529273c4c/signature-1 | gpg --verify
gpg: Signature made Wed 14 Oct 2020 02:58:42 AM PDT using RSA key ID FD431D51
gpg: Good signature from "Red Hat, Inc. (release key 2) <security@redhat.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 567E 347A D004 4ADE 55BA  8A5F 199E 2F91 FD43 1D51

[1]: https://github.com/openshift/cluster-update-keys/blob/4cc42fb333592c974070c7112577806549d3f039/stores/store-openshift-official-release

Comment 5 Christian LaPolt 2020-10-19 20:09:08 UTC
This is on s390x

Comment 7 Christian LaPolt 2020-10-20 13:44:07 UTC
This is working now via GUI install.

Working towards 4.5.15: 77% complete

Comment 8 W. Trevor King 2020-10-20 19:38:44 UTC
> Ah, actually it should be https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/sha256=1df294ebe5b84f0eeceaa85b2162862c390143f5e84cda5acc22cc4529273c4c/signature-1 , which also 404s

Grr.  Reading [1] again, it should be:

$ curl -s https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/sha256=1df294ebe5b84f0eeceaa85b2162862c390143f5e84cda5acc22cc4529273c4c/signature-1 | gpg --decrypt
{"critical": {"image": {"docker-manifest-digest": "sha256:1df294ebe5b84f0eeceaa85b2162862c390143f5e84cda5acc22cc4529273c4c"}, "type": "atomic container signature", "identity": {"docker-reference": "quay.io/openshift-release-dev/ocp-release:4.5.15-x86_64"}}, "optional": {"creator": "Red Hat OpenShift Signing Authority 0.0.1"}}gpg: Signature made Wed 14 Oct 2020 02:58:42 AM PDT using RSA key ID FD431D51
gpg: Good signature from "Red Hat, Inc. (release key 2) <security@redhat.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 567E 347A D004 4ADE 55BA  8A5F 199E 2F91 FD43 1D51

But yeah, as Thiago pointed out the issue here was the lack of signatures for s390x and ppc64le, since fixed.

[1]: https://github.com/openshift/cluster-update-keys/blob/4cc42fb333592c974070c7112577806549d3f039/stores/store-openshift-official-release-mirror

Comment 9 Luke Meyer 2020-10-20 21:13:21 UTC
Closing as it seems solved - reopen if that's incorrect.

Comment 10 Christian LaPolt 2020-10-21 00:30:52 UTC
I agree.  Cool to close.


Note You need to log in before you can comment on or make changes to this bug.