Bug 1789765 - Failed to upgrade to 4.2.14 due to image is not signed well
Summary: Failed to upgrade to 4.2.14 due to image is not signed well
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Release
Version: 4.2.z
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.2.z
Assignee: Luke Meyer
QA Contact: Wei Sun
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-01-10 11:46 UTC by weiwei jiang
Modified: 2020-02-27 13:20 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-01-22 10:46:40 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:0107 None None None 2020-01-22 10:46:57 UTC

Description weiwei jiang 2020-01-10 11:46:02 UTC
Description of problem:
trigger an upgrade from 4.2.13 to 4.2.14, and got 
I0110 09:19:46.673790       1 cvo.go:398] Desired version from spec is v1.Update{Version:"4.2.14", Image:"registry.svc.ci.openshift.org/ocp/release@sha256:3fabe939da31f9a31f509251b9f73d321e367aba2d09ff392c2f452f6433a95a", Force:false}
I0110 09:19:46.673912       1 cvo.go:371] Finished syncing cluster version "openshift-cluster-version/version" (148.896µs)
I0110 09:19:46.673951       1 sync_worker.go:453] Running sync 4.2.14 (force=false) on generation 12 in state Updating at attempt 0
I0110 09:19:46.673966       1 sync_worker.go:459] Loading payload
I0110 09:19:46.890747       1 verify.go:281] Unable to verify sha256:3fabe939da31f9a31f509251b9f73d321e367aba2d09ff392c2f452f6433a95a against keyring verifier-public-key-redhat

Version-Release number of selected component (if applicable):
quay.io/openshift-release-dev/ocp-release:4.2.14-x86_64

How reproducible:
Always

Steps to Reproduce:
1. Install a quay.io/openshift-release-dev/ocp-release:4.2.13 cluster
2. Upgrade to quay.io/openshift-release-dev/ocp-release:4.2.14-x86_64 without --force
3. Check clusterversion resource and also clusterversion-operator pods log

Actual results:
I0110 09:19:46.673790       1 cvo.go:398] Desired version from spec is v1.Update{Version:"4.2.14", Image:"registry.svc.ci.openshift.org/ocp/release@sha256:3fabe939da31f9a31f509251b9f73d321e367aba2d09ff392c2f452f6433a95a", Force:false}
I0110 09:19:46.673912       1 cvo.go:371] Finished syncing cluster version "openshift-cluster-version/version" (148.896µs)
I0110 09:19:46.673951       1 sync_worker.go:453] Running sync 4.2.14 (force=false) on generation 12 in state Updating at attempt 0
I0110 09:19:46.673966       1 sync_worker.go:459] Loading payload
I0110 09:19:46.890747       1 verify.go:281] Unable to verify sha256:3fabe939da31f9a31f509251b9f73d321e367aba2d09ff392c2f452f6433a95a against keyring verifier-public-key-redhat

Expected results:
The image should pass the signature verification.

Additional info:

Comment 1 Luke Meyer 2020-01-10 14:56:40 UTC
Thanks for bringing this to our attention. Our signing job was broken. It should be signed now, please confirm!

Comment 2 W. Trevor King 2020-01-10 19:12:05 UTC
$ oc adm release info quay.io/openshift-release-dev/ocp-release:4.2.14-x86_64 | grep Digest
Digest:        sha256:3fabe939da31f9a31f509251b9f73d321e367aba2d09ff392c2f452f6433a95a
$ oc adm release info quay.io/openshift-release-dev/ocp-release:4.2.14-s390x | grep Digest
Digest:        sha256:26ba0c17618872c68a2e9a5002c3f3e48f75ae844fcf66e83063bd72a104cfc7
$ oc adm release info quay.io/openshift-release-dev/ocp-release:4.1.30-x86_64 | grep Digest
Digest:    sha256:c811f484faeefa469492b583033da759ca3323e9810471185a579baab187052c
$ for DIG in sha256=3fabe939da31f9a31f509251b9f73d321e367aba2d09ff392c2f452f6433a95a sha256=26ba0c17618872c68a2e9a5002c3f3e48f75ae844fcf66e83063bd72a104cfc7 sha256=c811f484faeefa469492b583033da759ca3323e9810471185a579baab187052c; do curl -s "https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/${DIG}/signature-1" >sig-1 && gpg --verify sig-1 && gpg --decrypt sig-1; done
gpg: Signature made Fri 10 Jan 2020 04:32:11 AM PST using RSA key ID FD431D51
gpg: Good signature from "Red Hat, Inc. (release key 2) <security@redhat.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 567E 347A D004 4ADE 55BA  8A5F 199E 2F91 FD43 1D51
{"critical": {"image": {"docker-manifest-digest": "sha256:3fabe939da31f9a31f509251b9f73d321e367aba2d09ff392c2f452f6433a95a"}, "type": "atomic container signature", "identity": {"docker-reference": "quay.io/openshift-release-dev/ocp-release:4.2.14-x86_64"}}}gpg: Signature made Fri 10 Jan 2020 04:32:11 AM PST using RSA key ID FD431D51
gpg: Good signature from "Red Hat, Inc. (release key 2) <security@redhat.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 567E 347A D004 4ADE 55BA  8A5F 199E 2F91 FD43 1D51
gpg: Signature made Fri 10 Jan 2020 04:33:57 AM PST using RSA key ID FD431D51
gpg: Good signature from "Red Hat, Inc. (release key 2) <security@redhat.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 567E 347A D004 4ADE 55BA  8A5F 199E 2F91 FD43 1D51
{"critical": {"image": {"docker-manifest-digest": "sha256:26ba0c17618872c68a2e9a5002c3f3e48f75ae844fcf66e83063bd72a104cfc7"}, "type": "atomic container signature", "identity": {"docker-reference": "quay.io/openshift-release-dev/ocp-release:4.2.14-s390x"}}}gpg: Signature made Fri 10 Jan 2020 04:33:57 AM PST using RSA key ID FD431D51
gpg: Good signature from "Red Hat, Inc. (release key 2) <security@redhat.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 567E 347A D004 4ADE 55BA  8A5F 199E 2F91 FD43 1D51
gpg: Signature made Fri 10 Jan 2020 05:39:21 AM PST using RSA key ID FD431D51
gpg: Good signature from "Red Hat, Inc. (release key 2) <security@redhat.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 567E 347A D004 4ADE 55BA  8A5F 199E 2F91 FD43 1D51
{"critical": {"image": {"docker-manifest-digest": "sha256:c811f484faeefa469492b583033da759ca3323e9810471185a579baab187052c"}, "type": "atomic container signature", "identity": {"docker-reference": "quay.io/openshift-release-dev/ocp-release:4.1.30-x86_64"}}}gpg: Signature made Fri 10 Jan 2020 05:39:21 AM PST using RSA key ID FD431D51
gpg: Good signature from "Red Hat, Inc. (release key 2) <security@redhat.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 567E 347A D004 4ADE 55BA  8A5F 199E 2F91 FD43 1D51

Looks good to me :).  I'd personally rather have the docker-reference name match the compiled in release name, but that would break folks who expect it to match the tag name (unless we signed twice with both names), and Clayton has said he doesn't feel like signature names checks are the way to guard against substitute-image attacks [1].

[1]: https://github.com/openshift/cluster-version-operator/pull/293#issuecomment-571780361

Comment 3 Lalatendu Mohanty 2020-01-10 19:17:40 UTC
Don't we use some kind of automation for this signing process?

Comment 4 W. Trevor King 2020-01-10 19:44:19 UTC
> Don't we use some kind of automation for this signing process?

We do [1].

[1]: https://github.com/openshift/aos-cd-jobs/blob/a8ab46843e83fe17319b0de17d0f2cd266d9506d/pipeline-scripts/release.groovy#L418

Comment 5 weiwei jiang 2020-01-13 03:08:50 UTC
Confirmed that, it's working well now. thanks. Move to verified.

Comment 7 errata-xmlrpc 2020-01-22 10:46:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0107


Note You need to log in before you can comment on or make changes to this bug.