Description of problem: trigger an upgrade from 4.2.13 to 4.2.14, and got I0110 09:19:46.673790 1 cvo.go:398] Desired version from spec is v1.Update{Version:"4.2.14", Image:"registry.svc.ci.openshift.org/ocp/release@sha256:3fabe939da31f9a31f509251b9f73d321e367aba2d09ff392c2f452f6433a95a", Force:false} I0110 09:19:46.673912 1 cvo.go:371] Finished syncing cluster version "openshift-cluster-version/version" (148.896µs) I0110 09:19:46.673951 1 sync_worker.go:453] Running sync 4.2.14 (force=false) on generation 12 in state Updating at attempt 0 I0110 09:19:46.673966 1 sync_worker.go:459] Loading payload I0110 09:19:46.890747 1 verify.go:281] Unable to verify sha256:3fabe939da31f9a31f509251b9f73d321e367aba2d09ff392c2f452f6433a95a against keyring verifier-public-key-redhat Version-Release number of selected component (if applicable): quay.io/openshift-release-dev/ocp-release:4.2.14-x86_64 How reproducible: Always Steps to Reproduce: 1. Install a quay.io/openshift-release-dev/ocp-release:4.2.13 cluster 2. Upgrade to quay.io/openshift-release-dev/ocp-release:4.2.14-x86_64 without --force 3. Check clusterversion resource and also clusterversion-operator pods log Actual results: I0110 09:19:46.673790 1 cvo.go:398] Desired version from spec is v1.Update{Version:"4.2.14", Image:"registry.svc.ci.openshift.org/ocp/release@sha256:3fabe939da31f9a31f509251b9f73d321e367aba2d09ff392c2f452f6433a95a", Force:false} I0110 09:19:46.673912 1 cvo.go:371] Finished syncing cluster version "openshift-cluster-version/version" (148.896µs) I0110 09:19:46.673951 1 sync_worker.go:453] Running sync 4.2.14 (force=false) on generation 12 in state Updating at attempt 0 I0110 09:19:46.673966 1 sync_worker.go:459] Loading payload I0110 09:19:46.890747 1 verify.go:281] Unable to verify sha256:3fabe939da31f9a31f509251b9f73d321e367aba2d09ff392c2f452f6433a95a against keyring verifier-public-key-redhat Expected results: The image should pass the signature verification. Additional info:
Thanks for bringing this to our attention. Our signing job was broken. It should be signed now, please confirm!
$ oc adm release info quay.io/openshift-release-dev/ocp-release:4.2.14-x86_64 | grep Digest Digest: sha256:3fabe939da31f9a31f509251b9f73d321e367aba2d09ff392c2f452f6433a95a $ oc adm release info quay.io/openshift-release-dev/ocp-release:4.2.14-s390x | grep Digest Digest: sha256:26ba0c17618872c68a2e9a5002c3f3e48f75ae844fcf66e83063bd72a104cfc7 $ oc adm release info quay.io/openshift-release-dev/ocp-release:4.1.30-x86_64 | grep Digest Digest: sha256:c811f484faeefa469492b583033da759ca3323e9810471185a579baab187052c $ for DIG in sha256=3fabe939da31f9a31f509251b9f73d321e367aba2d09ff392c2f452f6433a95a sha256=26ba0c17618872c68a2e9a5002c3f3e48f75ae844fcf66e83063bd72a104cfc7 sha256=c811f484faeefa469492b583033da759ca3323e9810471185a579baab187052c; do curl -s "https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/${DIG}/signature-1" >sig-1 && gpg --verify sig-1 && gpg --decrypt sig-1; done gpg: Signature made Fri 10 Jan 2020 04:32:11 AM PST using RSA key ID FD431D51 gpg: Good signature from "Red Hat, Inc. (release key 2) <security>" gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 567E 347A D004 4ADE 55BA 8A5F 199E 2F91 FD43 1D51 {"critical": {"image": {"docker-manifest-digest": "sha256:3fabe939da31f9a31f509251b9f73d321e367aba2d09ff392c2f452f6433a95a"}, "type": "atomic container signature", "identity": {"docker-reference": "quay.io/openshift-release-dev/ocp-release:4.2.14-x86_64"}}}gpg: Signature made Fri 10 Jan 2020 04:32:11 AM PST using RSA key ID FD431D51 gpg: Good signature from "Red Hat, Inc. (release key 2) <security>" gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 567E 347A D004 4ADE 55BA 8A5F 199E 2F91 FD43 1D51 gpg: Signature made Fri 10 Jan 2020 04:33:57 AM PST using RSA key ID FD431D51 gpg: Good signature from "Red Hat, Inc. (release key 2) <security>" gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 567E 347A D004 4ADE 55BA 8A5F 199E 2F91 FD43 1D51 {"critical": {"image": {"docker-manifest-digest": "sha256:26ba0c17618872c68a2e9a5002c3f3e48f75ae844fcf66e83063bd72a104cfc7"}, "type": "atomic container signature", "identity": {"docker-reference": "quay.io/openshift-release-dev/ocp-release:4.2.14-s390x"}}}gpg: Signature made Fri 10 Jan 2020 04:33:57 AM PST using RSA key ID FD431D51 gpg: Good signature from "Red Hat, Inc. (release key 2) <security>" gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 567E 347A D004 4ADE 55BA 8A5F 199E 2F91 FD43 1D51 gpg: Signature made Fri 10 Jan 2020 05:39:21 AM PST using RSA key ID FD431D51 gpg: Good signature from "Red Hat, Inc. (release key 2) <security>" gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 567E 347A D004 4ADE 55BA 8A5F 199E 2F91 FD43 1D51 {"critical": {"image": {"docker-manifest-digest": "sha256:c811f484faeefa469492b583033da759ca3323e9810471185a579baab187052c"}, "type": "atomic container signature", "identity": {"docker-reference": "quay.io/openshift-release-dev/ocp-release:4.1.30-x86_64"}}}gpg: Signature made Fri 10 Jan 2020 05:39:21 AM PST using RSA key ID FD431D51 gpg: Good signature from "Red Hat, Inc. (release key 2) <security>" gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 567E 347A D004 4ADE 55BA 8A5F 199E 2F91 FD43 1D51 Looks good to me :). I'd personally rather have the docker-reference name match the compiled in release name, but that would break folks who expect it to match the tag name (unless we signed twice with both names), and Clayton has said he doesn't feel like signature names checks are the way to guard against substitute-image attacks [1]. [1]: https://github.com/openshift/cluster-version-operator/pull/293#issuecomment-571780361
Don't we use some kind of automation for this signing process?
> Don't we use some kind of automation for this signing process? We do [1]. [1]: https://github.com/openshift/aos-cd-jobs/blob/a8ab46843e83fe17319b0de17d0f2cd266d9506d/pipeline-scripts/release.groovy#L418
Confirmed that, it's working well now. thanks. Move to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0107