Created attachment 1887340 [details] CVO log file Description of problem: Clearing upgrade after signature verification fails, ReleaseAccepted=False keeps complaining about the update cannot be verified blah blah. # oc get clusterversion/version -ojson | jq -r '.spec, .status.conditions' { "channel": "stable-4.11", "clusterID": "d740b8f3-bb49-40cf-86e8-5df4a755111a" } [ { "lastTransitionTime": "2022-06-07T01:31:43Z", "message": "Unable to retrieve available updates: currently reconciling cluster version 4.11.0-0.nightly-2022-06-06-025509 not found in the \"stable-4.11\" channel", "reason": "VersionNotFound", "status": "False", "type": "RetrievedUpdates" }, { "lastTransitionTime": "2022-06-07T01:31:43Z", "message": "Capabilities match configured spec", "reason": "AsExpected", "status": "False", "type": "ImplicitlyEnabledCapabilities" }, { "lastTransitionTime": "2022-06-07T02:44:54Z", "message": "Retrieving payload failed version=\"\" image=\"registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100\" failure=The update cannot be verified: unable to verify sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100 against keyrings: verifier-public-key-redhat", "reason": "RetrievePayload", "status": "False", "type": "ReleaseAccepted" }, { "lastTransitionTime": "2022-06-07T01:56:17Z", "message": "Done applying 4.11.0-0.nightly-2022-06-06-025509", "status": "True", "type": "Available" }, { "lastTransitionTime": "2022-06-07T01:55:47Z", "status": "False", "type": "Failing" }, { "lastTransitionTime": "2022-06-07T01:56:17Z", "message": "Cluster version is 4.11.0-0.nightly-2022-06-06-025509", "status": "False", "type": "Progressing" } ] Version-Release number of the following components: 4.11.0-0.nightly-2022-06-06-025509 How reproducible: 1/1 Steps to Reproduce: 1. Upgrade to a fake release # oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100 --allow-explicit-upgrade warning: The requested upgrade image is not one of the available updates.You have used --allow-explicit-upgrade for the update to proceed anyway Requesting update to release image registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100 2. Check ReleaseAccepted=False due to target image signature verification failure # oc adm upgrade Cluster version is 4.11.0-0.nightly-2022-06-04-014713 ReleaseAccepted=False Reason: RetrievePayload Message: Retrieving payload failed version="" image="registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100" failure=The update cannot be verified: unable to verify sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100 against keyrings: verifier-public-key-redhat Upstream is unset, so the cluster will use an appropriate default. Channel: stable-4.11 warning: Cannot display available updates: Reason: VersionNotFound Message: Unable to retrieve available updates: currently reconciling cluster version 4.11.0-0.nightly-2022-06-04-014713 not found in the "stable-4.11" channel 3. Clear the upgrade # oc adm upgrade --clear Cancelled requested upgrade to registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100 4. Check oc adm upgrade info # oc adm upgrade Cluster version is 4.11.0-0.nightly-2022-06-04-014713 ReleaseAccepted=False Reason: RetrievePayload Message: Retrieving payload failed version="" image="registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100" failure=The update cannot be verified: unable to verify sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100 against keyrings: verifier-public-key-redhat Upstream is unset, so the cluster will use an appropriate default. Channel: stable-4.11 warning: Cannot display available updates: Reason: VersionNotFound Message: Unable to retrieve available updates: currently reconciling cluster version 4.11.0-0.nightly-2022-06-04-014713 not found in the "stable-4.11" channel Actual results: After upgrade is cleared, cv condition ReleaseAccepted keeps to false with message The update cannot be verified Expected results: After upgrade is cleared, cv condition ReleaseAccepted should stop complaining about the target image Additional info: Please attach logs from ansible-playbook with the -vvv flag
We need something similar to [1] at [2]. [1] https://github.com/openshift/cluster-version-operator/blob/0d63b9bbced8c782b109855aea4cb43a4e87c083/pkg/cvo/sync_worker.go#L250 [2] https://github.com/openshift/cluster-version-operator/blob/0d63b9bbced8c782b109855aea4cb43a4e87c083/pkg/cvo/sync_worker.go#L446
It's reproduced with 4.10.18. So changing the version to 4.10 # oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100 --allow-explicit-upgrade warning: The requested upgrade image is not one of the available updates.You have used --allow-explicit-upgrade for the update to proceed anyway Requesting update to release image registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100 # oc adm upgrade Cluster version is 4.10.18 ReleaseAccepted=False Reason: RetrievePayload Message: Retrieving payload failed version="" image="registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100" failure=The update cannot be verified: unable to locate a valid signature for one or more sources Upstream is unset, so the cluster will use an appropriate default. Channel: candidate-4.11 (available channels: candidate-4.10, candidate-4.11) No updates available. You may force an upgrade to a specific release image, but doing so may not be supported and may result in downtime or data loss. # oc adm upgrade --clear Cancelled requested upgrade to registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100 # oc adm upgrade Cluster version is 4.10.18 ReleaseAccepted=False Reason: RetrievePayload Message: Retrieving payload failed version="" image="registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100" failure=The update cannot be verified: unable to locate a valid signature for one or more sources Upstream is unset, so the cluster will use an appropriate default. Channel: candidate-4.11 (available channels: candidate-4.10, candidate-4.11) No updates available. You may force an upgrade to a specific release image, but doing so may not be supported and may result in downtime or data loss.
Verifying before PR is merged: 1. Install a cluster with the PR using cluster-bot # oc adm upgrade Cluster version is 4.11.0-0.ci.test-2022-07-27-053923-ci-ln-b77hqs2-latest warning: Cannot display available updates: Reason: NoChannel Message: The update channel has not been configured. 2. Upgrade to an unsigned payload # oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:5cdba1294e32ebc1a0140426faf41e2757416955dee25f5caca1a100b6fdea34 --allow-explicit-upgrade warning: The requested upgrade image is not one of the available updates.You have used --allow-explicit-upgrade for the update to proceed anyway Requesting update to release image registry.ci.openshift.org/ocp/release@sha256:5cdba1294e32ebc1a0140426faf41e2757416955dee25f5caca1a100b6fdea34 # oc adm upgrade Cluster version is 4.11.0-0.ci.test-2022-07-27-053923-ci-ln-b77hqs2-latest ReleaseAccepted=False Reason: RetrievePayload Message: Retrieving payload failed version="" image="registry.ci.openshift.org/ocp/release@sha256:5cdba1294e32ebc1a0140426faf41e2757416955dee25f5caca1a100b6fdea34" failure=The update cannot be verified: unable to verify sha256:5cdba1294e32ebc1a0140426faf41e2757416955dee25f5caca1a100b6fdea34 against keyrings: verifier-public-key-redhat warning: Cannot display available updates: Reason: NoChannel Message: The update channel has not been configured. 3. Clear the upgrade # oc adm upgrade --clear Cancelled requested upgrade to registry.ci.openshift.org/ocp/release@sha256:5cdba1294e32ebc1a0140426faf41e2757416955dee25f5caca1a100b6fdea34 # oc adm upgrade Cluster version is 4.11.0-0.ci.test-2022-07-27-053923-ci-ln-b77hqs2-latest warning: Cannot display available updates: Reason: NoChannel Message: The update channel has not been configured. After upgrade is cleared, cvo doesn't complain about previous desired target any more. Looks good to me.
Moving to verified state based on comment#3.
Jack, will we introduce it back to 4.11 or 4.10?
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:7399