Bug 2094174 - ReleaseAccepted=False keeps complaining about the update cannot be verified after the upgrade is cleared
Summary: ReleaseAccepted=False keeps complaining about the update cannot be verified a...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Version Operator
Version: 4.10
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.12.0
Assignee: Over the Air Updates
QA Contact: Yang Yang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-06-07 03:44 UTC by Yang Yang
Modified: 2023-01-17 19:50 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-01-17 19:49:52 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
CVO log file (8.18 MB, text/plain)
2022-06-07 03:44 UTC, Yang Yang
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-version-operator pull 788 0 None open Bug 2094174: pkg/cvo: reset payload load status 2022-06-08 15:58:57 UTC
Red Hat Product Errata RHSA-2022:7399 0 None None None 2023-01-17 19:50:00 UTC

Description Yang Yang 2022-06-07 03:44:31 UTC
Created attachment 1887340 [details]
CVO log file

Description of problem:
Clearing upgrade after signature verification fails, ReleaseAccepted=False keeps complaining about the update cannot be verified blah blah.

# oc get clusterversion/version -ojson | jq -r '.spec, .status.conditions'
{
  "channel": "stable-4.11",
  "clusterID": "d740b8f3-bb49-40cf-86e8-5df4a755111a"
}
[
  {
    "lastTransitionTime": "2022-06-07T01:31:43Z",
    "message": "Unable to retrieve available updates: currently reconciling cluster version 4.11.0-0.nightly-2022-06-06-025509 not found in the \"stable-4.11\" channel",
    "reason": "VersionNotFound",
    "status": "False",
    "type": "RetrievedUpdates"
  },
  {
    "lastTransitionTime": "2022-06-07T01:31:43Z",
    "message": "Capabilities match configured spec",
    "reason": "AsExpected",
    "status": "False",
    "type": "ImplicitlyEnabledCapabilities"
  },
  {
    "lastTransitionTime": "2022-06-07T02:44:54Z",
    "message": "Retrieving payload failed version=\"\" image=\"registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100\" failure=The update cannot be verified: unable to verify sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100 against keyrings: verifier-public-key-redhat",
    "reason": "RetrievePayload",
    "status": "False",
    "type": "ReleaseAccepted"
  },
  {
    "lastTransitionTime": "2022-06-07T01:56:17Z",
    "message": "Done applying 4.11.0-0.nightly-2022-06-06-025509",
    "status": "True",
    "type": "Available"
  },
  {
    "lastTransitionTime": "2022-06-07T01:55:47Z",
    "status": "False",
    "type": "Failing"
  },
  {
    "lastTransitionTime": "2022-06-07T01:56:17Z",
    "message": "Cluster version is 4.11.0-0.nightly-2022-06-06-025509",
    "status": "False",
    "type": "Progressing"
  }
]

Version-Release number of the following components:
4.11.0-0.nightly-2022-06-06-025509

How reproducible:
1/1

Steps to Reproduce:
1. Upgrade to a fake release

# oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100 --allow-explicit-upgrade
warning: The requested upgrade image is not one of the available updates.You have used --allow-explicit-upgrade for the update to proceed anyway
Requesting update to release image registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100

2. Check ReleaseAccepted=False due to target image signature verification failure
# oc adm upgrade 
Cluster version is 4.11.0-0.nightly-2022-06-04-014713

ReleaseAccepted=False

  Reason: RetrievePayload
  Message: Retrieving payload failed version="" image="registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100" failure=The update cannot be verified: unable to verify sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100 against keyrings: verifier-public-key-redhat

Upstream is unset, so the cluster will use an appropriate default.
Channel: stable-4.11
warning: Cannot display available updates:
  Reason: VersionNotFound
  Message: Unable to retrieve available updates: currently reconciling cluster version 4.11.0-0.nightly-2022-06-04-014713 not found in the "stable-4.11" channel

3. Clear the upgrade 
# oc adm upgrade --clear
Cancelled requested upgrade to registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100

4. Check oc adm upgrade info
# oc adm upgrade 
Cluster version is 4.11.0-0.nightly-2022-06-04-014713

ReleaseAccepted=False

  Reason: RetrievePayload
  Message: Retrieving payload failed version="" image="registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100" failure=The update cannot be verified: unable to verify sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100 against keyrings: verifier-public-key-redhat

Upstream is unset, so the cluster will use an appropriate default.
Channel: stable-4.11
warning: Cannot display available updates:
  Reason: VersionNotFound
  Message: Unable to retrieve available updates: currently reconciling cluster version 4.11.0-0.nightly-2022-06-04-014713 not found in the "stable-4.11" channel

Actual results:
After upgrade is cleared, cv condition ReleaseAccepted keeps to false with message The update cannot be verified

Expected results:
After upgrade is cleared, cv condition ReleaseAccepted should stop complaining about the target image


Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 2 Yang Yang 2022-06-09 08:19:31 UTC
It's reproduced with 4.10.18. So changing the version to 4.10

# oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100 --allow-explicit-upgrade
warning: The requested upgrade image is not one of the available updates.You have used --allow-explicit-upgrade for the update to proceed anyway
Requesting update to release image registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100

# oc adm upgrade 
Cluster version is 4.10.18

ReleaseAccepted=False

  Reason: RetrievePayload
  Message: Retrieving payload failed version="" image="registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100" failure=The update cannot be verified: unable to locate a valid signature for one or more sources

Upstream is unset, so the cluster will use an appropriate default.
Channel: candidate-4.11 (available channels: candidate-4.10, candidate-4.11)
No updates available. You may force an upgrade to a specific release image, but doing so may not be supported and may result in downtime or data loss.

# oc adm upgrade --clear
Cancelled requested upgrade to registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100

# oc adm upgrade 
Cluster version is 4.10.18

ReleaseAccepted=False

  Reason: RetrievePayload
  Message: Retrieving payload failed version="" image="registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100" failure=The update cannot be verified: unable to locate a valid signature for one or more sources

Upstream is unset, so the cluster will use an appropriate default.
Channel: candidate-4.11 (available channels: candidate-4.10, candidate-4.11)
No updates available. You may force an upgrade to a specific release image, but doing so may not be supported and may result in downtime or data loss.

Comment 3 Yang Yang 2022-07-27 06:20:31 UTC
Verifying before PR is merged:

1. Install a cluster with the PR using cluster-bot
# oc adm upgrade
Cluster version is 4.11.0-0.ci.test-2022-07-27-053923-ci-ln-b77hqs2-latest

warning: Cannot display available updates:
  Reason: NoChannel
  Message: The update channel has not been configured.

2. Upgrade to an unsigned payload
# oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:5cdba1294e32ebc1a0140426faf41e2757416955dee25f5caca1a100b6fdea34 --allow-explicit-upgrade
warning: The requested upgrade image is not one of the available updates.You have used --allow-explicit-upgrade for the update to proceed anyway
Requesting update to release image registry.ci.openshift.org/ocp/release@sha256:5cdba1294e32ebc1a0140426faf41e2757416955dee25f5caca1a100b6fdea34

# oc adm upgrade 
Cluster version is 4.11.0-0.ci.test-2022-07-27-053923-ci-ln-b77hqs2-latest

ReleaseAccepted=False

  Reason: RetrievePayload
  Message: Retrieving payload failed version="" image="registry.ci.openshift.org/ocp/release@sha256:5cdba1294e32ebc1a0140426faf41e2757416955dee25f5caca1a100b6fdea34" failure=The update cannot be verified: unable to verify sha256:5cdba1294e32ebc1a0140426faf41e2757416955dee25f5caca1a100b6fdea34 against keyrings: verifier-public-key-redhat

warning: Cannot display available updates:
  Reason: NoChannel
  Message: The update channel has not been configured.

3. Clear the upgrade
# oc adm upgrade --clear
Cancelled requested upgrade to registry.ci.openshift.org/ocp/release@sha256:5cdba1294e32ebc1a0140426faf41e2757416955dee25f5caca1a100b6fdea34

# oc adm upgrade 
Cluster version is 4.11.0-0.ci.test-2022-07-27-053923-ci-ln-b77hqs2-latest

warning: Cannot display available updates:
  Reason: NoChannel
  Message: The update channel has not been configured.

After upgrade is cleared, cvo doesn't complain about previous desired target any more. Looks good to me.

Comment 5 Yang Yang 2022-08-01 02:13:30 UTC
Moving to verified state based on comment#3.

Comment 6 Yang Yang 2022-08-01 02:37:26 UTC
Jack, will we introduce it back to 4.11 or 4.10?

Comment 10 errata-xmlrpc 2023-01-17 19:49:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7399


Note You need to log in before you can comment on or make changes to this bug.