Bug 2087103

Summary: "Updating to release image" from 'oc' should point out that the cluster-version operator hasn't accepted the update
Product: OpenShift Container Platform Reporter: Polina Rabinovich <prabinov>
Component: ocAssignee: W. Trevor King <wking>
oc sub component: oc QA Contact: Yang Yang <yanyang>
Status: CLOSED ERRATA Docs Contact:
Severity: low    
Priority: medium CC: aos-team-ota, jack.ottofaro, mfojtik, wking, yanyang
Version: 4.6   
Target Milestone: ---   
Target Release: 4.11.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-08-10 11:12:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Polina Rabinovich 2022-05-17 10:29:33 UTC
Description of problem:
Incorrect message when upgrade shouldn't start:

I'm testing the poison pill operator and new release of the PP operator will contains changes which prevent OCP upgrades to 4.11.

When I install poison pill on the cluster I get this messages:

[kni@provisionhost-0-0 ~]$ oc describe clusteroperator operator-lifecycle-manager | grep block
    Message:               ClusterServiceVersions blocking cluster upgrade: openshift-operators/poison-pill.v0.3.1 is incompatible with OpenShift minor versions greater than 4.10
[kni@provisionhost-0-0 ~]$ oc adm upgrade
Cluster version is 4.10.15

Upgradeable=False

  Reason: IncompatibleOperatorsInstalled
  Message: Cluster operator operator-lifecycle-manager should not be upgraded between minor versions: ClusterServiceVersions blocking cluster upgrade: openshift-operators/poison-pill.v0.3.1 is incompatible with OpenShift minor versions greater than 4.10

Upstream is unset, so the cluster will use an appropriate default.
Channel: stable-4.10
warning: Cannot display available updates:
  Reason: VersionNotFound
  Message: Unable to retrieve available updates: currently reconciling cluster version 4.10.15 not found in the "stable-4.10" channel

When I run the upgrade command the upgrade indeed not started (as expected), but I get message that upgrade started:

$ oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f107 --allow-explicit-upgrade

warning: The requested upgrade image is not one of the available updates.You have used --allow-explicit-upgrade for the update to proceed anyway
Updating to release image registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f107
--------------------------------------------------
Version-Release number of the following components:

[kni@provisionhost-0-0 ~]$ oc get csv -n openshift-operators
NAME                 DISPLAY                VERSION   REPLACES   PHASE
poison-pill.v0.3.1   Poison Pill Operator   0.3.1                Succeeded


[kni@provisionhost-0-0 ~]$ oc version
Client Version: 4.10.15
Server Version: 4.10.15
Kubernetes Version: v1.23.5+3afdacb
--------------------------------------------------
How reproducible:

Steps to Reproduce:
1. $ oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f107 --allow-explicit-upgrade
2.
3.
--------------------------------------------------
Actual results:
I get incorrect message when upgrade shouldn't start
--------------------------------------------------
Expected results:
I get correct message when upgrade shouldn't start
--------------------------------------------------

Comment 2 Jack Ottofaro 2022-05-26 19:25:27 UTC
The behaviour described is how the client app "oc" currently operates.

As described in "oc adm upgrade"'s help:

"...If no arguments are passed the command will retrieve the current version info and display whether an upgrade is in progress or whether any errors might prevent an upgrade, as well as show the suggested updates available to the cluster."

However when arguments are passed the command becomes a request for the cluster to begin an upgrade so the message "Updating to ..." is correct. It does not know at this point how CVO will handle that request. To know that you would run "oc adm upgrade" again.

If any changes were to made in this area I believe they would be made in "oc" so changing the component however it appears to be operating as designed.

Comment 3 W. Trevor King 2022-05-26 23:30:31 UTC
How do we feel about changing "Updating to release image" to "Requesting update to release image" or some such?  To make it clearer that oc is satisfied, but that oc is not going to vouch for whether the cluster-version operator will be satisfied.

Comment 4 W. Trevor King 2022-06-01 04:22:39 UTC
Shifting the Version back to 4.6, because that's our oldest supported version [1], and oc has always used the outgoing wording.  Dropping severity to low, because I'm just going to fiddle with a stdout string, and not change any functionality.

[1]: https://access.redhat.com/support/policy/updates/openshift/#dates

Comment 7 Yang Yang 2022-06-06 03:37:33 UTC
Verifying with 4.11.0-0.nightly-2022-06-04-014713

# oc version
Client Version: 4.11.0-0.nightly-2022-06-04-014713

Upgrading to a fake release

# oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100 --allow-explicit-upgrade
warning: The requested upgrade image is not one of the available updates.You have used --allow-explicit-upgrade for the update to proceed anyway
Requesting update to release image registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100

# oc adm upgrade 
Cluster version is 4.11.0-0.nightly-2022-06-04-014713

ReleaseAccepted=False

  Reason: RetrievePayload
  Message: Retrieving payload failed version="" image="registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100" failure=The update cannot be verified: unable to verify sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100 against keyrings: verifier-public-key-redhat

Upstream is unset, so the cluster will use an appropriate default.
Channel: stable-4.11
warning: Cannot display available updates:
  Reason: VersionNotFound
  Message: Unable to retrieve available updates: currently reconciling cluster version 4.11.0-0.nightly-2022-06-04-014713 not found in the "stable-4.11" channel

# oc adm upgrade --clear
Cancelled requested upgrade to registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100

# oc adm upgrade 
Cluster version is 4.11.0-0.nightly-2022-06-04-014713

ReleaseAccepted=False

  Reason: RetrievePayload
  Message: Retrieving payload failed version="" image="registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100" failure=The update cannot be verified: unable to verify sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100 against keyrings: verifier-public-key-redhat

Upstream is unset, so the cluster will use an appropriate default.
Channel: stable-4.11
warning: Cannot display available updates:
  Reason: VersionNotFound
  Message: Unable to retrieve available updates: currently reconciling cluster version 4.11.0-0.nightly-2022-06-04-014713 not found in the "stable-4.11" channel

# oc patch clusterversion/version --patch '{"spec":{"upstream":"https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/graph"}}' --type=merge
clusterversion.config.openshift.io/version patched

# oc adm upgrade 
Cluster version is 4.11.0-0.nightly-2022-06-04-014713

ReleaseAccepted=False

  Reason: RetrievePayload
  Message: Retrieving payload failed version="" image="registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100" failure=The update cannot be verified: unable to verify sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100 against keyrings: verifier-public-key-redhat

Upstream: https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/graph
Channel: stable-4.11

Recommended updates:

  VERSION                            IMAGE
  4.11.0-0.nightly-2022-06-05-215804 registry.ci.openshift.org/ocp/release@sha256:eb40aae6e09863357f85367456c18484f3c552930c698ed07e341b5022147b3b
  4.11.0-0.nightly-2022-06-05-073515 registry.ci.openshift.org/ocp/release@sha256:20962b3a99a6ad8bdbee1e79c02593d081d79e7fc514dada3860ec49d5b5383e
  4.11.0-0.nightly-2022-06-05-003417 registry.ci.openshift.org/ocp/release@sha256:474d87884bc6bb0fc5c6e99835afd22bc992039e48413da1e1c92a116f647d5c
  4.11.0-0.nightly-2022-06-04-180008 registry.ci.openshift.org/ocp/release@sha256:54f775170ea8323770ba9501a9556ffb570de05856f28abde58a540e94be8903
  4.11.0-0.nightly-2022-06-04-105325 registry.ci.openshift.org/ocp/release@sha256:0dd4037677719b2837a9615016ab9bdd359775ad81d33e888bf0f5485426c5a9

# oc adm upgrade --to 4.11.0-0.nightly-2022-06-04-105325 --force
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Requesting update to 4.11.0-0.nightly-2022-06-04-105325

# oc adm upgrade 
info: An upgrade is in progress. Working towards 4.11.0-0.nightly-2022-06-04-105325: 9 of 802 done (1% complete)

Upstream: https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/graph
Channel: stable-4.11

Recommended updates:

  VERSION                            IMAGE
  4.11.0-0.nightly-2022-06-05-215804 registry.ci.openshift.org/ocp/release@sha256:eb40aae6e09863357f85367456c18484f3c552930c698ed07e341b5022147b3b
  4.11.0-0.nightly-2022-06-05-073515 registry.ci.openshift.org/ocp/release@sha256:20962b3a99a6ad8bdbee1e79c02593d081d79e7fc514dada3860ec49d5b5383e
  4.11.0-0.nightly-2022-06-05-003417 registry.ci.openshift.org/ocp/release@sha256:474d87884bc6bb0fc5c6e99835afd22bc992039e48413da1e1c92a116f647d5c
  4.11.0-0.nightly-2022-06-04-180008 registry.ci.openshift.org/ocp/release@sha256:54f775170ea8323770ba9501a9556ffb570de05856f28abde58a540e94be8903
  4.11.0-0.nightly-2022-06-04-105325 registry.ci.openshift.org/ocp/release@sha256:0dd4037677719b2837a9615016ab9bdd359775ad81d33e888bf0f5485426c5a9

oc outputs message "Requesting update to ...". Moving it to verified state.

Comment 9 errata-xmlrpc 2022-08-10 11:12:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069