Bug 1992680 - oc adm upgrade is unclear about minor and patch upgrade blocking
Summary: oc adm upgrade is unclear about minor and patch upgrade blocking
Keywords:
Status: POST
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: oc
Version: 4.8
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.10.0
Assignee: W. Trevor King
QA Contact: Yang Yang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-08-11 14:44 UTC by Kevin Rizza
Modified: 2021-10-20 10:50 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift oc pull 899 0 None None None 2021-08-16 18:14:14 UTC
Github openshift oc pull 900 0 None None None 2021-08-17 00:47:05 UTC
Github openshift oc pull 903 0 None None None 2021-08-18 21:10:31 UTC

Description Kevin Rizza 2021-08-11 14:44:49 UTC
Description of problem:

Components that set their clusteroperator's upgradeable status to false prevent minor version upgrades, but they do not prevent z-stream updates. The `oc adm upgrade` command provides some information to a user about what upgrades are allowed. However, this command makes it seem like any upgrade is disallowed if Upgradeable = False for a given component. For example:

$ oc adm upgrade
Cluster version is 4.8.4

Upgradeable=False

  Reason: IncompatibleOperatorsInstalled
  Message: Cluster operator operator-lifecycle-manager cannot be upgraded between minor versions: The following operators block OpenShift upgrades: Operator openshift-gitops-operator.v1.2.0 in namespace openshift-operators is not compatible with OpenShift versions greater than 4.8.0

While this status message could be improved (and certainly exacerbates this problem given the fact that it references 4.8.0 rather than 4.8.z), it is unclear from a user's perspective that we allow z-stream updates in this state. This message should be improved so that oc adm upgrade explains what subset of upgrades are and are not allowed.

How reproducible:

Always

Steps to Reproduce:
1. Get a clusteroperator into a state where upgradeable is set to false
2. run `oc adm upgrade` without any arguments

Actual results:

The upgradeable status is written to the console output without any indication that certain upgrades are still allowed.

Expected results:

A delineation between minor and patch upgrades.

Comment 1 Ben Parees 2021-08-11 15:11:44 UTC
There is another bug here that the "oc adm upgrade" command is supposed to output available upgrade versions, but if there are no available upgrade versions(regardless of the upgradeable status, just no actual edges that exist), the command doesn't provide any indication of that, it just prints nothing.  That lead to more confusion for this particular user since they thought they were being denied an available upgrade, but the reality is there just are no available upgrades for them.

I think both things can be fixed as part of this bug for improving the message output of "oc adm upgrade"

Comment 2 Ben Parees 2021-08-11 15:14:33 UTC
also this help text from "oc adm upgrade --help" is confusing:

This command will request that the cluster begin an upgrade. If no arguments
are passed the command will retrieve the current version info and display
whether an upgrade is in progress or whether any errors might prevent an
upgrade, as well as show the suggested updates available to the cluster.
Information about compatible updates is periodically retrieved from the update
server and cached on the cluster - these are updates that are known to be
supported as upgrades from the current version.


The first sentence says the command will begin an upgrade.  The following sentences immediately contradict that (the command does *not* start an upgrade, without additional explicit arguments, it just prints upgrade status/availability).

Too many people are going to read that first sentence and not understand the nuance that the command *can* be used to start an upgrade(with the right args), or it can be used to get status info(with no args).  I think it can be improved, again as part of this bug.

Comment 3 Jack Ottofaro 2021-08-11 19:58:21 UTC
The condition message states "Cluster operator operator-lifecycle-manager cannot be upgraded between minor versions" so it is delineating, minor version upgrades. Are we saying that customers don't know what minor level means and CVO needs to add something like "but you still can do z-level upgrades" or whatever we want to call them? I wouldn't think that's necessary.

Comment 4 Ben Parees 2021-08-11 20:51:15 UTC
> The condition message states "Cluster operator operator-lifecycle-manager cannot be upgraded between minor versions" so it is delineating, minor version upgrades. 

we're saying this user read "upgradeable=false" and didn't read or understand the nuance of the condition message.

I don't think the CVO needs to do anything, i think the oc adm upgrade command needs to massage its output to make it clearer what "upgradeable=false" means, independent of what the CVO conditions may say.  (Unless we think the implications of "upgradeable=false" are going to change at some point)

Comment 5 Jack Ottofaro 2021-08-12 12:41:30 UTC
> (Unless we think the implications of "upgradeable=false" are going to change at some point)

Unfortunately Upgradeable=False when ClusterVersion overrides are set does block all updates, including z version updates. So we do have that exceptional case which was a change made to the original meaning of Upgradeable=false. The only thing that distinguishes that is the message so perhaps "oc adm upgrade" should put the message first.

Comment 6 Jack Ottofaro 2021-08-16 15:41:11 UTC
Based on comments https://bugzilla.redhat.com/show_bug.cgi?id=1992680#c4 and https://bugzilla.redhat.com/show_bug.cgi?id=1992680#c5 this bug requires a change to "oc adm upgrade" so changing component.

Comment 7 W. Trevor King 2021-08-16 17:02:32 UTC
From the current OLM message:

> ...not compatible with OpenShift versions greater than 4.8.0

The fact that this is just minor-bump-blocking and so should be softened to "versions greater than 4.8" or some such is covered in bug 1994038.

Comment 8 W. Trevor King 2021-08-16 17:58:57 UTC
(In reply to Ben Parees from comment #1)
> ...but if there are no available upgrade versions(regardless of the upgradeable status, just no actual edges that exist), the command doesn't provide any indication of that, it just prints nothing.

Really?  Since way back in 2019, when the oc repo was created, we should be printing [1]:

  No updates available. You may force an upgrade to a specific release image, but doing so may not be supported and result in downtime or data loss.

Confirming with a recent oc client:

  $ oc version --client
  Client Version: 4.9.0-0.nightly-arm64-2021-07-08-160356
  $ oc adm upgrade
  Cluster version is 4.8.4

  Upstream is unset, so the cluster will use an appropriate default.
  Channel: stable-4.8 (available channels: candidate-4.8, fast-4.8, stable-4.8)
  No updates available. You may force an upgrade to a specific release image, but doing so may not be supported and result in downtime or data loss.

Looks like we could tweak that to say "... and may result in...", but we are at least explicitly saying "No updates available" for this case today, and as far as I can tell, have been since 2019.

[1]: https://github.com/openshift/oc/blob/3abf60a31e41f2af6da6b7e5c946d8bcf6976193/pkg/cli/admin/upgrade/upgrade.go#L281

Comment 9 Ben Parees 2021-08-16 18:05:03 UTC
> Really?  Since way back in 2019, when the oc repo was created, we should be printing [1]:

I was going off the output that was reported in the bug by the user who was confused about not being able to upgrade their cluster.  If that message was there, but they didn't read it/understand it, and then truncated it when they opened their issue and showed us their output, then I guess we can ignore that aspect of this bug/feedback.

That said, is it possible we don't print out that information when upgradeable=false?

we appear to short-circuit it when the cluster is degraded:
https://github.com/openshift/oc/blob/3abf60a31e41f2af6da6b7e5c946d8bcf6976193/pkg/cli/admin/upgrade/upgrade.go#L243-L251

Comment 10 W. Trevor King 2021-08-16 23:47:09 UTC
Huh.  I don't think we ever set Degraded in ClusterVersion; that might be dead code.  More fixups to hang on this bug.  But the only Upgradeable check is the non-fatal print in [1].

[1]: https://github.com/openshift/oc/blob/681248e6d3b6c3d965ea04f75b6ce03ca19671d0/pkg/cli/admin/upgrade/upgrade.go#L332-L334

Comment 11 W. Trevor King 2021-08-17 17:19:52 UTC
On the doc front, out openshift-docs rendering [1] just says that the condition type is a string, and doesn't include a rendering of the  ClusterStatusConditionType values.  The godocs do render those values [2], but are Go-oriented, and not pitched as generic API docs.  I'd really like to avoid duplicating in two locations the API information we are trying to cover in openshift/api, but maybe there's currently no way around that?

[1]: https://docs.openshift.com/container-platform/4.8/rest_api/config_apis/clusterversion-config-openshift-io-v1.html#specification
[2]: https://pkg.go.dev/github.com/openshift/api/config/v1#ClusterStatusConditionType

Comment 12 Lalatendu Mohanty 2021-08-18 21:30:46 UTC
Here are the examples what OC outputs when there is no available updates

$ oc adm upgrade
Cluster version is 4.8.5

warning: Cannot display available updates:
  Reason: NoChannel
  Message: The update channel has not been configured.

$ oc patch clusterversion/version -p '{"spec":{"channel":"fast-4.8"}}' --type merge 
clusterversion.config.openshift.io/version patched

$ oc adm upgrade
Cluster version is 4.8.5

No updates available. You may force an upgrade to a specific release image, but doing so may not be supported and result in downtime or data loss.

Comment 13 W. Trevor King 2021-09-15 16:58:57 UTC
Probably not getting more in before 4.9 GAs.  Punting the remaining work to 4.10.

Comment 14 W. Trevor King 2021-10-08 02:58:07 UTC
In discussion today, it seems like we can clear up some of the minor/patch confusion by replacing "...cannot be upgraded between minor versions..." with "...cannot be upgraded to 4.9..." or "...cannot be upgraded to any release..." or some such to distinguish the two cases.  This will require some internal refactoring, but will remove our current reliance on SemVer's terminology [1].

[1]: https://semver.org/spec/v2.0.0.html#summary

Comment 15 Yang Yang 2021-10-20 09:22:45 UTC
Reproducing it by installing a 4.8 cluster,

# oc adm upgrade 
Cluster version is 4.8.13

Upgradeable=False

  Reason: IncompatibleOperatorsInstalled
  Message: Cluster operator operator-lifecycle-manager cannot be upgraded between minor versions: ClusterServiceVersions blocking cluster upgrade: openshift-operators-redhat/elasticsearch-operator.5.1.2-7 is incompatible with OpenShift minor versions greater than 4.8

Upstream is unset, so the cluster will use an appropriate default.
Channel: stable-4.8 (available channels: candidate-4.8, candidate-4.9, fast-4.8, fast-4.9, stable-4.8)
Updates:

VERSION IMAGE
4.8.14  quay.io/openshift-release-dev/ocp-release@sha256:bf48faa639523b73131ec7c91637d5c94d33a4afe09ac8bdad672862f5e86ccb

Upgradeable is set to False to block minor upgrade, but the z-stream upgrade is available. So, the issue described in comment#0 is reproduced.

The help info has fixed.

# oc adm upgrade -h
Check on upgrade status or upgrade the cluster to a newer version

 This command assists with cluster upgrades. If no arguments are passed the command will retrieve the current version
info and display whether an upgrade is in progress or whether any errors might prevent an upgrade, as well as show the
suggested updates available to the cluster. Information about compatible updates is periodically retrieved from the
update server and cached on the cluster - these are updates that are known to be supported as upgrades from the current
version.

Comment 16 Yang Yang 2021-10-20 10:50:02 UTC
# oc adm upgrade channel fast-4.9
warning: No channels known to be compatible with the current version "4.9.0"; unable to validate "fast-4.9". Setting the update channel to "fast-4.9" anyway.
[root@preserve-yangyangmerrn-1 tmp]# oc adm upgrade 
Cluster version is 4.9.0

Upstream is unset, so the cluster will use an appropriate default.
Channel: fast-4.9 (available channels: candidate-4.9, fast-4.9, stable-4.9)
No updates available. You may force an upgrade to a specific release image, but doing so may not be supported and may result in downtime or data loss.

So PR#899 is verified and passed.


Note You need to log in before you can comment on or make changes to this bug.