Description of problem: Components that set their clusteroperator's upgradeable status to false prevent minor version upgrades, but they do not prevent z-stream updates. The `oc adm upgrade` command provides some information to a user about what upgrades are allowed. However, this command makes it seem like any upgrade is disallowed if Upgradeable = False for a given component. For example: $ oc adm upgrade Cluster version is 4.8.4 Upgradeable=False Reason: IncompatibleOperatorsInstalled Message: Cluster operator operator-lifecycle-manager cannot be upgraded between minor versions: The following operators block OpenShift upgrades: Operator openshift-gitops-operator.v1.2.0 in namespace openshift-operators is not compatible with OpenShift versions greater than 4.8.0 While this status message could be improved (and certainly exacerbates this problem given the fact that it references 4.8.0 rather than 4.8.z), it is unclear from a user's perspective that we allow z-stream updates in this state. This message should be improved so that oc adm upgrade explains what subset of upgrades are and are not allowed. How reproducible: Always Steps to Reproduce: 1. Get a clusteroperator into a state where upgradeable is set to false 2. run `oc adm upgrade` without any arguments Actual results: The upgradeable status is written to the console output without any indication that certain upgrades are still allowed. Expected results: A delineation between minor and patch upgrades.
There is another bug here that the "oc adm upgrade" command is supposed to output available upgrade versions, but if there are no available upgrade versions(regardless of the upgradeable status, just no actual edges that exist), the command doesn't provide any indication of that, it just prints nothing. That lead to more confusion for this particular user since they thought they were being denied an available upgrade, but the reality is there just are no available upgrades for them. I think both things can be fixed as part of this bug for improving the message output of "oc adm upgrade"
also this help text from "oc adm upgrade --help" is confusing: This command will request that the cluster begin an upgrade. If no arguments are passed the command will retrieve the current version info and display whether an upgrade is in progress or whether any errors might prevent an upgrade, as well as show the suggested updates available to the cluster. Information about compatible updates is periodically retrieved from the update server and cached on the cluster - these are updates that are known to be supported as upgrades from the current version. The first sentence says the command will begin an upgrade. The following sentences immediately contradict that (the command does *not* start an upgrade, without additional explicit arguments, it just prints upgrade status/availability). Too many people are going to read that first sentence and not understand the nuance that the command *can* be used to start an upgrade(with the right args), or it can be used to get status info(with no args). I think it can be improved, again as part of this bug.
The condition message states "Cluster operator operator-lifecycle-manager cannot be upgraded between minor versions" so it is delineating, minor version upgrades. Are we saying that customers don't know what minor level means and CVO needs to add something like "but you still can do z-level upgrades" or whatever we want to call them? I wouldn't think that's necessary.
> The condition message states "Cluster operator operator-lifecycle-manager cannot be upgraded between minor versions" so it is delineating, minor version upgrades. we're saying this user read "upgradeable=false" and didn't read or understand the nuance of the condition message. I don't think the CVO needs to do anything, i think the oc adm upgrade command needs to massage its output to make it clearer what "upgradeable=false" means, independent of what the CVO conditions may say. (Unless we think the implications of "upgradeable=false" are going to change at some point)
> (Unless we think the implications of "upgradeable=false" are going to change at some point) Unfortunately Upgradeable=False when ClusterVersion overrides are set does block all updates, including z version updates. So we do have that exceptional case which was a change made to the original meaning of Upgradeable=false. The only thing that distinguishes that is the message so perhaps "oc adm upgrade" should put the message first.
Based on comments https://bugzilla.redhat.com/show_bug.cgi?id=1992680#c4 and https://bugzilla.redhat.com/show_bug.cgi?id=1992680#c5 this bug requires a change to "oc adm upgrade" so changing component.
From the current OLM message: > ...not compatible with OpenShift versions greater than 4.8.0 The fact that this is just minor-bump-blocking and so should be softened to "versions greater than 4.8" or some such is covered in bug 1994038.
(In reply to Ben Parees from comment #1) > ...but if there are no available upgrade versions(regardless of the upgradeable status, just no actual edges that exist), the command doesn't provide any indication of that, it just prints nothing. Really? Since way back in 2019, when the oc repo was created, we should be printing [1]: No updates available. You may force an upgrade to a specific release image, but doing so may not be supported and result in downtime or data loss. Confirming with a recent oc client: $ oc version --client Client Version: 4.9.0-0.nightly-arm64-2021-07-08-160356 $ oc adm upgrade Cluster version is 4.8.4 Upstream is unset, so the cluster will use an appropriate default. Channel: stable-4.8 (available channels: candidate-4.8, fast-4.8, stable-4.8) No updates available. You may force an upgrade to a specific release image, but doing so may not be supported and result in downtime or data loss. Looks like we could tweak that to say "... and may result in...", but we are at least explicitly saying "No updates available" for this case today, and as far as I can tell, have been since 2019. [1]: https://github.com/openshift/oc/blob/3abf60a31e41f2af6da6b7e5c946d8bcf6976193/pkg/cli/admin/upgrade/upgrade.go#L281
> Really? Since way back in 2019, when the oc repo was created, we should be printing [1]: I was going off the output that was reported in the bug by the user who was confused about not being able to upgrade their cluster. If that message was there, but they didn't read it/understand it, and then truncated it when they opened their issue and showed us their output, then I guess we can ignore that aspect of this bug/feedback. That said, is it possible we don't print out that information when upgradeable=false? we appear to short-circuit it when the cluster is degraded: https://github.com/openshift/oc/blob/3abf60a31e41f2af6da6b7e5c946d8bcf6976193/pkg/cli/admin/upgrade/upgrade.go#L243-L251
Huh. I don't think we ever set Degraded in ClusterVersion; that might be dead code. More fixups to hang on this bug. But the only Upgradeable check is the non-fatal print in [1]. [1]: https://github.com/openshift/oc/blob/681248e6d3b6c3d965ea04f75b6ce03ca19671d0/pkg/cli/admin/upgrade/upgrade.go#L332-L334
On the doc front, out openshift-docs rendering [1] just says that the condition type is a string, and doesn't include a rendering of the ClusterStatusConditionType values. The godocs do render those values [2], but are Go-oriented, and not pitched as generic API docs. I'd really like to avoid duplicating in two locations the API information we are trying to cover in openshift/api, but maybe there's currently no way around that? [1]: https://docs.openshift.com/container-platform/4.8/rest_api/config_apis/clusterversion-config-openshift-io-v1.html#specification [2]: https://pkg.go.dev/github.com/openshift/api/config/v1#ClusterStatusConditionType
Here are the examples what OC outputs when there is no available updates $ oc adm upgrade Cluster version is 4.8.5 warning: Cannot display available updates: Reason: NoChannel Message: The update channel has not been configured. $ oc patch clusterversion/version -p '{"spec":{"channel":"fast-4.8"}}' --type merge clusterversion.config.openshift.io/version patched $ oc adm upgrade Cluster version is 4.8.5 No updates available. You may force an upgrade to a specific release image, but doing so may not be supported and result in downtime or data loss.
Probably not getting more in before 4.9 GAs. Punting the remaining work to 4.10.
In discussion today, it seems like we can clear up some of the minor/patch confusion by replacing "...cannot be upgraded between minor versions..." with "...cannot be upgraded to 4.9..." or "...cannot be upgraded to any release..." or some such to distinguish the two cases. This will require some internal refactoring, but will remove our current reliance on SemVer's terminology [1]. [1]: https://semver.org/spec/v2.0.0.html#summary
Reproducing it by installing a 4.8 cluster, # oc adm upgrade Cluster version is 4.8.13 Upgradeable=False Reason: IncompatibleOperatorsInstalled Message: Cluster operator operator-lifecycle-manager cannot be upgraded between minor versions: ClusterServiceVersions blocking cluster upgrade: openshift-operators-redhat/elasticsearch-operator.5.1.2-7 is incompatible with OpenShift minor versions greater than 4.8 Upstream is unset, so the cluster will use an appropriate default. Channel: stable-4.8 (available channels: candidate-4.8, candidate-4.9, fast-4.8, fast-4.9, stable-4.8) Updates: VERSION IMAGE 4.8.14 quay.io/openshift-release-dev/ocp-release@sha256:bf48faa639523b73131ec7c91637d5c94d33a4afe09ac8bdad672862f5e86ccb Upgradeable is set to False to block minor upgrade, but the z-stream upgrade is available. So, the issue described in comment#0 is reproduced. The help info has fixed. # oc adm upgrade -h Check on upgrade status or upgrade the cluster to a newer version This command assists with cluster upgrades. If no arguments are passed the command will retrieve the current version info and display whether an upgrade is in progress or whether any errors might prevent an upgrade, as well as show the suggested updates available to the cluster. Information about compatible updates is periodically retrieved from the update server and cached on the cluster - these are updates that are known to be supported as upgrades from the current version.
# oc adm upgrade channel fast-4.9 warning: No channels known to be compatible with the current version "4.9.0"; unable to validate "fast-4.9". Setting the update channel to "fast-4.9" anyway. [root@preserve-yangyangmerrn-1 tmp]# oc adm upgrade Cluster version is 4.9.0 Upstream is unset, so the cluster will use an appropriate default. Channel: fast-4.9 (available channels: candidate-4.9, fast-4.9, stable-4.9) No updates available. You may force an upgrade to a specific release image, but doing so may not be supported and may result in downtime or data loss. So PR#899 is verified and passed.
PR#903 is now outdated. We are using Recommended updates instead. # oc adm upgrade Cluster version is 4.10.25 Upstream is unset, so the cluster will use an appropriate default. Channel: stable-4.10 (available channels: candidate-4.10, candidate-4.11, eus-4.10, fast-4.10, fast-4.11, stable-4.10) Recommended updates: VERSION IMAGE 4.10.26 quay.io/openshift-release-dev/ocp-release@sha256:e1fa1f513068082d97d78be643c369398b0e6820afab708d26acda2262940954
I've picked [1] back up, and since it doesn't have much to do with this bug's Upgradeable focus, I've spun out a new bug to track oc#900 [2]. [1]: https://github.com/openshift/oc/pull/900 [2]: https://issues.redhat.com/browse/OCPBUGS-3714