Bug 1992677 - OLM upgradeable condition message unclear with MaxOpenShiftVersion set
Summary: OLM upgradeable condition message unclear with MaxOpenShiftVersion set
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.8
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.9.0
Assignee: Ankita Thomas
QA Contact: Jian Zhang
Padraig O'Grady
URL:
Whiteboard:
: 1992809 (view as bug list)
Depends On:
Blocks: 1994038
TreeView+ depends on / blocked
 
Reported: 2021-08-11 14:36 UTC by Kevin Rizza
Modified: 2022-05-29 11:54 UTC (History)
24 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: MaxOpenShiftVersion specifies only <major>.<minor>.0 version Consequence: Error message on upgrade says upgrades are not allowed past <major>.<minor>.0, when what it should say is that upgrades are blocked only to greated major/minor versions: patch version upgrades are allowed. Fix: Clarify error message Result: Error message specifies only minor/major version upgrades are blocked.
Clone Of:
: 1994038 (view as bug list)
Environment:
Last Closed: 2021-10-18 17:45:58 UTC
Target Upstream Version:
Embargoed:
davegord: needinfo-


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift operator-framework-olm pull 169 0 None None None 2021-08-18 14:44:36 UTC
Red Hat Knowledge Base (Solution) 6267791 0 None None None 2021-08-18 09:21:35 UTC
Red Hat Product Errata RHSA-2021:3759 0 None None None 2021-10-18 17:46:14 UTC

Description Kevin Rizza 2021-08-11 14:36:31 UTC
Description of problem:

When OLM reconciles the clusteroperator upgradeable condition, it sets that condition to false when installed operators set the maxopenshiftversion to a minor ocp version less than or equal to the current ocp version. This blocks minor version upgrades until those workloads are handled correctly.

However, when we set that condition we also propogate a message to the cluster that describes this. It includes a reference to the maxocpversion, but it appends the patch version in order to describe the semantic version of the cluster. Ex:

Reason: IncompatibleOperatorsInstalled
  Message: Cluster operator operator-lifecycle-manager cannot be upgraded between minor versions: The following operators block OpenShift upgrades: Operator openshift-gitops-operator.v1.2.0 in namespace openshift-operators is not compatible with OpenShift versions greater than 4.8.0

This message is incorrect. This status does not block upgrades for versions greater than 4.8.0, it blocks upgrades for versions >=4.9.0 when the operator sets MaxOpenShiftVersion = "4.8".

This message needs to be updated so that it is clear to end users that this does not block z-stream updates in the 4.8 upgrade path.

Version-Release number of selected component (if applicable):

4.8

How reproducible:

always

Steps to Reproduce:
1. Install an operator on 4.8 that sets the maxopenshiftversion=4.8
2. Look at the clusteroperator status field for the operator-lifecycle-manager clusteroperator object
3.

Actual results:

Status looks like:

Cluster operator operator-lifecycle-manager cannot be upgraded between minor versions: The following operators block OpenShift upgrades: Operator $OperatorName in namespace $OperatorNamespace is not compatible with OpenShift versions greater than 4.8.0

Expected results:

The message should be updated to be clear that z-streams for the current version can still be installed instead of referencing 4.8.0

Additional info:

Comment 1 Kevin Rizza 2021-08-12 12:18:35 UTC
*** Bug 1992809 has been marked as a duplicate of this bug. ***

Comment 2 Jimmy Scott 2021-08-13 07:57:19 UTC
Hi there,

Currently upgrading to 4.8.4 and bumping into a similar issue:

> $ oc get clusterversion
>
> NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
> version   4.7.21    True        True          10h     Unable to apply 4.8.4: the cluster operator authentication has not yet successfully rolled out

> $ oc get clusterversion -o yaml
> ...
>     - lastTransitionTime: "2021-08-12T10:31:28Z"
>       message: 'Cluster operator operator-lifecycle-manager cannot be upgraded between minor versions: The following operators block OpenShift upgrades: Operator ocs-operator.v4.8.0 in namespace openshift-storage is not compatible with OpenShift versions greater than 4.8.0,Operator kubevirt-hyperconverged-operator.v4.8.0 in namespace openshift-cnv is not compatible with OpenShift versions greater than 4.8.0'
>       reason: IncompatibleOperatorsInstalled
>       status: "False"
>       type: Upgradeable
> ...

What would be a valid workaround?

I'm thinking of updating the operator yaml from:

> olm.properties: '[{"type": "olm.maxOpenShiftVersion", "value": "4.8"}]'

to:

> olm.properties: '[{"type": "olm.maxOpenShiftVersion", "value": "4.8.4"}]'

But suggestions are welcome!

Kind regards,
Jimmy Scott

Comment 3 Scott Dodson 2021-08-16 14:40:03 UTC
This bug doesn't affect cluster availability but the messaging is confusing to the point that I believe we should fix this before we promote 4.7 to 4.8 upgrades to the stable channel. So marking this as Upgrades, UpgradeBlocker to help track that and I'll go ahead and clone this.

Comment 5 Jian Zhang 2021-08-19 07:56:04 UTC
Since there is no 4.9.z payload, the users won't meet this bug on 4.9 for now. After reading the code, looks good. I will verify it so that we can backport it to the 4.8.z quickly.
FYI, I also created a cluster that contains the unmerged PR via cluster-bot, but it's a 4.8 cluster, not 4.9.

launch openshift/operator-framework-olm#169 aws

[cloud-user@preserve-olm-env jian]$ oc get clusterversion
NAME      VERSION                                                  AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.ci.test-2021-08-19-063248-ci-ln-mk8cbdk-latest   True        False         35m     Cluster version is 4.8.0-0.ci.test-2021-08-19-063248-ci-ln-mk8cbdk-latest

[cloud-user@preserve-olm-env jian]$ oc exec catalog-operator-5f464ccbdc-dkzx6 -- olm --version
OLM version: 0.18.3
git commit: 17061749dd8f8c2f0e669f07b93f35a92c6c95d1

Comment 7 Jian Zhang 2021-08-23 08:35:34 UTC
Changed it to VERIFIED based on comment 5.

Comment 9 W. Trevor King 2021-09-15 15:58:20 UTC
Per comment 3, UpgradeBlocker here was reminding us that we wanted this fixed in 4.8.z before we enabled stable 4.7->4.8 edges.  But the backport of this fix went out with 4.8.10 [1], so we can drop that marker now.

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1994038#c17

Comment 11 Dave Gordon 2021-10-06 11:13:43 UTC
@

Comment 15 errata-xmlrpc 2021-10-18 17:45:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759

Comment 16 jdee 2022-02-24 21:06:52 UTC
I also have a CU having the same issue in 4.9.15

Case linked 03143352

ClusterServiceVersions blocking cluster upgrade: sebshift-powerscale-csi/dell-csi-operator.v1.6.0 is incompatible with OpenShift minor versions greater than 4.9


Note You need to log in before you can comment on or make changes to this bug.