Bug 1992677

Summary: OLM upgradeable condition message unclear with MaxOpenShiftVersion set
Product: OpenShift Container Platform Reporter: Kevin Rizza <krizza>
Component: OLMAssignee: Ankita Thomas <ankithom>
OLM sub component: OLM QA Contact: Jian Zhang <jiazha>
Status: CLOSED ERRATA Docs Contact: Padraig O'Grady <pogrady>
Severity: urgent    
Priority: urgent CC: acandelp, aharchin, ankithom, brault, chdeshpa, davegord, dcain, fdeutsch, jdee, jmontleo, jsaucier, jsco, kbon, nkaushik, pmagotra, pogrady, rbobek, rjohnson, r.levensalor, rsandu, ssankar, tflannag, vkochuku, wking
Version: 4.8Keywords: FastFix, Upgrades
Target Milestone: ---Flags: davegord: needinfo-
Target Release: 4.9.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: MaxOpenShiftVersion specifies only <major>.<minor>.0 version Consequence: Error message on upgrade says upgrades are not allowed past <major>.<minor>.0, when what it should say is that upgrades are blocked only to greated major/minor versions: patch version upgrades are allowed. Fix: Clarify error message Result: Error message specifies only minor/major version upgrades are blocked.
Story Points: ---
Clone Of:
: 1994038 (view as bug list) Environment:
Last Closed: 2021-10-18 17:45:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1994038    

Description Kevin Rizza 2021-08-11 14:36:31 UTC
Description of problem:

When OLM reconciles the clusteroperator upgradeable condition, it sets that condition to false when installed operators set the maxopenshiftversion to a minor ocp version less than or equal to the current ocp version. This blocks minor version upgrades until those workloads are handled correctly.

However, when we set that condition we also propogate a message to the cluster that describes this. It includes a reference to the maxocpversion, but it appends the patch version in order to describe the semantic version of the cluster. Ex:

Reason: IncompatibleOperatorsInstalled
  Message: Cluster operator operator-lifecycle-manager cannot be upgraded between minor versions: The following operators block OpenShift upgrades: Operator openshift-gitops-operator.v1.2.0 in namespace openshift-operators is not compatible with OpenShift versions greater than 4.8.0

This message is incorrect. This status does not block upgrades for versions greater than 4.8.0, it blocks upgrades for versions >=4.9.0 when the operator sets MaxOpenShiftVersion = "4.8".

This message needs to be updated so that it is clear to end users that this does not block z-stream updates in the 4.8 upgrade path.

Version-Release number of selected component (if applicable):

4.8

How reproducible:

always

Steps to Reproduce:
1. Install an operator on 4.8 that sets the maxopenshiftversion=4.8
2. Look at the clusteroperator status field for the operator-lifecycle-manager clusteroperator object
3.

Actual results:

Status looks like:

Cluster operator operator-lifecycle-manager cannot be upgraded between minor versions: The following operators block OpenShift upgrades: Operator $OperatorName in namespace $OperatorNamespace is not compatible with OpenShift versions greater than 4.8.0

Expected results:

The message should be updated to be clear that z-streams for the current version can still be installed instead of referencing 4.8.0

Additional info:

Comment 1 Kevin Rizza 2021-08-12 12:18:35 UTC
*** Bug 1992809 has been marked as a duplicate of this bug. ***

Comment 2 Jimmy Scott 2021-08-13 07:57:19 UTC
Hi there,

Currently upgrading to 4.8.4 and bumping into a similar issue:

> $ oc get clusterversion
>
> NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
> version   4.7.21    True        True          10h     Unable to apply 4.8.4: the cluster operator authentication has not yet successfully rolled out

> $ oc get clusterversion -o yaml
> ...
>     - lastTransitionTime: "2021-08-12T10:31:28Z"
>       message: 'Cluster operator operator-lifecycle-manager cannot be upgraded between minor versions: The following operators block OpenShift upgrades: Operator ocs-operator.v4.8.0 in namespace openshift-storage is not compatible with OpenShift versions greater than 4.8.0,Operator kubevirt-hyperconverged-operator.v4.8.0 in namespace openshift-cnv is not compatible with OpenShift versions greater than 4.8.0'
>       reason: IncompatibleOperatorsInstalled
>       status: "False"
>       type: Upgradeable
> ...

What would be a valid workaround?

I'm thinking of updating the operator yaml from:

> olm.properties: '[{"type": "olm.maxOpenShiftVersion", "value": "4.8"}]'

to:

> olm.properties: '[{"type": "olm.maxOpenShiftVersion", "value": "4.8.4"}]'

But suggestions are welcome!

Kind regards,
Jimmy Scott

Comment 3 Scott Dodson 2021-08-16 14:40:03 UTC
This bug doesn't affect cluster availability but the messaging is confusing to the point that I believe we should fix this before we promote 4.7 to 4.8 upgrades to the stable channel. So marking this as Upgrades, UpgradeBlocker to help track that and I'll go ahead and clone this.

Comment 5 Jian Zhang 2021-08-19 07:56:04 UTC
Since there is no 4.9.z payload, the users won't meet this bug on 4.9 for now. After reading the code, looks good. I will verify it so that we can backport it to the 4.8.z quickly.
FYI, I also created a cluster that contains the unmerged PR via cluster-bot, but it's a 4.8 cluster, not 4.9.

launch openshift/operator-framework-olm#169 aws

[cloud-user@preserve-olm-env jian]$ oc get clusterversion
NAME      VERSION                                                  AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.ci.test-2021-08-19-063248-ci-ln-mk8cbdk-latest   True        False         35m     Cluster version is 4.8.0-0.ci.test-2021-08-19-063248-ci-ln-mk8cbdk-latest

[cloud-user@preserve-olm-env jian]$ oc exec catalog-operator-5f464ccbdc-dkzx6 -- olm --version
OLM version: 0.18.3
git commit: 17061749dd8f8c2f0e669f07b93f35a92c6c95d1

Comment 7 Jian Zhang 2021-08-23 08:35:34 UTC
Changed it to VERIFIED based on comment 5.

Comment 9 W. Trevor King 2021-09-15 15:58:20 UTC
Per comment 3, UpgradeBlocker here was reminding us that we wanted this fixed in 4.8.z before we enabled stable 4.7->4.8 edges.  But the backport of this fix went out with 4.8.10 [1], so we can drop that marker now.

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1994038#c17

Comment 11 Dave Gordon 2021-10-06 11:13:43 UTC
@

Comment 15 errata-xmlrpc 2021-10-18 17:45:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759

Comment 16 jdee 2022-02-24 21:06:52 UTC
I also have a CU having the same issue in 4.9.15

Case linked 03143352

ClusterServiceVersions blocking cluster upgrade: sebshift-powerscale-csi/dell-csi-operator.v1.6.0 is incompatible with OpenShift minor versions greater than 4.9