Bug 2213452

Summary: Set ​​maxOpenShiftVersion to block OpenShift that didn't upgrade ODF version
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Eran Tamir <etamir>
Component: odf-operatorAssignee: Nitin Goyal <nigoyal>
Status: CLOSED ERRATA QA Contact: Vijay Avuthu <vavuthu>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.12CC: ebenahar, kramdoss, muagarwa, nigoyal, ocs-bugs, odf-bz-bot, vavuthu
Target Milestone: ---   
Target Release: ODF 4.12.5   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 4.12.5-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 2213451
: 2213456 (view as bug list) Environment:
Last Closed: 2023-07-26 16:57:57 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2213456    
Bug Blocks: 2213451    

Description Eran Tamir 2023-06-08 08:00:29 UTC 
Description of problem (please be detailed as possible and provide log
snippests):

Currently, we don't block OpenShift upgrades when the ODF version is not compatible. For example, when the ODF version is 4.12 and the OpenShift version is 4.13, we should block an upgrade to 4.14 as ODF 4.12 is incompatible. 



Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

Yes. It may cause and issue and ends with a support call. 

Is there any workaround available to the best of your knowledge?

Sometimes manual changes in the CSV can help, but it involves support and may help just to upgrade to the relevant ODF version, while part of the service may be impacted. 

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?

1

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?

Yes
Comment 5 Nitin Goyal 2023-07-03 06:11:52 UTC 
PR is merged.
Comment 7 Vijay Avuthu 2023-07-14 13:50:38 UTC 
Update:
========

1. install odf 4.12.5-1 with openshift installer (4.12.0-0.nightly-2023-07-06-110941)
2. upgrade OCP 4.12 to 4.13 which is successful
3. upgrade OCP 4.13 to 4.14 which is also successful ( expected to fail )

> csv
$  oc get csv
NAME                                    DISPLAY                       VERSION        REPLACES                                PHASE
mcg-operator.v4.12.5-rhodf              NooBaa Operator               4.12.5-rhodf   mcg-operator.v4.12.4-rhodf              Succeeded
ocs-operator.v4.12.5-rhodf              OpenShift Container Storage   4.12.5-rhodf   ocs-operator.v4.12.4-rhodf              Succeeded
odf-csi-addons-operator.v4.12.5-rhodf   CSI Addons                    4.12.5-rhodf   odf-csi-addons-operator.v4.12.4-rhodf   Succeeded
odf-operator.v4.12.5-rhodf              OpenShift Data Foundation     4.12.5-rhodf   odf-operator.v4.12.4-rhodf              Succeeded

> OCP upgrade history

$ oc get clusterversion -o json|jq ".items[0].status.history"
[
  {
    "acceptedRisks": "Target release version=\"\" image=\"registry.ci.openshift.org/ocp/release:4.14.0-0.nightly-2023-07-11-092038\" cannot be verified, but continuing anyway because the update was forced: release images that are not accessed via digest cannot be verified\nPrecondition \"ClusterVersionRecommendedUpdate\" failed because of \"UnknownUpdate\": RetrievedUpdates=False (VersionNotFound), so the recommended status of updating from 4.13.0-0.nightly-2023-07-12-170413 to 4.14.0-0.nightly-2023-07-11-092038 is unknown.",
    "completionTime": "2023-07-14T07:04:44Z",
    "image": "registry.ci.openshift.org/ocp/release:4.14.0-0.nightly-2023-07-11-092038",
    "startedTime": "2023-07-14T05:55:13Z",
    "state": "Completed",
    "verified": false,
    "version": "4.14.0-0.nightly-2023-07-11-092038"
  },
  {
    "acceptedRisks": "Target release version=\"\" image=\"registry.ci.openshift.org/ocp/release:4.13.0-0.nightly-2023-07-12-170413\" cannot be verified, but continuing anyway because the update was forced: release images that are not accessed via digest cannot be verified\nForced through blocking failures: Multiple precondition checks failed:\n* Precondition \"ClusterVersionUpgradeable\" failed because of \"AdminAckRequired\": Kubernetes 1.26 and therefore OpenShift 4.13 remove several APIs which require admin consideration. Please see the knowledge article https://access.redhat.com/articles/6958394 for details and instructions.\n* Precondition \"EtcdRecentBackup\" failed because of \"ControllerStarted\": RecentBackup: The etcd backup controller is starting, and will decide if recent backups are available or if a backup is required\n* Precondition \"ClusterVersionRecommendedUpdate\" failed because of \"UnknownUpdate\": RetrievedUpdates=False (VersionNotFound), so the recommended status of updating from 4.12.0-0.nightly-2023-07-06-110941 to 4.13.0-0.nightly-2023-07-12-170413 is unknown.",
    "completionTime": "2023-07-13T12:09:31Z",
    "image": "registry.ci.openshift.org/ocp/release:4.13.0-0.nightly-2023-07-12-170413",
    "startedTime": "2023-07-13T10:51:00Z",
    "state": "Completed",
    "verified": false,
    "version": "4.13.0-0.nightly-2023-07-12-170413"
  },
  {
    "completionTime": "2023-07-13T10:26:09Z",
    "image": "registry.ci.openshift.org/ocp/release@sha256:7396e806e21fa058413bb7baa9df00037dbeaab2d1d5de36e435a1c1b7a0fbb8",
    "startedTime": "2023-07-13T10:07:50Z",
    "state": "Completed",
    "verified": false,
    "version": "4.12.0-0.nightly-2023-07-06-110941"
  }
]

> job1 ( installation of odf 4.12.5-1 + OCP 4.12 and upgrade of OCP from 4.12 to 4.13 ) :  https://url.corp.redhat.com/2495551
> job2 ( upgrade from OCP 4.13 to OCP 4.14 ): https://url.corp.redhat.com/3ceb4df
> must gather: https://url.corp.redhat.com/1a3310e
Comment 10 Vijay Avuthu 2023-07-19 09:16:30 UTC 
Update:
============

1. Installed OCP 4.12 ( 4.12.23 ) + ODF 4.12 ( ocs-registry:4.12.5-2 )

$ oc get csv
NAME                                    DISPLAY                       VERSION        REPLACES                                PHASE
mcg-operator.v4.12.5-rhodf              NooBaa Operator               4.12.5-rhodf   mcg-operator.v4.12.4-rhodf              Succeeded
ocs-operator.v4.12.5-rhodf              OpenShift Container Storage   4.12.5-rhodf   ocs-operator.v4.12.4-rhodf              Succeeded
odf-csi-addons-operator.v4.12.5-rhodf   CSI Addons                    4.12.5-rhodf   odf-csi-addons-operator.v4.12.4-rhodf   Succeeded
odf-operator.v4.12.5-rhodf              OpenShift Data Foundation     4.12.5-rhodf   odf-operator.v4.12.4-rhodf              Succeeded


$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.12.23   True        False         13h     Cluster version is 4.12.23

2. changed channel to "candidate-4.13" and upgrade to latest

$ oc patch clusterversions/version -p '{"spec":{"channel":"candidate-4.13"}}' --type=merge
clusterversion.config.openshift.io/version patched

$ oc adm upgrade --to-image=quay.io/openshift-release-dev/ocp-release@sha256:af19e94813478382e36ae1fa2ae7bbbff1f903dded6180f4eb0624afe6fc6cd4 --allow-explicit-upgrade
Requesting update to 4.13.5

oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.13.5    True        False         9m20s   Cluster version is 4.13.5

3. chnaged channel to candidate-4.14 and tried upgrading to latest image available and its not triggered the upgrade which is expected

$ oc patch clusterversions/version -p '{"spec":{"channel":"candidate-4.14"}}' --type=merge
clusterversion.config.openshift.io/version patched

$ oc adm upgrade
Cluster version is 4.13.5

Upgradeable=False

  Reason: IncompatibleOperatorsInstalled
  Message: Cluster operator operator-lifecycle-manager should not be upgraded between minor versions: ClusterServiceVersions blocking cluster upgrade: openshift-storage/odf-operator.v4.12.5-rhodf is incompatible with OpenShift minor versions greater than 4.13

Upstream is unset, so the cluster will use an appropriate default.
Channel: candidate-4.14 (available channels: candidate-4.13, candidate-4.14)

Recommended updates:

  VERSION     IMAGE
  4.14.0-ec.3 quay.io/openshift-release-dev/ocp-release@sha256:3c050cb52fdd3e65c518d4999d238ec026ef724503f275377fee6bf0d33093ab
 
$ oc adm upgrade --to-image=quay.io/openshift-release-dev/ocp-release@sha256:3c050cb52fdd3e65c518d4999d238ec026ef724503f275377fee6bf0d33093ab --allow-explicit-upgrade
Requesting update to 4.14.0-ec.3

$ oc adm upgrade
Cluster version is 4.13.5

Upgradeable=False

  Reason: IncompatibleOperatorsInstalled
  Message: Cluster operator operator-lifecycle-manager should not be upgraded between minor versions: ClusterServiceVersions blocking cluster upgrade: openshift-storage/odf-operator.v4.12.5-rhodf is incompatible with OpenShift minor versions greater than 4.13

ReleaseAccepted=False

  Reason: PreconditionChecks
  Message: Preconditions failed for payload loaded version="4.14.0-ec.3" image="quay.io/openshift-release-dev/ocp-release@sha256:3c050cb52fdd3e65c518d4999d238ec026ef724503f275377fee6bf0d33093ab": Precondition "ClusterVersionUpgradeable" failed because of "IncompatibleOperatorsInstalled": Cluster operator operator-lifecycle-manager should not be upgraded between minor versions: ClusterServiceVersions blocking cluster upgrade: openshift-storage/odf-operator.v4.12.5-rhodf is incompatible with OpenShift minor versions greater than 4.13

Upstream is unset, so the cluster will use an appropriate default.
Channel: candidate-4.14 (available channels: candidate-4.13, candidate-4.14)

Recommended updates:

  VERSION     IMAGE
  4.14.0-ec.3 quay.io/openshift-release-dev/ocp-release@sha256:3c050cb52fdd3e65c518d4999d238ec026ef724503f275377fee6bf0d33093ab

4. tried upgrading using --to-latest flag and upgrade is not triggered as expected

$ oc adm upgrade --to-latest
Requesting update to 4.14.0-ec.3
vijay ~/rem $ oc adm upgrade
Cluster version is 4.13.5

Upgradeable=False

  Reason: IncompatibleOperatorsInstalled
  Message: Cluster operator operator-lifecycle-manager should not be upgraded between minor versions: ClusterServiceVersions blocking cluster upgrade: openshift-storage/odf-operator.v4.12.5-rhodf is incompatible with OpenShift minor versions greater than 4.13

ReleaseAccepted=False

  Reason: PreconditionChecks
  Message: Preconditions failed for payload loaded version="4.14.0-ec.3" image="quay.io/openshift-release-dev/ocp-release@sha256:3c050cb52fdd3e65c518d4999d238ec026ef724503f275377fee6bf0d33093ab": Precondition "ClusterVersionUpgradeable" failed because of "IncompatibleOperatorsInstalled": Cluster operator operator-lifecycle-manager should not be upgraded between minor versions: ClusterServiceVersions blocking cluster upgrade: openshift-storage/odf-operator.v4.12.5-rhodf is incompatible with OpenShift minor versions greater than 4.13

Upstream is unset, so the cluster will use an appropriate default.
Channel: candidate-4.14 (available channels: candidate-4.13, candidate-4.14)

Recommended updates:

  VERSION     IMAGE
  4.14.0-ec.3 quay.io/openshift-release-dev/ocp-release@sha256:3c050cb52fdd3e65c518d4999d238ec026ef724503f275377fee6bf0d33093ab
$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.13.5    True        False         30m     Cluster version is 4.13.5

> checked olm.properties and maxOpenShiftVersion version is set properly 

oc get csv odf-operator.v4.12.5-rhodf -o json | jq '.metadata.annotations."olm.properties"'
"[{\"type\": \"olm.maxOpenShiftVersion\", \"value\": \"4.13\"}]"
Comment 15 errata-xmlrpc 2023-07-26 16:57:57 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat OpenShift Data Foundation 4.12.5 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:4287