Description of problem (please be detailed as possible and provide log snippests): Currently, we don't block OpenShift upgrades when the ODF version is not compatible. For example, when the ODF version is 4.12 and the OpenShift version is 4.13, we should block an upgrade to 4.14 as ODF 4.12 is incompatible. Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Yes. It may cause and issue and ends with a support call. Is there any workaround available to the best of your knowledge? Sometimes manual changes in the CSV can help, but it involves support and may help just to upgrade to the relevant ODF version, while part of the service may be impacted. Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 1 Can this issue reproducible? Yes Can this issue reproduce from the UI? Yes
PR is merged.
Update: ======== 1. install odf 4.12.5-1 with openshift installer (4.12.0-0.nightly-2023-07-06-110941) 2. upgrade OCP 4.12 to 4.13 which is successful 3. upgrade OCP 4.13 to 4.14 which is also successful ( expected to fail ) > csv $ oc get csv NAME DISPLAY VERSION REPLACES PHASE mcg-operator.v4.12.5-rhodf NooBaa Operator 4.12.5-rhodf mcg-operator.v4.12.4-rhodf Succeeded ocs-operator.v4.12.5-rhodf OpenShift Container Storage 4.12.5-rhodf ocs-operator.v4.12.4-rhodf Succeeded odf-csi-addons-operator.v4.12.5-rhodf CSI Addons 4.12.5-rhodf odf-csi-addons-operator.v4.12.4-rhodf Succeeded odf-operator.v4.12.5-rhodf OpenShift Data Foundation 4.12.5-rhodf odf-operator.v4.12.4-rhodf Succeeded > OCP upgrade history $ oc get clusterversion -o json|jq ".items[0].status.history" [ { "acceptedRisks": "Target release version=\"\" image=\"registry.ci.openshift.org/ocp/release:4.14.0-0.nightly-2023-07-11-092038\" cannot be verified, but continuing anyway because the update was forced: release images that are not accessed via digest cannot be verified\nPrecondition \"ClusterVersionRecommendedUpdate\" failed because of \"UnknownUpdate\": RetrievedUpdates=False (VersionNotFound), so the recommended status of updating from 4.13.0-0.nightly-2023-07-12-170413 to 4.14.0-0.nightly-2023-07-11-092038 is unknown.", "completionTime": "2023-07-14T07:04:44Z", "image": "registry.ci.openshift.org/ocp/release:4.14.0-0.nightly-2023-07-11-092038", "startedTime": "2023-07-14T05:55:13Z", "state": "Completed", "verified": false, "version": "4.14.0-0.nightly-2023-07-11-092038" }, { "acceptedRisks": "Target release version=\"\" image=\"registry.ci.openshift.org/ocp/release:4.13.0-0.nightly-2023-07-12-170413\" cannot be verified, but continuing anyway because the update was forced: release images that are not accessed via digest cannot be verified\nForced through blocking failures: Multiple precondition checks failed:\n* Precondition \"ClusterVersionUpgradeable\" failed because of \"AdminAckRequired\": Kubernetes 1.26 and therefore OpenShift 4.13 remove several APIs which require admin consideration. Please see the knowledge article https://access.redhat.com/articles/6958394 for details and instructions.\n* Precondition \"EtcdRecentBackup\" failed because of \"ControllerStarted\": RecentBackup: The etcd backup controller is starting, and will decide if recent backups are available or if a backup is required\n* Precondition \"ClusterVersionRecommendedUpdate\" failed because of \"UnknownUpdate\": RetrievedUpdates=False (VersionNotFound), so the recommended status of updating from 4.12.0-0.nightly-2023-07-06-110941 to 4.13.0-0.nightly-2023-07-12-170413 is unknown.", "completionTime": "2023-07-13T12:09:31Z", "image": "registry.ci.openshift.org/ocp/release:4.13.0-0.nightly-2023-07-12-170413", "startedTime": "2023-07-13T10:51:00Z", "state": "Completed", "verified": false, "version": "4.13.0-0.nightly-2023-07-12-170413" }, { "completionTime": "2023-07-13T10:26:09Z", "image": "registry.ci.openshift.org/ocp/release@sha256:7396e806e21fa058413bb7baa9df00037dbeaab2d1d5de36e435a1c1b7a0fbb8", "startedTime": "2023-07-13T10:07:50Z", "state": "Completed", "verified": false, "version": "4.12.0-0.nightly-2023-07-06-110941" } ] > job1 ( installation of odf 4.12.5-1 + OCP 4.12 and upgrade of OCP from 4.12 to 4.13 ) : https://url.corp.redhat.com/2495551 > job2 ( upgrade from OCP 4.13 to OCP 4.14 ): https://url.corp.redhat.com/3ceb4df > must gather: https://url.corp.redhat.com/1a3310e
Update: ============ 1. Installed OCP 4.12 ( 4.12.23 ) + ODF 4.12 ( ocs-registry:4.12.5-2 ) $ oc get csv NAME DISPLAY VERSION REPLACES PHASE mcg-operator.v4.12.5-rhodf NooBaa Operator 4.12.5-rhodf mcg-operator.v4.12.4-rhodf Succeeded ocs-operator.v4.12.5-rhodf OpenShift Container Storage 4.12.5-rhodf ocs-operator.v4.12.4-rhodf Succeeded odf-csi-addons-operator.v4.12.5-rhodf CSI Addons 4.12.5-rhodf odf-csi-addons-operator.v4.12.4-rhodf Succeeded odf-operator.v4.12.5-rhodf OpenShift Data Foundation 4.12.5-rhodf odf-operator.v4.12.4-rhodf Succeeded $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.12.23 True False 13h Cluster version is 4.12.23 2. changed channel to "candidate-4.13" and upgrade to latest $ oc patch clusterversions/version -p '{"spec":{"channel":"candidate-4.13"}}' --type=merge clusterversion.config.openshift.io/version patched $ oc adm upgrade --to-image=quay.io/openshift-release-dev/ocp-release@sha256:af19e94813478382e36ae1fa2ae7bbbff1f903dded6180f4eb0624afe6fc6cd4 --allow-explicit-upgrade Requesting update to 4.13.5 oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.13.5 True False 9m20s Cluster version is 4.13.5 3. chnaged channel to candidate-4.14 and tried upgrading to latest image available and its not triggered the upgrade which is expected $ oc patch clusterversions/version -p '{"spec":{"channel":"candidate-4.14"}}' --type=merge clusterversion.config.openshift.io/version patched $ oc adm upgrade Cluster version is 4.13.5 Upgradeable=False Reason: IncompatibleOperatorsInstalled Message: Cluster operator operator-lifecycle-manager should not be upgraded between minor versions: ClusterServiceVersions blocking cluster upgrade: openshift-storage/odf-operator.v4.12.5-rhodf is incompatible with OpenShift minor versions greater than 4.13 Upstream is unset, so the cluster will use an appropriate default. Channel: candidate-4.14 (available channels: candidate-4.13, candidate-4.14) Recommended updates: VERSION IMAGE 4.14.0-ec.3 quay.io/openshift-release-dev/ocp-release@sha256:3c050cb52fdd3e65c518d4999d238ec026ef724503f275377fee6bf0d33093ab $ oc adm upgrade --to-image=quay.io/openshift-release-dev/ocp-release@sha256:3c050cb52fdd3e65c518d4999d238ec026ef724503f275377fee6bf0d33093ab --allow-explicit-upgrade Requesting update to 4.14.0-ec.3 $ oc adm upgrade Cluster version is 4.13.5 Upgradeable=False Reason: IncompatibleOperatorsInstalled Message: Cluster operator operator-lifecycle-manager should not be upgraded between minor versions: ClusterServiceVersions blocking cluster upgrade: openshift-storage/odf-operator.v4.12.5-rhodf is incompatible with OpenShift minor versions greater than 4.13 ReleaseAccepted=False Reason: PreconditionChecks Message: Preconditions failed for payload loaded version="4.14.0-ec.3" image="quay.io/openshift-release-dev/ocp-release@sha256:3c050cb52fdd3e65c518d4999d238ec026ef724503f275377fee6bf0d33093ab": Precondition "ClusterVersionUpgradeable" failed because of "IncompatibleOperatorsInstalled": Cluster operator operator-lifecycle-manager should not be upgraded between minor versions: ClusterServiceVersions blocking cluster upgrade: openshift-storage/odf-operator.v4.12.5-rhodf is incompatible with OpenShift minor versions greater than 4.13 Upstream is unset, so the cluster will use an appropriate default. Channel: candidate-4.14 (available channels: candidate-4.13, candidate-4.14) Recommended updates: VERSION IMAGE 4.14.0-ec.3 quay.io/openshift-release-dev/ocp-release@sha256:3c050cb52fdd3e65c518d4999d238ec026ef724503f275377fee6bf0d33093ab 4. tried upgrading using --to-latest flag and upgrade is not triggered as expected $ oc adm upgrade --to-latest Requesting update to 4.14.0-ec.3 vijay ~/rem $ oc adm upgrade Cluster version is 4.13.5 Upgradeable=False Reason: IncompatibleOperatorsInstalled Message: Cluster operator operator-lifecycle-manager should not be upgraded between minor versions: ClusterServiceVersions blocking cluster upgrade: openshift-storage/odf-operator.v4.12.5-rhodf is incompatible with OpenShift minor versions greater than 4.13 ReleaseAccepted=False Reason: PreconditionChecks Message: Preconditions failed for payload loaded version="4.14.0-ec.3" image="quay.io/openshift-release-dev/ocp-release@sha256:3c050cb52fdd3e65c518d4999d238ec026ef724503f275377fee6bf0d33093ab": Precondition "ClusterVersionUpgradeable" failed because of "IncompatibleOperatorsInstalled": Cluster operator operator-lifecycle-manager should not be upgraded between minor versions: ClusterServiceVersions blocking cluster upgrade: openshift-storage/odf-operator.v4.12.5-rhodf is incompatible with OpenShift minor versions greater than 4.13 Upstream is unset, so the cluster will use an appropriate default. Channel: candidate-4.14 (available channels: candidate-4.13, candidate-4.14) Recommended updates: VERSION IMAGE 4.14.0-ec.3 quay.io/openshift-release-dev/ocp-release@sha256:3c050cb52fdd3e65c518d4999d238ec026ef724503f275377fee6bf0d33093ab $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.13.5 True False 30m Cluster version is 4.13.5 > checked olm.properties and maxOpenShiftVersion version is set properly oc get csv odf-operator.v4.12.5-rhodf -o json | jq '.metadata.annotations."olm.properties"' "[{\"type\": \"olm.maxOpenShiftVersion\", \"value\": \"4.13\"}]"
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat OpenShift Data Foundation 4.12.5 security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:4287