Description of problem: During upgrade to v2.0.5 ocs-osd-deployer.v2.0.4 csv got stuck in phase Replacing and ocs-osd-deployer.v2.0.5 in phase Pending. Version-Release number of selected component (if applicable): oc get csv -n openshift-storage NAME DISPLAY VERSION REPLACES PHASE mcg-operator.v4.10.5 NooBaa Operator 4.10.5 mcg-operator.v4.10.4 Succeeded ocs-operator.v4.10.5 OpenShift Container Storage 4.10.5 ocs-operator.v4.10.4 Succeeded ocs-osd-deployer.v2.0.4 OCS OSD Deployer 2.0.4 ocs-osd-deployer.v2.0.3 Replacing ocs-osd-deployer.v2.0.5 OCS OSD Deployer 2.0.5 ocs-osd-deployer.v2.0.4 Pending odf-csi-addons-operator.v4.10.5 CSI Addons 4.10.5 odf-csi-addons-operator.v4.10.4 Succeeded odf-operator.v4.10.5 OpenShift Data Foundation 4.10.5 odf-operator.v4.10.4 Succeeded ose-prometheus-operator.4.10.0 Prometheus Operator 4.10.0 ose-prometheus-operator.4.8.0 Succeeded route-monitor-operator.v0.1.422-151be96 Route Monitor Operator 0.1.422-151be96 route-monitor-operator.v0.1.420-b65f47e Succeeded How reproducible: 5/5 clusters Steps to Reproduce: 1. install odf addon v2.0.4 2. upgrade it to v2.0.5 3. Actual results: v2.0.5 ocs-osd-deployer.v2.0.4 csv got stuck in phase Replacing and ocs-osd-deployer.v2.0.5 in phase Pending Expected results: deployer.v2.0.5 is in Succeeded phase after 5-10 minutes Additional info: n odf-operator-controller-manager pod I see the following errors: ERROR controllers.StorageSystem failed to validate CSV {"instance": "openshift-storage/ocs-storagecluster-storagesystem", "ClusterServiceVersion": "ocs-operator.v4.10.5", "error": "Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"ocs-operator.v4.10.5\": the object has been modified; please apply your changes to the latest version and try again"} github.com/red-hat-data-services/odf-operator/controllers.(*StorageSystemReconciler).reconcile /remote-source/app/controllers/storagesystem_controller.go:163 github.com/red-hat-data-services/odf-operator/controllers.(*StorageSystemReconciler).Reconcile /remote-source/app/controllers/storagesystem_controller.go:87 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214 2022-08-26T13:39:55.740Z INFO controllers.StorageSystem vendor CSV is installed and ready {"instance": "openshift-storage/ocs-storagecluster-storagesystem", "ClusterServiceVersion": "odf-csi-addons-operator.v4.10.5"} 2022-08-26T13:39:55.761Z ERROR controller-runtime.manager.controller.subscription Reconciler error {"reconciler group": "operators.coreos.com", "reconciler kind": "Subscription", "name": "odf-operator-stable-4.10-redhat-operators-openshift-marketplace", "namespace": "openshift-storage", "error": "Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"odf-csi-addons-operator.v4.10.5\": the object has been modified; please apply your changes to the latest version and try again"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214 2022-08-26T13:39:55.841Z ERROR controller-runtime.manager.controller.storagesystem Reconciler error {"reconciler group": "odf.openshift.io", "reconciler kind": "StorageSystem", "name": "ocs-storagecluster-storagesystem", "namespace": "openshift-storage", "error": "Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"ocs-operator.v4.10.5\": the object has been modified; please apply your changes to the latest version and try again"}
$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.10.26 True False 11h Cluster version is 4.10.26 $ oc get storagecluster NAME AGE PHASE EXTERNAL CREATED AT VERSION ocs-storagecluster 11h Ready 2022-08-26T04:32:57Z $ oc get managedocs managedocs -n openshift-storage -oyaml apiVersion: ocs.openshift.io/v1alpha1 kind: ManagedOCS metadata: creationTimestamp: "2022-08-26T04:32:28Z" finalizers: - managedocs.ocs.openshift.io generation: 1 name: managedocs namespace: openshift-storage resourceVersion: "1160588" uid: ef45a009-f3a1-4b73-9e9b-db32043b49ba spec: {} status: components: alertmanager: state: Ready prometheus: state: Ready storageCluster: state: Ready reconcileStrategy: strict $ rosa list addon -c sgatfane-26-pr | grep ocs-provider-qe ocs-provider-qe Red Hat OpenShift Data Foundation Managed Service Provider (QE) ready
The next attempt to upgrade provider addon on September 1st was successful. I'll update the bz if we hit the issue again.
Upgrade problem got resolved and the rollout is released. --> VERIFIED Tested with: ocs-osd-deployer.v2.0.5