Description of problem (please be detailed as possible and provide log snippests): 1] Issue during ODF upgrade. 2] Noobaa core and db are reporting the incorrect images causing the storagecluster to be in a progressing state 3] Steps to rebuild noobaa from article : https://access.redhat.com/solutions/5948631 was run with no success 4] The actual and desired image of noobaaCore and noobaaDB are different images: ceph: actualImage: registry.redhat.io/rhceph/rhceph-5-rhel8@sha256:2296c19fbd3a0be84d6030dff789ce3e79b38cc30c39f45913aec97967b65cce desiredImage: registry.redhat.io/rhceph/rhceph-5-rhel8@sha256:2296c19fbd3a0be84d6030dff789ce3e79b38cc30c39f45913aec97967b65cce noobaaCore: actualImage: registry.redhat.io/odf4/mcg-core-rhel8@sha256:fbb89796c5adfee97bf64d29f72ee40857d91f8c065c5b9b96bff40dbb4931aa desiredImage: registry.redhat.io/odf4/mcg-core-rhel8@sha256:04209248c71e5176e631644b47d37a8e28724961ae948af5ea239db66c356ebb noobaaDB: actualImage: registry.redhat.io/rhel8/postgresql-12@sha256:81d9bf20387ecfa85bf24dd53167242393075544a4368636a4bbda79d8769f49 desiredImage: registry.redhat.io/rhel8/postgresql-12@sha256:78ed1e1f454c49664ae653b3d52af3d77ef1e9cad37a7b0fff09feeaa8294e01 Logs: [sarreddy@scdvlmcd13 ~]$ oc get deployments -n openshift-storage NAME READY UP-TO-DATE AVAILABLE AGE csi-cephfsplugin-provisioner 2/2 2 2 2y15d csi-rbdplugin-provisioner 2/2 2 2 2y15d noobaa-operator 1/1 1 1 230d ocs-metrics-exporter 1/1 1 1 2y16d ocs-operator 1/1 1 1 2y16d odf-console 1/1 1 1 230d odf-operator-controller-manager 1/1 1 1 230d rook-ceph-crashcollector-infra-0.jawwy-ocs-sit-dev.stc.com.sa 1/1 1 1 33h rook-ceph-crashcollector-infra-1.jawwy-ocs-sit-dev.stc.com.sa 1/1 1 1 33h rook-ceph-crashcollector-infra-2.jawwy-ocs-sit-dev.stc.com.sa 1/1 1 1 34h rook-ceph-mds-ocs-storagecluster-cephfilesystem-a 1/1 1 1 2y15d rook-ceph-mds-ocs-storagecluster-cephfilesystem-b 1/1 1 1 2y15d rook-ceph-mgr-a 1/1 1 1 2y15d rook-ceph-mon-e 1/1 1 1 697d rook-ceph-mon-f 1/1 1 1 324d rook-ceph-mon-g 1/1 1 1 232d rook-ceph-operator 1/1 1 1 2y16d rook-ceph-osd-0 1/1 1 1 2y15d rook-ceph-osd-1 1/1 1 1 2y15d rook-ceph-osd-2 1/1 1 1 2y15d rook-ceph-osd-3 1/1 1 1 2y15d rook-ceph-osd-4 1/1 1 1 2y15d rook-ceph-osd-5 1/1 1 1 2y15d rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a 1/1 1 1 2y15d rook-ceph-tools 1/1 1 1 2y15d [sarreddy@scdvlmcd13 ~]$ [sarreddy@scdvlmcd13 ~]$ oc get storagecluster NAME AGE PHASE EXTERNAL CREATED AT VERSION ocs-storagecluster 2y15d Progressing 2021-03-02T16:54:04Z 4.6.0 [sarreddy@scdvlmcd13 ~]$ [sarreddy@scdvlmcd13 ~]$ [sarreddy@scdvlmcd13 ~]$ [sarreddy@scdvlmcd13 ~]$ oc get pods NAME READY STATUS RESTARTS AGE csi-cephfsplugin-89jf9 3/3 Running 0 3h53m csi-cephfsplugin-9dc8x 3/3 Running 0 3h55m csi-cephfsplugin-dhm8q 3/3 Running 0 3h52m csi-cephfsplugin-ffsbf 3/3 Running 0 3h56m csi-cephfsplugin-m2jbx 3/3 Running 0 3h55m csi-cephfsplugin-n2l7b 3/3 Running 0 3h55m csi-cephfsplugin-provisioner-d8c5c5bc4-l6bpc 6/6 Running 0 3h58m csi-cephfsplugin-provisioner-d8c5c5bc4-llrtk 6/6 Running 0 3h58m csi-cephfsplugin-wmpt9 3/3 Running 0 3h58m csi-cephfsplugin-xzsfc 3/3 Running 0 3h55m csi-rbdplugin-bqs8j 3/3 Running 0 3h53m csi-rbdplugin-hv99f 3/3 Running 0 3h54m csi-rbdplugin-lh886 3/3 Running 0 3h53m csi-rbdplugin-mmf56 3/3 Running 0 3h56m csi-rbdplugin-p5lrf 3/3 Running 0 3h55m csi-rbdplugin-provisioner-676c856bd4-c646z 6/6 Running 0 3h58m csi-rbdplugin-provisioner-676c856bd4-ftsff 6/6 Running 0 3h58m csi-rbdplugin-r2crc 3/3 Running 0 3h53m csi-rbdplugin-v5x5s 3/3 Running 0 3h58m csi-rbdplugin-vkbtk 3/3 Running 0 3h55m noobaa-endpoint-fb4b66487-f8kzj 1/1 Running 0 33h noobaa-operator-6977f79d8d-hm4qt 1/1 Running 0 160m ocs-metrics-exporter-7fbbc99c96-fxgkh 1/1 Running 0 4h ocs-operator-54f644cfbf-mrxxj 1/1 Running 0 161m odf-console-f54f898df-gj5l9 1/1 Running 0 27h odf-operator-controller-manager-5c565c5b58-4cdxl 2/2 Running 0 161m rook-ceph-crashcollector-infra-0.jawwy-ocs-sit-dev.stc.com9zzmj 1/1 Running 0 33h rook-ceph-crashcollector-infra-0.jawwy-ocs-sit-dev.stc.comjgrhv 1/1 Running 0 33h rook-ceph-crashcollector-infra-0.jawwy-ocs-sit-dev.stc.comm5g5c 1/1 Running 0 3h58m rook-ceph-crashcollector-infra-0.jawwy-ocs-sit-dev.stc.comx8xqq 1/1 Running 0 33h rook-ceph-crashcollector-infra-1.jawwy-ocs-sit-dev.stc.com2fm2q 1/1 Running 0 33h rook-ceph-crashcollector-infra-1.jawwy-ocs-sit-dev.stc.com42hfm 1/1 Running 0 33h rook-ceph-crashcollector-infra-1.jawwy-ocs-sit-dev.stc.comcvmdr 1/1 Running 0 3h58m rook-ceph-crashcollector-infra-1.jawwy-ocs-sit-dev.stc.comqzd88 1/1 Running 0 33h rook-ceph-crashcollector-infra-1.jawwy-ocs-sit-dev.stc.comw8t8m 1/1 Running 0 33h rook-ceph-crashcollector-infra-2.jawwy-ocs-sit-dev.stc.comcb7s9 1/1 Running 0 36h rook-ceph-crashcollector-infra-2.jawwy-ocs-sit-dev.stc.comdggrj 1/1 Running 0 36h rook-ceph-crashcollector-infra-2.jawwy-ocs-sit-dev.stc.comz5lhn 1/1 Running 0 3h58m rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-57f867c7nxjcq 2/2 Running 0 33h rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-5c996df48kg96 2/2 Running 0 33h rook-ceph-mgr-a-5946474f67-xjvxw 2/2 Running 0 33h rook-ceph-mon-e-6989f7c778-mtcrz 2/2 Running 0 33h rook-ceph-mon-f-65db4c474b-vmrmp 2/2 Running 0 33h rook-ceph-mon-g-7d44688797-4x7hm 2/2 Running 0 36h rook-ceph-operator-7df8bd997b-zhhsj 1/1 Running 0 4h rook-ceph-osd-0-66557bf8f4-s5c68 2/2 Running 0 36h rook-ceph-osd-1-64fb9599d7-c5gs5 2/2 Running 0 33h rook-ceph-osd-2-5f6cdb546f-jbbpz 2/2 Running 0 33h rook-ceph-osd-3-74567c8494-dldwr 2/2 Running 0 34h rook-ceph-osd-4-8cbf76749-529jx 2/2 Running 0 33h rook-ceph-osd-5-b55b6d4b-vssjc 2/2 Running 0 33h rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-566fff4sk5z9 2/2 Running 0 33h rook-ceph-tools-8579d5599c-hpbmr 1/1 Running 0 3h59m [sarreddy@scdvlmcd13 ~]$ sarreddy@scdvlmcd13 ~]$ oc logs noobaa-operator-6977f79d8d-wqfm5 time="2023-03-19T03:39:40Z" level=info msg="CLI version: 5.9.2\n" time="2023-03-19T03:39:40Z" level=info msg="noobaa-image: noobaa/noobaa-core:5.9.0\n" time="2023-03-19T03:39:40Z" level=info msg="operator-image: noobaa/noobaa-operator:5.9.2\n" I0319 03:39:41.268357 1 request.go:668] Waited for 1.032685445s due to client-side throttling, not priority and fairness, request: GET:https://10.29.128.1:443/apis/policy/v1?timeout=32s [sarreddy@scdvlmcd13 ~]$ [sarreddy@scdvlmcd13 ~]$ oc logs ocs-operator-54f644cfbf-tmtkf {"level":"info","ts":1679197183.7147424,"logger":"cmd","msg":"Go Version: go1.16.12"} {"level":"info","ts":1679197183.7156096,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"} I0319 03:39:44.766062 1 request.go:668] Waited for 1.037647659s due to client-side throttling, not priority and fairness, request: GET:https://10.29.128.1:443/apis/cert-manager.io/v1alpha2?timeout=32s {"level":"info","ts":1679197187.7708323,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"} {"level":"info","ts":1679197187.7849748,"logger":"cmd","msg":"OCSInitialization resource already exists"} {"level":"info","ts":1679197191.84642,"logger":"cmd","msg":"starting manager"} I0319 03:39:51.846930 1 leaderelection.go:243] attempting to acquire leader lease openshift-storage/ab76f4c9.openshift.io... {"level":"info","ts":1679197191.846939,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"} I0319 03:40:09.270292 1 leaderelection.go:253] successfully acquired lease openshift-storage/ab76f4c9.openshift.io {"level":"info","ts":1679197209.2705457,"logger":"controller-runtime.manager.controller.persistentvolume","msg":"Starting EventSource","reconciler group":"","reconciler kind":"PersistentVolume","source":"kind source: /, Kind="} {"level":"info","ts":1679197209.270645,"logger":"controller-runtime.manager.controller.persistentvolume","msg":"Starting Controller","reconciler group":"","reconciler kind":"PersistentVolume"} {"level":"info","ts":1679197209.2706227,"logger":"controller-runtime.manager.controller.ocsinitialization","msg":"Starting EventSource","reconciler group":"ocs.openshift.io","reconciler kind":"OCSInitialization","source":"kind source: /, Kind="} [sarreddy@scdvlmcd13 ~]$ oc get events -n openshift-storage LAST SEEN TYPE REASON OBJECT MESSAGE 128m Normal LeaderElection configmap/4fd470de.openshift.io odf-operator-controller-manager-5c565c5b58-z7ch7_8a94ef37-495c-4478-a027-994b6b934bb7 became leader 128m Normal LeaderElection lease/4fd470de.openshift.io odf-operator-controller-manager-5c565c5b58-z7ch7_8a94ef37-495c-4478-a027-994b6b934bb7 became leader 128m Normal LeaderElection configmap/ab76f4c9.openshift.io ocs-operator-54f644cfbf-tmtkf_205bfda0-7165-47de-9fa4-b5ffa962c192 became leader 128m Normal LeaderElection lease/ab76f4c9.openshift.io ocs-operator-54f644cfbf-tmtkf_205bfda0-7165-47de-9fa4-b5ffa962c192 became leader 128m Normal InstallSucceeded clusterserviceversion/mcg-operator.v4.9.11 install strategy completed with no errors 128m Warning ComponentUnhealthy clusterserviceversion/mcg-operator.v4.9.11 installing: waiting for deployment noobaa-operator to become ready: deployment "noobaa-operator" not available: Deployment does not have minimum availability. 2m43s Warning FailedGetScale horizontalpodautoscaler/noobaa-endpoint deployments/scale.apps "noobaa-endpoint" not found 131m Normal Killing pod/noobaa-operator-6977f79d8d-hpmqz Stopping container noobaa-operator 128m Normal Scheduled pod/noobaa-operator-6977f79d8d-wqfm5 Successfully assigned openshift-storage/noobaa-operator-6977f79d8d-wqfm5 to worker-4.jawwy-ocs-sit-dev.stc.com.sa 128m Normal AddedInterface pod/noobaa-operator-6977f79d8d-wqfm5 Add eth0 [10.29.19.29/23] from openshift-sdn 128m Normal Pulled pod/noobaa-operator-6977f79d8d-wqfm5 Container image "registry.redhat.io/odf4/mcg-rhel8-operator@sha256:376cdd149b926d12575caa90f391f1f5d221eee545a79ffedafca2d4fa296d00" already present on machine 128m Normal Created pod/noobaa-operator-6977f79d8d-wqfm5 Created container noobaa-operator 128m Normal Started pod/noobaa-operator-6977f79d8d-wqfm5 Started container noobaa-operator 131m Normal SuccessfulDelete replicaset/noobaa-operator-6977f79d8d Deleted pod: noobaa-operator-6977f79d8d-hpmqz 128m Normal SuccessfulCreate replicaset/noobaa-operator-6977f79d8d Created pod: noobaa-operator-6977f79d8d-wqfm5 128m Normal ScalingReplicaSet deployment/noobaa-operator Scaled up replica set noobaa-operator-6977f79d8d to 1 131m Normal ScalingReplicaSet deployment/noobaa-operator Scaled down replica set noobaa-operator-6977f79d8d to 0 131m Normal Killing pod/ocs-operator-54f644cfbf-9qpqs Stopping container ocs-operator 128m Normal Scheduled pod/ocs-operator-54f644cfbf-tmtkf Successfully assigned openshift-storage/ocs-operator-54f644cfbf-tmtkf to worker-4.jawwy-ocs-sit-dev.stc.com.sa 128m Normal AddedInterface pod/ocs-operator-54f644cfbf-tmtkf Add eth0 [10.29.19.28/23] from openshift-sdn 128m Normal Pulling pod/ocs-operator-54f644cfbf-tmtkf Pulling image "registry.redhat.io/odf4/ocs-rhel8-operator@sha256:0b3bb17855057eab6612881b4dccabf472d64cfb5ec8b3db7a799c5ce60e9f29" 128m Normal Pulled pod/ocs-operator-54f644cfbf-tmtkf Successfully pulled image "registry.redhat.io/odf4/ocs-rhel8-operator@sha256:0b3bb17855057eab6612881b4dccabf472d64cfb5ec8b3db7a799c5ce60e9f29" in 4.207312021s 128m Normal Created pod/ocs-operator-54f644cfbf-tmtkf Created container ocs-operator 128m Normal Started pod/ocs-operator-54f644cfbf-tmtkf Started container ocs-operator 131m Normal SuccessfulDelete replicaset/ocs-operator-54f644cfbf Deleted pod: ocs-operator-54f644cfbf-9qpqs 128m Normal SuccessfulCreate replicaset/ocs-operator-54f644cfbf Created pod: ocs-operator-54f644cfbf-tmtkf 128m Normal ScalingReplicaSet deployment/ocs-operator Scaled up replica set ocs-operator-54f644cfbf to 1 131m Normal ScalingReplicaSet deployment/ocs-operator Scaled down replica set ocs-operator-54f644cfbf to 0 128m Warning ComponentUnhealthy clusterserviceversion/ocs-operator.v4.9.11 installing: waiting for deployment ocs-operator to become ready: deployment "ocs-operator" not available: Deployment does not have minimum availability. 128m Normal NeedsReinstall clusterserviceversion/ocs-operator.v4.9.11 installing: waiting for deployment ocs-operator to become ready: deployment "ocs-operator" not available: Deployment does not have minimum availability. 7m48s Warning ReconcileFailed cephcluster/ocs-storagecluster-cephcluster failed to reconcile cluster "ocs-storagecluster-cephcluster": failed to configure local ceph cluster: failed the ceph version check: failed to complete ceph version job: failed to run CmdReporter rook-ceph-detect-version successfully. failed to delete existing results ConfigMap rook-ceph-detect-version. failed to delete ConfigMap rook-ceph-detect-version. gave up waiting after 20 retries every 2ns seconds. <nil> Version of all relevant components (if applicable): NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.10.53 True False 44h Cluster version is 4.10.53 NAME DISPLAY VERSION REPLACES PHASE elasticsearch-operator.5.5.8 OpenShift Elasticsearch Operator 5.5.8 elasticsearch-operator.5.2.13 Succeeded mcg-operator.v4.9.11 NooBaa Operator 4.9.11 mcg-operator.v4.9.10 Succeeded nginx-ingress-operator.v1.1.0 Nginx Ingress Operator 1.1.0 Installing ocs-operator.v4.9.11 OpenShift Container Storage 4.9.11 ocs-operator.v4.9.10 Succeeded odf-operator.v4.9.11 OpenShift Data Foundation 4.9.11 odf-operator.v4.9.10 Succeeded Does this issue impact your ability to continue to work with the product Yes, customer is unable to use their cluster and start application Is there any workaround available to the best of your knowledge? Workaround to rebuild noobaa was run with no success Rate from 1 - 5 the complexity of the scenario you performed that caused this 4 Can this issue reproducible? N/A Can this issue reproduce from the UI? No If this is a regression, please provide more details to justify this: N/A Steps to Reproduce: N/A Actual results: Noobaa deployment stuck during ODF upgrade Expected results: Noobaa deployment should be successful along with ODF upgrade Additional info:
Closing it as per comment 6