> cephobjectstore is in connected state

$ oc describe cephobjectstore ocs-external-storagecluster-cephobjectstore
Name:         ocs-external-storagecluster-cephobjectstore
Namespace:    openshift-storage
Labels:       <none>
Annotations:  <none>
API Version:  ceph.rook.io/v1
Kind:         CephObjectStore

Status:
  Bucket Status:
    Health:        Connected
    Last Changed:  2022-03-14T20:07:49Z
    Last Checked:  2022-03-15T04:50:51Z
  Info:
    Endpoint:  http://rook-ceph-rgw-ocs-external-storagecluster-cephobjectstore.openshift-storage.svc:8080
  Phase:       Connected
Events:        <none>


> pods are in running state

$ oc get pods
NAME                                               READY   STATUS    RESTARTS      AGE
csi-addons-controller-manager-57966b99d-ltm8n      2/2     Running   0             13h
csi-cephfsplugin-dxh4d                             3/3     Running   0             14h
csi-cephfsplugin-m4ns5                             3/3     Running   0             14h
csi-cephfsplugin-provisioner-688699dc8-tbrnf       6/6     Running   0             14h
csi-cephfsplugin-provisioner-688699dc8-wh8vz       6/6     Running   0             14h
csi-cephfsplugin-t9fzj                             3/3     Running   0             14h
csi-rbdplugin-8l6wl                                3/3     Running   0             14h
csi-rbdplugin-kchwc                                3/3     Running   0             14h
csi-rbdplugin-provisioner-58cb769786-5rg56         6/6     Running   0             14h
csi-rbdplugin-provisioner-58cb769786-wdjbb         6/6     Running   0             14h
csi-rbdplugin-vjpkj                                3/3     Running   0             14h
noobaa-core-0                                      1/1     Running   0             14h
noobaa-db-pg-0                                     1/1     Running   4 (13h ago)   13h
noobaa-endpoint-6d46d6875b-6cfv8                   1/1     Running   0             14h
noobaa-operator-644bf9bb74-hqckj                   1/1     Running   0             13h
ocs-metrics-exporter-7d87cdfcd-rlpsw               1/1     Running   0             14h
ocs-operator-5d7b75df8f-tdf6t                      1/1     Running   0             14h
odf-console-74f4bc8b99-2f8ws                       1/1     Running   0             13h
odf-operator-controller-manager-5f5db6bf97-x4snd   2/2     Running   0             13h
rook-ceph-operator-5758fc4894-jtwrs                1/1     Running   0             14h
rook-ceph-tools-external-65cfc7bbd5-jmzs7          1/1     Running   0             14h

> odf-operator-controller-manager-5f5db6bf97-x4snd log

2022-03-15T04:59:26.184Z	INFO	controllers.StorageSystem	vendor CSV is installed and ready	{"instance": "openshift-storage/ocs-external-storagecluster-storagesystem", "ClusterServiceVersion": "odf-csi-addons-operator.v4.10.0"}
2022-03-15T04:59:26.189Z	ERROR	controller-runtime.manager.controller.storagesystem	Reconciler error	{"reconciler group": "odf.openshift.io", "reconciler kind": "StorageSystem", "name": "ocs-external-storagecluster-storagesystem", "namespace": "openshift-storage", "error": "CSV is not successfully installed"}

$ oc describe storagesystem 
Name:         ocs-external-storagecluster-storagesystem
Namespace:    openshift-storage
Labels:       <none>
Annotations:  <none>
API Version:  odf.openshift.io/v1alpha1
Kind:         StorageSystem

Status:
  Conditions:
    Last Heartbeat Time:   2022-03-15T04:59:26Z
    Last Transition Time:  2022-03-14T15:00:53Z
    Message:               Reconcile is in progress
    Reason:                Reconciling
    Status:                False
    Type:                  Available
    Last Heartbeat Time:   2022-03-15T04:59:26Z
    Last Transition Time:  2022-03-14T15:00:53Z
    Message:               Reconcile is in progress
    Reason:                Reconciling
    Status:                True
    Type:                  Progressing
    Last Heartbeat Time:   2022-03-15T04:59:26Z
    Last Transition Time:  2022-03-14T14:44:07Z
    Message:               StorageSystem CR is valid
    Reason:                Valid
    Status:                False
    Type:                  StorageSystemInvalid
    Last Heartbeat Time:   2022-03-15T04:59:26Z
    Last Transition Time:  2022-03-14T15:00:53Z
    Message:               CSV is not successfully installed
    Reason:                NotReady
    Status:                False
    Type:                  VendorCsvReady
    Last Heartbeat Time:   2022-03-14T14:44:07Z
    Last Transition Time:  2022-03-14T14:44:07Z
    Reason:                Found
    Status:                True
    Type:                  VendorSystemPresent
Events:
  Type     Reason           Age                 From                      Message
  ----     ------           ----                ----                      -------
  Warning  ReconcileFailed  39m (x13 over 14h)  StorageSystem controller  CSV is not successfully installed

> cluster still exists in case if it needed for live debugging

This is just happening with external cluster only?
If yes, have we tested upgrade previously for external cluster?

(In reply to Mudit Agarwal from comment #4)
> This is just happening with external cluster only?
> If yes, have we tested upgrade previously for external cluster?

It happened only in external cluster. and in previous version upgrade is passed ( v4.8.4 to ocs-registry:4.9.0-249.ci )

old run log file: http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/j-037vu1ce33-ua/j-037vu1ce33-ua_20211125T130856/logs/ocs-ci-logs-1637848904/tests/ecosystem/upgrade/test_upgrade.py/test_upgrade/logs

old job: https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-trigger-vsphere-upi-1az-rhcos-external-3m-3w-upgrade-ocs-auto/37/console

Thanks but I want to know if this was ever run for 4.9 to 4.10 for external mode or not?

(In reply to Mudit Agarwal from comment #6)
> Thanks but I want to know if this was ever run for 4.9 to 4.10 for external
> mode or not?

nope, this was the 1st time we are running from 4.9 to 4.10 in external mode

Looks like https://github.com/red-hat-storage/ocs-operator/pull/1291 change which was merged in ocs-operator v4.9.z is causing this issue and only for external mode clusters.
When external mode clusters were upgraded from 4.8 to 4.9 (with this change) it was marked as "Not Upgradable". So all future upgrades will be blocked including upgrade to newer 4.9.z.
We'd have to somehow fix this in 4.9 and then upgrade to 4.10.

Is it possible to get an external mode cluster upgraded from 4.8 to 4.9(latest) to verify this?

This issue wasn't reproduced in cluster upgraded from 4.8 to 4.9.
So, it can be further upgraded to 4.10.

But, there is one similarity in CephCluster status. On both clusters I see this status
```
  - lastHeartbeatTime: "2022-03-14T14:44:10Z"
    lastTransitionTime: "2022-03-14T14:44:10Z"
    message: Attempting to connect to an external Ceph cluster
    reason: ClusterConnecting
    status: "True"
    type: Connecting
  - lastHeartbeatTime: "2022-03-14T15:15:20Z"
    lastTransitionTime: "2022-03-14T14:44:33Z"
    message: Cluster connected successfully
    reason: ClusterConnected
    status: "True"
    type: Connected
```
Both "connected" and "connecting" status can't be true at the same time.
This along with change pointed in Comment #8 might be causing this issue.

Still trying to pin point when and why exactly this happens.

Since this issue was reproduced multiple times (for 4.9 to 4.10 upgrades), adding devel_ack+ .
Posted a fix here: https://github.com/red-hat-storage/ocs-operator/pull/1597.
Once we test this, it needs to be backported to 4.10 and 4.9.

Despite this PR being merged into release-4.10, I don't believe it is an adequate solution. As per the original design, an external StorageCluster should be reporting a Phase of "Connected" instead of "Ready" if reconciliation completed without any errors. @vavuthu when you next test this please check if the StorageCluster is showing "Ready" after upgrade.

That said, this is probably a regression introduced in ODF 4.9 that we've just never caught until now. As such, I do think it would be fair to open another BZ against it, but I will leave it up to QE whether they want to consider it a blocker or not.

upgrade from 4.9.5-4 to ocs-registry:4.10.0-210 is successful
 	
 	https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster-prod/3629/console

(In reply to Jose A. Rivera from comment #18)
> Despite this PR being merged into release-4.10, I don't believe it is an
> adequate solution. As per the original design, an external StorageCluster
> should be reporting a Phase of "Connected" instead of "Ready" if
> reconciliation completed without any errors. @vavuthu when you
> next test this please check if the StorageCluster is showing "Ready" after
> upgrade.
> 
> That said, this is probably a regression introduced in ODF 4.9 that we've
> just never caught until now. As such, I do think it would be fair to open
> another BZ against it, but I will leave it up to QE whether they want to
> consider it a blocker or not.

(In reply to Jose A. Rivera from comment #18)
> Despite this PR being merged into release-4.10, I don't believe it is an
> adequate solution. As per the original design, an external StorageCluster
> should be reporting a Phase of "Connected" instead of "Ready" if
> reconciliation completed without any errors. @vavuthu when you
> next test this please check if the StorageCluster is showing "Ready" after
> upgrade.
> 
> That said, this is probably a regression introduced in ODF 4.9 that we've
> just never caught until now. As such, I do think it would be fair to open
> another BZ against it, but I will leave it up to QE whether they want to
> consider it a blocker or not.

is it storagecluster or cephcluster? I can see StorageCluster status as Ready and CephCluster as Connected 

http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/vavuthuext-upgrade/vavuthuext-upgrade_20220325T061303/logs/testcases_1648191511/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-2657d3206a2ff2a742043b6432387fe63fba26c056bf8afd0296ffc42cf29fe1/namespaces/openshift-storage/oc_output/storagecluster

Name:         ocs-external-storagecluster
Namespace:    openshift-storage
Labels:       <none>
Annotations:  storagesystem.odf.openshift.io/watched-by: ocs-external-storagecluster-storagesystem
              uninstall.ocs.openshift.io/cleanup-policy: delete
              uninstall.ocs.openshift.io/mode: graceful
API Version:  ocs.openshift.io/v1
Kind:         StorageCluster

  Images:
    Ceph:
      Desired Image:  quay.io/rhceph-dev/rhceph@sha256:82acc1ae5b6ee7f4c9100dbc803054b8edd7b77c64461966ecba621b3380f14b
    Noobaa Core:
      Actual Image:   quay.io/rhceph-dev/odf4-mcg-core-rhel8@sha256:805be8933e81fa2b037a843f9bf521f46ce25b828b28854c1d377dedc23f7f08
      Desired Image:  quay.io/rhceph-dev/odf4-mcg-core-rhel8@sha256:805be8933e81fa2b037a843f9bf521f46ce25b828b28854c1d377dedc23f7f08
    Noobaa DB:
      Actual Image:   quay.io/rhceph-dev/rhel8-postgresql-12@sha256:be7212e938d1ef314a75aca070c28b6433cd0346704d0d3523c8ef403ff0c69e
      Desired Image:  quay.io/rhceph-dev/rhel8-postgresql-12@sha256:be7212e938d1ef314a75aca070c28b6433cd0346704d0d3523c8ef403ff0c69e
  Phase:              Ready



http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/vavuthuext-upgrade/vavuthuext-upgrade_20220325T061303/logs/testcases_1648191511/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-2657d3206a2ff2a742043b6432387fe63fba26c056bf8afd0296ffc42cf29fe1/ceph/namespaces/openshift-storage/ceph.rook.io/cephclusters/

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  creationTimestamp: "2022-03-25T06:47:08Z"
  finalizers:
  - cephcluster.ceph.rook.io
  generation: 1
  labels:
    app: ocs-external-storagecluster
  name: ocs-external-storagecluster-cephcluster
  namespace: openshift-storage


  - lastHeartbeatTime: "2022-03-25T07:12:30Z"
    lastTransitionTime: "2022-03-25T06:47:17Z"
    message: Cluster connected successfully
    reason: ClusterConnected
    status: "True"
    type: Connected
  message: Cluster connected successfully
  phase: Connected
  state: Connected

Since fix is there in 4.9.5-4, we need to upgrade from 4.9.5-4 to ocs-registry:4.10.0-210

1. upgrade from 4.9.5-4 to ocs-registry:4.10.0-210

Result: PASS
 	
 	https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster-prod/3629/console

Marking as verified