Description of problem (please be detailed as possible and provide log
snippests):

After upgrade from 4.8 to 4.9.1, storagesystem is not ready


Version of all relevant components (if applicable):

upgrade from : 4.8.6
upgrade to image: quay.io/rhceph-dev/ocs-registry:4.9.1-252.ci


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes

Is there any workaround available to the best of your knowledge?
NA

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
Not tried

If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. upgrade ODF from 4.8 to 4.9
2. check storagesystem status
3.


Actual results:

storage system is not in expected state


Expected results:

storage system should be in ready state


Additional info:

storage system state:

$ oc get storagesystem -o yaml
apiVersion: v1
items:
- apiVersion: odf.openshift.io/v1alpha1
  kind: StorageSystem
  metadata:
    creationTimestamp: "2021-12-18T00:58:29Z"
    finalizers:
    - storagesystem.odf.openshift.io
    generation: 1
    managedFields:
    - apiVersion: odf.openshift.io/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:finalizers:
            .: {}
            v:"storagesystem.odf.openshift.io": {}
        f:spec:
          .: {}
          f:kind: {}
          f:name: {}
          f:namespace: {}
      manager: manager
      operation: Update
      time: "2021-12-18T00:58:29Z"
    - apiVersion: odf.openshift.io/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          .: {}
          f:conditions: {}
      manager: manager
      operation: Update
      subresource: status
      time: "2021-12-18T00:58:29Z"
    name: ocs-storagecluster-storagesystem
    namespace: openshift-storage
    resourceVersion: "2287583"
    uid: 2902a8fe-0e6f-4a57-9171-92a012becb23
  spec:
    kind: storagecluster.ocs.openshift.io/v1
    name: ocs-storagecluster
    namespace: openshift-storage
  status:
    conditions:
    - lastHeartbeatTime: "2021-12-20T06:07:03Z"
      lastTransitionTime: "2021-12-18T00:58:29Z"
      message: Reconcile is in progress
      reason: Reconciling
      status: "False"
      type: Available
    - lastHeartbeatTime: "2021-12-20T06:07:03Z"
      lastTransitionTime: "2021-12-18T00:58:29Z"
      message: Reconcile is in progress
      reason: Reconciling
      status: "True"
      type: Progressing
    - lastHeartbeatTime: "2021-12-20T06:07:03Z"
      lastTransitionTime: "2021-12-18T00:58:29Z"
      message: StorageSystem CR is valid
      reason: Valid
      status: "False"
      type: StorageSystemInvalid
    - lastHeartbeatTime: "2021-12-18T00:58:39Z"
      lastTransitionTime: "2021-12-18T00:58:29Z"
      message: ClusterServiceVersion.operators.coreos.com "mcg-operator.v4.9.1" not found; ClusterServiceVersion.operators.coreos.com "ocs-operator.v4.9.1" not found
      reason: NotReady
      status: "False"
      type: VendorCsvReady
    - lastHeartbeatTime: "2021-12-18T00:58:29Z"
      lastTransitionTime: "2021-12-18T00:58:29Z"
      message: Initializing StorageSystem
      reason: Init
      status: Unknown
      type: VendorSystemPresent
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
$ 

> All csv are in succeeded state
$ oc get csv
NAME                  DISPLAY                       VERSION   REPLACES              PHASE
mcg-operator.v4.9.1   NooBaa Operator               4.9.1     mcg-operator.v4.9.0   Succeeded
ocs-operator.v4.9.1   OpenShift Container Storage   4.9.1     ocs-operator.v4.8.6   Succeeded
odf-operator.v4.9.1   OpenShift Data Foundation     4.9.1     odf-operator.v4.9.0   Succeeded

> Job: https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster-prod/2647//consoleFull

Must gather: http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/j-124vu1cs33-ua/j-124vu1cs33-ua_20211217T233900/logs/failed_testcase_ocs_logs_1639786979/test_upgrade_ocs_logs/