Description of problem (please be detailed as possible and provide log
snippests):
Consumer storagecluster remains in "Progressing" state after upgrading the provider cluster to ODF 4.11.0. The CephClient and CephBlockPool status in the storageconsumer on provider is showing as failure. The actual status of CephClients and CephBlockPool is Ready.

From consumer:

$ oc get csv
NAME                                      DISPLAY                       VERSION           REPLACES                                  PHASE
mcg-operator.v4.10.5                      NooBaa Operator               4.10.5            mcg-operator.v4.10.4                      Succeeded
ocs-operator.v4.10.4                      OpenShift Container Storage   4.10.4            ocs-operator.v4.10.3                      Succeeded
ocs-osd-deployer.v2.0.4                   OCS OSD Deployer              2.0.4             ocs-osd-deployer.v2.0.3                   Installing
odf-csi-addons-operator.v4.10.4           CSI Addons                    4.10.4            odf-csi-addons-operator.v4.10.3           Succeeded
odf-operator.v4.10.4                      OpenShift Data Foundation     4.10.4            odf-operator.v4.10.3                      Succeeded
ose-prometheus-operator.4.10.0            Prometheus Operator           4.10.0            ose-prometheus-operator.4.8.0             Succeeded
route-monitor-operator.v0.1.422-151be96   Route Monitor Operator        0.1.422-151be96   route-monitor-operator.v0.1.420-b65f47e   Succeeded


$ oc get storagecluster
NAME                 AGE   PHASE         EXTERNAL   CREATED AT             VERSION
ocs-storagecluster   9h    Progressing   true       2022-08-02T08:18:35Z  


$ oc get managedocs managedocs -o yaml
apiVersion: ocs.openshift.io/v1alpha1
kind: ManagedOCS
metadata:
  creationTimestamp: "2022-08-02T08:17:50Z"
  finalizers:
  - managedocs.ocs.openshift.io
  generation: 1
  name: managedocs
  namespace: openshift-storage
  resourceVersion: "165640"
  uid: 182e0112-2496-4a18-91ae-261255fab2ec
spec: {}
status:
  components:
    alertmanager:
      state: Ready
    prometheus:
      state: Ready
    storageCluster:
      state: Pending
  reconcileStrategy: strict


--------------------------------------------------


From provider:

$ oc get csv
NAME                                      DISPLAY                       VERSION           REPLACES                                  PHASE
mcg-operator.v4.11.0                      NooBaa Operator               4.11.0            mcg-operator.v4.10.5                      Succeeded
ocs-operator.v4.11.0                      OpenShift Container Storage   4.11.0            ocs-operator.v4.10.5                      Succeeded
ocs-osd-deployer.v2.0.4                   OCS OSD Deployer              2.0.4             ocs-osd-deployer.v2.0.3                   Succeeded
odf-csi-addons-operator.v4.11.0           CSI Addons                    4.11.0            odf-csi-addons-operator.v4.10.5           Succeeded
odf-operator.v4.11.0                      OpenShift Data Foundation     4.11.0            odf-operator.v4.10.4                      Succeeded
ose-prometheus-operator.4.10.0            Prometheus Operator           4.10.0            ose-prometheus-operator.4.8.0             Succeeded
route-monitor-operator.v0.1.422-151be96   Route Monitor Operator        0.1.422-151be96   route-monitor-operator.v0.1.420-b65f47e   Succeeded



$ oc get storagecluster
NAME                 AGE   PHASE   EXTERNAL   CREATED AT             VERSION
ocs-storagecluster   10h   Ready              2022-08-02T06:53:20Z   



$ oc get cephclients
NAME                               PHASE
0037ad9b0c9d74e76a8a2f3d3f3f7549   Ready
16a1a588c6db0c40627b6ab1565ebe9b   Ready
478d082a002c4687176fcabe4907f724   Ready
8b3bedb61c8e50393fb56e0ffd89ae61   Ready
9d23b4bd2bbcb0df4df528384b7a1926   Ready
f6ae77a3f949da74e24b982903cdf49c   Ready


$ oc get cephblockpool
NAME                                                                 PHASE
cephblockpool-storageconsumer-c1073a0a-bfe6-493a-8d92-8aee7cb3e5e9   Ready


$ oc get storageconsumer storageconsumer-c1073a0a-bfe6-493a-8d92-8aee7cb3e5e9 -o yaml
apiVersion: ocs.openshift.io/v1alpha1
kind: StorageConsumer
metadata:
  annotations:
    ocs.openshift.io/provider-onboarding-ticket: |
      eyJpZCI6ImZiZTRhOTMyLTNhYjQtNDkwMi04NTQ5LTEzMmZmOTY1NDFhMiIsImV4cGlyYXRpb25EYXRlIjoiMTY1OTYwMDkzMyJ9.McRHPwee7OBnncHPPCuGHTeNKhU88w5KgmHL+XpVgyk32OVnIsuucN3QrxlEaQXZN8S+eZfmmi1Y/7f/9GJWmRFaKJJ3dub7Oe58/u0bUbUUmdFgU891LDlxsnoJI6H3ENm0KesmXsXhjR71YW5BQ1OcEq/JQCQOgnUVetorjZt7UD/pS5owwMdFWKSsRHGJh3/G2FCSbyi4u5SwNuKBp3mJXezc8VQc06oWaHwUMRZXr25jb6yNU4vlEN57ifSNF37PQfc7Kv3UgETz/ZShdb/vxS/YuzVG4YKchZ3mAAQak2+r01fUTmiiw09hblW9R12g3MqpLr3f/B0XVzF+B9L6GggBXk5KwLtFWoAYzE4ajZM4Yof+O6mmSudqfMiqNOMcuAzTzj5m9kLWLPUPwrucVVMjsxfIIaECMa42KXwTv2czjcF+miViF6FaqsMCcVBIFzFYG1DhmmB3l6uM8JkoL3eEZiLzNTpPKe/kvtudZQDTr84Pi5/lH2PXIGjn9HPdFLazRC/od4lGGrYDvigjE7mSTyy83cYpsOO6xTxRg29nwPzZTITa4peJrjpUuOG0YOq+QzX6BY6VtqoYdyMK1XKWFfbCq9NDbW+Gz79MPBs82+6vZ/bBqiB1r5KQKORnWsiU6b3f2NX8wOmU00mFFDLZgXNkl9rWtvXf1dA=
  creationTimestamp: "2022-08-02T08:18:35Z"
  finalizers:
  - storagesconsumer.ocs.openshift.io
  generation: 2
  name: storageconsumer-c1073a0a-bfe6-493a-8d92-8aee7cb3e5e9
  namespace: openshift-storage
  resourceVersion: "1076524"
  uid: c891b5b4-b117-4c89-880b-f2c0ea6813be
spec:
  capacity: 1Pi
  enable: true
status:
  cephResources:
  - kind: CephClient
    name: 16a1a588c6db0c40627b6ab1565ebe9b
    status: Ready
  - kind: CephClient
    name: f6ae77a3f949da74e24b982903cdf49c
    status: Ready
  - cephClients:
      node: f6ae77a3f949da74e24b982903cdf49c
      provisioner: 16a1a588c6db0c40627b6ab1565ebe9b
    kind: CephBlockPool
    name: cephblockpool-storageconsumer-c1073a0a-bfe6-493a-8d92-8aee7cb3e5e9
    status: Failure
  - kind: CephClient
    name: 9d23b4bd2bbcb0df4df528384b7a1926
    status: Failure
  - kind: CephClient
    name: 8b3bedb61c8e50393fb56e0ffd89ae61
    status: Ready
  - cephClients:
      node: 8b3bedb61c8e50393fb56e0ffd89ae61
      provisioner: 9d23b4bd2bbcb0df4df528384b7a1926
    kind: CephFilesystemSubVolumeGroup
    name: cephfilesystemsubvolumegroup-storageconsumer-c1073a0a-bfe6-493a-8d92-8aee7cb3e5e9
    status: Ready
  - kind: CephClient
    name: 478d082a002c4687176fcabe4907f724
    status: Ready
  - kind: CephClient
    name: 0037ad9b0c9d74e76a8a2f3d3f3f7549
    status: Ready
  grantedCapacity: 1Pi
  state: Configuring
  
  
$ oc get managedocs managedocs -o yaml
apiVersion: ocs.openshift.io/v1alpha1
kind: ManagedOCS
metadata:
  creationTimestamp: "2022-08-02T06:52:02Z"
  finalizers:
  - managedocs.ocs.openshift.io
  generation: 1
  name: managedocs
  namespace: openshift-storage
  resourceVersion: "1116403"
  uid: f3f4c792-35f9-4f43-820f-438c5b47b2c2
spec: {}
status:
  components:
    alertmanager:
      state: Ready
    prometheus:
      state: Ready
    storageCluster:
      state: Ready
  reconcileStrategy: strict

--------------------------------------------------
logs:
provider - http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jijoy-a2-pr/jijoy-a2-pr_20220802T061435/logs/testcases_1659445641/

consumer - http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jijoy-a2-c1/jijoy-a2-c1_20220802T073528/logs/testcases_1659445696/


=======================================================================

Version of all relevant components (if applicable):

Provider:
ODF 4.11.0-129
OCP 4.10.24
ocs-osd-deployer.v2.0.4

Consumer:
ODF 4.10.4-2
OCP 4.10.24
ocs-osd-deployer.v2.0.4

=================================================================
Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes. Unable to test 4.11 features on MS.
==================================================
Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
2

Can this issue reproducible?
2/2

Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:
Upgrade was working in the older builds of 4.11.0 (eg: ODF 4.11.0-113)

===============================================================
Steps to Reproduce:
1. Create MS provider and consumer cluster(ODF 4.10)
2. Upgrade the provider cluster to ODF 4.11.0
3. Check the status of storagecluster on consumer.
4. Check the status of cephResources in the storageconsumer on provider cluster. 

==========================================
Actual results:
Storagecluster on consumer remains in Progressing state.
Status of some CephClients and cephBlockpool are showing as Failure in the storageconsumer on provider cluster.

===============================================
Expected results:
Storagecluster on consumer should be Ready.
Status of cephResources in the storageconsumer on provider cluster should be Ready.


Additional info: