Description of problem (please be detailed as possible and provide log
snippests):

platform: VSPHERE IPI 1AZ RHCOS VSAN 3M 3W 


Version of all relevant components (if applicable):
upgraded from 4.14.5 to 4.15.0-157


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes

Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
1/1

Can this issue reproduce from the UI?
not tried

If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. upgrade odf from 4.14.5 to 4.15.0-157
2. check storagecluster satus
3.


Actual results:

Status:
  Conditions:
    Last Heartbeat Time:   2024-03-09T09:03:52Z
    Last Transition Time:  2024-03-09T09:03:52Z
    Message:               Version check successful
    Reason:                VersionMatched
    Status:                False
    Type:                  VersionMismatch
    Last Heartbeat Time:   2024-03-09T14:20:13Z
    Last Transition Time:  2024-03-09T13:38:36Z
    Message:               Reconcile completed successfully
    Reason:                ReconcileCompleted
    Status:                True
    Type:                  ReconcileComplete
    Last Heartbeat Time:   2024-03-09T13:29:41Z
    Last Transition Time:  2024-03-09T09:12:33Z
    Message:               Reconcile completed successfully
    Reason:                ReconcileCompleted
    Status:                True
    Type:                  Available
    Last Heartbeat Time:   2024-03-09T14:20:13Z
    Last Transition Time:  2024-03-09T13:30:28Z
    Message:               Waiting on Nooba instance to finish initialization
    Reason:                NoobaaInitializing
    Status:                True
    Type:                  Progressing
    Last Heartbeat Time:   2024-03-09T13:29:41Z
    Last Transition Time:  2024-03-09T11:47:32Z
    Message:               Reconcile completed successfully
    Reason:                ReconcileCompleted
    Status:                False
    Type:                  Degraded
    Last Heartbeat Time:   2024-03-09T13:39:07Z
    Last Transition Time:  2024-03-09T13:30:38Z
    Message:               CephCluster is creating: Processing OSD 2 on PVC "ocs-deviceset-0-data-0w4lzl"
    Reason:                ClusterStateCreating
    Status:                False
    Type:                  Upgradeable
  Current Mon Count:       3
  Failure Domain:          rack
 
.
.
.

  Phase:  Progressing


Expected results:

storagecluster should be in Ready state


Additional info:

> operator log

time="2024-03-09T14:23:14Z" level=error msg="âš ï¸  RPC: system.read_system() Response Error: Code=UNAUTHORIZED Message=account not found 65ec27f52e541700269b2156"
time="2024-03-09T14:23:14Z" level=info msg="SetPhase: temporary error during phase \"Connecting\"" backingstore=openshift-storage/noobaa-default-backing-store
time="2024-03-09T14:23:14Z" level=warning msg="â³ Temporary Error: account not found 65ec27f52e541700269b2156" backingstore=openshift-storage/noobaa-default-backing-store
time="2024-03-09T14:23:14Z" level=info msg="Update event detected for noobaa-default-backing-store (openshift-storage), queuing Reconcile"

> noobaa-core logs

2024-03-09T14:19:37.401306082Z [32mMar-9 14:19:37.401[35m [WebServer/38] [31m[ERROR][39m CONSOLE:: RPC._on_request: ERROR srv system_api.read_system reqid wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/-5043 connid ws://[::ffff:10.128.4.12]:52282/(59puv4u.zzxm) Error: account not found 65ec27f52e541700269b2156
2024-03-09T14:19:37.401306082Z     at req.check_auth (/root/node_modules/noobaa-core/src/server/common_services/auth_server.js:544:23)
2024-03-09T14:19:37.401306082Z     at authorize (/root/node_modules/noobaa-core/src/server/common_services/auth_server.js:434:17)
2024-03-09T14:19:37.401306082Z     at RPC._on_request (/root/node_modules/noobaa-core/src/rpc/rpc.js:323:27)
2024-03-09T14:19:37.401306082Z     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)


job: https://url.corp.redhat.com/1248f21
must gather: https://url.corp.redhat.com/94a5771