Description of problem (please be detailed as possible and provide log
snippests):

noobaa-default-backing-store in Rejected state in MCG Only deployment.

Version of all relevant components (if applicable):
ocs-operator.v4.16.0-90.stable

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
not able to install MCG only deployment

Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
Yes, 2/2

Can this issue reproduce from the UI?
not tried

If this is a regression, please provide more details to justify this:
Yes

Steps to Reproduce:
1. Install MCG only deployment using ocs-ci
2. check backingstore status
3.


Actual results:
$ oc get backingstore
NAME                           TYPE      PHASE      AGE
noobaa-default-backing-store   pv-pool   Rejected   31m


Expected results:


Additional info:

$ oc describe backingstore noobaa-default-backing-store
Name:         noobaa-default-backing-store
Namespace:    openshift-storage
Labels:       app=noobaa
Annotations:  <none>
API Version:  noobaa.io/v1alpha1
Kind:         BackingStore
.
.
.
Status:
  Conditions:
    Last Heartbeat Time:   2024-04-30T09:14:03Z
    Last Transition Time:  2024-04-30T09:14:03Z
    Message:               Current failing: 1 from requested: 1
    Reason:                Failed connecting all pods in backingstore for more than 10 minutes
    Status:                Unknown
    Type:                  Available
    Last Heartbeat Time:   2024-04-30T09:14:03Z
    Last Transition Time:  2024-04-30T09:14:03Z
    Message:               Current failing: 1 from requested: 1
    Reason:                Failed connecting all pods in backingstore for more than 10 minutes
    Status:                False
    Type:                  Progressing
    Last Heartbeat Time:   2024-04-30T09:14:03Z
    Last Transition Time:  2024-04-30T09:14:03Z
    Message:               Current failing: 1 from requested: 1
    Reason:                Failed connecting all pods in backingstore for more than 10 minutes
    Status:                True
    Type:                  Degraded
    Last Heartbeat Time:   2024-04-30T09:14:03Z
    Last Transition Time:  2024-04-30T09:14:03Z
    Message:               Current failing: 1 from requested: 1
    Reason:                Failed connecting all pods in backingstore for more than 10 minutes
    Status:                Unknown
    Type:                  Upgradeable


> noobaa-default-backing-store-noobaa-pod is in CLBO

$ oc get pods
NAME                                                      READY   STATUS             RESTARTS         AGE
csi-addons-controller-manager-67499dd99d-5dcdl            2/2     Running            0                44m
noobaa-core-0                                             2/2     Running            0                44m
noobaa-db-pg-0                                            1/1     Running            0                44m
noobaa-default-backing-store-noobaa-pod-8b0ee01e          0/1     CrashLoopBackOff   11 (3m17s ago)   42m
noobaa-endpoint-7dcc44f954-4gl2x                          1/1     Running            0                43m
noobaa-operator-555c8784c8-5bng4                          1/1     Running            0                46m
ocs-client-operator-console-75fc966869-5zzw2              1/1     Running            0                45m
ocs-client-operator-controller-manager-7944f448b5-5zzrp   2/2     Running            0                45m
ocs-metrics-exporter-6558b7585b-dd6vf                     1/1     Running            0                44m
ocs-operator-776bbff6d4-r642w                             1/1     Running            0                46m
odf-console-6d446c5b48-ml46z                              1/1     Running            0                46m
odf-operator-controller-manager-699bdc69ff-6nhmn          2/2     Running            0                46m
rook-ceph-operator-7f598695f5-pfjsn                       1/1     Running            0                46m
ux-backend-server-5f586bdd95-r6fww                        2/2     Running            0                46m

> 

$ oc describe pod noobaa-default-backing-store-noobaa-pod-8b0ee01e
Name:             noobaa-default-backing-store-noobaa-pod-8b0ee01e
Namespace:        openshift-storage
Priority:         0
Service Account:  default
Node:             j-024vi1rmcg33-t1-kbk95-worker-0-c6mwt/10.1.113.160
Start Time:       Tue, 30 Apr 2024 14:03:05 +0530
Labels:           app=noobaa
                  pool=noobaa-default-backing-store
.
.
.
Events:
  Type     Reason                  Age                    From                     Message
  ----     ------                  ----                   ----                     -------
  Normal   Scheduled               42m                    default-scheduler        Successfully assigned openshift-storage/noobaa-default-backing-store-noobaa-pod-8b0ee01e to j-024vi1rmcg33-t1-kbk95-worker-0-c6mwt
  Normal   SuccessfulAttachVolume  42m                    attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-e9ad53fe-36ad-4192-9ac6-79b139409bd4"
  Normal   AddedInterface          42m                    multus                   Add eth0 [10.131.0.23/23] from ovn-kubernetes
  Normal   Pulled                  38m (x5 over 42m)      kubelet                  Container image "registry.redhat.io/odf4/mcg-core-rhel9@sha256:53f62e85f733d38b511b4b2f0ebb0da40e606c55f10c592781f751c960bf9b3c" already present on machine
  Normal   Created                 38m (x5 over 42m)      kubelet                  Created container noobaa-agent
  Normal   Started                 38m (x5 over 42m)      kubelet                  Started container noobaa-agent
  Warning  BackOff                 2m22s (x153 over 41m)  kubelet                  Back-off restarting failed container noobaa-agent in pod noobaa-default-backing-store-noobaa-pod-8b0ee01e_openshift-storage(15b08b26-c3ae-426b-b3be-4c24d6df9bef)


> noobaa operator log

time="2024-04-30T09:05:59Z" level=info msg="SetPhase: Rejected" backingstore=openshift-storage/noobaa-default-backing-store
time="2024-04-30T09:05:59Z" level=error msg="❌ Persistent Error: Current failing: 1 from requested: 1" backingstore=openshift-storage/noobaa-default-backing-store
time="2024-04-30T09:05:59Z" level=info msg="Update event detected for noobaa-default-backing-store (openshift-storage), queuing Reconcile"
time="2024-04-30T09:05:59Z" level=info msg="checking which bucketclasses to reconcile. mapping backingstore openshift-storage/noobaa-default-backing-store to bucketclasses"
time="2024-04-30T09:05:59Z" level=info msg="UpdateStatus: Done" backingstore=openshift-storage/noobaa-default-backing-store
time="2024-04-30T09:05:59Z" level=info msg="will reconcile these bucketclasses: [openshift-storage/noobaa-default-bucket-class]"
time="2024-04-30T09:05:59Z" level=info msg="checking which bucketclasses to reconcile. mapping backingstore openshift-storage/noobaa-default-backing-store to bucketclasses"
time="2024-04-30T09:05:59Z" level=info msg="Start BucketClass Reconcile..." bucketclass=openshift-storage/noobaa-default-bucket-class
time="2024-04-30T09:05:59Z" level=info msg="will reconcile these bucketclasses: [openshift-storage/noobaa-default-bucket-class]"
time="2024-04-30T09:05:59Z" level=info msg="✅ Exists: NooBaa \"noobaa\"\n"
time="2024-04-30T09:05:59Z" level=info msg="✅ Exists: BucketClass \"noobaa-default-bucket-class\"\n"
time="2024-04-30T09:05:59Z" level=info msg="SetPhase: Verifying" bucketclass=openshift-storage/noobaa-default-bucket-class
time="2024-04-30T09:05:59Z" level=info msg="validating placement policy &{Tiers:[{Placement: BackingStores:[noobaa-default-backing-store]}]}"
time="2024-04-30T09:05:59Z" level=info msg="✅ Exists: BackingStore \"noobaa-default-backing-store\"\n"
time="2024-04-30T09:05:59Z" level=info msg="SetPhase: Rejected" bucketclass=openshift-storage/noobaa-default-bucket-class
time="2024-04-30T09:05:59Z" level=error msg="❌ Persistent Error: NooBaa BackingStore \"noobaa-default-backing-store\" is in rejected phase" bucketclass=openshift-storage/noobaa-default-bucket-class
time="2024-04-30T09:05:59Z" level=info msg="UpdateStatus: Done" bucketclass=openshift-storage/noobaa-default-bucket-class


must gather: https://url.corp.redhat.com/02e543e
job: https://url.corp.redhat.com/2789162