Description of problem (please be detailed as possible and provide log
snippests):

Backing Store failed to initialize and never reached OPTIMAL state.

Backingstore  error: 
--------------
uid change has been identified - will change from uid: 0 to new uid: 10001
Error: opendir(/noobaa_storage/lost+found) failed Permission denied

real	0m0.096s
user	0m0.001s
sys	0m0.003s
Got base64 agent_conf: eyJhZGRyZXNzIjoid3NzOi8vbm9vYmFhLW1nbXQub3BlbnNoaWZ0LXN0b3JhZ2Uuc3ZjLmNsdXN0ZXIubG9jYWwvIiwicm91dGluZ19oaW50IjoiSU5URVJOQUwiLCJzeXN0ZW0iOiJub29iYWEiLCJjcmVhdGVfbm9kZV90b2tlbiI6ImV5SmhiR2NpT2lKSVV6STFOaUlzSW5SNWNDSTZJa3BYVkNKOS5leUpoWTJOdmRXNTBYMmxrSWpvaU5qWXpZMkZtTW1SaU9EQTJPV1F3TURJMk5qSTJaV1JpSWl3aWMzbHpkR1Z0WDJsa0lqb2lOall6WTJGbU1tSmlPREEyT1dRd01ESTJOakkyWldOaklpd2ljbTlzWlNJNkltTnlaV0YwWlY5dWIyUmxJaXdpWlhoMGNtRWlPbnNpWVdkbGJuUmZZMjl1Wm1sblgybGtJam9pTmpZelpEUTNZelF5WlRkaFlUSXdNREl5WVROa01UQmpJbjBzSW1GMWRHaHZjbWw2WldSZllua2lPaUp1YjI5aVlXRWlMQ0pwWVhRaU9qRTNNVFV5T1RJeE1EQjkuTW5Sc29lSVN2by1yWTFKTGk0Mldhckx3UnpnNHFTUzJKN2NkWmJBTFo1dyIsInJvb3RfcGF0aCI6Ii4vbm9vYmFhX3N0b3JhZ2UvIn0=
/noobaa_init_files/noobaa_init.sh: line 133: /noobaa_storage/agent_conf.json: Permission denied
cat: /noobaa_storage/agent_conf.json: No such file or directory
Written agent_conf.json: 
Version is: 5.16.0-6b86620
Running: node --unhandled-rejections=warn ./src/agent/agent_cli
Error accrued while getting the data from /etc/noobaa-server/jwt: Error: ENOENT: no such file or directory, open '/etc/noobaa-server/jwt'
Error accrued while getting the data from /etc/noobaa-server/server_secret: Error: ENOENT: no such file or directory, open '/etc/noobaa-server/server_secret'
Error accrued while getting the data from /etc/noobaa-auth-token/auth_token: Error: ENOENT: no such file or directory, open '/etc/noobaa-auth-token/auth_token'
load_config_local: LOADED {
  DEFAULT_ACCOUNT_PREFERENCES: { ui_theme: 'LIGHT' },
  REMOTE_NOOAA_NAMESPACE: 'openshift-storage',
  ALLOW_BUCKET_CREATE_ON_INTERNAL: false
}
detect_fips_mode: found /proc/sys/crypto/fips_enabled with value 0


Noobaa operator logs: 
------------------------
....
time="2024-05-09T22:09:17Z" level=info msg="Start BackingStore Reconcile ..." backingstore=openshift-storage/pv-backingstore-6425a392ee68464983e3fe3a
time="2024-05-09T22:09:17Z" level=info msg="✅ Exists: NooBaa \"noobaa\"\n"
time="2024-05-09T22:09:18Z" level=info msg="✅ Exists: BackingStore \"pv-backingstore-6425a392ee68464983e3fe3a\"\n"
time="2024-05-09T22:09:18Z" level=info msg="✅ Exists: Secret \"backing-store-pv-pool-pv-backingstore-6425a392ee68464983e3fe3a\"\n"
time="2024-05-09T22:09:18Z" level=info msg="✅ Exists:  \"backing-store-pv-pool-pv-backingstore-6425a392ee68464983e3fe3a\"\n"
time="2024-05-09T22:09:18Z" level=info msg="SetPhase: Verifying" backingstore=openshift-storage/pv-backingstore-6425a392ee68464983e3fe3a
time="2024-05-09T22:09:18Z" level=info msg="SetPhase: Connecting" backingstore=openshift-storage/pv-backingstore-6425a392ee68464983e3fe3a
time="2024-05-09T22:09:18Z" level=info msg="✅ Exists: NooBaa \"noobaa\"\n"
time="2024-05-09T22:09:18Z" level=info msg="✅ Exists: Service \"noobaa-mgmt\"\n"
time="2024-05-09T22:09:18Z" level=info msg="✅ Exists: Secret \"noobaa-operator\"\n"
time="2024-05-09T22:09:18Z" level=info msg="✅ Exists: Secret \"noobaa-admin\"\n"
time="2024-05-09T22:09:18Z" level=info msg="âœˆï¸  RPC: system.read_system() Request: <nil>"
time="2024-05-09T22:09:18Z" level=info msg="✅ RPC: system.read_system() Response OK: took 15.3ms"
time="2024-05-09T22:09:18Z" level=info msg="âœˆï¸  RPC: host.list_hosts() Request: {Query:{Pools:[pv-backingstore-6425a392ee68464983e3fe3a]}}"
time="2024-05-09T22:09:18Z" level=info msg="✅ RPC: host.list_hosts() Response OK: took 1.3ms"
time="2024-05-09T22:09:18Z" level=info msg="SetPhase: Creating" backingstore=openshift-storage/pv-backingstore-6425a392ee68464983e3fe3a
time="2024-05-09T22:09:18Z" level=info msg="✅ Exists: ServiceAccount \"default\"\n"
time="2024-05-09T22:09:18Z" level=warning msg="Pod pv-backingstore-6425a392ee68464983e3fe3a-noobaa-pod-64af0b5f didn't attach yet to noobaa system" backingstore=openshift-storage/pv-backingstore-6425a392ee68464983e3fe3a
time="2024-05-09T22:09:18Z" level=info msg="SetPhase: temporary error during phase \"Creating\"" backingstore=openshift-storage/pv-backingstore-6425a392ee68464983e3fe3a
time="2024-05-09T22:09:18Z" level=warning msg="â³ Temporary Error: BackingStore Still didn't connect all requested pods. 1 from 1 are pending" backingstore=openshift-storage/pv-backingstore-6425a392ee68464983e3fe3a
time="2024-05-09T22:09:18Z" level=info msg="Update event detected for pv-backingstore-6425a392ee68464983e3fe3a (openshift-storage), queuing Reconcile"
time="2024-05-09T22:09:18Z" level=info msg="checking which bucketclasses to reconcile. mapping backingstore openshift-storage/pv-backingstore-6425a392ee68464983e3fe3a to bucketclasses"
time="2024-05-09T22:09:18Z" level=info msg="UpdateStatus: Done" backingstore=openshift-storage/pv-backingstore-6425a392ee68464983e3fe3a
time="2024-05-09T22:09:18Z" level=info msg="will reconcile these bucketclasses: []"
time="2024-05-09T22:09:18Z" level=info msg="checking which bucketclasses to reconcile. mapping backingstore openshift-storage/pv-backingstore-6425a392ee68464983e3fe3a to bucketclasses"
time="2024-05-09T22:09:18Z" level=info msg="will reconcile these bucketclasses: []"


Version of all relevant components (if applicable):
4.16.0-95


Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
N/A

If this is a regression, please provide more details to justify this:
This failure has seen on multiple build in 4.16.
Same test case is passing in 4.15


Actual results:
Backing Store failed to create

Expected results:
Backing Store should able to create without any issue.

Additional info:
Test failure logs:
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/j-020zi3c33-uba/j-020zi3c33-uba_20240509T055225/logs/ocs-ci-logs-1715275314/tests/functional/object/mcg/test_bucket_creation_deletion.py/TestBucketCreationAndDeletion/test_bucket_creation_deletion-1-OC-PVPOOL/

Must gather logs:
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/j-020zi3c33-uba/j-020zi3c33-uba_20240509T055225/logs/failed_testcase_ocs_logs_1715275314/test_bucket_creation_deletion%5b1-OC-PVPOOL%5d_ocs_logs/j-020zi3c33-uba/

Noobaa operator logs: 
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/j-020zi3c33-uba/j-020zi3c33-uba_20240509T055225/logs/failed_testcase_ocs_logs_1715275314/test_bucket_creation_deletion%5b1-OC-PVPOOL%5d_ocs_logs/j-020zi3c33-uba/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-4d932e1c422c3cd3f36c31af95bf7e3831ad54b26d3428fbcfcff933157f65cf/noobaa/logs/openshift-storage/noobaa-operator-9776d79bc-p7kn4.log