Description of problem (please be detailed as possible and provide log
snippests):
Cephobjectstore stuck on Progressing state because " failed to create or retrieve rgw admin ops user"

Version of all relevant components (if applicable):
OCP Version: 4.13.0-0.nightly-2023-05-22-181752
ODF Version: 4.13.0-203.stable
Platform: Vsphere

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1.Install OCP4.13 + ODF4.13 with multus [public-net+cluster-net and bridge mode]

NAD configuration:
---
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
 name: public-net
 namespace: default
 labels: {}
 annotations: {}
spec:
 config: '{ "cniVersion": "0.3.1", "type": "macvlan", "master": "br-ex", "mode": "bridge", "ipam": { "type": "whereabouts", "range": "192.168.20.0/24" } }'
---
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
 name: cluster-net
 namespace: default
 labels: {}
 annotations: {}
spec:
 config: '{ "cniVersion": "0.3.1", "type": "macvlan", "master": "br-ex", "ipam": { "type": "whereabouts", "range": "192.168.30.0/24" } }'


2.Check  CephObjectStore status:
$ oc get CephObjectStore 
NAME                                 PHASE
ocs-storagecluster-cephobjectstore   Progressing
  
Warning  ReconcileFailed  11m (x90 over 21h)  rook-ceph-object-controller  failed to reconcile CephObjectStore "openshift-storage/ocs-storagecluster-cephobjectstore". failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "ocs-storagecluster-cephobjectstore": failed to create s3 user. . : signal: interrupt

3.Check  RGW pod:
$ oc get pod rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-9d4d788l9h8k
NAME                                                              READY   STATUS    RESTARTS        AGE
rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-9d4d788l9h8k   1/2     Running   288 (35s ago)   21h

Events:
  Type     Reason     Age                     From     Message
  ----     ------     ----                    ----     -------
  Normal   Pulled     4h47m (x225 over 21h)   kubelet  Container image "quay.io/rhceph-dev/rhceph@sha256:fa6d01cdef17bc32d2b95b8121b02f4d41adccc5ba8a9b95f38c97797ff6621f" already present on machine
  Warning  BackOff    7m14s (x2226 over 21h)  kubelet  Back-off restarting failed container rgw in pod rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-9d4d788l9h8k_openshift-storage(64db8bd3-2327-4aa1-9c01-3f70476a475e)
  Warning  Unhealthy  2m13s (x4741 over 21h)  kubelet  Startup probe failed: RGW health check failed with error code: 7. the RGW likely cannot be reached by clients

for more info on deployment process:
https://docs.google.com/document/d/1q2OzxGGUgM9R8TUWEbb2ulRUw_GtHcJs_SmxfSE5iXM/edit

Actual results:
CephObjectStore in Progressing state

Expected results:
CephObjectStore in Ready state

Additional info:
OCS MG:
http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/bz-2209643.tar.gz