Bug 1981849

Description of problem (please be detailed as possible and provide log
snippests):
For OCS with Multus enabled cluster, OBCs provisioning doesn't succeed and are stuck in Pending state when using "ocs-storagecluster-ceph-rgw" storageclass
This issue is observed in both VMware and Baremetal clusters.

$ oc -n openshift-storage get obc
NAME   STORAGE-CLASS                 PHASE     AGE
obc1   openshift-storage.noobaa.io   Bound     3m52s
obc2   ocs-storagecluster-ceph-rgw   Pending   3m38s

rook-ceph-operator have following error logs:
```
2021-07-13 14:33:13.358266 I | exec: timeout waiting for process radosgw-admin to return. Sending interrupt signal to the process
E0713 14:33:13.360489       8 controller.go:199] error syncing 'openshift-storage/obc2': error provisioning bucket: failed to set admin ops api client: failed to retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "ocs-storagecluster-cephobjectstore": failed to create s3 user. . : signal: interrupt, requeuing
```

>> Status of other underlying resources:

$ oc -n openshift-storage get csv,storagecluster,noobaa,backingstore,bucketclass
NAME                                                                    DISPLAY                       VERSION        REPLACES   PHASE
clusterserviceversion.operators.coreos.com/ocs-operator.v4.8.0-452.ci   OpenShift Container Storage   4.8.0-452.ci              Succeeded

NAME                                                 AGE   PHASE   EXTERNAL   CREATED AT             VERSION
storagecluster.ocs.openshift.io/ocs-storagecluster   14m   Ready              2021-07-13T14:20:30Z   4.8.0

NAME                      MGMT-ENDPOINTS                   S3-ENDPOINTS                     IMAGE                                                                                                 PHASE   AGE
noobaa.noobaa.io/noobaa   ["https://10.1.161.201:31601"]   ["https://10.1.161.201:31272"]   quay.io/rhceph-dev/mcg-core@sha256:1569f6f70c235385a86a9fdadf632a3a188648de948404d352ed4679685ee8f0   Ready   11m

NAME                                                  TYPE            PHASE   AGE
backingstore.noobaa.io/noobaa-default-backing-store   s3-compatible   Ready   9m40s

NAME                                                PLACEMENT                                                        NAMESPACEPOLICY   PHASE   AGE
bucketclass.noobaa.io/noobaa-default-bucket-class   {"tiers":[{"backingStores":["noobaa-default-backing-store"]}]}                     Ready   9m40s


Version of all relevant components (if applicable):
VMware:
    OCP: 4.8.0-0.nightly-2021-07-09-181248
    OCS: ocs-operator.v4.8.0-452.ci
Bare Metal: 
    OCP: 4.8.0-0.nightly-2021-06-13-101614
    OCS: ocs-operator.v4.8.0-452.ci


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes, it is not possible to use new OBC because it's stuck in Pending

Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
Yes (2/2)

Can this issue reproduce from the UI?
yes

If this is a regression, please provide more details to justify this:
Deployment itself was blocked till now due to Bug 1978722

Steps to Reproduce:
1. Install OCS operator
2. Create Storage Cluster with Multus
3. Create an OBC using "ocs-storagecluster-ceph-rgw" StorageClass


Actual results:
OBC is stuck in a Pending state

Expected results:
OBC is Bound