Bug 2209171

Description of problem (please be detailed as possible and provide log
snippests):

- I have a customer trying to create a new namestorespace but it is stuck in 'creating'. 

- The name of the namespace store "uat-ns" is being re-used here, they had deleted the old one and now attempting to create a new one with the same name. 

[amenon@supportshell-1 03511299]$ cat 0060-namespacestore.txt
NAME         TYPE            PHASE      AGE
azure        azure-blob      Ready      94d
dynamic-ns   s3-compatible   Ready      6d3h
pp-ns        s3-compatible   Ready      6d3h
uat-ns       s3-compatible   Creating   6d3h <----------------

- From the Nooba operator logs, we can see 2 main errors 

1. Temporary Error: Connection name already exists: uat-ns" namespace=openshift-storage/uat-ns

2. Temporary Error: NooBaa NamespaceStore \"uat-ns\" is not yet ready" bucketclass=openshift-storage/uat-bc

- More details have been added in private comment #1

- It looks like delete of the previous Namespacestore uat-ns  didn't clean up properly. 

- Need help from engineering on the correct steps to delete and recreate a namespacestore

Version of all relevant components (if applicable):

ODF v4.11.7

[amenon@supportshell-1 oc_output]$ cat get_clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.22   True        False         117d    Cluster version is 4.11.22

mcg-operator.v4.11.7                           NooBaa Operator                            4.11.7                  mcg-operator.v4.11.6                           Succeeded
namespace-configuration-operator.v1.2.4        Namespace Configuration Operator           1.2.4                   namespace-configuration-operator.v1.2.3        Succeeded
ocs-operator.v4.11.7                           OpenShift Container Storage                4.11.7                  ocs-operator.v4.11.6                           Succeeded
odf-csi-addons-operator.v4.11.7                CSI Addons                                 4.11.7                  odf-csi-addons-operator.v4.11.6                Succeeded
odf-operator.v4.11.7                           OpenShift Data Foundation                  4.11.7                  odf-operator.v4.11.6                           Succeeded

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

- Cu is unable to create namespacestore which is preventing them from rolling out the object storage solution

Is there any workaround available to the best of your knowledge?

No

Can this issue reproducible?

Yes

Can this issue reproduce from the UI?

- Steps performed by cu to remove the 'uat-ns' namespacestore from OpenShift GUI:

1. Delete the uat-ns namespacestore
2. Delete the backing bucket
3. Delete the bucketclass.

If this is a regression, please provide more details to justify this:
N/A

Steps to Reproduce:

- Tried to test this via cli but not successful

$ noobaa namespacestore status <namespacestore>

# noobaa namespacestore status namespacestoretest
INFO[0000]  Exists: NamespaceStore "namespacestoretest"
INFO[0000]  Exists: Secret "namespace-store-aws-s3-namespacestoretest"
INFO[0000]  NamespaceStore "namespacestoretest" Phase is Ready
# NamespaceStore spec:
accessMode: ReadWrite
awsS3:
  secret:
    name: namespace-store-aws-s3-namespacestoretest
    namespace: openshift-storage
  targetBucket: namespacestorebucket
type: aws-s3


$ noobaa namespacestore delete <namespacestore>

# noobaa namespacestore delete namespacestoretest
INFO[0000]  Exists: NooBaa "noobaa"
INFO[0000]  NooBaa system deleted at 2023-04-10 02:04:37 -0500 CDT
FATA[0000]  Connect(): System not found
3:14
#--------------------#
#- Namespace Stores -#
#--------------------#
NAME                 TYPE     TARGET-BUCKET          PHASE   AGE
namespacestoretest   aws-s3   namespacestorebucket   Ready   29m24s

Actual results:

Namespacestore creation is stuck

Expected results:

Expecting the namespacestore to create successfully

Additional info:

- OCP and ODF m-g is available in supportshell under ~/03511299/