Bug 2155553

Description of problem (please be detailed as possible and provide log
snippests):
When trying to create a NamespaceStore with valid AWS S3 bucket, there is an error on platforms IBM Cloud IPI and VSPHERE UPI Disconnected:

http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/j-002ici3c33-t4a/j-002ici3c33-t4a_20221213T163311/logs/ocs-ci-logs-1670954128/by_outcome/failed/tests/manage/monitoring/prometheus/test_noobaa.py/test_noobaa_ns_bucket/logs

http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/j-001vud1cs33-t4a/j-001vud1cs33-t4a_20221125T174052/logs/ocs-ci-logs-1669404527/by_outcome/failed/tests/manage/monitoring/prometheus/test_noobaa.py/test_noobaa_ns_bucket/logs

The same test creates the NamespaceStore correctly on VSPHERE UPI 1AZ RHCOS EXTERNAL platform (https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster-prod/6112/).

Following errors appear in NamespaceStore status:
VSPHERE UPI Disconnected:

kind: NamespaceStore
metadata:
  creationTimestamp: "2022-11-25T19:53:08Z"
  finalizers:
  - noobaa.io/finalizer
  generation: 1
  labels:
    app: noobaa
  name: aws-ns-store-f3fbf2a775ce42f4944900093f7
  namespace: openshift-storage
  resourceVersion: "84008"
  uid: a4aa3dce-5385-47a8-a886-b2e95ea38b5b
spec:
  awsS3:
    secret:
      name: secret-cldmgr-aws-0d66310c61d1445083cc19
      namespace: openshift-storage
    targetBucket: aws-uls-16fe0e2b7fda4067bdac06bae4547e45
  type: aws-s3
status:
  conditions:
  - lastHeartbeatTime: "2022-11-25T19:53:08Z"
    lastTransitionTime: "2022-11-25T19:53:08Z"
    message: CheckExternalConnection Status=TIMEOUT Error=OperationTimeout Message=Operation
      timeout
    reason: TemporaryError
    status: "False"
    type: Available
  - lastHeartbeatTime: "2022-11-25T19:53:08Z"
    lastTransitionTime: "2022-11-25T19:53:08Z"
    message: CheckExternalConnection Status=TIMEOUT Error=OperationTimeout Message=Operation
      timeout
    reason: TemporaryError
    status: "True"
    type: Progressing
  - lastHeartbeatTime: "2022-11-25T19:53:08Z"
    lastTransitionTime: "2022-11-25T19:53:08Z"
    message: CheckExternalConnection Status=TIMEOUT Error=OperationTimeout Message=Operation
      timeout
    reason: TemporaryError
    status: "False"
    type: Degraded
  - lastHeartbeatTime: "2022-11-25T19:53:08Z"
    lastTransitionTime: "2022-11-25T19:53:08Z"
    message: CheckExternalConnection Status=TIMEOUT Error=OperationTimeout Message=Operation
      timeout
    reason: TemporaryError
    status: "False"
    type: Upgradeable
  mode: {}
  phase: Creating

IBM Cloud IPI 3AZ RHCOS:
kind: NamespaceStore
metadata:
  creationTimestamp: "2022-12-13T18:24:58Z"
  finalizers:
  - noobaa.io/finalizer
  generation: 1
  labels:
    app: noobaa
  name: aws-ns-store-79f13954b56641ac83e0672caae
  namespace: openshift-storage
  resourceVersion: "66402"
  uid: f01d71fc-341a-4195-91ea-e69268aa0bd8
spec:
  awsS3:
    secret:
      name: secret-cldmgr-aws-e6a9239c2bcb48468dcaff
      namespace: openshift-storage
    targetBucket: aws-uls-2e8eed0e5ffa4c9f983d2877ed19351a
  type: aws-s3
status:
  conditions:
  - lastHeartbeatTime: "2022-12-13T18:24:58Z"
    lastTransitionTime: "2022-12-13T18:24:58Z"
    message: CheckExternalConnection Status=UNKNOWN_FAILURE Error=AccessDenied Message=Access
      Denied
    reason: TemporaryError
    status: "False"
    type: Available
  - lastHeartbeatTime: "2022-12-13T18:24:58Z"
    lastTransitionTime: "2022-12-13T18:24:58Z"
    message: CheckExternalConnection Status=UNKNOWN_FAILURE Error=AccessDenied Message=Access
      Denied
    reason: TemporaryError
    status: "True"
    type: Progressing
  - lastHeartbeatTime: "2022-12-13T18:24:58Z"
    lastTransitionTime: "2022-12-13T18:24:58Z"
    message: CheckExternalConnection Status=UNKNOWN_FAILURE Error=AccessDenied Message=Access
      Denied
    reason: TemporaryError
    status: "False"
    type: Degraded
  - lastHeartbeatTime: "2022-12-13T18:24:58Z"
    lastTransitionTime: "2022-12-13T18:24:58Z"
    message: CheckExternalConnection Status=UNKNOWN_FAILURE Error=AccessDenied Message=Access
      Denied
    reason: TemporaryError
    status: "False"
    type: Upgradeable
  mode: {}
  phase: Creating

Version of all relevant components (if applicable):
ocs-registry:4.12.0-116

Can this issue reproducible?
?

If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Create AWS S3 bucket
2. Create NamespaceStore that uses the bucket


Actual results:
There are issues during creation of the namespacestore

Expected results:
Namespacestore should be in healthy state

Additional info:
Issue was found with automated test https://github.com/red-hat-storage/ocs-ci/blob/master/tests/manage/monitoring/prometheus/test_noobaa.py#L94
Results:
https://reportportal-ocs4.apps.ocp-c1.prod.psi.redhat.com/ui/#ocs/launches/362/7549/331695/331703/331704/log
https://reportportal-ocs4.apps.ocp-c1.prod.psi.redhat.com/ui/#ocs/launches/362/6596/271263/271271/271272/log
Test that passed on different platform (there was a system issue with timing of the alert):
https://reportportal-ocs4.apps.ocp-c1.prod.psi.redhat.com/ui/#ocs/launches/362/6524/268695/268703/268704/log