Bug 2265051 - ocs-storagecluster is in progressing state due to noobaa in configuring state due to tls: failed to verify certificate: x509: certificate signed by unknown authority
Summary: ocs-storagecluster is in progressing state due to noobaa in configuring state...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenShift Data Foundation
Classification: Red Hat Storage
Component: Multi-Cloud Object Gateway
Version: 4.15
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: ODF 4.15.0
Assignee: Ben Eli
QA Contact: Vijay Avuthu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2024-02-20 04:43 UTC by Vijay Avuthu
Modified: 2024-06-24 14:50 UTC (History)
6 users (show)

Fixed In Version: 4.15.0-155
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2024-03-19 15:32:51 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github noobaa noobaa-operator pull 1304 0 None Merged Utilize the refreshing global CA bundle for the Azure and Google clients 2024-03-05 10:01:35 UTC
Github noobaa noobaa-operator pull 1308 0 None Merged [5.15 BACKPORT] Utilize the refreshing global CA bundle for the Azure and Google clients 2024-03-01 16:49:39 UTC
Red Hat Product Errata RHSA-2024:1383 0 None None None 2024-03-19 15:32:54 UTC

Description Vijay Avuthu 2024-02-20 04:43:35 UTC
Description of problem (please be detailed as possible and provide log
snippests):

Deploymnet Type: AZURE IPI FIPS ENCRYPTION 1AZ RHCOS 3M 3W 

ocs-storagecluster is in progressing state due to noobaa in configuring state due to tls: failed to verify certificate: x509: certificate signed by unknown authority

Version of all relevant components (if applicable):
ocs-registry:4.15.0-144

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes

Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
2/2

Can this issue reproduce from the UI?
Not tried

If this is a regression, please provide more details to justify this:
Yes

Steps to Reproduce:
1. install ODF using ocs-ci and check storagecluster status



Actual results:

storagecluster status:

Status:
  Conditions:
    Last Heartbeat Time:   2024-02-19T09:33:36Z
    Last Transition Time:  2024-02-19T09:33:36Z
    Message:               Version check successful
    Reason:                VersionMatched
    Status:                False
    Type:                  VersionMismatch
    Last Heartbeat Time:   2024-02-19T09:40:39Z
    Last Transition Time:  2024-02-19T09:38:15Z
    Message:               Reconcile completed successfully
    Reason:                ReconcileCompleted
    Status:                True
    Type:                  ReconcileComplete
    Last Heartbeat Time:   2024-02-19T09:38:12Z
    Last Transition Time:  2024-02-19T09:38:12Z
    Message:               Reconcile completed successfully
    Reason:                ReconcileCompleted
    Status:                True
    Type:                  Available
    Last Heartbeat Time:   2024-02-19T09:40:39Z
    Last Transition Time:  2024-02-19T09:38:13Z
    Message:               Waiting on Nooba instance to finish initialization
    Reason:                NoobaaInitializing
    Status:                True
    Type:                  Progressing
    Last Heartbeat Time:   2024-02-19T09:38:12Z
    Last Transition Time:  2024-02-19T09:38:12Z
    Message:               Reconcile completed successfully
    Reason:                ReconcileCompleted
    Status:                False
    Type:                  Degraded
    Last Heartbeat Time:   2024-02-19T09:38:40Z
    Last Transition Time:  2024-02-19T09:38:13Z
    Message:               CephCluster is creating: Processing OSD 2 on PVC "ocs-deviceset-managed-csi-0-data-0gnxqg"
    Reason:                ClusterStateCreating
    Status:                False
    Type:                  Upgradeable
  Current Mon Count:       3
  Failure Domain:          rack
  Failure Domain Key:      topology.rook.io/rack
.
.
.
  Phase:  Progressing


Expected results:

storagecluster should be in Ready state


Additional info:

$ oc get noobaa 
NAME     S3-ENDPOINTS                   STS-ENDPOINTS                  IMAGE                                                                                                            PHASE         AGE
noobaa   ["https://10.0.128.5:30107"]   ["https://10.0.128.5:30373"]   registry.redhat.io/odf4/mcg-core-rhel9@sha256:bfd4cb6d5c7526fdc8f35d4fedb1ff9ecb39ac7564c8f05c2d5e12aea41048e9   Configuring   19h

$ oc get noobaa noobaa -o yaml
apiVersion: noobaa.io/v1alpha1
kind: NooBaa
metadata:
  creationTimestamp: "2024-02-19T09:37:36Z"
  finalizers:
  - noobaa.io/graceful_finalizer
  generation: 1
  labels:
    app: noobaa
  name: noobaa
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: ocs.openshift.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: StorageCluster
    name: ocs-storagecluster
    uid: d4c313f2-e6e6-4efc-9bb1-6af0b720bd21
  resourceVersion: "633764"
  uid: 388e1f36-bc7b-451e-b2fc-70b6e064b793

.
.
.
status:
  accounts:
    admin:
      secretRef:
        name: noobaa-admin
        namespace: openshift-storage
  actualImage: registry.redhat.io/odf4/mcg-core-rhel9@sha256:bfd4cb6d5c7526fdc8f35d4fedb1ff9ecb39ac7564c8f05c2d5e12aea41048e9
  conditions:
  - lastHeartbeatTime: "2024-02-20T04:33:08Z"
    lastTransitionTime: "2024-02-19T09:37:36Z"
    message: 'failed to start creating storage account: azure.BearerAuthorizer#WithAuthorization:
      Failed to refresh the Token for request to https://management.azure.com/subscriptions/9bef6367-8ff5-4f08-84c9-3da195c53762/resourceGroups/j-156zife1c33-d-b9t2d-rg/providers/Microsoft.Storage/storageAccounts/noobaaaccountssm3a?api-version=2019-06-01:
      StatusCode=0 -- Original Error: adal: Failed to execute the refresh request.
      Error = ''Post "https://login.microsoftonline.com/9cf78105-e3e9-4321-b88d-b001b66c762b/oauth2/token?api-version=1.0":
      tls: failed to verify certificate: x509: certificate signed by unknown authority'''
    reason: TemporaryError
    status: "False"
    type: Available
  - lastHeartbeatTime: "2024-02-20T04:33:08Z"
    lastTransitionTime: "2024-02-19T09:37:36Z"
    message: 'failed to start creating storage account: azure.BearerAuthorizer#WithAuthorization:
      Failed to refresh the Token for request to https://management.azure.com/subscriptions/9bef6367-8ff5-4f08-84c9-3da195c53762/resourceGroups/j-156zife1c33-d-b9t2d-rg/providers/Microsoft.Storage/storageAccounts/noobaaaccountssm3a?api-version=2019-06-01:
      StatusCode=0 -- Original Error: adal: Failed to execute the refresh request.
      Error = ''Post "https://login.microsoftonline.com/9cf78105-e3e9-4321-b88d-b001b66c762b/oauth2/token?api-version=1.0":
      tls: failed to verify certificate: x509: certificate signed by unknown authority'''
    reason: TemporaryError
    status: "True"
    type: Progressing
  - lastHeartbeatTime: "2024-02-20T04:33:08Z"
    lastTransitionTime: "2024-02-19T09:37:36Z"
    message: 'failed to start creating storage account: azure.BearerAuthorizer#WithAuthorization:
      Failed to refresh the Token for request to https://management.azure.com/subscriptions/9bef6367-8ff5-4f08-84c9-3da195c53762/resourceGroups/j-156zife1c33-d-b9t2d-rg/providers/Microsoft.Storage/storageAccounts/noobaaaccountssm3a?api-version=2019-06-01:
      StatusCode=0 -- Original Error: adal: Failed to execute the refresh request.
      Error = ''Post "https://login.microsoftonline.com/9cf78105-e3e9-4321-b88d-b001b66c762b/oauth2/token?api-version=1.0":
      tls: failed to verify certificate: x509: certificate signed by unknown authority'''
    reason: TemporaryError
    status: "False"
    type: Degraded
  - lastHeartbeatTime: "2024-02-20T04:33:08Z"
    lastTransitionTime: "2024-02-19T09:37:36Z"
    message: 'failed to start creating storage account: azure.BearerAuthorizer#WithAuthorization:
      Failed to refresh the Token for request to https://management.azure.com/subscriptions/9bef6367-8ff5-4f08-84c9-3da195c53762/resourceGroups/j-156zife1c33-d-b9t2d-rg/providers/Microsoft.Storage/storageAccounts/noobaaaccountssm3a?api-version=2019-06-01:
      StatusCode=0 -- Original Error: adal: Failed to execute the refresh request.
      Error = ''Post "https://login.microsoftonline.com/9cf78105-e3e9-4321-b88d-b001b66c762b/oauth2/token?api-version=1.0":
      tls: failed to verify certificate: x509: certificate signed by unknown authority'''
    reason: TemporaryError
    status: "False"
    type: Upgradeable
  - lastHeartbeatTime: "2024-02-20T04:25:37Z"
    lastTransitionTime: "2024-02-19T09:37:37Z"
    status: k8s
    type: KMS-Type
  - lastHeartbeatTime: "2024-02-20T04:25:37Z"
    lastTransitionTime: "2024-02-19T09:37:38Z"
    status: Sync
    type: KMS-Status
  observedGeneration: 1
  phase: Configuring

must gather logs: https://url.corp.redhat.com/45bc1ca
job: https://url.corp.redhat.com/cc698ab

Comment 8 Vijay Avuthu 2024-03-05 11:45:28 UTC
verified with build: ocs-registry:4.15.0-155

job:  https://url.corp.redhat.com/b896aca

logs: https://url.corp.redhat.com/745a5b4

2024-03-04 20:54:42  15:24:42 - MainThread - ocs_ci.ocs.resources.storage_cluster - INFO  - Verifying status of storage cluster: ocs-storagecluster
2024-03-04 20:54:42  15:24:42 - MainThread - ocs_ci.ocs.resources.storage_cluster - INFO  - Check if StorageCluster: ocs-storagecluster is in Succeeded phase
2024-03-04 20:54:42  15:24:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc --kubeconfig /home/jenkins/current-cluster-dir/openshift-cluster-dir/auth/kubeconfig -n openshift-storage get StorageCluster ocs-storagecluster -n openshift-storage -o yaml
2024-03-04 20:54:42  15:24:42 - MainThread - ocs_ci.ocs.ocp - INFO  - Resource ocs-storagecluster is in phase: Ready!

Comment 9 errata-xmlrpc 2024-03-19 15:32:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.15.0 security, enhancement, & bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:1383


Note You need to log in before you can comment on or make changes to this bug.