Bug 2269379 - [GCP]: Noobaa instance fails to finish initialization due to "oauth2: cannot fetch token: Post \"https://oauth2.googleapis.com/token\": tls: failed to verify certificate: x509: certificate signed by unknown authority" sys=openshift-storage/noobaa"
Summary: [GCP]: Noobaa instance fails to finish initialization due to "oauth2: cannot...
Keywords:
Status: CLOSED DUPLICATE of bug 2271580
Alias: None
Product: Red Hat OpenShift Data Foundation
Classification: Red Hat Storage
Component: Multi-Cloud Object Gateway
Version: 4.15
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: ---
Assignee: Nimrod Becker
QA Contact: krishnaram Karthick
URL:
Whiteboard:
Depends On:
Blocks: 2246375
TreeView+ depends on / blocked
 
Reported: 2024-03-13 14:19 UTC by Vijay Avuthu
Modified: 2024-05-27 10:55 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
.Multicloud Object Gateway instance fails to finish initialization Due to a race in timing between the pod code run and OpenShift loading the Certificate Authority (CA) bundle into the pod, the pod is unable to communicate with the cloud storage service. As a result, default backing store cannot be created. Workaround: Restart the Multicloud Object Gateway (MCG) operator pod: ---- $ oc delete pod noobaa-operator-<ID ---- With the workaround the backing store is reconciled and works.
Clone Of:
Environment:
Last Closed: 2024-05-27 10:55:32 UTC
Embargoed:


Attachments (Terms of Use)

Description Vijay Avuthu 2024-03-13 14:19:07 UTC
Description of problem (please be detailed as possible and provide log
snippests):

platform: GCP

Version of all relevant components (if applicable):
ocs-registry:4.15.0-158


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
not able to install ODF fully

Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
1/1

Can this issue reproduce from the UI?
Not tried

If this is a regression, please provide more details to justify this:
Yes


Steps to Reproduce:
1. install odf using ocs-ci on GCP  platform
2. check storagecluster satus


Actual results:

storage cluster status

Status:
  Conditions:
    Last Heartbeat Time:   2024-03-13T10:11:15Z
    Last Transition Time:  2024-03-13T10:11:15Z
    Message:               Version check successful
    Reason:                VersionMatched
    Status:                False
    Type:                  VersionMismatch
    Last Heartbeat Time:   2024-03-13T10:18:31Z
    Last Transition Time:  2024-03-13T10:17:42Z
    Message:               Reconcile completed successfully
    Reason:                ReconcileCompleted
    Status:                True
    Type:                  ReconcileComplete
    Last Heartbeat Time:   2024-03-13T10:11:15Z
    Last Transition Time:  2024-03-13T10:11:15Z
    Message:               Initializing StorageCluster
    Reason:                Init
    Status:                False
    Type:                  Available
    Last Heartbeat Time:   2024-03-13T10:18:31Z
    Last Transition Time:  2024-03-13T10:11:15Z
    Message:               Waiting on Nooba instance to finish initialization
    Reason:                NoobaaInitializing
    Status:                True
    Type:                  Progressing
    Last Heartbeat Time:   2024-03-13T10:11:15Z
    Last Transition Time:  2024-03-13T10:11:15Z
    Message:               Initializing StorageCluster
    Reason:                Init
    Status:                False
    Type:                  Degraded
    Last Heartbeat Time:   2024-03-13T10:18:01Z
    Last Transition Time:  2024-03-13T10:14:38Z
    Message:               CephCluster is creating: Processing OSD 2 on PVC "ocs-deviceset-0-data-0hs5bk"
    Reason:                ClusterStateCreating
    Status:                False
    Type:                  Upgradeable
  Current Mon Count:       3
.
.
.
  Phase:  Progressing


Expected results:

storagecluster should be in Ready state


Additional info:

> noobaa 

$ oc get noobaa noobaa -o yaml
apiVersion: noobaa.io/v1alpha1
kind: NooBaa
metadata:
  creationTimestamp: "2024-03-13T10:14:43Z"
  finalizers:
  - noobaa.io/graceful_finalizer
  generation: 1
  labels:
    app: noobaa
  name: noobaa
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: ocs.openshift.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: StorageCluster
    name: ocs-storagecluster
    uid: 374a238b-40f5-4ebc-ab38-c2dc174acd5a
  resourceVersion: "202191"
  uid: ab5d0aaf-c663-4d11-8af1-2e6b9e196a57

status:
  accounts:
    admin:
      secretRef:
        name: noobaa-admin
        namespace: openshift-storage
  actualImage: registry.redhat.io/odf4/mcg-core-rhel9@sha256:79ca4ebf33fc91115fa5d5aa79c08c81c3df7df4f302b85ce6e8f8eba9d9e1bc
  conditions:
  - lastHeartbeatTime: "2024-03-13T14:14:10Z"
    lastTransitionTime: "2024-03-13T10:14:43Z"
    message: 'Post "https://storage.googleapis.com/storage/v1/b?alt=json&prettyPrint=false&project=odf-qe":
      oauth2: cannot fetch token: Post "https://oauth2.googleapis.com/token": tls:
      failed to verify certificate: x509: certificate signed by unknown authority'
    reason: TemporaryError
    status: "False"
    type: Available
  - lastHeartbeatTime: "2024-03-13T14:14:10Z"
    lastTransitionTime: "2024-03-13T10:14:43Z"
    message: 'Post "https://storage.googleapis.com/storage/v1/b?alt=json&prettyPrint=false&project=odf-qe":
      oauth2: cannot fetch token: Post "https://oauth2.googleapis.com/token": tls:
      failed to verify certificate: x509: certificate signed by unknown authority'
    reason: TemporaryError
    status: "True"
    type: Progressing
  - lastHeartbeatTime: "2024-03-13T14:14:10Z"
    lastTransitionTime: "2024-03-13T10:14:43Z"
    message: 'Post "https://storage.googleapis.com/storage/v1/b?alt=json&prettyPrint=false&project=odf-qe":
      oauth2: cannot fetch token: Post "https://oauth2.googleapis.com/token": tls:
      failed to verify certificate: x509: certificate signed by unknown authority'
    reason: TemporaryError
    status: "False"
    type: Degraded
  - lastHeartbeatTime: "2024-03-13T14:14:10Z"
    lastTransitionTime: "2024-03-13T10:14:43Z"
    message: 'Post "https://storage.googleapis.com/storage/v1/b?alt=json&prettyPrint=false&project=odf-qe":
      oauth2: cannot fetch token: Post "https://oauth2.googleapis.com/token": tls:
      failed to verify certificate: x509: certificate signed by unknown authority'
    reason: TemporaryError
    status: "False"
    type: Upgradeable
  - lastHeartbeatTime: "2024-03-13T14:14:10Z"
    lastTransitionTime: "2024-03-13T10:14:43Z"
    status: k8s
    type: KMS-Type
  - lastHeartbeatTime: "2024-03-13T14:14:10Z"
    lastTransitionTime: "2024-03-13T10:14:44Z"
    status: Sync
    type: KMS-Status
  observedGeneration: 1
  phase: Configuring
  postgresUpdatePhase: NoUpgrade
  readme: "\n\n\tNooBaa operator is still working to reconcile this system.\n\tCheck
    out the system status.phase, status.conditions, and events with:\n\n\t\tkubectl
    -n openshift-storage describe noobaa\n\t\tkubectl -n openshift-storage get noobaa
    -o yaml\n\t\tkubectl -n openshift-storage get events --sort-by=metadata.creationTimestamp\n\n\tYou
    can wait for a specific condition with:\n\n\t\tkubectl -n openshift-storage wait
    noobaa/noobaa --for condition=available --timeout -1s\n\n\tNooBaa Core Version:
    \    master-20230920\n\tNooBaa Operator Version: 5.15.0\n"


> noobaa operator log

2024-03-13T10:18:24.351573751Z time="2024-03-13T10:18:24Z" level=info msg="Post \"https://storage.googleapis.com/storage/v1/b?alt=json&prettyPrint=false&project=odf-qe\": oauth2: cannot fetch token: Post \"https://oauth2.googleapis.com/token\": tls: failed to verify certificate: x509: certificate signed by unknown authority" sys=openshift-storage/noobaa
2024-03-13T10:18:24.351573751Z time="2024-03-13T10:18:24Z" level=info msg="SetPhase: temporary error during phase \"Configuring\"" sys=openshift-storage/noobaa
2024-03-13T10:18:24.351612396Z time="2024-03-13T10:18:24Z" level=warning msg="â³ Temporary Error: Post \"https://storage.googleapis.com/storage/v1/b?alt=json&prettyPrint=false&project=odf-qe\": oauth2: cannot fetch token: Post \"https://oauth2.googleapis.com/token\": tls: failed to verify certificate: x509: certificate signed by unknown authority" sys=openshift-storage/noobaa
2024-03-13T10:18:24.363413531Z time="2024-03-13T10:18:24Z" level=info msg="UpdateStatus: Done generation 1" sys=openshift-storage/noobaa


job: https://url.corp.redhat.com/e274e22
must gather: https://url.corp.redhat.com/b7d4175

Comment 5 Nimrod Becker 2024-03-18 13:08:12 UTC
Known issue, provided text
moving actual BZ to 4.16

Comment 6 Ben Eli 2024-05-27 10:55:32 UTC

*** This bug has been marked as a duplicate of bug 2271580 ***


Note You need to log in before you can comment on or make changes to this bug.