[GCP]: Noobaa instance fails to finish initialization due to "oauth2: cannot fetch token: Post \"https://oauth2.googleapis.com/token\": tls: failed to verify certificate: x509: certificate signed by unknown authority" sys=openshift-storage/noobaa"
Product:
[Red Hat Storage] Red Hat OpenShift Data Foundation
.Multicloud Object Gateway instance fails to finish initialization
Due to a race in timing between the pod code run and OpenShift loading the Certificate Authority (CA) bundle into the pod, the pod is unable to communicate with the cloud storage service. As a result, default backing store cannot be created.
Workaround: Restart the Multicloud Object Gateway (MCG) operator pod:
----
$ oc delete pod noobaa-operator-<ID
----
With the workaround the backing store is reconciled and works.
Description of problem (please be detailed as possible and provide log
snippests):
platform: GCP
Version of all relevant components (if applicable):
ocs-registry:4.15.0-158
Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
not able to install ODF fully
Is there any workaround available to the best of your knowledge?
No
Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1
Can this issue reproducible?
1/1
Can this issue reproduce from the UI?
Not tried
If this is a regression, please provide more details to justify this:
Yes
Steps to Reproduce:
1. install odf using ocs-ci on GCP platform
2. check storagecluster satus
Actual results:
storage cluster status
Status:
Conditions:
Last Heartbeat Time: 2024-03-13T10:11:15Z
Last Transition Time: 2024-03-13T10:11:15Z
Message: Version check successful
Reason: VersionMatched
Status: False
Type: VersionMismatch
Last Heartbeat Time: 2024-03-13T10:18:31Z
Last Transition Time: 2024-03-13T10:17:42Z
Message: Reconcile completed successfully
Reason: ReconcileCompleted
Status: True
Type: ReconcileComplete
Last Heartbeat Time: 2024-03-13T10:11:15Z
Last Transition Time: 2024-03-13T10:11:15Z
Message: Initializing StorageCluster
Reason: Init
Status: False
Type: Available
Last Heartbeat Time: 2024-03-13T10:18:31Z
Last Transition Time: 2024-03-13T10:11:15Z
Message: Waiting on Nooba instance to finish initialization
Reason: NoobaaInitializing
Status: True
Type: Progressing
Last Heartbeat Time: 2024-03-13T10:11:15Z
Last Transition Time: 2024-03-13T10:11:15Z
Message: Initializing StorageCluster
Reason: Init
Status: False
Type: Degraded
Last Heartbeat Time: 2024-03-13T10:18:01Z
Last Transition Time: 2024-03-13T10:14:38Z
Message: CephCluster is creating: Processing OSD 2 on PVC "ocs-deviceset-0-data-0hs5bk"
Reason: ClusterStateCreating
Status: False
Type: Upgradeable
Current Mon Count: 3
.
.
.
Phase: Progressing
Expected results:
storagecluster should be in Ready state
Additional info:
> noobaa
$ oc get noobaa noobaa -o yaml
apiVersion: noobaa.io/v1alpha1
kind: NooBaa
metadata:
creationTimestamp: "2024-03-13T10:14:43Z"
finalizers:
- noobaa.io/graceful_finalizer
generation: 1
labels:
app: noobaa
name: noobaa
namespace: openshift-storage
ownerReferences:
- apiVersion: ocs.openshift.io/v1
blockOwnerDeletion: true
controller: true
kind: StorageCluster
name: ocs-storagecluster
uid: 374a238b-40f5-4ebc-ab38-c2dc174acd5a
resourceVersion: "202191"
uid: ab5d0aaf-c663-4d11-8af1-2e6b9e196a57
status:
accounts:
admin:
secretRef:
name: noobaa-admin
namespace: openshift-storage
actualImage: registry.redhat.io/odf4/mcg-core-rhel9@sha256:79ca4ebf33fc91115fa5d5aa79c08c81c3df7df4f302b85ce6e8f8eba9d9e1bc
conditions:
- lastHeartbeatTime: "2024-03-13T14:14:10Z"
lastTransitionTime: "2024-03-13T10:14:43Z"
message: 'Post "https://storage.googleapis.com/storage/v1/b?alt=json&prettyPrint=false&project=odf-qe":
oauth2: cannot fetch token: Post "https://oauth2.googleapis.com/token": tls:
failed to verify certificate: x509: certificate signed by unknown authority'
reason: TemporaryError
status: "False"
type: Available
- lastHeartbeatTime: "2024-03-13T14:14:10Z"
lastTransitionTime: "2024-03-13T10:14:43Z"
message: 'Post "https://storage.googleapis.com/storage/v1/b?alt=json&prettyPrint=false&project=odf-qe":
oauth2: cannot fetch token: Post "https://oauth2.googleapis.com/token": tls:
failed to verify certificate: x509: certificate signed by unknown authority'
reason: TemporaryError
status: "True"
type: Progressing
- lastHeartbeatTime: "2024-03-13T14:14:10Z"
lastTransitionTime: "2024-03-13T10:14:43Z"
message: 'Post "https://storage.googleapis.com/storage/v1/b?alt=json&prettyPrint=false&project=odf-qe":
oauth2: cannot fetch token: Post "https://oauth2.googleapis.com/token": tls:
failed to verify certificate: x509: certificate signed by unknown authority'
reason: TemporaryError
status: "False"
type: Degraded
- lastHeartbeatTime: "2024-03-13T14:14:10Z"
lastTransitionTime: "2024-03-13T10:14:43Z"
message: 'Post "https://storage.googleapis.com/storage/v1/b?alt=json&prettyPrint=false&project=odf-qe":
oauth2: cannot fetch token: Post "https://oauth2.googleapis.com/token": tls:
failed to verify certificate: x509: certificate signed by unknown authority'
reason: TemporaryError
status: "False"
type: Upgradeable
- lastHeartbeatTime: "2024-03-13T14:14:10Z"
lastTransitionTime: "2024-03-13T10:14:43Z"
status: k8s
type: KMS-Type
- lastHeartbeatTime: "2024-03-13T14:14:10Z"
lastTransitionTime: "2024-03-13T10:14:44Z"
status: Sync
type: KMS-Status
observedGeneration: 1
phase: Configuring
postgresUpdatePhase: NoUpgrade
readme: "\n\n\tNooBaa operator is still working to reconcile this system.\n\tCheck
out the system status.phase, status.conditions, and events with:\n\n\t\tkubectl
-n openshift-storage describe noobaa\n\t\tkubectl -n openshift-storage get noobaa
-o yaml\n\t\tkubectl -n openshift-storage get events --sort-by=metadata.creationTimestamp\n\n\tYou
can wait for a specific condition with:\n\n\t\tkubectl -n openshift-storage wait
noobaa/noobaa --for condition=available --timeout -1s\n\n\tNooBaa Core Version:
\ master-20230920\n\tNooBaa Operator Version: 5.15.0\n"
> noobaa operator log
2024-03-13T10:18:24.351573751Z time="2024-03-13T10:18:24Z" level=info msg="Post \"https://storage.googleapis.com/storage/v1/b?alt=json&prettyPrint=false&project=odf-qe\": oauth2: cannot fetch token: Post \"https://oauth2.googleapis.com/token\": tls: failed to verify certificate: x509: certificate signed by unknown authority" sys=openshift-storage/noobaa
2024-03-13T10:18:24.351573751Z time="2024-03-13T10:18:24Z" level=info msg="SetPhase: temporary error during phase \"Configuring\"" sys=openshift-storage/noobaa
2024-03-13T10:18:24.351612396Z time="2024-03-13T10:18:24Z" level=warning msg="â³ Temporary Error: Post \"https://storage.googleapis.com/storage/v1/b?alt=json&prettyPrint=false&project=odf-qe\": oauth2: cannot fetch token: Post \"https://oauth2.googleapis.com/token\": tls: failed to verify certificate: x509: certificate signed by unknown authority" sys=openshift-storage/noobaa
2024-03-13T10:18:24.363413531Z time="2024-03-13T10:18:24Z" level=info msg="UpdateStatus: Done generation 1" sys=openshift-storage/noobaa
job: https://url.corp.redhat.com/e274e22
must gather: https://url.corp.redhat.com/b7d4175
Description of problem (please be detailed as possible and provide log snippests): platform: GCP Version of all relevant components (if applicable): ocs-registry:4.15.0-158 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? not able to install ODF fully Is there any workaround available to the best of your knowledge? No Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 1 Can this issue reproducible? 1/1 Can this issue reproduce from the UI? Not tried If this is a regression, please provide more details to justify this: Yes Steps to Reproduce: 1. install odf using ocs-ci on GCP platform 2. check storagecluster satus Actual results: storage cluster status Status: Conditions: Last Heartbeat Time: 2024-03-13T10:11:15Z Last Transition Time: 2024-03-13T10:11:15Z Message: Version check successful Reason: VersionMatched Status: False Type: VersionMismatch Last Heartbeat Time: 2024-03-13T10:18:31Z Last Transition Time: 2024-03-13T10:17:42Z Message: Reconcile completed successfully Reason: ReconcileCompleted Status: True Type: ReconcileComplete Last Heartbeat Time: 2024-03-13T10:11:15Z Last Transition Time: 2024-03-13T10:11:15Z Message: Initializing StorageCluster Reason: Init Status: False Type: Available Last Heartbeat Time: 2024-03-13T10:18:31Z Last Transition Time: 2024-03-13T10:11:15Z Message: Waiting on Nooba instance to finish initialization Reason: NoobaaInitializing Status: True Type: Progressing Last Heartbeat Time: 2024-03-13T10:11:15Z Last Transition Time: 2024-03-13T10:11:15Z Message: Initializing StorageCluster Reason: Init Status: False Type: Degraded Last Heartbeat Time: 2024-03-13T10:18:01Z Last Transition Time: 2024-03-13T10:14:38Z Message: CephCluster is creating: Processing OSD 2 on PVC "ocs-deviceset-0-data-0hs5bk" Reason: ClusterStateCreating Status: False Type: Upgradeable Current Mon Count: 3 . . . Phase: Progressing Expected results: storagecluster should be in Ready state Additional info: > noobaa $ oc get noobaa noobaa -o yaml apiVersion: noobaa.io/v1alpha1 kind: NooBaa metadata: creationTimestamp: "2024-03-13T10:14:43Z" finalizers: - noobaa.io/graceful_finalizer generation: 1 labels: app: noobaa name: noobaa namespace: openshift-storage ownerReferences: - apiVersion: ocs.openshift.io/v1 blockOwnerDeletion: true controller: true kind: StorageCluster name: ocs-storagecluster uid: 374a238b-40f5-4ebc-ab38-c2dc174acd5a resourceVersion: "202191" uid: ab5d0aaf-c663-4d11-8af1-2e6b9e196a57 status: accounts: admin: secretRef: name: noobaa-admin namespace: openshift-storage actualImage: registry.redhat.io/odf4/mcg-core-rhel9@sha256:79ca4ebf33fc91115fa5d5aa79c08c81c3df7df4f302b85ce6e8f8eba9d9e1bc conditions: - lastHeartbeatTime: "2024-03-13T14:14:10Z" lastTransitionTime: "2024-03-13T10:14:43Z" message: 'Post "https://storage.googleapis.com/storage/v1/b?alt=json&prettyPrint=false&project=odf-qe": oauth2: cannot fetch token: Post "https://oauth2.googleapis.com/token": tls: failed to verify certificate: x509: certificate signed by unknown authority' reason: TemporaryError status: "False" type: Available - lastHeartbeatTime: "2024-03-13T14:14:10Z" lastTransitionTime: "2024-03-13T10:14:43Z" message: 'Post "https://storage.googleapis.com/storage/v1/b?alt=json&prettyPrint=false&project=odf-qe": oauth2: cannot fetch token: Post "https://oauth2.googleapis.com/token": tls: failed to verify certificate: x509: certificate signed by unknown authority' reason: TemporaryError status: "True" type: Progressing - lastHeartbeatTime: "2024-03-13T14:14:10Z" lastTransitionTime: "2024-03-13T10:14:43Z" message: 'Post "https://storage.googleapis.com/storage/v1/b?alt=json&prettyPrint=false&project=odf-qe": oauth2: cannot fetch token: Post "https://oauth2.googleapis.com/token": tls: failed to verify certificate: x509: certificate signed by unknown authority' reason: TemporaryError status: "False" type: Degraded - lastHeartbeatTime: "2024-03-13T14:14:10Z" lastTransitionTime: "2024-03-13T10:14:43Z" message: 'Post "https://storage.googleapis.com/storage/v1/b?alt=json&prettyPrint=false&project=odf-qe": oauth2: cannot fetch token: Post "https://oauth2.googleapis.com/token": tls: failed to verify certificate: x509: certificate signed by unknown authority' reason: TemporaryError status: "False" type: Upgradeable - lastHeartbeatTime: "2024-03-13T14:14:10Z" lastTransitionTime: "2024-03-13T10:14:43Z" status: k8s type: KMS-Type - lastHeartbeatTime: "2024-03-13T14:14:10Z" lastTransitionTime: "2024-03-13T10:14:44Z" status: Sync type: KMS-Status observedGeneration: 1 phase: Configuring postgresUpdatePhase: NoUpgrade readme: "\n\n\tNooBaa operator is still working to reconcile this system.\n\tCheck out the system status.phase, status.conditions, and events with:\n\n\t\tkubectl -n openshift-storage describe noobaa\n\t\tkubectl -n openshift-storage get noobaa -o yaml\n\t\tkubectl -n openshift-storage get events --sort-by=metadata.creationTimestamp\n\n\tYou can wait for a specific condition with:\n\n\t\tkubectl -n openshift-storage wait noobaa/noobaa --for condition=available --timeout -1s\n\n\tNooBaa Core Version: \ master-20230920\n\tNooBaa Operator Version: 5.15.0\n" > noobaa operator log 2024-03-13T10:18:24.351573751Z time="2024-03-13T10:18:24Z" level=info msg="Post \"https://storage.googleapis.com/storage/v1/b?alt=json&prettyPrint=false&project=odf-qe\": oauth2: cannot fetch token: Post \"https://oauth2.googleapis.com/token\": tls: failed to verify certificate: x509: certificate signed by unknown authority" sys=openshift-storage/noobaa 2024-03-13T10:18:24.351573751Z time="2024-03-13T10:18:24Z" level=info msg="SetPhase: temporary error during phase \"Configuring\"" sys=openshift-storage/noobaa 2024-03-13T10:18:24.351612396Z time="2024-03-13T10:18:24Z" level=warning msg="â³ Temporary Error: Post \"https://storage.googleapis.com/storage/v1/b?alt=json&prettyPrint=false&project=odf-qe\": oauth2: cannot fetch token: Post \"https://oauth2.googleapis.com/token\": tls: failed to verify certificate: x509: certificate signed by unknown authority" sys=openshift-storage/noobaa 2024-03-13T10:18:24.363413531Z time="2024-03-13T10:18:24Z" level=info msg="UpdateStatus: Done generation 1" sys=openshift-storage/noobaa job: https://url.corp.redhat.com/e274e22 must gather: https://url.corp.redhat.com/b7d4175