Bug 2267857

Summary:	noobaa instance is initializating due to panic: reflect.Value.Interface: cannot return value obtained from unexported field or method
Product:	[Red Hat Storage] Red Hat OpenShift Data Foundation	Reporter:	Vijay Avuthu <vavuthu>
Component:	Multi-Cloud Object Gateway	Assignee:	Ben Eli <belimele>
Status:	CLOSED ERRATA	QA Contact:	Vijay Avuthu <vavuthu>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	4.15	CC:	belimele, dzaken, ebenahar, muagarwa, odf-bz-bot
Target Milestone:	---	Keywords:	Automation, Regression
Target Release:	ODF 4.15.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	verification-blocked
Fixed In Version:	4.15.0-157	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2024-03-19 15:33:21 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Vijay Avuthu 2024-03-05 06:34:58 UTC

Description of problem (please be detailed as possible and provide log
snippests):

GCP IPI platform, build: ocs-registry:4.15.0-155 

Resource ocs-storagecluster is in phase: Progressing

Version of all relevant components (if applicable):
ocs-registry:4.15.0-155 


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes

Is there any workaround available to the best of your knowledge?
NA

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1/1

Can this issue reproducible?
not sure

Can this issue reproduce from the UI?
Not tried

If this is a regression, please provide more details to justify this:
Yes

Steps to Reproduce:
1. install odf using ocs-ci on GCP  platform
2. check storagecluster satus
3.


Actual results:

Status:
  Conditions:
    Last Heartbeat Time:   2024-03-04T15:14:30Z
    Last Transition Time:  2024-03-04T15:14:30Z
    Message:               Version check successful
    Reason:                VersionMatched
    Status:                False
    Type:                  VersionMismatch
    Last Heartbeat Time:   2024-03-04T15:20:55Z
    Last Transition Time:  2024-03-04T15:18:57Z
    Message:               Reconcile completed successfully
    Reason:                ReconcileCompleted
    Status:                True
    Type:                  ReconcileComplete
    Last Heartbeat Time:   2024-03-04T15:18:10Z
    Last Transition Time:  2024-03-04T15:18:10Z
    Message:               Reconcile completed successfully
    Reason:                ReconcileCompleted
    Status:                True
    Type:                  Available
    Last Heartbeat Time:   2024-03-04T15:20:55Z
    Last Transition Time:  2024-03-04T15:18:11Z
    Message:               Waiting on Nooba instance to finish initialization
    Reason:                NoobaaInitializing
    Status:                True
    Type:                  Progressing
    Last Heartbeat Time:   2024-03-04T15:18:10Z
    Last Transition Time:  2024-03-04T15:14:30Z
    Message:               Reconcile completed successfully
    Reason:                ReconcileCompleted
    Status:                False
    Type:                  Degraded
    Last Heartbeat Time:   2024-03-04T15:19:15Z
    Last Transition Time:  2024-03-04T15:18:11Z
    Message:               CephCluster is creating: Processing OSD 2 on PVC "ocs-deviceset-1-data-0g67hr"
    Reason:                ClusterStateCreating
    Status:                False
    Type:                  Upgradeable
  Current Mon Count:       3


Expected results:

storagecluster should be in Ready state


Additional info:

noobaa operator log:

2024-03-04T15:20:51.229604636Z time="2024-03-04T15:20:51Z" level=info msg="âœ… Exists: Secret \"noobaa-gcp-bucket-creds\"\n"
2024-03-04T15:20:51.232068718Z panic: reflect.Value.Interface: cannot return value obtained from unexported field or method [recovered]
2024-03-04T15:20:51.232068718Z 	panic: reflect.Value.Interface: cannot return value obtained from unexported field or method
2024-03-04T15:20:51.232068718Z 
2024-03-04T15:20:51.232068718Z goroutine 2413 [running]:
2024-03-04T15:20:51.232068718Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
2024-03-04T15:20:51.232068718Z 	/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.3/pkg/internal/controller/controller.go:116 +0x1fa
2024-03-04T15:20:51.232096516Z panic({0x22946c0, 0x2d73ae0})
2024-03-04T15:20:51.232096516Z 	/usr/lib/golang/src/runtime/panic.go:884 +0x213
2024-03-04T15:20:51.232096516Z reflect.valueInterface({0x2608920?, 0xc001ccdbc0?, 0xc001eed470?}, 0x66?)
2024-03-04T15:20:51.232109747Z 	/usr/lib/golang/src/reflect/value.go:1491 +0xd8
2024-03-04T15:20:51.232109747Z reflect.Value.Interface(...)
2024-03-04T15:20:51.232118542Z 	/usr/lib/golang/src/reflect/value.go:1480
2024-03-04T15:20:51.232126381Z github.com/noobaa/noobaa-operator/v5/pkg/system.(*Reconciler).prepareGCPBackingStore(0xc000832280)
2024-03-04T15:20:51.232134578Z 	/remote-source/app/pkg/system/phase4_configuring.go:1034 +0x765
2024-03-04T15:20:51.232134578Z github.com/noobaa/noobaa-operator/v5/pkg/system.(*Reconciler).ReconcileDefaultBackingStore(0xc000832280)
2024-03-04T15:20:51.232143559Z 	/remote-source/app/pkg/system/phase4_configuring.go:720 +0x35a
2024-03-04T15:20:51.232152147Z github.com/noobaa/noobaa-operator/v5/pkg/system.(*Reconciler).ReconcilePhaseConfiguring(0xc000832280)
2024-03-04T15:20:51.232152147Z 	/remote-source/app/pkg/system/phase4_configuring.go:80 +0x12e
2024-03-04T15:20:51.232171402Z github.com/noobaa/noobaa-operator/v5/pkg/system.(*Reconciler).ReconcilePhases(0x2815542?)
2024-03-04T15:20:51.232171402Z 	/remote-source/app/pkg/system/reconciler.go:547 +0x65
2024-03-04T15:20:51.232171402Z github.com/noobaa/noobaa-operator/v5/pkg/system.(*Reconciler).Reconcile(0xc000832280)
2024-03-04T15:20:51.232171402Z 	/remote-source/app/pkg/system/reconciler.go:422 +0x33b
2024-03-04T15:20:51.232181675Z github.com/noobaa/noobaa-operator/v5/pkg/controller/noobaa.Add.func1({0xc001370ce0?, 0x40dd8a?}, {{{0xc00110c3f0?, 0x30?}, {0xc0015fe7d6?, 0x25affa0?}}})
2024-03-04T15:20:51.232198620Z 	/remote-source/app/pkg/controller/noobaa/noobaa_controller.go:53 +0xe5
2024-03-04T15:20:51.232207649Z sigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile(0x7f6c21bc61d8?, {0x2d9a008?, 0xc0020ca300?}, {{{0xc00110c3f0?, 0x0?}, {0xc0015fe7d6?, 0xc001370d48?}}})
2024-03-04T15:20:51.232240234Z 	/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.3/pkg/reconcile/reconcile.go:111 +0x43
2024-03-04T15:20:51.232240234Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x2d9eb48?, {0x2d9a008?, 0xc0020ca300?}, {{{0xc00110c3f0?, 0xb?}, {0xc0015fe7d6?, 0x0?}}})
2024-03-04T15:20:51.232272791Z 	/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.3/pkg/internal/controller/controller.go:119 +0xc8
2024-03-04T15:20:51.232272791Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000acb540, {0x2d99f60, 0xc0002f16d0}, {0x24d5d00?, 0xc00105b6a0?})
2024-03-04T15:20:51.232290081Z 	/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.3/pkg/internal/controller/controller.go:316 +0x3ca
2024-03-04T15:20:51.232298206Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000acb540, {0x2d99f60, 0xc0002f16d0})
2024-03-04T15:20:51.232306292Z 	/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.3/pkg/internal/controller/controller.go:266 +0x1d9
2024-03-04T15:20:51.232314545Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
2024-03-04T15:20:51.232314545Z 	/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.3/pkg/internal/controller/controller.go:227 +0x85
2024-03-04T15:20:51.232322429Z created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
2024-03-04T15:20:51.232322429Z 	/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.3/pkg/internal/controller/controller.go:223 +0x587

job: https://url.corp.redhat.com/4bca560
must gather: https://url.corp.redhat.com/aae2e6b

Comment 7 Vijay Avuthu 2024-03-14 15:55:20 UTC

Update
========
build: ocs-registry:4.15.0-158

Didn't see the panic in build 158 ( out of 2 times, first time deployment is successfull and second hit the https://bugzilla.redhat.com/show_bug.cgi?id=2269379 )

job: https://url.corp.redhat.com/21f8765
logs: https://url.corp.redhat.com/1033e76

2024-03-14 12:14:22  06:44:22 - MainThread - ocs_ci.ocs.resources.storage_cluster - INFO  - Check if StorageCluster: ocs-storagecluster is in Succeeded phase
2024-03-14 12:14:22  06:44:22 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc --kubeconfig /home/jenkins/current-cluster-dir/openshift-cluster-dir/auth/kubeconfig -n openshift-storage get StorageCluster ocs-storagecluster -n openshift-storage -o yaml
2024-03-14 12:14:22  06:44:22 - MainThread - ocs_ci.ocs.ocp - INFO  - Resource ocs-storagecluster is in phase: Ready!

Marking as Verified

Comment 9 errata-xmlrpc 2024-03-19 15:33:21 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.15.0 security, enhancement, & bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:1383