Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1814328

Summary:	Cloud credential operator goes into crash loop when a 4.1 cluster is upgraded
Product:	OpenShift Container Platform	Reporter:	Clayton Coleman <ccoleman>
Component:	Cloud Credential Operator	Assignee:	Devan Goodwin <dgoodwin>
Status:	CLOSED DUPLICATE	QA Contact:	wang lin <lwan>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	4.4
Target Milestone:	---
Target Release:	4.4.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-03-17 20:44:09 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Clayton Coleman 2020-03-17 16:42:30 UTC

Release blocker

https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.1-to-4.2-to-4.3-to-4.4-nightly/22

E0317 06:49:40.767169       1 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/go/src/github.com/openshift/cloud-credential-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/go/src/github.com/openshift/cloud-credential-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/github.com/openshift/cloud-credential-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/opt/rh/go-toolset-1.13/root/usr/lib/go-toolset-1.13-golang/src/runtime/panic.go:679
/opt/rh/go-toolset-1.13/root/usr/lib/go-toolset-1.13-golang/src/runtime/panic.go:199
/opt/rh/go-toolset-1.13/root/usr/lib/go-toolset-1.13-golang/src/runtime/signal_unix.go:394
/go/src/github.com/openshift/cloud-credential-operator/pkg/controller/utils/utils.go:73
/go/src/github.com/openshift/cloud-credential-operator/pkg/controller/secretannotator/aws/reconciler.go:137
/go/src/github.com/openshift/cloud-credential-operator/pkg/controller/secretannotator/aws/reconciler.go:115
/go/src/github.com/openshift/cloud-credential-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:216
/go/src/github.com/openshift/cloud-credential-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:192
/go/src/github.com/openshift/cloud-credential-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:171
/go/src/github.com/openshift/cloud-credential-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152
/go/src/github.com/openshift/cloud-credential-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153
/go/src/github.com/openshift/cloud-credential-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/opt/rh/go-toolset-1.13/root/usr/lib/go-toolset-1.13-golang/src/runtime/asm_amd64.s:1357
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x1143d59]

goroutine 824 [running]:
github.com/openshift/cloud-credential-operator/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/openshift/cloud-credential-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x105
panic(0x174a740, 0x2ad5810)
	/opt/rh/go-toolset-1.13/root/usr/lib/go-toolset-1.13-golang/src/runtime/panic.go:679 +0x1b2
github.com/openshift/cloud-credential-operator/pkg/controller/utils.LoadInfrastructureRegion(0x1c8ebc0, 0xc00082c7b0, 0x1cad480, 0xc00036d560, 0xc0027fdac0, 0x1, 0x1596c71, 0x6)
	/go/src/github.com/openshift/cloud-credential-operator/pkg/controller/utils/utils.go:73 +0xf9
github.com/openshift/cloud-credential-operator/pkg/controller/secretannotator/aws.(*ReconcileCloudCredSecret).validateCloudCredsSecret(0xc00070aea0, 0xc0062012c0, 0xc0000420e0, 0xc003028750)
	/go/src/github.com/openshift/cloud-credential-operator/pkg/controller/secretannotator/aws/reconciler.go:137 +0x133
github.com/openshift/cloud-credential-operator/pkg/controller/secretannotator/aws.(*ReconcileCloudCredSecret).Reconcile(0xc00070aea0, 0xc003028750, 0xb, 0xc003028730, 0x9, 0xc0067ffc00, 0x0, 0x0, 0x0)
	/go/src/github.com/openshift/cloud-credential-operator/pkg/controller/secretannotator/aws/reconciler.go:115 +0x352

Comment 1 Clayton Coleman 2020-03-17 16:45:43 UTC

There are 300 clusters that started at 4.1 in telemetry

count(count by (_id) (cluster_version{type="cluster",version=~"4\\.1.*"} or cluster_version{type="initial",version=~"4\\.1.*"})) = 300

Comment 2 Scott Dodson 2020-03-17 20:44:09 UTC


*** This bug has been marked as a duplicate of bug 1813998 ***

Comment 3 W. Trevor King 2021-04-05 17:47:18 UTC

Removing UpgradeBlocker from this older bug, to remove it from the suspect queue described in [1].  If you feel like this bug still needs to be a suspect, please add keyword again.

[1]: https://github.com/openshift/enhancements/pull/475