Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2007229

Summary:	4.8. to 4.9 upgrade hung due to cluster-version-operator - invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
Product:	OpenShift Container Platform	Reporter:	Abhishek Banerjee <abhbaner>
Component:	Cluster Version Operator	Assignee:	Over the Air Updates <aos-team-ota>
Status:	CLOSED DUPLICATE	QA Contact:	Johnny Liu <jialiu>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	4.9	CC:	aos-bugs, jokerman, lmohanty
Target Milestone:	---
Target Release:	4.10.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-09-29 16:54:45 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Abhishek Banerjee 2021-09-23 11:36:25 UTC

Description of problem:
When upgrade from 4.8.0-0.nightly-2021-09-20-145947 to 4.9.0-0.nightly-2021-09-20-203004, the upgrade hung at Working towards 4.9.0-0.nightly-2021-09-20-203004: 9 of 734 done (1% complete), check the cluster-version-operator pod

invalid configuration: no configuration has been provided, try setting 

NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-09-20-145947   True        True          32m     Working towards 4.9.0-0.nightly-2021-09-20-203004: 9 of 734 done (1% complete)


Version-Release number of the following components:
rpm -q openshift-ansible
rpm -q ansible
ansible --version
4.8.0-0.nightly-2021-09-20-145947 to 4.9.0-0.nightly-2021-09-20-203004

How reproducible:
3

Steps to Reproduce:
1. Upgrade vSphere cluster from 4.8.0-0.nightly-2021-09-20-145947 to 4.9.0-0.nightly-2021-09-20-203004
2. The cluster-version-operator pod
abhbaner@abhbaner-mac ~ % oc -n openshift-cluster-version logs cluster-version-operator-68d48647fb-dcb6s
I0923 11:15:25.357444       1 start.go:21] ClusterVersionOperator 4.9.0-202109161743.p0.git.43d63b8.assembly.stream-43d63b8
F0923 11:15:25.357893       1 start.go:24] error: error creating clients: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0xc0001ac001, 0xc0000c4000, 0xb8, 0xd0)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/klog/v2/klog.go:1021 +0xb9
k8s.io/klog/v2.(*loggingT).output(0x2ad7120, 0xc000000003, 0x0, 0x0, 0xc0003be230, 0x22e7998, 0x8, 0x18, 0x0)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/klog/v2/klog.go:970 +0x191
k8s.io/klog/v2.(*loggingT).printf(0x2ad7120, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x1c6ecc9, 0x9, 0xc000190a50, 0x1, ...)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/klog/v2/klog.go:751 +0x191
k8s.io/klog/v2.Fatalf(...)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/klog/v2/klog.go:1509
main.init.3.func1(0xc0000a6000, 0xc0003be1c0, 0x0, 0x7)
	/go/src/github.com/openshift/cluster-version-operator/cmd/start.go:24 +0x1ed
github.com/spf13/cobra.(*Command).execute(0xc0000a6000, 0xc0003be150, 0x7, 0x7, 0xc0000a6000, 0xc0003be150)
	/go/src/github.com/openshift/cluster-version-operator/vendor/github.com/spf13/cobra/command.go:854 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0x2ac3380, 0xc000000180, 0xc000066740, 0x46ef85)
	/go/src/github.com/openshift/cluster-version-operator/vendor/github.com/spf13/cobra/command.go:958 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
	/go/src/github.com/openshift/cluster-version-operator/vendor/github.com/spf13/cobra/command.go:895
main.main()
	/go/src/github.com/openshift/cluster-version-operator/cmd/main.go:26 +0x53

goroutine 19 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x2ad7120)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/klog/v2/klog.go:1164 +0x8b
created by k8s.io/klog/v2.init.0
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/klog/v2/klog.go:418 +0xdf

Actual results:
Upgrade hung

Expected results:
Upgrade should be successful

Comment 1 Lalatendu Mohanty 2021-09-29 16:54:45 UTC


*** This bug has been marked as a duplicate of bug 2006145 ***