1870548 – panic: "invalid memory address or nil pointer dereference" for machine-config operator

Bug 1870548 - panic: "invalid memory address or nil pointer dereference" for machine-config operator

Summary: panic: "invalid memory address or nil pointer dereference" for machine-confi...

Keywords:
Status:	CLOSED DUPLICATE of bug 1861404
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Machine Config Operator
Sub Component:
Version:	4.6
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	4.6.0
Assignee:	Antonio Murdaca
QA Contact:	Michael Nguyen
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2020-08-20 10:30 UTC by Junqi Zhao
Modified:	2021-04-05 17:46 UTC (History)
CC List:	0 users
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-08-20 11:13:15 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Junqi Zhao 2020-08-20 10:30:57 UTC

Description of problem:
upgrade from 4.1.38-x86_64 to 4.2.36-x86_64,4.3.33-x86_64,4.4.17-x86_64,4.5.6-x86_64,4.6.0-0.nightly-2020-08-18-165040
the upgrade is successful before upgrade to 4.6.0-0.nightly-2020-08-18-165040
meet the panic: "invalid memory address or nil pointer dereference" for machine-config operator when upgrade from 4.5.6 to 4.6.0-0.nightly-2020-08-18-165040

# oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.6     True        True          154m    Unable to apply 4.6.0-0.nightly-2020-08-18-165040: the cluster operator machine-config has not yet successfully rolled out

# oc -n openshift-machine-config-operator  get pod
NAME                                         READY   STATUS             RESTARTS   AGE
etcd-quorum-guard-798bc99559-2jthg           1/1     Running            0          3h4m
etcd-quorum-guard-798bc99559-clwbt           1/1     Running            0          3h4m
etcd-quorum-guard-798bc99559-kwx57           1/1     Running            0          3h4m
machine-config-controller-7cb4d7cf6b-pvv5g   1/1     Running            0          104m
machine-config-daemon-n27t6                  2/2     Running            0          105m
machine-config-daemon-pffhc                  2/2     Running            0          104m
machine-config-daemon-pmqgr                  2/2     Running            0          105m
machine-config-daemon-rqzpj                  2/2     Running            0          104m
machine-config-daemon-tq6k5                  2/2     Running            0          104m
machine-config-daemon-wv7rj                  2/2     Running            0          105m
machine-config-operator-d9f77ddf5-fghdg      0/1     CrashLoopBackOff   18         107m
machine-config-server-bt8ld                  1/1     Running            0          3h22m
machine-config-server-vqh56                  1/1     Running            0          3h22m
machine-config-server-wv4lg                  1/1     Running            0          3h22m


# oc -n openshift-machine-config-operator logs machine-config-operator-d9f77ddf5-fghdg -c machine-config-operator
I0820 10:13:38.626542       1 start.go:46] Version: 4.6.0-0.nightly-2020-08-18-165040 (Raw: v4.6.0-202008180246.p0-dirty, Hash: b112e80508e3389a556e7c88e7e74608977a871a)
I0820 10:13:38.628476       1 leaderelection.go:242] attempting to acquire leader lease  openshift-machine-config-operator/machine-config...
I0820 10:15:36.440300       1 leaderelection.go:252] successfully acquired lease openshift-machine-config-operator/machine-config
I0820 10:15:37.270556       1 operator.go:270] Starting MachineConfigOperator
E0820 10:15:39.471065       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 247 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x18151e0, 0x2a37200)
	/go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa3
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x82
panic(0x18151e0, 0x2a37200)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/runtime/panic.go:969 +0x166
github.com/openshift/machine-config-operator/lib/resourcemerge.ensureControllerConfigSpec(0xc000cbb80f, 0xc00003d018, 0xc000ed9660, 0xb, 0x0, 0x0, 0xc000ed9788, 0x3, 0xc000fdc510, 0x29, ...)
	/go/src/github.com/openshift/machine-config-operator/lib/resourcemerge/machineconfig.go:83 +0x19f
github.com/openshift/machine-config-operator/lib/resourcemerge.EnsureControllerConfig(0xc000cbb80f, 0xc00003cf00, 0x16c3311, 0x10, 0xc000fdc5a0, 0x24, 0xc000fb5c40, 0x19, 0x0, 0x0, ...)
	/go/src/github.com/openshift/machine-config-operator/lib/resourcemerge/machineconfig.go:19 +0xd4
github.com/openshift/machine-config-operator/lib/resourceapply.ApplyControllerConfig(0x7f1547e7efd8, 0xc0002fda40, 0xc00003cc80, 0x7f1547e7efd8, 0xc0002fda40, 0x5ac9, 0x5b42)
	/go/src/github.com/openshift/machine-config-operator/lib/resourceapply/machineconfig.go:67 +0x185
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).syncMachineConfigController(0xc000182900, 0xc0009d0000, 0xc01959b7dc, 0xa1e5d8cb54b)
	/go/src/github.com/openshift/machine-config-operator/pkg/operator/sync.go:468 +0x438
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).syncAll(0xc000182900, 0xc000cbbca8, 0x6, 0x6, 0xc0003b4c01, 0x413893)
	/go/src/github.com/openshift/machine-config-operator/pkg/operator/sync.go:69 +0x177
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).sync(0xc000182900, 0xc0006f53b0, 0x30, 0x0, 0x0)
	/go/src/github.com/openshift/machine-config-operator/pkg/operator/operator.go:362 +0x40a
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).processNextWorkItem(0xc000182900, 0x203000)
	/go/src/github.com/openshift/machine-config-operator/pkg/operator/operator.go:318 +0xd2
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).worker(0xc000182900)
	/go/src/github.com/openshift/machine-config-operator/pkg/operator/operator.go:307 +0x2b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0008be020)
	/go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0008be020, 0x1cc05a0, 0xc000b66030, 0xc00041a001, 0xc0004581e0)
	/go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xa3
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0008be020, 0x3b9aca00, 0x0, 0x4946495452454301, 0xc0004581e0)
	/go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0xe2
k8s.io/apimachinery/pkg/util/wait.Until(0xc0008be020, 0x3b9aca00, 0xc0004581e0)
	/go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
created by github.com/openshift/machine-config-operator/pkg/operator.(*Operator).Run
	/go/src/github.com/openshift/machine-config-operator/pkg/operator/operator.go:276 +0x3dc
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1a8 pc=0x13ab99f]

goroutine 247 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x105
panic(0x18151e0, 0x2a37200)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/runtime/panic.go:969 +0x166
github.com/openshift/machine-config-operator/lib/resourcemerge.ensureControllerConfigSpec(0xc000cbb80f, 0xc00003d018, 0xc000ed9660, 0xb, 0x0, 0x0, 0xc000ed9788, 0x3, 0xc000fdc510, 0x29, ...)
	/go/src/github.com/openshift/machine-config-operator/lib/resourcemerge/machineconfig.go:83 +0x19f
github.com/openshift/machine-config-operator/lib/resourcemerge.EnsureControllerConfig(0xc000cbb80f, 0xc00003cf00, 0x16c3311, 0x10, 0xc000fdc5a0, 0x24, 0xc000fb5c40, 0x19, 0x0, 0x0, ...)
	/go/src/github.com/openshift/machine-config-operator/lib/resourcemerge/machineconfig.go:19 +0xd4
github.com/openshift/machine-config-operator/lib/resourceapply.ApplyControllerConfig(0x7f1547e7efd8, 0xc0002fda40, 0xc00003cc80, 0x7f1547e7efd8, 0xc0002fda40, 0x5ac9, 0x5b42)
	/go/src/github.com/openshift/machine-config-operator/lib/resourceapply/machineconfig.go:67 +0x185
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).syncMachineConfigController(0xc000182900, 0xc0009d0000, 0xc01959b7dc, 0xa1e5d8cb54b)
	/go/src/github.com/openshift/machine-config-operator/pkg/operator/sync.go:468 +0x438
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).syncAll(0xc000182900, 0xc000cbbca8, 0x6, 0x6, 0xc0003b4c01, 0x413893)
	/go/src/github.com/openshift/machine-config-operator/pkg/operator/sync.go:69 +0x177
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).sync(0xc000182900, 0xc0006f53b0, 0x30, 0x0, 0x0)
	/go/src/github.com/openshift/machine-config-operator/pkg/operator/operator.go:362 +0x40a
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).processNextWorkItem(0xc000182900, 0x203000)
	/go/src/github.com/openshift/machine-config-operator/pkg/operator/operator.go:318 +0xd2
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).worker(0xc000182900)
	/go/src/github.com/openshift/machine-config-operator/pkg/operator/operator.go:307 +0x2b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0008be020)
	/go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0008be020, 0x1cc05a0, 0xc000b66030, 0xc00041a001, 0xc0004581e0)
	/go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xa3
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0008be020, 0x3b9aca00, 0x0, 0x4946495452454301, 0xc0004581e0)
	/go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0xe2
k8s.io/apimachinery/pkg/util/wait.Until(0xc0008be020, 0x3b9aca00, 0xc0004581e0)
	/go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
created by github.com/openshift/machine-config-operator/pkg/operator.(*Operator).Run
	/go/src/github.com/openshift/machine-config-operator/pkg/operator/operator.go:276 +0x3dc

Version-Release number of selected component (if applicable):
upgrade from 4.5.6 to 4.6.0-0.nightly-2020-08-18-165040

How reproducible:
not sure

Steps to Reproduce:
1. see the description
2.
3.

Actual results:
meet the panic: "invalid memory address or nil pointer dereference" for machine-config operator

Expected results:
no error

Additional info:

Comment 2 Antonio Murdaca 2020-08-20 11:13:15 UTC


*** This bug has been marked as a duplicate of bug 1861404 ***

Comment 3 W. Trevor King 2021-04-05 17:46:47 UTC

Removing UpgradeBlocker from this older bug, to remove it from the suspect queue described in [1].  If you feel like this bug still needs to be a suspect, please add keyword again.

[1]: https://github.com/openshift/enhancements/pull/475

Note You need to log in before you can comment on or make changes to this bug.