Bug 1870548

Summary: panic: "invalid memory address or nil pointer dereference" for machine-config operator
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: Machine Config OperatorAssignee: Antonio Murdaca <amurdaca>
Status: CLOSED DUPLICATE QA Contact: Michael Nguyen <mnguyen>
Severity: high Docs Contact:
Priority: high    
Version: 4.6   
Target Milestone: ---   
Target Release: 4.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-08-20 11:13:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Junqi Zhao 2020-08-20 10:30:57 UTC 
Description of problem:
upgrade from 4.1.38-x86_64 to 4.2.36-x86_64,4.3.33-x86_64,4.4.17-x86_64,4.5.6-x86_64,4.6.0-0.nightly-2020-08-18-165040
the upgrade is successful before upgrade to 4.6.0-0.nightly-2020-08-18-165040
meet the panic: "invalid memory address or nil pointer dereference" for machine-config operator when upgrade from 4.5.6 to 4.6.0-0.nightly-2020-08-18-165040

# oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.6     True        True          154m    Unable to apply 4.6.0-0.nightly-2020-08-18-165040: the cluster operator machine-config has not yet successfully rolled out

# oc -n openshift-machine-config-operator  get pod
NAME                                         READY   STATUS             RESTARTS   AGE
etcd-quorum-guard-798bc99559-2jthg           1/1     Running            0          3h4m
etcd-quorum-guard-798bc99559-clwbt           1/1     Running            0          3h4m
etcd-quorum-guard-798bc99559-kwx57           1/1     Running            0          3h4m
machine-config-controller-7cb4d7cf6b-pvv5g   1/1     Running            0          104m
machine-config-daemon-n27t6                  2/2     Running            0          105m
machine-config-daemon-pffhc                  2/2     Running            0          104m
machine-config-daemon-pmqgr                  2/2     Running            0          105m
machine-config-daemon-rqzpj                  2/2     Running            0          104m
machine-config-daemon-tq6k5                  2/2     Running            0          104m
machine-config-daemon-wv7rj                  2/2     Running            0          105m
machine-config-operator-d9f77ddf5-fghdg      0/1     CrashLoopBackOff   18         107m
machine-config-server-bt8ld                  1/1     Running            0          3h22m
machine-config-server-vqh56                  1/1     Running            0          3h22m
machine-config-server-wv4lg                  1/1     Running            0          3h22m


# oc -n openshift-machine-config-operator logs machine-config-operator-d9f77ddf5-fghdg -c machine-config-operator
I0820 10:13:38.626542       1 start.go:46] Version: 4.6.0-0.nightly-2020-08-18-165040 (Raw: v4.6.0-202008180246.p0-dirty, Hash: b112e80508e3389a556e7c88e7e74608977a871a)
I0820 10:13:38.628476       1 leaderelection.go:242] attempting to acquire leader lease  openshift-machine-config-operator/machine-config...
I0820 10:15:36.440300       1 leaderelection.go:252] successfully acquired lease openshift-machine-config-operator/machine-config
I0820 10:15:37.270556       1 operator.go:270] Starting MachineConfigOperator
E0820 10:15:39.471065       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 247 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x18151e0, 0x2a37200)
	/go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa3
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x82
panic(0x18151e0, 0x2a37200)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/runtime/panic.go:969 +0x166
github.com/openshift/machine-config-operator/lib/resourcemerge.ensureControllerConfigSpec(0xc000cbb80f, 0xc00003d018, 0xc000ed9660, 0xb, 0x0, 0x0, 0xc000ed9788, 0x3, 0xc000fdc510, 0x29, ...)
	/go/src/github.com/openshift/machine-config-operator/lib/resourcemerge/machineconfig.go:83 +0x19f
github.com/openshift/machine-config-operator/lib/resourcemerge.EnsureControllerConfig(0xc000cbb80f, 0xc00003cf00, 0x16c3311, 0x10, 0xc000fdc5a0, 0x24, 0xc000fb5c40, 0x19, 0x0, 0x0, ...)
	/go/src/github.com/openshift/machine-config-operator/lib/resourcemerge/machineconfig.go:19 +0xd4
github.com/openshift/machine-config-operator/lib/resourceapply.ApplyControllerConfig(0x7f1547e7efd8, 0xc0002fda40, 0xc00003cc80, 0x7f1547e7efd8, 0xc0002fda40, 0x5ac9, 0x5b42)
	/go/src/github.com/openshift/machine-config-operator/lib/resourceapply/machineconfig.go:67 +0x185
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).syncMachineConfigController(0xc000182900, 0xc0009d0000, 0xc01959b7dc, 0xa1e5d8cb54b)
	/go/src/github.com/openshift/machine-config-operator/pkg/operator/sync.go:468 +0x438
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).syncAll(0xc000182900, 0xc000cbbca8, 0x6, 0x6, 0xc0003b4c01, 0x413893)
	/go/src/github.com/openshift/machine-config-operator/pkg/operator/sync.go:69 +0x177
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).sync(0xc000182900, 0xc0006f53b0, 0x30, 0x0, 0x0)
	/go/src/github.com/openshift/machine-config-operator/pkg/operator/operator.go:362 +0x40a
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).processNextWorkItem(0xc000182900, 0x203000)
	/go/src/github.com/openshift/machine-config-operator/pkg/operator/operator.go:318 +0xd2
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).worker(0xc000182900)
	/go/src/github.com/openshift/machine-config-operator/pkg/operator/operator.go:307 +0x2b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0008be020)
	/go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0008be020, 0x1cc05a0, 0xc000b66030, 0xc00041a001, 0xc0004581e0)
	/go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xa3
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0008be020, 0x3b9aca00, 0x0, 0x4946495452454301, 0xc0004581e0)
	/go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0xe2
k8s.io/apimachinery/pkg/util/wait.Until(0xc0008be020, 0x3b9aca00, 0xc0004581e0)
	/go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
created by github.com/openshift/machine-config-operator/pkg/operator.(*Operator).Run
	/go/src/github.com/openshift/machine-config-operator/pkg/operator/operator.go:276 +0x3dc
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1a8 pc=0x13ab99f]

goroutine 247 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x105
panic(0x18151e0, 0x2a37200)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/runtime/panic.go:969 +0x166
github.com/openshift/machine-config-operator/lib/resourcemerge.ensureControllerConfigSpec(0xc000cbb80f, 0xc00003d018, 0xc000ed9660, 0xb, 0x0, 0x0, 0xc000ed9788, 0x3, 0xc000fdc510, 0x29, ...)
	/go/src/github.com/openshift/machine-config-operator/lib/resourcemerge/machineconfig.go:83 +0x19f
github.com/openshift/machine-config-operator/lib/resourcemerge.EnsureControllerConfig(0xc000cbb80f, 0xc00003cf00, 0x16c3311, 0x10, 0xc000fdc5a0, 0x24, 0xc000fb5c40, 0x19, 0x0, 0x0, ...)
	/go/src/github.com/openshift/machine-config-operator/lib/resourcemerge/machineconfig.go:19 +0xd4
github.com/openshift/machine-config-operator/lib/resourceapply.ApplyControllerConfig(0x7f1547e7efd8, 0xc0002fda40, 0xc00003cc80, 0x7f1547e7efd8, 0xc0002fda40, 0x5ac9, 0x5b42)
	/go/src/github.com/openshift/machine-config-operator/lib/resourceapply/machineconfig.go:67 +0x185
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).syncMachineConfigController(0xc000182900, 0xc0009d0000, 0xc01959b7dc, 0xa1e5d8cb54b)
	/go/src/github.com/openshift/machine-config-operator/pkg/operator/sync.go:468 +0x438
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).syncAll(0xc000182900, 0xc000cbbca8, 0x6, 0x6, 0xc0003b4c01, 0x413893)
	/go/src/github.com/openshift/machine-config-operator/pkg/operator/sync.go:69 +0x177
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).sync(0xc000182900, 0xc0006f53b0, 0x30, 0x0, 0x0)
	/go/src/github.com/openshift/machine-config-operator/pkg/operator/operator.go:362 +0x40a
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).processNextWorkItem(0xc000182900, 0x203000)
	/go/src/github.com/openshift/machine-config-operator/pkg/operator/operator.go:318 +0xd2
github.com/openshift/machine-config-operator/pkg/operator.(*Operator).worker(0xc000182900)
	/go/src/github.com/openshift/machine-config-operator/pkg/operator/operator.go:307 +0x2b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0008be020)
	/go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0008be020, 0x1cc05a0, 0xc000b66030, 0xc00041a001, 0xc0004581e0)
	/go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xa3
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0008be020, 0x3b9aca00, 0x0, 0x4946495452454301, 0xc0004581e0)
	/go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0xe2
k8s.io/apimachinery/pkg/util/wait.Until(0xc0008be020, 0x3b9aca00, 0xc0004581e0)
	/go/src/github.com/openshift/machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
created by github.com/openshift/machine-config-operator/pkg/operator.(*Operator).Run
	/go/src/github.com/openshift/machine-config-operator/pkg/operator/operator.go:276 +0x3dc

Version-Release number of selected component (if applicable):
upgrade from 4.5.6 to 4.6.0-0.nightly-2020-08-18-165040

How reproducible:
not sure

Steps to Reproduce:
1. see the description
2.
3.

Actual results:
meet the panic: "invalid memory address or nil pointer dereference" for machine-config operator

Expected results:
no error

Additional info:
Comment 2 Antonio Murdaca 2020-08-20 11:13:15 UTC 

*** This bug has been marked as a duplicate of bug 1861404 ***
Comment 3 W. Trevor King 2021-04-05 17:46:47 UTC 
Removing UpgradeBlocker from this older bug, to remove it from the suspect queue described in [1].  If you feel like this bug still needs to be a suspect, please add keyword again.

[1]: https://github.com/openshift/enhancements/pull/475