Bug 1921275

Summary: Panic in authentication-operator in (*deploymentController).updateOperatorDeploymentInfo
Product: OpenShift Container Platform Reporter: Clayton Coleman <ccoleman>
Component: apiserver-authAssignee: Standa Laznicka <slaznick>
Status: CLOSED ERRATA QA Contact: pmali
Severity: medium Docs Contact:
Priority: medium    
Version: 4.7CC: aos-bugs, mfojtik, sttts, wking
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-24 15:56:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Clayton Coleman 2021-01-27 20:14:11 UTC
Observed a panic in a 4.7 e2e run:

https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-sdn-network-stress-4.7/1353930135601942528

E0126 05:18:03.270532       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 4915 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x2087c40, 0x3723060)
	/go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x95
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x89
panic(0x2087c40, 0x3723060)
	/usr/lib/golang/src/runtime/panic.go:969 +0x1b9
github.com/openshift/cluster-authentication-operator/pkg/controllers/deployment.(*deploymentController).updateOperatorDeploymentInfo.func1(0x0, 0x0)
	/go/src/github.com/openshift/cluster-authentication-operator/pkg/controllers/deployment/deployment_controller.go:373 +0x7f
k8s.io/client-go/util/retry.OnError.func1(0xc003ca0b40, 0xc004442970, 0xf)
	/go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/retry/util.go:51 +0x3c
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection(0xc001ac97b8, 0x0, 0x0, 0x0)
	/go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:211 +0x69
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff(0x989680, 0x3ff0000000000000, 0x3fb999999999999a, 0x5, 0x0, 0xc001ac97b8, 0x3, 0x3)
	/go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:399 +0x55
k8s.io/client-go/util/retry.OnError(0x989680, 0x3ff0000000000000, 0x3fb999999999999a, 0x5, 0x0, 0x2511eb8, 0xc001ac9878, 0x0, 0x0)
	/go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/retry/util.go:50 +0xa6
k8s.io/client-go/util/retry.RetryOnConflict(...)
	/go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/retry/util.go:104
github.com/openshift/cluster-authentication-operator/pkg/controllers/deployment.(*deploymentController).updateOperatorDeploymentInfo(0xc000c92cc0, 0x271b8c0, 0xc001b21040, 0x2716080, 0xc003d1df50, 0xc004529200, 0xc00443fb00, 0x0, 0x0)
	/go/src/github.com/openshift/cluster-authentication-operator/pkg/controllers/deployment/deployment_controller.go:368 +0x13e
github.com/openshift/cluster-authentication-operator/pkg/controllers/deployment.(*deploymentController).sync(0xc000c92cc0, 0x271b8c0, 0xc001b21040, 0x2716080, 0xc003d1df50, 0x26f1fc9e8c6d218e, 0x408ccd)
	/go/src/github.com/openshift/cluster-authentication-operator/pkg/controllers/deployment/deployment_controller.go:178 +0x10c7
github.com/openshift/library-go/pkg/controller/factory.(*baseController).reconcile(0xc000a27ef0, 0x271b8c0, 0xc001b21040, 0x2716080, 0xc003d1df50, 0x0, 0x0)
	/go/src/github.com/openshift/cluster-authentication-operator/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:205 +0x76
github.com/openshift/library-go/pkg/controller/factory.(*baseController).processNextWorkItem(0xc000a27ef0, 0x271b8c0, 0xc001b21040)
	/go/src/github.com/openshift/cluster-authentication-operator/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:245 +0x230
github.com/openshift/library-go/pkg/controller/factory.(*baseController).runWorker.func1(0xc00234dc40, 0x271b8c0, 0xc001b21040, 0xc000a27ef0)
	/go/src/github.com/openshift/cluster-authentication-operator/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:196 +0x99
created by github.com/openshift/library-go/pkg/controller/factory.(*baseController).runWorker
	/go/src/github.com/openshift/cluster-authentication-operator/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:188 +0x8d

Comment 1 Standa Laznicka 2021-01-28 08:57:33 UTC
Looks like authentication.operator.openshift.io/cluster was not found. The panic at that point is not great, yet not terrible as something else is probably very wrong in that cluster.

Comment 3 Clayton Coleman 2021-01-30 18:25:20 UTC
https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-4.7/1355501970990829568

Separate panic

E0130 13:57:28.518155       1 runtime.go:76] Observed a panic: runtime error: invalid memory address or nil pointer dereference
goroutine 31873 [running]:
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1(0xc004315140)
	/go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:106 +0x113
panic(0x2088fe0, 0x3726080)
	/usr/lib/golang/src/runtime/panic.go:969 +0x1b9
k8s.io/apiserver/plugin/pkg/authorizer/webhook.(*WebhookAuthorizer).Authorize(0xc000adf3e0, 0x271d280, 0xc005f57530, 0x273cb40, 0xc0022bd2c0, 0x0, 0xc003e50d00, 0x16f80cf, 0x271d880, 0xc00275b940)
	/go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apiserver/plugin/pkg/authorizer/webhook/webhook.go:208 +0x8b9
k8s.io/apiserver/pkg/authorization/union.unionAuthzHandler.Authorize(0xc0007eb910, 0x1, 0x1, 0x271d280, 0xc005f57530, 0x273cb40, 0xc0022bd2c0, 0x1, 0x1, 0x23ad1e2, ...)
	/go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apiserver/pkg/authorization/union/union.go:52 +0xfe
k8s.io/apiserver/pkg/authorization/union.unionAuthzHandler.Authorize(0xc0005c8ea0, 0x2, 0x2, 0x271d280, 0xc005f57530, 0x273cb40, 0xc0022bd2c0, 0x268f820, 0x1f4dd20, 0xc0044f9340, ...)
	/go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apiserver/pkg/authorization/union/union.go:52 +0xfe
k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7f93d5cf1d00, 0xc0060b8068, 0xc004283900)
	/go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:59 +0x165
net/http.HandlerFunc.ServeHTTP(0xc000533d00, 0x7f93d5cf1d00, 0xc0060b8068, 0xc004283900)
	/usr/lib/golang/src/net/http/server.go:2054 +0x44
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1(0x7f93d5cf1d00, 0xc0060b8068, 0xc004283900)
	/go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:71 +0x186
net/http.HandlerFunc.ServeHTTP(0xc000533d40, 0x7f93d5cf1d00, 0xc0060b8068, 0xc004283900)
	/usr/lib/golang/src/net/http/server.go:2054 +0x44
k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7f93d5cf1d00, 0xc0060b8068, 0xc004283900)
	/go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:184 +0x4cf
net/http.HandlerFunc.ServeHTTP(0xc00098dbc0, 0x7f93d5cf1d00, 0xc0060b8068, 0xc004283900)
	/usr/lib/golang/src/net/http/server.go:2054 +0x44
k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1(0x7f93d5cf1d00, 0xc0060b8068, 0xc004283900)
	/go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:95 +0x165
net/http.HandlerFunc.ServeHTTP(0xc00098dbf0, 0x7f93d5cf1d00, 0xc0060b8068, 0xc004283900)
	/usr/lib/golang/src/net/http/server.go:2054 +0x44
k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7f93d5cf1d00, 0xc0060b8068, 0xc004283900)

Looks like the same failure as https://bugzilla.redhat.com/show_bug.cgi?id=1913525

Opening a separate bug https://bugzilla.redhat.com/show_bug.cgi?id=1922646

Comment 7 errata-xmlrpc 2021-02-24 15:56:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633