Observed a panic in a 4.7 e2e run: https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-sdn-network-stress-4.7/1353930135601942528 E0126 05:18:03.270532 1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) goroutine 4915 [running]: k8s.io/apimachinery/pkg/util/runtime.logPanic(0x2087c40, 0x3723060) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x95 k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x89 panic(0x2087c40, 0x3723060) /usr/lib/golang/src/runtime/panic.go:969 +0x1b9 github.com/openshift/cluster-authentication-operator/pkg/controllers/deployment.(*deploymentController).updateOperatorDeploymentInfo.func1(0x0, 0x0) /go/src/github.com/openshift/cluster-authentication-operator/pkg/controllers/deployment/deployment_controller.go:373 +0x7f k8s.io/client-go/util/retry.OnError.func1(0xc003ca0b40, 0xc004442970, 0xf) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/retry/util.go:51 +0x3c k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection(0xc001ac97b8, 0x0, 0x0, 0x0) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:211 +0x69 k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff(0x989680, 0x3ff0000000000000, 0x3fb999999999999a, 0x5, 0x0, 0xc001ac97b8, 0x3, 0x3) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:399 +0x55 k8s.io/client-go/util/retry.OnError(0x989680, 0x3ff0000000000000, 0x3fb999999999999a, 0x5, 0x0, 0x2511eb8, 0xc001ac9878, 0x0, 0x0) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/retry/util.go:50 +0xa6 k8s.io/client-go/util/retry.RetryOnConflict(...) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/retry/util.go:104 github.com/openshift/cluster-authentication-operator/pkg/controllers/deployment.(*deploymentController).updateOperatorDeploymentInfo(0xc000c92cc0, 0x271b8c0, 0xc001b21040, 0x2716080, 0xc003d1df50, 0xc004529200, 0xc00443fb00, 0x0, 0x0) /go/src/github.com/openshift/cluster-authentication-operator/pkg/controllers/deployment/deployment_controller.go:368 +0x13e github.com/openshift/cluster-authentication-operator/pkg/controllers/deployment.(*deploymentController).sync(0xc000c92cc0, 0x271b8c0, 0xc001b21040, 0x2716080, 0xc003d1df50, 0x26f1fc9e8c6d218e, 0x408ccd) /go/src/github.com/openshift/cluster-authentication-operator/pkg/controllers/deployment/deployment_controller.go:178 +0x10c7 github.com/openshift/library-go/pkg/controller/factory.(*baseController).reconcile(0xc000a27ef0, 0x271b8c0, 0xc001b21040, 0x2716080, 0xc003d1df50, 0x0, 0x0) /go/src/github.com/openshift/cluster-authentication-operator/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:205 +0x76 github.com/openshift/library-go/pkg/controller/factory.(*baseController).processNextWorkItem(0xc000a27ef0, 0x271b8c0, 0xc001b21040) /go/src/github.com/openshift/cluster-authentication-operator/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:245 +0x230 github.com/openshift/library-go/pkg/controller/factory.(*baseController).runWorker.func1(0xc00234dc40, 0x271b8c0, 0xc001b21040, 0xc000a27ef0) /go/src/github.com/openshift/cluster-authentication-operator/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:196 +0x99 created by github.com/openshift/library-go/pkg/controller/factory.(*baseController).runWorker /go/src/github.com/openshift/cluster-authentication-operator/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:188 +0x8d
Looks like authentication.operator.openshift.io/cluster was not found. The panic at that point is not great, yet not terrible as something else is probably very wrong in that cluster.
https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-4.7/1355501970990829568 Separate panic E0130 13:57:28.518155 1 runtime.go:76] Observed a panic: runtime error: invalid memory address or nil pointer dereference goroutine 31873 [running]: k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1(0xc004315140) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:106 +0x113 panic(0x2088fe0, 0x3726080) /usr/lib/golang/src/runtime/panic.go:969 +0x1b9 k8s.io/apiserver/plugin/pkg/authorizer/webhook.(*WebhookAuthorizer).Authorize(0xc000adf3e0, 0x271d280, 0xc005f57530, 0x273cb40, 0xc0022bd2c0, 0x0, 0xc003e50d00, 0x16f80cf, 0x271d880, 0xc00275b940) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apiserver/plugin/pkg/authorizer/webhook/webhook.go:208 +0x8b9 k8s.io/apiserver/pkg/authorization/union.unionAuthzHandler.Authorize(0xc0007eb910, 0x1, 0x1, 0x271d280, 0xc005f57530, 0x273cb40, 0xc0022bd2c0, 0x1, 0x1, 0x23ad1e2, ...) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apiserver/pkg/authorization/union/union.go:52 +0xfe k8s.io/apiserver/pkg/authorization/union.unionAuthzHandler.Authorize(0xc0005c8ea0, 0x2, 0x2, 0x271d280, 0xc005f57530, 0x273cb40, 0xc0022bd2c0, 0x268f820, 0x1f4dd20, 0xc0044f9340, ...) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apiserver/pkg/authorization/union/union.go:52 +0xfe k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7f93d5cf1d00, 0xc0060b8068, 0xc004283900) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:59 +0x165 net/http.HandlerFunc.ServeHTTP(0xc000533d00, 0x7f93d5cf1d00, 0xc0060b8068, 0xc004283900) /usr/lib/golang/src/net/http/server.go:2054 +0x44 k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1(0x7f93d5cf1d00, 0xc0060b8068, 0xc004283900) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:71 +0x186 net/http.HandlerFunc.ServeHTTP(0xc000533d40, 0x7f93d5cf1d00, 0xc0060b8068, 0xc004283900) /usr/lib/golang/src/net/http/server.go:2054 +0x44 k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7f93d5cf1d00, 0xc0060b8068, 0xc004283900) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:184 +0x4cf net/http.HandlerFunc.ServeHTTP(0xc00098dbc0, 0x7f93d5cf1d00, 0xc0060b8068, 0xc004283900) /usr/lib/golang/src/net/http/server.go:2054 +0x44 k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1(0x7f93d5cf1d00, 0xc0060b8068, 0xc004283900) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:95 +0x165 net/http.HandlerFunc.ServeHTTP(0xc00098dbf0, 0x7f93d5cf1d00, 0xc0060b8068, 0xc004283900) /usr/lib/golang/src/net/http/server.go:2054 +0x44 k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7f93d5cf1d00, 0xc0060b8068, 0xc004283900) Looks like the same failure as https://bugzilla.redhat.com/show_bug.cgi?id=1913525 Opening a separate bug https://bugzilla.redhat.com/show_bug.cgi?id=1922646
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633