This bug was initially created as a copy of Bug #1720770 I am copying this bug because: Document URL: https://docs.openshift.com/container-platform/4.1/authentication/certificates/api-server.html#add-default-api-server_api-server-certificates Section Number and Name: Describe the issue: Customer followed instructions for setting the default API server serving certificate. The API server is not presenting the certificate and one of the kube-apiserver pods is stuck in CrashLoopBackoff. Suggestions for improvement: Additional information: On the customer's cluster, they have 3 masters. In the process of applying the change, the first instance of the kube-apiserver pod is stuck in CrashLoopBackoff. Investigating the logs of the kube-apiserver container reveals that the container is unable to find the serving cert. See stack trace below. I can reproduce the same problem on my AWS cluster as well. /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/cmd/hypershift/main.go:45 +0x2b8 E0614 15:04:19.542054 1 pathrecorder.go:107] registered "/readyz/crd-informer-synced" from goroutine 1 [running]: runtime/debug.Stack(0x4e8a780, 0xc008619fb0, 0xc002860260) /opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/debug/stack.go:24 +0xa7 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).trackCallers(0xc001ece690, 0xc002860260, 0x1b) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:109 +0x89 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).Handle(0xc001ece690, 0xc002860260, 0x1b, 0xaa2ed40, 0xc00b8c3880) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:173 +0x86 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/healthz.InstallPathHandler(0xaa27f80, 0xc001ece690, 0x5969c5c, 0x7, 0xc0056ede00, 0x1d, 0x1e) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:120 +0x3c1 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*GenericAPIServer).installReadyz(0xc00638edc0, 0xc0003bc6c0) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/healthz.go:55 +0x1a8 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.preparedGenericAPIServer.Run(0xc00638edc0, 0xc0003bc6c0, 0xc00638edc0, 0x0) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/genericapiserver.go:290 +0x88 github.com/openshift/origin/vendor/k8s.io/kubernetes/cmd/kube-apiserver/app.Run(0xc0001ff8c0, 0xc0003bc6c0, 0x0, 0x0) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:153 +0x145 github.com/openshift/origin/vendor/k8s.io/kubernetes/cmd/kube-apiserver/app.NewAPIServerCommand.func1(0xc001228780, 0x0, 0x0, 0x0, 0x1, 0x0) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:115 +0x109 github.com/openshift/origin/pkg/cmd/openshift-kube-apiserver.RunOpenShiftKubeAPIServerServer(0xc001334800, 0xc0003bc6c0, 0x24, 0x7ffda2d38c27) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/openshift-kube-apiserver/server.go:70 +0x57e github.com/openshift/origin/pkg/cmd/openshift-kube-apiserver.(*OpenShiftKubeAPIServerServer).RunAPIServer(0xc0003dde60, 0xc0003bc6c0, 0xc000aa38c0, 0xc000519800) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/openshift-kube-apiserver/cmd.go:129 +0x817 github.com/openshift/origin/pkg/cmd/openshift-kube-apiserver.NewOpenShiftKubeAPIServerServerCommand.func1(0xc00074b900, 0xc000394380, 0x0, 0x2) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/openshift-kube-apiserver/cmd.go:61 +0x10c github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).execute(0xc00074b900, 0xc0003942a0, 0x2, 0x2, 0xc00074b900, 0xc0003942a0) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:760 +0x2cc github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc00074b180, 0xc001228000, 0xc00074bb80, 0xc00074b900) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:846 +0x2fd github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).Execute(0xc00074b180, 0xc00074b180, 0x0) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:794 +0x2b main.main() /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/cmd/hypershift/main.go:45 +0x2b8 F0614 15:04:19.543371 1 cmd.go:71] open /etc/kubernetes/static-pod-certs/secrets/user-serving-cert/tls.crt: no such file or directory
Verified in 4.1.0-0.nightly-2019-07-18-023612. Now defaultServingCertificate setting is removed (not allowed) oc edit apiserver cluster ... spec: servingCerts: defaultServingCertificate: name: some-secret ... Attempted to save and exit, it failed and warned the setting is forbidden. And the Doc already removed it too.
Another bug 1731105 is reported after this.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1809
*** Bug 1711447 has been marked as a duplicate of this bug. ***