Bug 1728754

Summary: Instructions for adding default API server default certificate fails to configure serving cert
Product: OpenShift Container Platform Reporter: David Eads <deads>
Component: kube-apiserverAssignee: David Eads <deads>
Status: CLOSED ERRATA QA Contact: Xingxing Xia <xxia>
Severity: high Docs Contact:
Priority: high    
Version: 4.1.0CC: aos-bugs, calfonso, mfojtik, sponnaga
Target Milestone: ---   
Target Release: 4.1.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: 4.1.7
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-07-25 05:32:32 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description David Eads 2019-07-10 15:33:49 UTC 
This bug was initially created as a copy of Bug #1720770

I am copying this bug because: 



Document URL: https://docs.openshift.com/container-platform/4.1/authentication/certificates/api-server.html#add-default-api-server_api-server-certificates

Section Number and Name: 

Describe the issue: 

Customer followed instructions for setting the default API server serving certificate.  The API server is not presenting the certificate and one of the kube-apiserver pods is stuck in CrashLoopBackoff.

Suggestions for improvement: 

Additional information: 

On the customer's cluster, they have 3 masters.  In the process of applying the change, the first instance of the kube-apiserver pod is stuck in CrashLoopBackoff.  Investigating the logs of the kube-apiserver container reveals that the container is unable to find the serving cert.  See stack trace below.  I can reproduce the same problem on my AWS cluster as well.

	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/cmd/hypershift/main.go:45 +0x2b8
E0614 15:04:19.542054       1 pathrecorder.go:107] registered "/readyz/crd-informer-synced" from goroutine 1 [running]:
runtime/debug.Stack(0x4e8a780, 0xc008619fb0, 0xc002860260)
	/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/debug/stack.go:24 +0xa7
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).trackCallers(0xc001ece690, 0xc002860260, 0x1b)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:109 +0x89
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).Handle(0xc001ece690, 0xc002860260, 0x1b, 0xaa2ed40, 0xc00b8c3880)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:173 +0x86
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/healthz.InstallPathHandler(0xaa27f80, 0xc001ece690, 0x5969c5c, 0x7, 0xc0056ede00, 0x1d, 0x1e)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:120 +0x3c1
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*GenericAPIServer).installReadyz(0xc00638edc0, 0xc0003bc6c0)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/healthz.go:55 +0x1a8
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.preparedGenericAPIServer.Run(0xc00638edc0, 0xc0003bc6c0, 0xc00638edc0, 0x0)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/genericapiserver.go:290 +0x88
github.com/openshift/origin/vendor/k8s.io/kubernetes/cmd/kube-apiserver/app.Run(0xc0001ff8c0, 0xc0003bc6c0, 0x0, 0x0)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:153 +0x145
github.com/openshift/origin/vendor/k8s.io/kubernetes/cmd/kube-apiserver/app.NewAPIServerCommand.func1(0xc001228780, 0x0, 0x0, 0x0, 0x1, 0x0)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:115 +0x109
github.com/openshift/origin/pkg/cmd/openshift-kube-apiserver.RunOpenShiftKubeAPIServerServer(0xc001334800, 0xc0003bc6c0, 0x24, 0x7ffda2d38c27)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/openshift-kube-apiserver/server.go:70 +0x57e
github.com/openshift/origin/pkg/cmd/openshift-kube-apiserver.(*OpenShiftKubeAPIServerServer).RunAPIServer(0xc0003dde60, 0xc0003bc6c0, 0xc000aa38c0, 0xc000519800)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/openshift-kube-apiserver/cmd.go:129 +0x817
github.com/openshift/origin/pkg/cmd/openshift-kube-apiserver.NewOpenShiftKubeAPIServerServerCommand.func1(0xc00074b900, 0xc000394380, 0x0, 0x2)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/openshift-kube-apiserver/cmd.go:61 +0x10c
github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).execute(0xc00074b900, 0xc0003942a0, 0x2, 0x2, 0xc00074b900, 0xc0003942a0)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:760 +0x2cc
github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc00074b180, 0xc001228000, 0xc00074bb80, 0xc00074b900)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:846 +0x2fd
github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).Execute(0xc00074b180, 0xc00074b180, 0x0)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:794 +0x2b
main.main()
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/cmd/hypershift/main.go:45 +0x2b8
F0614 15:04:19.543371       1 cmd.go:71] open /etc/kubernetes/static-pod-certs/secrets/user-serving-cert/tls.crt: no such file or directory
Comment 2 Xingxing Xia 2019-07-18 09:48:54 UTC 
Verified in 4.1.0-0.nightly-2019-07-18-023612. Now defaultServingCertificate setting is removed (not allowed)
oc edit apiserver cluster
...
spec:
  servingCerts:
    defaultServingCertificate:
      name: some-secret
...

Attempted to save and exit, it failed and warned the setting is forbidden. And the Doc already removed it too.
Comment 3 Xingxing Xia 2019-07-18 10:03:54 UTC 
Another bug 1731105 is reported after this.
Comment 5 errata-xmlrpc 2019-07-25 05:32:32 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1809
Comment 6 Luis Sanchez 2019-11-12 15:26:16 UTC 
*** Bug 1711447 has been marked as a duplicate of this bug. ***