Bug 1728754

Summary:	Instructions for adding default API server default certificate fails to configure serving cert
Product:	OpenShift Container Platform	Reporter:	David Eads <deads>
Component:	kube-apiserver	Assignee:	David Eads <deads>
Status:	CLOSED ERRATA	QA Contact:	Xingxing Xia <xxia>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.1.0	CC:	aos-bugs, calfonso, mfojtik, sponnaga
Target Milestone:	---
Target Release:	4.1.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	4.1.7
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-07-25 05:32:32 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description David Eads 2019-07-10 15:33:49 UTC

This bug was initially created as a copy of Bug #1720770

I am copying this bug because: 



Document URL: https://docs.openshift.com/container-platform/4.1/authentication/certificates/api-server.html#add-default-api-server_api-server-certificates

Section Number and Name: 

Describe the issue: 

Customer followed instructions for setting the default API server serving certificate.  The API server is not presenting the certificate and one of the kube-apiserver pods is stuck in CrashLoopBackoff.

Suggestions for improvement: 

Additional information: 

On the customer's cluster, they have 3 masters.  In the process of applying the change, the first instance of the kube-apiserver pod is stuck in CrashLoopBackoff.  Investigating the logs of the kube-apiserver container reveals that the container is unable to find the serving cert.  See stack trace below.  I can reproduce the same problem on my AWS cluster as well.

	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/cmd/hypershift/main.go:45 +0x2b8
E0614 15:04:19.542054       1 pathrecorder.go:107] registered "/readyz/crd-informer-synced" from goroutine 1 [running]:
runtime/debug.Stack(0x4e8a780, 0xc008619fb0, 0xc002860260)
	/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/debug/stack.go:24 +0xa7
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).trackCallers(0xc001ece690, 0xc002860260, 0x1b)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:109 +0x89
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).Handle(0xc001ece690, 0xc002860260, 0x1b, 0xaa2ed40, 0xc00b8c3880)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:173 +0x86
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/healthz.InstallPathHandler(0xaa27f80, 0xc001ece690, 0x5969c5c, 0x7, 0xc0056ede00, 0x1d, 0x1e)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:120 +0x3c1
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*GenericAPIServer).installReadyz(0xc00638edc0, 0xc0003bc6c0)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/healthz.go:55 +0x1a8
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.preparedGenericAPIServer.Run(0xc00638edc0, 0xc0003bc6c0, 0xc00638edc0, 0x0)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/genericapiserver.go:290 +0x88
github.com/openshift/origin/vendor/k8s.io/kubernetes/cmd/kube-apiserver/app.Run(0xc0001ff8c0, 0xc0003bc6c0, 0x0, 0x0)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:153 +0x145
github.com/openshift/origin/vendor/k8s.io/kubernetes/cmd/kube-apiserver/app.NewAPIServerCommand.func1(0xc001228780, 0x0, 0x0, 0x0, 0x1, 0x0)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:115 +0x109
github.com/openshift/origin/pkg/cmd/openshift-kube-apiserver.RunOpenShiftKubeAPIServerServer(0xc001334800, 0xc0003bc6c0, 0x24, 0x7ffda2d38c27)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/openshift-kube-apiserver/server.go:70 +0x57e
github.com/openshift/origin/pkg/cmd/openshift-kube-apiserver.(*OpenShiftKubeAPIServerServer).RunAPIServer(0xc0003dde60, 0xc0003bc6c0, 0xc000aa38c0, 0xc000519800)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/openshift-kube-apiserver/cmd.go:129 +0x817
github.com/openshift/origin/pkg/cmd/openshift-kube-apiserver.NewOpenShiftKubeAPIServerServerCommand.func1(0xc00074b900, 0xc000394380, 0x0, 0x2)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/openshift-kube-apiserver/cmd.go:61 +0x10c
github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).execute(0xc00074b900, 0xc0003942a0, 0x2, 0x2, 0xc00074b900, 0xc0003942a0)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:760 +0x2cc
github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc00074b180, 0xc001228000, 0xc00074bb80, 0xc00074b900)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:846 +0x2fd
github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).Execute(0xc00074b180, 0xc00074b180, 0x0)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:794 +0x2b
main.main()
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/cmd/hypershift/main.go:45 +0x2b8
F0614 15:04:19.543371       1 cmd.go:71] open /etc/kubernetes/static-pod-certs/secrets/user-serving-cert/tls.crt: no such file or directory

Comment 2 Xingxing Xia 2019-07-18 09:48:54 UTC

Verified in 4.1.0-0.nightly-2019-07-18-023612. Now defaultServingCertificate setting is removed (not allowed)
oc edit apiserver cluster
...
spec:
  servingCerts:
    defaultServingCertificate:
      name: some-secret
...

Attempted to save and exit, it failed and warned the setting is forbidden. And the Doc already removed it too.

Comment 3 Xingxing Xia 2019-07-18 10:03:54 UTC

Another bug 1731105 is reported after this.

Comment 5 errata-xmlrpc 2019-07-25 05:32:32 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1809

Comment 6 Luis Sanchez 2019-11-12 15:26:16 UTC

*** Bug 1711447 has been marked as a duplicate of this bug. ***