Bug 1900446

Summary:	OCP4.7 nightly installation failed while kube-apiserver co is not available
Product:	OpenShift Container Platform	Reporter:	Wei Duan <wduan>
Component:	kube-apiserver	Assignee:	Stefan Schimanski <sttts>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Ke Wang <kewang>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	4.7	CC:	aos-bugs, knarra, mfojtik, piqin, vlaad, xxia
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-08-25 21:53:39 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1877681

Description Wei Duan 2020-11-23 03:11:38 UTC

Version:
4.7.0-0.nightly-2020-11-22-123106

Platform:
openstack
(https://mastern-jenkins-csb-openshift-qe.cloud.paas.psi.redhat.com/job/Launch%20Environment%20Flexy/123767/)

Please specify:
IPI

What happened?
See log-bundle-20201123013249 in http://virt-openshift-05.lab.eng.nay.redhat.com/wduan/logs/log-bundle-20201123013249.tar.gz. 

OCP4.7 nightly(2020-11-22-123106) installation failed on OSP while kube-apiserver co not available.   
Checking the api related pod:
$ oc -n openshift-kube-apiserver get pod
NAME                                         READY   STATUS             RESTARTS   AGE
installer-2-wduan-11203b-mg84j-master-2      0/1     Completed          0          43m
installer-3-wduan-11203b-mg84j-master-1      0/1     Completed          0          28m
installer-4-wduan-11203b-mg84j-master-0      0/1     Completed          0          7m54s
kube-apiserver-wduan-11203b-mg84j-master-0   3/5     CrashLoopBackOff   11         7m42s
kube-apiserver-wduan-11203b-mg84j-master-1   3/5     CrashLoopBackOff   23         28m
kube-apiserver-wduan-11203b-mg84j-master-2   3/5     CrashLoopBackOff   33         43m
I1123 01:49:45.066711      18 dynamic_cafile_content.go:129] Loaded a new CA Bundle and Verifier for "request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt"
Error: error reading public key file /etc/kubernetes/static-pod-resources/configmaps/bound-sa-token-signing-certs/service-account-001.pub: data does not contain any valid RSA or ECDSA public keys
$ oc get cm -n openshift-kube-apiserver bound-sa-token-signing-certs -oyaml
apiVersion: v1
data:
  service-account-001.pub: ""
  service-account-002.pub: |
    -----BEGIN RSA PUBLIC KEY-----
    MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtFbFF4YdTrNT8arjKxiY
    n7hgab6QGPFwhP81P6wRb3L2TgEET63SmU50eGcftMNkdGYxwTv747lDEOkiXgoe
    Gl8bNZV8kaUtAH3e1lnLybLqldkR77qiXhEJbjScMt+IfuZsMfux5EcsojcRcOYS
    MCSZBKuONbL+KHvPxlDcbae1J/5YII4EXsm9Jk2w8/mGIFfe5A3+BnKUoVG/EqE1
    8s/xCoQ+6mmgLeGBtG8dKiflxAEKJDF9g5+AKCclCUVAkek0PBh/VnYZD+ZsdjOu
    quQHOifgq7zPk1RQctA8s2J8QL5WHCUmD7SehHRpxN3ksXLf0yDHOPUqsQHT3IPc
    XwIDAQAB
    -----END RSA PUBLIC KEY-----

What did you expect to happen?
Installation should be successful.


How to reproduce it (as minimally and precisely as possible)?
2/2

Comment 1 RamaKasturi 2020-11-23 10:36:56 UTC

Hit similar issue with aws as well, but looks like not reproducible always.

payload used : 4.7.0-0.nightly-2020-11-22-204912

logs can be found here: http://virt-openshift-05.lab.eng.nay.redhat.com/knarra/1900446/

[knarra@knarra verification-tests]$ oc get pods -n openshift-kube-apiserver
NAME                                                        READY   STATUS             RESTARTS   AGE
installer-6-ip-10-0-211-74.us-east-2.compute.internal       0/1     Completed          0          3h17m
installer-7-ip-10-0-141-133.us-east-2.compute.internal      0/1     Completed          0          3h15m
kube-apiserver-ip-10-0-141-133.us-east-2.compute.internal   3/5     CrashLoopBackOff   130        3h15m
kube-apiserver-ip-10-0-175-218.us-east-2.compute.internal   3/5     CrashLoopBackOff   121        3h21m
kube-apiserver-ip-10-0-211-74.us-east-2.compute.internal    3/5     CrashLoopBackOff   118        3h16m

I1123 10:23:51.134953      18 server.go:201] Version: v1.19.2+13d6aa9
I1123 10:23:51.135523      18 dynamic_serving_content.go:111] Loaded a new cert/key pair for "serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key"
I1123 10:23:51.135734      18 dynamic_serving_content.go:111] Loaded a new cert/key pair for "sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.key"
I1123 10:23:51.136050      18 dynamic_serving_content.go:111] Loaded a new cert/key pair for "sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key"
I1123 10:23:51.136431      18 dynamic_serving_content.go:111] Loaded a new cert/key pair for "sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.key"
I1123 10:23:51.136775      18 dynamic_serving_content.go:111] Loaded a new cert/key pair for "sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.key"
I1123 10:23:51.137107      18 dynamic_serving_content.go:111] Loaded a new cert/key pair for "sni-serving-cert::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.crt::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.key"
I1123 10:23:51.685513      18 dynamic_cafile_content.go:129] Loaded a new CA Bundle and Verifier for "client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt"
I1123 10:23:51.685668      18 dynamic_cafile_content.go:129] Loaded a new CA Bundle and Verifier for "request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt"
Error: error reading public key file /etc/kubernetes/static-pod-resources/configmaps/bound-sa-token-signing-certs/service-account-001.pub: data does not contain any valid RSA or ECDSA public keys
I1123 10:23:51.687285       1 main.go:198] Termination finished with exit code 1
I1123 10:23:51.687311       1 main.go:151] Deleting termination lock file "/var/log/kube-apiserver/.terminating"

Comment 3 Xingxing Xia 2020-11-23 23:53:35 UTC

*** Bug 1900635 has been marked as a duplicate of this bug. ***

Comment 4 Xingxing Xia 2020-11-24 03:32:09 UTC

Verified in 4.7.0-0.nightly-2020-11-24-015807 OSP env because OSP env 100% hit yesterday.