Version: 4.7.0-0.nightly-2020-11-22-123106 Platform: openstack (https://mastern-jenkins-csb-openshift-qe.cloud.paas.psi.redhat.com/job/Launch%20Environment%20Flexy/123767/) Please specify: IPI What happened? See log-bundle-20201123013249 in http://virt-openshift-05.lab.eng.nay.redhat.com/wduan/logs/log-bundle-20201123013249.tar.gz. OCP4.7 nightly(2020-11-22-123106) installation failed on OSP while kube-apiserver co not available. Checking the api related pod: $ oc -n openshift-kube-apiserver get pod NAME READY STATUS RESTARTS AGE installer-2-wduan-11203b-mg84j-master-2 0/1 Completed 0 43m installer-3-wduan-11203b-mg84j-master-1 0/1 Completed 0 28m installer-4-wduan-11203b-mg84j-master-0 0/1 Completed 0 7m54s kube-apiserver-wduan-11203b-mg84j-master-0 3/5 CrashLoopBackOff 11 7m42s kube-apiserver-wduan-11203b-mg84j-master-1 3/5 CrashLoopBackOff 23 28m kube-apiserver-wduan-11203b-mg84j-master-2 3/5 CrashLoopBackOff 33 43m I1123 01:49:45.066711 18 dynamic_cafile_content.go:129] Loaded a new CA Bundle and Verifier for "request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt" Error: error reading public key file /etc/kubernetes/static-pod-resources/configmaps/bound-sa-token-signing-certs/service-account-001.pub: data does not contain any valid RSA or ECDSA public keys $ oc get cm -n openshift-kube-apiserver bound-sa-token-signing-certs -oyaml apiVersion: v1 data: service-account-001.pub: "" service-account-002.pub: | -----BEGIN RSA PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtFbFF4YdTrNT8arjKxiY n7hgab6QGPFwhP81P6wRb3L2TgEET63SmU50eGcftMNkdGYxwTv747lDEOkiXgoe Gl8bNZV8kaUtAH3e1lnLybLqldkR77qiXhEJbjScMt+IfuZsMfux5EcsojcRcOYS MCSZBKuONbL+KHvPxlDcbae1J/5YII4EXsm9Jk2w8/mGIFfe5A3+BnKUoVG/EqE1 8s/xCoQ+6mmgLeGBtG8dKiflxAEKJDF9g5+AKCclCUVAkek0PBh/VnYZD+ZsdjOu quQHOifgq7zPk1RQctA8s2J8QL5WHCUmD7SehHRpxN3ksXLf0yDHOPUqsQHT3IPc XwIDAQAB -----END RSA PUBLIC KEY----- What did you expect to happen? Installation should be successful. How to reproduce it (as minimally and precisely as possible)? 2/2
Hit similar issue with aws as well, but looks like not reproducible always. payload used : 4.7.0-0.nightly-2020-11-22-204912 logs can be found here: http://virt-openshift-05.lab.eng.nay.redhat.com/knarra/1900446/ [knarra@knarra verification-tests]$ oc get pods -n openshift-kube-apiserver NAME READY STATUS RESTARTS AGE installer-6-ip-10-0-211-74.us-east-2.compute.internal 0/1 Completed 0 3h17m installer-7-ip-10-0-141-133.us-east-2.compute.internal 0/1 Completed 0 3h15m kube-apiserver-ip-10-0-141-133.us-east-2.compute.internal 3/5 CrashLoopBackOff 130 3h15m kube-apiserver-ip-10-0-175-218.us-east-2.compute.internal 3/5 CrashLoopBackOff 121 3h21m kube-apiserver-ip-10-0-211-74.us-east-2.compute.internal 3/5 CrashLoopBackOff 118 3h16m I1123 10:23:51.134953 18 server.go:201] Version: v1.19.2+13d6aa9 I1123 10:23:51.135523 18 dynamic_serving_content.go:111] Loaded a new cert/key pair for "serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key" I1123 10:23:51.135734 18 dynamic_serving_content.go:111] Loaded a new cert/key pair for "sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.key" I1123 10:23:51.136050 18 dynamic_serving_content.go:111] Loaded a new cert/key pair for "sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key" I1123 10:23:51.136431 18 dynamic_serving_content.go:111] Loaded a new cert/key pair for "sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.key" I1123 10:23:51.136775 18 dynamic_serving_content.go:111] Loaded a new cert/key pair for "sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" I1123 10:23:51.137107 18 dynamic_serving_content.go:111] Loaded a new cert/key pair for "sni-serving-cert::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.crt::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.key" I1123 10:23:51.685513 18 dynamic_cafile_content.go:129] Loaded a new CA Bundle and Verifier for "client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt" I1123 10:23:51.685668 18 dynamic_cafile_content.go:129] Loaded a new CA Bundle and Verifier for "request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt" Error: error reading public key file /etc/kubernetes/static-pod-resources/configmaps/bound-sa-token-signing-certs/service-account-001.pub: data does not contain any valid RSA or ECDSA public keys I1123 10:23:51.687285 1 main.go:198] Termination finished with exit code 1 I1123 10:23:51.687311 1 main.go:151] Deleting termination lock file "/var/log/kube-apiserver/.terminating"
*** Bug 1900635 has been marked as a duplicate of this bug. ***
Verified in 4.7.0-0.nightly-2020-11-24-015807 OSP env because OSP env 100% hit yesterday.