Bug 1903586
| Summary: | kube-scheduler not scheduling pods for certificates not renewed automatically after nodes restoration | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Tomáš Nožička <tnozicka> | |
| Component: | kube-scheduler | Assignee: | Tomáš Nožička <tnozicka> | |
| Status: | CLOSED ERRATA | QA Contact: | RamaKasturi <knarra> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 4.6 | CC: | aarapov, aos-bugs, cpassare, dgautam, knarra, malonso, maszulik, mfojtik, mvardhan, ngirard, openshift-bugzilla-robot, tnozicka, tomek | |
| Target Milestone: | --- | Keywords: | UpcomingSprint | |
| Target Release: | 4.6.z | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | 1881322 | |||
| : | 1915363 (view as bug list) | Environment: | ||
| Last Closed: | 2021-01-25 20:02:12 UTC | Type: | --- | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1881322 | |||
| Bug Blocks: | 1903523, 1915363 | |||
|
Comment 1
Maciej Szulik
2020-12-03 10:37:00 UTC
Verified with the nightly payload below and did not see any pods in pending state but some operators are in degraded and looks like it is expected. Below are the steps followed to verify the bug [core@knarra46fix-7jx2c-compute-0 ~]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.6.0-0.nightly-2021-01-19-222345 True False 365d Error while reconciling 4.6.0-0.nightly-2021-01-19-222345: the cluster operator authentication is degraded In 4.6 cluster with fix: ========================== 1) Install 4.6 cluster with latest nightly 2) Now login to one of the master node and run the script https://gist.github.com/tnozicka/b1df897a905be8b6e22ab04ce5b9b90a#file-ocp-shifttime-sh 3) wait for the script to finish , export kubeconfig file 4) Approve Pending CSR for MCO, worker & master nodes 5) Now wait for co to return to correct status 6) check for any pending pods, could see that no pending pods [core@knarra46fix-7jx2c-compute-0 ~]$ oc get pods --all-namespaces |grep Pending [core@knarra46fix-7jx2c-compute-0 ~]$ 7) check for co status, some operators in degraded state due to no registry present in the cluster. [core@knarra46fix-7jx2c-compute-0 ~]$ oc get co | grep -v '.*True.*False.*False' NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.6.0-0.nightly-2021-01-19-222345 True True True 9h cloud-credential insights 4.6.0-0.nightly-2021-01-19-222345 True False True 9h monitoring False True True 9h network 4.6.0-0.nightly-2021-01-19-222345 True True False 9h Below logs are seen in kube-scheduler: ======================================= [core@knarra46fix-7jx2c-compute-0 ~]$ oc logs openshift-kube-scheduler-knarra46fix-7jx2c-control-plane-1 -n openshift-kube-scheduler | grep cert_rotation.go I0120 12:35:11.882624 1 cert_rotation.go:88] certificate rotation detected, shutting down client connections to start using new credentials I0120 12:35:50.419303 1 cert_rotation.go:88] certificate rotation detected, shutting down client connections to start using new credentials In 4.6 cluster with out fix: ============================== 1) Install 4.6 cluster with payload that does not contain fix 2) Now login to one of the master node and run the script https://gist.github.com/tnozicka/b1df897a905be8b6e22ab04ce5b9b90a#file-ocp-shifttime-sh ) wait for the script to finish , export kubeconfig file 4) Approve Pending CSR for MCO, worker & master nodes 5) Run 'oc get co' command and you can see that there is no status returned for operators which is expected and you can find all pods in pending state. [core@knarra463-9cnjd-compute-0 ~]$ oc get pods --all-namespaces | grep Pending openshift-apiserver-operator openshift-apiserver-operator-689cfc49c4-2ckts 0/1 Pending 0 9h openshift-apiserver apiserver-d8bcb55f9-4795s 0/2 Pending 0 9h openshift-apiserver apiserver-d8bcb55f9-pfp5j 0/2 Pending 0 9h openshift-apiserver apiserver-d8bcb55f9-xnbsg 0/2 Pending 0 9h openshift-authentication-operator authentication-operator-5bd5b48bc7-7zblb 0/1 Pending 0 9h openshift-authentication oauth-openshift-5b49dcb59f-fbm57 0/1 Pending 0 9h openshift-authentication oauth-openshift-5b49dcb59f-zdmls 0/1 Pending 0 9h openshift-cloud-credential-operator cloud-credential-operator-86b48b54f9-9rl9q 0/2 Pending 0 9h openshift-cluster-machine-approver machine-approver-77cd8fb58d-gmtn2 0/2 Pending 0 9h openshift-cluster-node-tuning-operator cluster-node-tuning-operator-7c9bdc4d9c-8bs5b 0/1 Pending 0 9h openshift-cluster-node-tuning-operator tuned-4zrhh 0/1 Pending 0 9h openshift-cluster-node-tuning-operator tuned-dmwf7 0/1 Pending 0 9h openshift-cluster-node-tuning-operator tuned-mtmm8 0/1 Pending 0 9h openshift-cluster-node-tuning-operator tuned-mxwhl 0/1 Pending 0 9h openshift-cluster-node-tuning-operator tuned-t2svs 0/1 Pending 0 9h openshift-cluster-node-tuning-operator tuned-x87pz 0/1 Pending 0 9h openshift-cluster-samples-operator cluster-samples-operator-5b5f588cfd-6w9ml 0/2 Pending 0 9h openshift-cluster-storage-operator cluster-storage-operator-796db8bb55-b59gw 0/1 Pending 0 9h openshift-cluster-storage-operator csi-snapshot-controller-5b97f7bcdb-2lqz9 0/1 Pending 0 9h openshift-cluster-storage-operator csi-snapshot-controller-operator-59df4999b6-zkdv9 0/1 Pending 0 9h openshift-cluster-version cluster-version-operator-6ffb655f5d-6sptk 0/1 Pending 0 9h openshift-config-operator openshift-config-operator-648cf8d767-hzgtd 0/1 Pending 0 9h openshift-console-operator console-operator-66c55d7484-dcrsr 0/1 Pending 0 9h openshift-console console-7599699b89-f2tkx 0/1 Pending 0 9h openshift-console console-7599699b89-k7kgj 0/1 Pending 0 9h openshift-console downloads-79f66c8cd9-2zgkm 0/1 Pending 0 9h openshift-console downloads-79f66c8cd9-dlkq7 0/1 Pending 0 9h openshift-controller-manager-operator openshift-controller-manager-operator-67f446c95c-k2hhf 0/1 Pending 0 9h openshift-controller-manager controller-manager-2wddn 0/1 Pending 0 9h openshift-controller-manager controller-manager-lq5hm 0/1 Pending 0 9h openshift-controller-manager controller-manager-swjhb 0/1 Pending 0 9h openshift-dns-operator dns-operator-74c4b88fdc-g26f6 0/2 Pending 0 9h openshift-dns dns-default-768hb 0/3 Pending 0 9h openshift-dns dns-default-9wlxh 0/3 Pending 0 9h openshift-dns dns-default-hcv75 0/3 Pending 0 9h openshift-dns dns-default-l9mvh 0/3 Pending 0 9h openshift-dns dns-default-spbtk 0/3 Pending 0 9h openshift-dns dns-default-xm8bd 0/3 Pending 0 9h openshift-etcd-operator etcd-operator-696fb7f94d-6wrbs 0/1 Pending 0 9h openshift-etcd etcd-quorum-guard-647b754d4c-7c95w 0/1 Pending 0 9h openshift-etcd etcd-quorum-guard-647b754d4c-l277q 0/1 Pending 0 9h openshift-etcd etcd-quorum-guard-647b754d4c-nwhqp 0/1 Pending 0 9h openshift-image-registry cluster-image-registry-operator-f744b7d56-pj6s5 0/1 Pending 0 9h openshift-image-registry image-registry-57d57d76bc-bztv7 0/1 Pending 0 9h openshift-image-registry node-ca-7k772 0/1 Pending 0 9h openshift-image-registry node-ca-hp78c 0/1 Pending 0 9h openshift-image-registry node-ca-hpszr 0/1 Pending 0 9h openshift-image-registry node-ca-jv2xd 0/1 Pending 0 9h openshift-image-registry node-ca-smzxx 0/1 Pending 0 9h openshift-image-registry node-ca-wrtsm 0/1 Pending 0 9h openshift-ingress-operator ingress-operator-54c46d4566-p8755 0/2 Pending 0 9h openshift-ingress router-default-5774bfcdff-bdml5 0/1 Pending 0 9h openshift-ingress router-default-5774bfcdff-gvvvd 0/1 Pending 0 9h openshift-insights insights-operator-d6b6f5486-7pv56 0/1 Pending 0 9h openshift-kube-apiserver-operator kube-apiserver-operator-68554d9f6f-4gpqx 0/1 Pending 0 9h openshift-kube-controller-manager-operator kube-controller-manager-operator-688fb78677-2c2x8 0/1 Pending 0 9h openshift-kube-scheduler-operator openshift-kube-scheduler-operator-67c8f54d5f-vzw5k 0/1 Pending 0 9h openshift-kube-storage-version-migrator-operator kube-storage-version-migrator-operator-5b66cb684b-vgrlw 0/1 Pending 0 9h openshift-kube-storage-version-migrator migrator-c85595df4-xzw2r 0/1 Pending 0 9h openshift-machine-api cluster-autoscaler-operator-5b6d5d7f9b-vg9dj 0/2 Pending 0 9h openshift-machine-api machine-api-operator-7fb55d47d9-cs9r4 0/2 Pending 0 9h openshift-machine-config-operator machine-config-controller-55685c9588-qfxp5 0/1 Pending 0 9h openshift-machine-config-operator machine-config-daemon-2csbg 0/2 Pending 0 9h openshift-machine-config-operator machine-config-daemon-7jpjr 0/2 Pending 0 9h openshift-machine-config-operator machine-config-daemon-9df8s 0/2 Pending 0 9h openshift-machine-config-operator machine-config-daemon-blt9p 0/2 Pending 0 9h openshift-machine-config-operator machine-config-daemon-hgb4t 0/2 Pending 0 9h openshift-machine-config-operator machine-config-daemon-tmc9l 0/2 Pending 0 9h openshift-machine-config-operator machine-config-operator-66f5464954-8fskt 0/1 Pending 0 9h openshift-machine-config-operator machine-config-server-2xclx 0/1 Pending 0 9h openshift-machine-config-operator machine-config-server-8pjk5 0/1 Pending 0 9h openshift-machine-config-operator machine-config-server-mlmrx 0/1 Pending 0 9h openshift-marketplace marketplace-operator-84f685dc4d-vrwxx 0/1 Pending 0 9h openshift-monitoring alertmanager-main-0 0/5 Pending 0 9h openshift-monitoring alertmanager-main-1 0/5 Pending 0 9h openshift-monitoring alertmanager-main-2 0/5 Pending 0 9h openshift-monitoring cluster-monitoring-operator-685f457c45-9mq5t 0/2 Pending 0 9h openshift-monitoring grafana-86765b6dcf-jgg4c 0/2 Pending 0 9h openshift-monitoring kube-state-metrics-56c8dc648c-8lk9m 0/3 Pending 0 9h openshift-monitoring node-exporter-2zf6k 0/2 Pending 0 9h openshift-monitoring node-exporter-65mvv 0/2 Pending 0 9h openshift-monitoring node-exporter-9crnv 0/2 Pending 0 9h openshift-monitoring node-exporter-jr6vz 0/2 Pending 0 9h openshift-monitoring node-exporter-r5x5j 0/2 Pending 0 9h openshift-monitoring node-exporter-wlgf7 0/2 Pending 0 9h openshift-monitoring openshift-state-metrics-6d5cf65975-qh4hm 0/3 Pending 0 9h openshift-monitoring prometheus-adapter-8548855c4b-62kbh 0/1 Pending 0 9h openshift-monitoring prometheus-adapter-8548855c4b-j2xch 0/1 Pending 0 9h openshift-monitoring prometheus-k8s-0 0/6 Pending 0 9h openshift-monitoring prometheus-k8s-1 0/6 Pending 0 9h openshift-monitoring prometheus-operator-79f7d5d6d9-bmsj8 0/2 Pending 0 9h openshift-monitoring telemeter-client-859856799-lcs9f 0/3 Pending 0 9h openshift-monitoring thanos-querier-7975db4f7d-476k8 0/5 Pending 0 9h openshift-monitoring thanos-querier-7975db4f7d-vpqn2 0/5 Pending 0 9h openshift-multus multus-27r8k 0/1 Pending 0 9h openshift-multus multus-6rpd2 0/1 Pending 0 9h openshift-multus multus-admission-controller-mh5jf 0/2 Pending 0 9h openshift-multus multus-admission-controller-sfz2g 0/2 Pending 0 9h openshift-multus multus-admission-controller-wkt65 0/2 Pending 0 9h openshift-multus multus-dxbh4 0/1 Pending 0 9h openshift-multus multus-lvlxq 0/1 Pending 0 9h openshift-multus multus-snr5s 0/1 Pending 0 9h openshift-multus multus-xpx5c 0/1 Pending 0 9h openshift-multus network-metrics-daemon-5m8d2 0/2 Pending 0 9h openshift-multus network-metrics-daemon-8hzhp 0/2 Pending 0 9h openshift-multus network-metrics-daemon-9wsc7 0/2 Pending 0 9h openshift-multus network-metrics-daemon-kzvsj 0/2 Pending 0 9h openshift-multus network-metrics-daemon-sm7k2 0/2 Pending 0 9h openshift-multus network-metrics-daemon-xsq7g 0/2 Pending 0 9h openshift-network-operator network-operator-7c586cf699-r7224 0/1 Pending 0 9h openshift-oauth-apiserver apiserver-59b9c9df94-g4sxg 0/1 Pending 0 9h openshift-oauth-apiserver apiserver-59b9c9df94-kjwdv 0/1 Pending 0 9h openshift-oauth-apiserver apiserver-59b9c9df94-w5cz4 0/1 Pending 0 9h openshift-operator-lifecycle-manager catalog-operator-559b6fd6bf-5cjcx 0/1 Pending 0 9h openshift-operator-lifecycle-manager olm-operator-8596dfd454-kkms8 0/1 Pending 0 9h openshift-operator-lifecycle-manager packageserver-bc67cf7d9-575rw 0/1 Pending 0 9h openshift-operator-lifecycle-manager packageserver-bc67cf7d9-5dwqr 0/1 Pending 0 9h openshift-sdn ovs-8hxrs 0/1 Pending 0 9h openshift-sdn ovs-bwbj8 0/1 Pending 0 9h openshift-sdn ovs-ltrsr 0/1 Pending 0 9h openshift-sdn ovs-m4sn4 0/1 Pending 0 9h openshift-sdn ovs-swp6z 0/1 Pending 0 9h openshift-sdn ovs-t7pvl 0/1 Pending 0 9h openshift-sdn sdn-4rrlj 0/2 Pending 0 9h openshift-sdn sdn-77vw9 0/2 Pending 0 9h openshift-sdn sdn-8q6c2 0/2 Pending 0 9h openshift-sdn sdn-controller-fp84s 0/1 Pending 0 9h openshift-sdn sdn-controller-tlhzx 0/1 Pending 0 9h openshift-sdn sdn-controller-wpz78 0/1 Pending 0 9h openshift-sdn sdn-g6k84 0/2 Pending 0 9h openshift-sdn sdn-k96kz 0/2 Pending 0 9h openshift-sdn sdn-pwzps 0/2 Pending 0 9h openshift-service-ca-operator service-ca-operator-8c5b688f7-lt8gx 0/1 Pending 0 9h openshift-service-ca service-ca-6885ffbf4c-szb7p 0/1 Pending 0 9h Based on the above moving bug to verified state Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.6.13 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:0171 |