Bug 1903586 - kube-scheduler not scheduling pods for certificates not renewed automatically after nodes restoration
Summary: kube-scheduler not scheduling pods for certificates not renewed automatically...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-scheduler
Version: 4.6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.6.z
Assignee: Tomáš Nožička
QA Contact: RamaKasturi
URL:
Whiteboard:
: 1903523 (view as bug list)
Depends On: 1881322
Blocks: 1903523 1915363
TreeView+ depends on / blocked
 
Reported: 2020-12-02 12:48 UTC by Tomáš Nožička
Modified: 2021-01-25 20:02 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1881322
: 1915363 (view as bug list)
Environment:
Last Closed: 2021-01-25 20:02:12 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-kube-scheduler-operator pull 315 0 None closed Bug 1903586: Sync new kube-scheduler-client-cert-key on recovery 2021-02-02 09:29:06 UTC
Red Hat Product Errata RHSA-2021:0171 0 None None None 2021-01-25 20:02:34 UTC

Comment 1 Maciej Szulik 2020-12-03 10:37:00 UTC
*** Bug 1903523 has been marked as a duplicate of this bug. ***

Comment 5 RamaKasturi 2021-01-20 15:58:09 UTC
Verified with the nightly payload below and did not see any pods in pending state but some operators are in degraded and looks like it is expected. Below are the steps followed to verify the bug

[core@knarra46fix-7jx2c-compute-0 ~]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2021-01-19-222345   True        False         365d    Error while reconciling 4.6.0-0.nightly-2021-01-19-222345: the cluster operator authentication is degraded


In 4.6 cluster with fix:
==========================
1) Install 4.6 cluster with latest nightly
2) Now login to one of the master node and run the script https://gist.github.com/tnozicka/b1df897a905be8b6e22ab04ce5b9b90a#file-ocp-shifttime-sh
3) wait for the script to finish , export kubeconfig file
4) Approve Pending CSR for MCO, worker & master nodes
5) Now wait for co to return to correct status
6) check for any pending pods, could see that no pending pods

[core@knarra46fix-7jx2c-compute-0 ~]$ oc get pods --all-namespaces |grep Pending
[core@knarra46fix-7jx2c-compute-0 ~]$ 
7) check for co status, some operators in degraded state due to no registry present in the cluster.

[core@knarra46fix-7jx2c-compute-0 ~]$ oc get co | grep -v '.*True.*False.*False'
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.6.0-0.nightly-2021-01-19-222345   True        True          True       9h
cloud-credential                                                                                                    
insights                                   4.6.0-0.nightly-2021-01-19-222345   True        False         True       9h
monitoring                                                                     False       True          True       9h
network                                    4.6.0-0.nightly-2021-01-19-222345   True        True          False      9h

Below logs are seen in kube-scheduler:
=======================================
[core@knarra46fix-7jx2c-compute-0 ~]$ oc logs openshift-kube-scheduler-knarra46fix-7jx2c-control-plane-1 -n openshift-kube-scheduler | grep cert_rotation.go
I0120 12:35:11.882624       1 cert_rotation.go:88] certificate rotation detected, shutting down client connections to start using new credentials
I0120 12:35:50.419303       1 cert_rotation.go:88] certificate rotation detected, shutting down client connections to start using new credentials


In 4.6 cluster with out fix:
==============================
1) Install 4.6 cluster with payload that does not contain  fix
2) Now login to one of the master node and run the script https://gist.github.com/tnozicka/b1df897a905be8b6e22ab04ce5b9b90a#file-ocp-shifttime-sh
) wait for the script to finish , export kubeconfig file
4) Approve Pending CSR for MCO, worker & master nodes
5) Run 'oc get co' command and you can see that there is no status returned for operators which is expected and you can find all pods in pending state.
[core@knarra463-9cnjd-compute-0 ~]$ oc get pods --all-namespaces | grep Pending
openshift-apiserver-operator                       openshift-apiserver-operator-689cfc49c4-2ckts              0/1     Pending            0          9h
openshift-apiserver                                apiserver-d8bcb55f9-4795s                                  0/2     Pending            0          9h
openshift-apiserver                                apiserver-d8bcb55f9-pfp5j                                  0/2     Pending            0          9h
openshift-apiserver                                apiserver-d8bcb55f9-xnbsg                                  0/2     Pending            0          9h
openshift-authentication-operator                  authentication-operator-5bd5b48bc7-7zblb                   0/1     Pending            0          9h
openshift-authentication                           oauth-openshift-5b49dcb59f-fbm57                           0/1     Pending            0          9h
openshift-authentication                           oauth-openshift-5b49dcb59f-zdmls                           0/1     Pending            0          9h
openshift-cloud-credential-operator                cloud-credential-operator-86b48b54f9-9rl9q                 0/2     Pending            0          9h
openshift-cluster-machine-approver                 machine-approver-77cd8fb58d-gmtn2                          0/2     Pending            0          9h
openshift-cluster-node-tuning-operator             cluster-node-tuning-operator-7c9bdc4d9c-8bs5b              0/1     Pending            0          9h
openshift-cluster-node-tuning-operator             tuned-4zrhh                                                0/1     Pending            0          9h
openshift-cluster-node-tuning-operator             tuned-dmwf7                                                0/1     Pending            0          9h
openshift-cluster-node-tuning-operator             tuned-mtmm8                                                0/1     Pending            0          9h
openshift-cluster-node-tuning-operator             tuned-mxwhl                                                0/1     Pending            0          9h
openshift-cluster-node-tuning-operator             tuned-t2svs                                                0/1     Pending            0          9h
openshift-cluster-node-tuning-operator             tuned-x87pz                                                0/1     Pending            0          9h
openshift-cluster-samples-operator                 cluster-samples-operator-5b5f588cfd-6w9ml                  0/2     Pending            0          9h
openshift-cluster-storage-operator                 cluster-storage-operator-796db8bb55-b59gw                  0/1     Pending            0          9h
openshift-cluster-storage-operator                 csi-snapshot-controller-5b97f7bcdb-2lqz9                   0/1     Pending            0          9h
openshift-cluster-storage-operator                 csi-snapshot-controller-operator-59df4999b6-zkdv9          0/1     Pending            0          9h
openshift-cluster-version                          cluster-version-operator-6ffb655f5d-6sptk                  0/1     Pending            0          9h
openshift-config-operator                          openshift-config-operator-648cf8d767-hzgtd                 0/1     Pending            0          9h
openshift-console-operator                         console-operator-66c55d7484-dcrsr                          0/1     Pending            0          9h
openshift-console                                  console-7599699b89-f2tkx                                   0/1     Pending            0          9h
openshift-console                                  console-7599699b89-k7kgj                                   0/1     Pending            0          9h
openshift-console                                  downloads-79f66c8cd9-2zgkm                                 0/1     Pending            0          9h
openshift-console                                  downloads-79f66c8cd9-dlkq7                                 0/1     Pending            0          9h
openshift-controller-manager-operator              openshift-controller-manager-operator-67f446c95c-k2hhf     0/1     Pending            0          9h
openshift-controller-manager                       controller-manager-2wddn                                   0/1     Pending            0          9h
openshift-controller-manager                       controller-manager-lq5hm                                   0/1     Pending            0          9h
openshift-controller-manager                       controller-manager-swjhb                                   0/1     Pending            0          9h
openshift-dns-operator                             dns-operator-74c4b88fdc-g26f6                              0/2     Pending            0          9h
openshift-dns                                      dns-default-768hb                                          0/3     Pending            0          9h
openshift-dns                                      dns-default-9wlxh                                          0/3     Pending            0          9h
openshift-dns                                      dns-default-hcv75                                          0/3     Pending            0          9h
openshift-dns                                      dns-default-l9mvh                                          0/3     Pending            0          9h
openshift-dns                                      dns-default-spbtk                                          0/3     Pending            0          9h
openshift-dns                                      dns-default-xm8bd                                          0/3     Pending            0          9h
openshift-etcd-operator                            etcd-operator-696fb7f94d-6wrbs                             0/1     Pending            0          9h
openshift-etcd                                     etcd-quorum-guard-647b754d4c-7c95w                         0/1     Pending            0          9h
openshift-etcd                                     etcd-quorum-guard-647b754d4c-l277q                         0/1     Pending            0          9h
openshift-etcd                                     etcd-quorum-guard-647b754d4c-nwhqp                         0/1     Pending            0          9h
openshift-image-registry                           cluster-image-registry-operator-f744b7d56-pj6s5            0/1     Pending            0          9h
openshift-image-registry                           image-registry-57d57d76bc-bztv7                            0/1     Pending            0          9h
openshift-image-registry                           node-ca-7k772                                              0/1     Pending            0          9h
openshift-image-registry                           node-ca-hp78c                                              0/1     Pending            0          9h
openshift-image-registry                           node-ca-hpszr                                              0/1     Pending            0          9h
openshift-image-registry                           node-ca-jv2xd                                              0/1     Pending            0          9h
openshift-image-registry                           node-ca-smzxx                                              0/1     Pending            0          9h
openshift-image-registry                           node-ca-wrtsm                                              0/1     Pending            0          9h
openshift-ingress-operator                         ingress-operator-54c46d4566-p8755                          0/2     Pending            0          9h
openshift-ingress                                  router-default-5774bfcdff-bdml5                            0/1     Pending            0          9h
openshift-ingress                                  router-default-5774bfcdff-gvvvd                            0/1     Pending            0          9h
openshift-insights                                 insights-operator-d6b6f5486-7pv56                          0/1     Pending            0          9h
openshift-kube-apiserver-operator                  kube-apiserver-operator-68554d9f6f-4gpqx                   0/1     Pending            0          9h
openshift-kube-controller-manager-operator         kube-controller-manager-operator-688fb78677-2c2x8          0/1     Pending            0          9h
openshift-kube-scheduler-operator                  openshift-kube-scheduler-operator-67c8f54d5f-vzw5k         0/1     Pending            0          9h
openshift-kube-storage-version-migrator-operator   kube-storage-version-migrator-operator-5b66cb684b-vgrlw    0/1     Pending            0          9h
openshift-kube-storage-version-migrator            migrator-c85595df4-xzw2r                                   0/1     Pending            0          9h
openshift-machine-api                              cluster-autoscaler-operator-5b6d5d7f9b-vg9dj               0/2     Pending            0          9h
openshift-machine-api                              machine-api-operator-7fb55d47d9-cs9r4                      0/2     Pending            0          9h
openshift-machine-config-operator                  machine-config-controller-55685c9588-qfxp5                 0/1     Pending            0          9h
openshift-machine-config-operator                  machine-config-daemon-2csbg                                0/2     Pending            0          9h
openshift-machine-config-operator                  machine-config-daemon-7jpjr                                0/2     Pending            0          9h
openshift-machine-config-operator                  machine-config-daemon-9df8s                                0/2     Pending            0          9h
openshift-machine-config-operator                  machine-config-daemon-blt9p                                0/2     Pending            0          9h
openshift-machine-config-operator                  machine-config-daemon-hgb4t                                0/2     Pending            0          9h
openshift-machine-config-operator                  machine-config-daemon-tmc9l                                0/2     Pending            0          9h
openshift-machine-config-operator                  machine-config-operator-66f5464954-8fskt                   0/1     Pending            0          9h
openshift-machine-config-operator                  machine-config-server-2xclx                                0/1     Pending            0          9h
openshift-machine-config-operator                  machine-config-server-8pjk5                                0/1     Pending            0          9h
openshift-machine-config-operator                  machine-config-server-mlmrx                                0/1     Pending            0          9h
openshift-marketplace                              marketplace-operator-84f685dc4d-vrwxx                      0/1     Pending            0          9h
openshift-monitoring                               alertmanager-main-0                                        0/5     Pending            0          9h
openshift-monitoring                               alertmanager-main-1                                        0/5     Pending            0          9h
openshift-monitoring                               alertmanager-main-2                                        0/5     Pending            0          9h
openshift-monitoring                               cluster-monitoring-operator-685f457c45-9mq5t               0/2     Pending            0          9h
openshift-monitoring                               grafana-86765b6dcf-jgg4c                                   0/2     Pending            0          9h
openshift-monitoring                               kube-state-metrics-56c8dc648c-8lk9m                        0/3     Pending            0          9h
openshift-monitoring                               node-exporter-2zf6k                                        0/2     Pending            0          9h
openshift-monitoring                               node-exporter-65mvv                                        0/2     Pending            0          9h
openshift-monitoring                               node-exporter-9crnv                                        0/2     Pending            0          9h
openshift-monitoring                               node-exporter-jr6vz                                        0/2     Pending            0          9h
openshift-monitoring                               node-exporter-r5x5j                                        0/2     Pending            0          9h
openshift-monitoring                               node-exporter-wlgf7                                        0/2     Pending            0          9h
openshift-monitoring                               openshift-state-metrics-6d5cf65975-qh4hm                   0/3     Pending            0          9h
openshift-monitoring                               prometheus-adapter-8548855c4b-62kbh                        0/1     Pending            0          9h
openshift-monitoring                               prometheus-adapter-8548855c4b-j2xch                        0/1     Pending            0          9h
openshift-monitoring                               prometheus-k8s-0                                           0/6     Pending            0          9h
openshift-monitoring                               prometheus-k8s-1                                           0/6     Pending            0          9h
openshift-monitoring                               prometheus-operator-79f7d5d6d9-bmsj8                       0/2     Pending            0          9h
openshift-monitoring                               telemeter-client-859856799-lcs9f                           0/3     Pending            0          9h
openshift-monitoring                               thanos-querier-7975db4f7d-476k8                            0/5     Pending            0          9h
openshift-monitoring                               thanos-querier-7975db4f7d-vpqn2                            0/5     Pending            0          9h
openshift-multus                                   multus-27r8k                                               0/1     Pending            0          9h
openshift-multus                                   multus-6rpd2                                               0/1     Pending            0          9h
openshift-multus                                   multus-admission-controller-mh5jf                          0/2     Pending            0          9h
openshift-multus                                   multus-admission-controller-sfz2g                          0/2     Pending            0          9h
openshift-multus                                   multus-admission-controller-wkt65                          0/2     Pending            0          9h
openshift-multus                                   multus-dxbh4                                               0/1     Pending            0          9h
openshift-multus                                   multus-lvlxq                                               0/1     Pending            0          9h
openshift-multus                                   multus-snr5s                                               0/1     Pending            0          9h
openshift-multus                                   multus-xpx5c                                               0/1     Pending            0          9h
openshift-multus                                   network-metrics-daemon-5m8d2                               0/2     Pending            0          9h
openshift-multus                                   network-metrics-daemon-8hzhp                               0/2     Pending            0          9h
openshift-multus                                   network-metrics-daemon-9wsc7                               0/2     Pending            0          9h
openshift-multus                                   network-metrics-daemon-kzvsj                               0/2     Pending            0          9h
openshift-multus                                   network-metrics-daemon-sm7k2                               0/2     Pending            0          9h
openshift-multus                                   network-metrics-daemon-xsq7g                               0/2     Pending            0          9h
openshift-network-operator                         network-operator-7c586cf699-r7224                          0/1     Pending            0          9h
openshift-oauth-apiserver                          apiserver-59b9c9df94-g4sxg                                 0/1     Pending            0          9h
openshift-oauth-apiserver                          apiserver-59b9c9df94-kjwdv                                 0/1     Pending            0          9h
openshift-oauth-apiserver                          apiserver-59b9c9df94-w5cz4                                 0/1     Pending            0          9h
openshift-operator-lifecycle-manager               catalog-operator-559b6fd6bf-5cjcx                          0/1     Pending            0          9h
openshift-operator-lifecycle-manager               olm-operator-8596dfd454-kkms8                              0/1     Pending            0          9h
openshift-operator-lifecycle-manager               packageserver-bc67cf7d9-575rw                              0/1     Pending            0          9h
openshift-operator-lifecycle-manager               packageserver-bc67cf7d9-5dwqr                              0/1     Pending            0          9h
openshift-sdn                                      ovs-8hxrs                                                  0/1     Pending            0          9h
openshift-sdn                                      ovs-bwbj8                                                  0/1     Pending            0          9h
openshift-sdn                                      ovs-ltrsr                                                  0/1     Pending            0          9h
openshift-sdn                                      ovs-m4sn4                                                  0/1     Pending            0          9h
openshift-sdn                                      ovs-swp6z                                                  0/1     Pending            0          9h
openshift-sdn                                      ovs-t7pvl                                                  0/1     Pending            0          9h
openshift-sdn                                      sdn-4rrlj                                                  0/2     Pending            0          9h
openshift-sdn                                      sdn-77vw9                                                  0/2     Pending            0          9h
openshift-sdn                                      sdn-8q6c2                                                  0/2     Pending            0          9h
openshift-sdn                                      sdn-controller-fp84s                                       0/1     Pending            0          9h
openshift-sdn                                      sdn-controller-tlhzx                                       0/1     Pending            0          9h
openshift-sdn                                      sdn-controller-wpz78                                       0/1     Pending            0          9h
openshift-sdn                                      sdn-g6k84                                                  0/2     Pending            0          9h
openshift-sdn                                      sdn-k96kz                                                  0/2     Pending            0          9h
openshift-sdn                                      sdn-pwzps                                                  0/2     Pending            0          9h
openshift-service-ca-operator                      service-ca-operator-8c5b688f7-lt8gx                        0/1     Pending            0          9h
openshift-service-ca                               service-ca-6885ffbf4c-szb7p                                0/1     Pending            0          9h

Based on the above moving bug to verified state

Comment 7 errata-xmlrpc 2021-01-25 20:02:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.6.13 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:0171


Note You need to log in before you can comment on or make changes to this bug.