Pick https://github.com/kubernetes/kubernetes/pull/86816 into origin to provide dynamic reloading for CSR signing controllers to match the kube-apiserver for cert/key pair reloading behavior.
Confirmed with payload: 4.5.0-0.nightly-2020-03-06-190457, the issue has fixed: One terminal: [root@dhcp-140-138 roottest]# oc delete secrets csr-signer -n openshift-kube-controller-manager-operator secret "csr-signer" deleted [root@dhcp-140-138 roottest]# oc get secrets -n openshift-kube-controller-manager-operator NAME TYPE DATA AGE builder-dockercfg-fqm9c kubernetes.io/dockercfg 1 67m builder-token-vcgqq kubernetes.io/service-account-token 4 68m builder-token-vnxqt kubernetes.io/service-account-token 4 68m csr-signer kubernetes.io/tls 2 15s Another terminal: [root@dhcp-140-138 roottest]# oc logs -f po/kube-controller-manager-ip-10-0-135-80.us-east-2.compute.internal Copying system trust bundle I0309 01:29:59.252226 1 feature_gate.go:244] feature gates: &{map[RotateKubeletServerCertificate:true]} I0309 01:29:59.252359 1 feature_gate.go:244] feature gates: &{map[RotateKubeletServerCertificate:true SupportPodPidsLimit:true]} ... I0309 02:31:59.866375 1 tlsconfig.go:179] loaded client CA [0/"client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt,request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt"]: "admin-kubeconfig-signer" [] issuer="<self>" (2020-03-09 01:01:23 +0000 UTC to 2030-03-07 01:01:23 +0000 UTC (now=2020-03-09 02:31:59.86635187 +0000 UTC)) I0309 02:31:59.866409 1 tlsconfig.go:179] loaded client CA [1/"client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt,request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt"]: "kube-csr-signer_@1583716679" [] issuer="kubelet-signer" (2020-03-09 01:17:59 +0000 UTC to 2020-03-10 01:01:37 +0000 UTC (now=2020-03-09 02:31:59.866398885 +0000 UTC)) I0309 02:31:59.866426 1 tlsconfig.go:179] loaded client CA [2/"client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt,request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt"]: "kubelet-signer" [] issuer="<self>" (2020-03-09 01:01:37 +0000 UTC to 2020-03-10 01:01:37 +0000 UTC (now=2020-03-09 02:31:59.866418712 +0000 UTC)) I0309 02:31:59.866443 1 tlsconfig.go:179] loaded client CA [3/"client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt,request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt"]: "kube-csr-signer_@1583721062" [] issuer="kubelet-signer" (2020-03-09 02:31:01 +0000 UTC to 2020-03-10 01:01:37 +0000 UTC (now=2020-03-09 02:31:59.866435024 +0000 UTC)) I0309 02:31:59.866461 1 tlsconfig.go:179] loaded client CA [4/"client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt,request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt"]: "kube-apiserver-to-kubelet-signer" [] issuer="<self>" (2020-03-09 01:01:38 +0000 UTC to 2021-03-09 01:01:38 +0000 UTC (now=2020-03-09 02:31:59.866453375 +0000 UTC)) I0309 02:31:59.866477 1 tlsconfig.go:179] loaded client CA [5/"client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt,request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt"]: "kube-control-plane-signer" [] issuer="<self>" (2020-03-09 01:01:38 +0000 UTC to 2021-03-09 01:01:38 +0000 UTC (now=2020-03-09 02:31:59.866469916 +0000 UTC)) I0309 02:31:59.866493 1 tlsconfig.go:179] loaded client CA [6/"client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt,request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt"]: "kubelet-bootstrap-kubeconfig-signer" [] issuer="<self>" (2020-03-09 01:01:26 +0000 UTC to 2030-03-07 01:01:26 +0000 UTC (now=2020-03-09 02:31:59.866485372 +0000 UTC)) I0309 02:31:59.866510 1 tlsconfig.go:179] loaded client CA [7/"client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt,request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt"]: "aggregator-signer" [] issuer="<self>" (2020-03-09 01:01:33 +0000 UTC to 2020-03-10 01:01:33 +0000 UTC (now=2020-03-09 02:31:59.866501495 +0000 UTC)) I0309 02:31:59.866782 1 tlsconfig.go:201] loaded serving cert ["serving-cert::/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.crt::/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.key"]: "kube-controller-manager.openshift-kube-controller-manager.svc" [serving] validServingFor=[kube-controller-manager.openshift-kube-controller-manager.svc,kube-controller-manager.openshift-kube-controller-manager.svc.cluster.local] issuer="openshift-service-serving-signer@1583716682" (2020-03-09 01:18:15 +0000 UTC to 2022-03-09 01:18:16 +0000 UTC (now=2020-03-09 02:31:59.866767461 +0000 UTC))
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409