Since kubelet doesn't respect the restart policy for static pods we need to store the logs in a file for future troubleshooting. For kube-apiserver monitor we decided to store the logs at /var/log/kube-apiserver/startup.log
Verification steps: $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2021-09-07-201519 True False 5h16m Cluster version is 4.9.0-0.nightly-2021-09-07-201519 $ oc debug node/<master node> sh-4.4# ls /var/log/kube-apiserver/startup.log /var/log/kube-apiserver/startup.log sh-4.4# tail -10 /var/log/kube-apiserver/startup.log I0908 05:44:28.967662 1 monitor.go:78] Waiting for readiness (interval 1s, timeout 5m0s)... I0908 05:44:29.278558 1 monitor.go:102] Watching kube-apiserver of revision 7: waiting for kube-apiserver static pod to listen on port 6443: Get "https://localhost:6443/healthz/etcd": dial tcp [::1]:6443: connect: connection refused (NetworkError) I0908 05:44:30.582332 1 monitor.go:102] Watching kube-apiserver of revision 7: waiting for kube-apiserver static pod to listen on port 6443: Get "https://localhost:6443/healthz/etcd": dial tcp [::1]:6443: connect: connection refused (NetworkError) I0908 05:44:35.760206 1 monitor.go:102] Watching kube-apiserver of revision 7: waiting for kube-apiserver static pod kube-apiserver-kewang-08sno1-bdvvm-master-0.c.openshift-qe.internal of revision 7, found 6 (UnexpectedRevision) I0908 05:44:47.105300 1 monitor.go:102] Watching kube-apiserver of revision 7: waiting for kube-apiserver static pod kube-apiserver-kewang-08sno1-bdvvm-master-0.c.openshift-qe.internal to be ready (PodNodReady) I0908 05:44:58.433282 1 fallback.go:198] Removed existing last known good revision manifest /etc/kubernetes/static-pod-resources/kube-apiserver-last-known-good I0908 05:44:58.433365 1 fallback.go:205] Created a symlink /etc/kubernetes/static-pod-resources/kube-apiserver-last-known-good for /etc/kubernetes/static-pod-resources/kube-apiserver-pod-7/kube-apiserver-pod.yaml I0908 05:44:58.449462 1 fallback.go:183] Successfully updated node kewang-08sno1-bdvvm-master-0.c.openshift-qe.internal to revision 7 I0908 05:44:58.449643 1 cmd.go:202] Waiting for SIGTERM... I0908 05:44:58.476078 1 signal.go:18] Received SIGTERM or SIGINT signal, shutting down the process. Worked as expected, so move the bug VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759