Bug 2000608 - static pod startup monitor should log to a log file in addition to stderr
Summary: static pod startup monitor should log to a log file in addition to stderr
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-apiserver
Version: 4.9
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.9.0
Assignee: Stefan Schimanski
QA Contact: Rahul Gangwar
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-09-02 13:38 UTC by Lukasz Szaszkiewicz
Modified: 2021-10-18 17:51 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-10-18 17:51:04 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-kube-apiserver-operator pull 1219 0 None None None 2021-09-02 13:40:00 UTC
Red Hat Product Errata RHSA-2021:3759 0 None None None 2021-10-18 17:51:19 UTC

Description Lukasz Szaszkiewicz 2021-09-02 13:38:58 UTC
Since kubelet doesn't respect the restart policy for static pods
we need to store the logs in a file for future troubleshooting.

For kube-apiserver monitor we decided to store the logs at /var/log/kube-apiserver/startup.log

Comment 3 Ke Wang 2021-09-08 11:02:59 UTC
Verification steps:

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.nightly-2021-09-07-201519   True        False         5h16m   Cluster version is 4.9.0-0.nightly-2021-09-07-201519

$ oc debug node/<master node>

sh-4.4# ls /var/log/kube-apiserver/startup.log 
/var/log/kube-apiserver/startup.log

sh-4.4# tail -10 /var/log/kube-apiserver/startup.log
I0908 05:44:28.967662       1 monitor.go:78] Waiting for readiness (interval 1s, timeout 5m0s)...
I0908 05:44:29.278558       1 monitor.go:102] Watching kube-apiserver of revision 7: waiting for kube-apiserver static pod to listen on port 6443: Get "https://localhost:6443/healthz/etcd": dial tcp [::1]:6443: connect: connection refused (NetworkError)
I0908 05:44:30.582332       1 monitor.go:102] Watching kube-apiserver of revision 7: waiting for kube-apiserver static pod to listen on port 6443: Get "https://localhost:6443/healthz/etcd": dial tcp [::1]:6443: connect: connection refused (NetworkError)
I0908 05:44:35.760206       1 monitor.go:102] Watching kube-apiserver of revision 7: waiting for kube-apiserver static pod kube-apiserver-kewang-08sno1-bdvvm-master-0.c.openshift-qe.internal of revision 7, found 6 (UnexpectedRevision)
I0908 05:44:47.105300       1 monitor.go:102] Watching kube-apiserver of revision 7: waiting for kube-apiserver static pod kube-apiserver-kewang-08sno1-bdvvm-master-0.c.openshift-qe.internal to be ready (PodNodReady)
I0908 05:44:58.433282       1 fallback.go:198] Removed existing last known good revision manifest /etc/kubernetes/static-pod-resources/kube-apiserver-last-known-good
I0908 05:44:58.433365       1 fallback.go:205] Created a symlink /etc/kubernetes/static-pod-resources/kube-apiserver-last-known-good for /etc/kubernetes/static-pod-resources/kube-apiserver-pod-7/kube-apiserver-pod.yaml
I0908 05:44:58.449462       1 fallback.go:183] Successfully updated node kewang-08sno1-bdvvm-master-0.c.openshift-qe.internal to revision 7
I0908 05:44:58.449643       1 cmd.go:202] Waiting for SIGTERM...
I0908 05:44:58.476078       1 signal.go:18] Received SIGTERM or SIGINT signal, shutting down the process.

Worked as expected, so move the bug VERIFIED.

Comment 6 errata-xmlrpc 2021-10-18 17:51:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759


Note You need to log in before you can comment on or make changes to this bug.