Bug 1896329 - Revert KUBELET_LOG_LEVEL back to level 3 (
Summary: Revert KUBELET_LOG_LEVEL back to level 3 (
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Machine Config Operator
Version: 4.6
Hardware: All
OS: Linux
urgent
urgent
Target Milestone: ---
: 4.6.z
Assignee: Kirsten Garrison
QA Contact: Michael Nguyen
URL:
Whiteboard:
: 1896332 (view as bug list)
Depends On: 1895385
Blocks: 1900013
TreeView+ depends on / blocked
 
Reported: 2020-11-10 09:59 UTC by Antonio Murdaca
Modified: 2021-01-11 13:45 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1895385
: 1900013 (view as bug list)
Environment:
Last Closed: 2020-11-30 16:46:09 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift machine-config-operator pull 2244 0 None closed [release-4.6] Bug 1896329: Drop kubelet logging back down to level 3 2021-02-02 17:52:01 UTC
Red Hat Product Errata RHBA-2020:5115 0 None None None 2020-11-30 16:46:29 UTC

Comment 1 Antonio Murdaca 2020-11-10 10:02:09 UTC
*** Bug 1896332 has been marked as a duplicate of this bug. ***

Comment 4 Michael Nguyen 2020-11-24 15:58:27 UTC
Verified on 4.6.0-0.nightly-2020-11-22-160856.  Log level is 3.

$ get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2020-11-22-160856   True        False         67s     Cluster version is 4.6.0-0.nightly-2020-11-22-160856
$ oc get nodes
NAME                                         STATUS   ROLES    AGE   VERSION
ip-10-0-130-200.us-west-2.compute.internal   Ready    master   26m   v1.19.0+43983cd
ip-10-0-137-216.us-west-2.compute.internal   Ready    worker   16m   v1.19.0+43983cd
ip-10-0-176-34.us-west-2.compute.internal    Ready    master   25m   v1.19.0+43983cd
ip-10-0-189-58.us-west-2.compute.internal    Ready    worker   16m   v1.19.0+43983cd
ip-10-0-196-76.us-west-2.compute.internal    Ready    master   26m   v1.19.0+43983cd
ip-10-0-209-11.us-west-2.compute.internal    Ready    worker   17m   v1.19.0+43983cd
$ oc debug node/ip-10-0-130-200.us-west-2.compute.internal 
Starting pod/ip-10-0-130-200us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# systemctl cat kubelet.service
# /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Wants=rpc-statd.service network-online.target crio.service
After=network-online.target crio.service

[Service]
Type=notify
ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests
ExecStartPre=/bin/rm -f /var/lib/kubelet/cpu_manager_state
Environment="KUBELET_LOG_LEVEL=3"
EnvironmentFile=/etc/os-release
EnvironmentFile=-/etc/kubernetes/kubelet-workaround
EnvironmentFile=-/etc/kubernetes/kubelet-env

ExecStart=/usr/bin/hyperkube \
    kubelet \
      --config=/etc/kubernetes/kubelet.conf \
      --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
      --kubeconfig=/var/lib/kubelet/kubeconfig \
      --container-runtime=remote \
      --container-runtime-endpoint=/var/run/crio/crio.sock \
      --runtime-cgroups=/system.slice/crio.service \
      --node-labels=node-role.kubernetes.io/master,node.openshift.io/os_id=${ID} \
      --node-ip=${KUBELET_NODE_IP} \
      --minimum-container-ttl-duration=6m0s \
      --cloud-provider=aws \
      --volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec \
       \
      --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
      --pod-infra-container-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:40e18ec81bab4d8f>
      --v=${KUBELET_LOG_LEVEL}

Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/kubelet.service.d/10-mco-default-env.conf
[Unit]
sh-4.4# exit
exit
sh-4.2# exit
exit

Removing debug pod ...
$ oc debug node/ip-10-0-137-216.us-west-2.compute.internal
Starting pod/ip-10-0-137-216us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# systemctl cat kubelet.service
# /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Wants=rpc-statd.service network-online.target crio.service
After=network-online.target crio.service

[Service]
Type=notify
ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests
ExecStartPre=/bin/rm -f /var/lib/kubelet/cpu_manager_state
Environment="KUBELET_LOG_LEVEL=3"
EnvironmentFile=/etc/os-release
EnvironmentFile=-/etc/kubernetes/kubelet-workaround
EnvironmentFile=-/etc/kubernetes/kubelet-env

ExecStart=/usr/bin/hyperkube \
    kubelet \
      --config=/etc/kubernetes/kubelet.conf \
      --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
      --kubeconfig=/var/lib/kubelet/kubeconfig \
      --container-runtime=remote \
      --container-runtime-endpoint=/var/run/crio/crio.sock \
      --runtime-cgroups=/system.slice/crio.service \
      --node-labels=node-role.kubernetes.io/worker,node.openshift.io/os_id=${ID} \
      --node-ip=${KUBELET_NODE_IP} \
      --minimum-container-ttl-duration=6m0s \
      --volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec \
      --cloud-provider=aws \
       \
      --pod-infra-container-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:40e18ec81bab4d8>
      --v=${KUBELET_LOG_LEVEL}

Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/kubelet.service.d/10-mco-default-env.conf
[Unit]
sh-4.4# exit
exit
sh-4.2# exit
exit

Removing debug pod ...

Comment 6 errata-xmlrpc 2020-11-30 16:46:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6.6 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:5115


Note You need to log in before you can comment on or make changes to this bug.