Bug 1896329

Summary:	Revert KUBELET_LOG_LEVEL back to level 3 (
Product:	OpenShift Container Platform	Reporter:	Antonio Murdaca <amurdaca>
Component:	Machine Config Operator	Assignee:	Kirsten Garrison <kgarriso>
Machine Config Operator sub component:	Machine Config Operator	QA Contact:	Michael Nguyen <mnguyen>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	urgent
Priority:	urgent	CC:	erich, john.johansson, kgarriso, mkrejci, mnguyen, mrobson, nnosenzo, rheinzma, rioliu, rphillips, scott.worthington, sreber, ssadhale, steven.barre
Version:	4.6
Target Milestone:	---
Target Release:	4.6.z
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1895385
Clones:	1900013 (view as bug list)		Environment:
Last Closed:	2020-11-30 16:46:09 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1895385
Bug Blocks:	1900013

Comment 1 Antonio Murdaca 2020-11-10 10:02:09 UTC

*** Bug 1896332 has been marked as a duplicate of this bug. ***

Comment 4 Michael Nguyen 2020-11-24 15:58:27 UTC

Verified on 4.6.0-0.nightly-2020-11-22-160856.  Log level is 3.

$ get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2020-11-22-160856   True        False         67s     Cluster version is 4.6.0-0.nightly-2020-11-22-160856
$ oc get nodes
NAME                                         STATUS   ROLES    AGE   VERSION
ip-10-0-130-200.us-west-2.compute.internal   Ready    master   26m   v1.19.0+43983cd
ip-10-0-137-216.us-west-2.compute.internal   Ready    worker   16m   v1.19.0+43983cd
ip-10-0-176-34.us-west-2.compute.internal    Ready    master   25m   v1.19.0+43983cd
ip-10-0-189-58.us-west-2.compute.internal    Ready    worker   16m   v1.19.0+43983cd
ip-10-0-196-76.us-west-2.compute.internal    Ready    master   26m   v1.19.0+43983cd
ip-10-0-209-11.us-west-2.compute.internal    Ready    worker   17m   v1.19.0+43983cd
$ oc debug node/ip-10-0-130-200.us-west-2.compute.internal 
Starting pod/ip-10-0-130-200us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# systemctl cat kubelet.service
# /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Wants=rpc-statd.service network-online.target crio.service
After=network-online.target crio.service

[Service]
Type=notify
ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests
ExecStartPre=/bin/rm -f /var/lib/kubelet/cpu_manager_state
Environment="KUBELET_LOG_LEVEL=3"
EnvironmentFile=/etc/os-release
EnvironmentFile=-/etc/kubernetes/kubelet-workaround
EnvironmentFile=-/etc/kubernetes/kubelet-env

ExecStart=/usr/bin/hyperkube \
    kubelet \
      --config=/etc/kubernetes/kubelet.conf \
      --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
      --kubeconfig=/var/lib/kubelet/kubeconfig \
      --container-runtime=remote \
      --container-runtime-endpoint=/var/run/crio/crio.sock \
      --runtime-cgroups=/system.slice/crio.service \
      --node-labels=node-role.kubernetes.io/master,node.openshift.io/os_id=${ID} \
      --node-ip=${KUBELET_NODE_IP} \
      --minimum-container-ttl-duration=6m0s \
      --cloud-provider=aws \
      --volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec \
       \
      --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
      --pod-infra-container-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:40e18ec81bab4d8f>
      --v=${KUBELET_LOG_LEVEL}

Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/kubelet.service.d/10-mco-default-env.conf
[Unit]
sh-4.4# exit
exit
sh-4.2# exit
exit

Removing debug pod ...
$ oc debug node/ip-10-0-137-216.us-west-2.compute.internal
Starting pod/ip-10-0-137-216us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# systemctl cat kubelet.service
# /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Wants=rpc-statd.service network-online.target crio.service
After=network-online.target crio.service

[Service]
Type=notify
ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests
ExecStartPre=/bin/rm -f /var/lib/kubelet/cpu_manager_state
Environment="KUBELET_LOG_LEVEL=3"
EnvironmentFile=/etc/os-release
EnvironmentFile=-/etc/kubernetes/kubelet-workaround
EnvironmentFile=-/etc/kubernetes/kubelet-env

ExecStart=/usr/bin/hyperkube \
    kubelet \
      --config=/etc/kubernetes/kubelet.conf \
      --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
      --kubeconfig=/var/lib/kubelet/kubeconfig \
      --container-runtime=remote \
      --container-runtime-endpoint=/var/run/crio/crio.sock \
      --runtime-cgroups=/system.slice/crio.service \
      --node-labels=node-role.kubernetes.io/worker,node.openshift.io/os_id=${ID} \
      --node-ip=${KUBELET_NODE_IP} \
      --minimum-container-ttl-duration=6m0s \
      --volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec \
      --cloud-provider=aws \
       \
      --pod-infra-container-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:40e18ec81bab4d8>
      --v=${KUBELET_LOG_LEVEL}

Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/kubelet.service.d/10-mco-default-env.conf
[Unit]
sh-4.4# exit
exit
sh-4.2# exit
exit

Removing debug pod ...

Comment 6 errata-xmlrpc 2020-11-30 16:46:09 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6.6 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:5115