Bug 1817430 - Machine Config operator parses kubelet.conf reducing usability.
Summary: Machine Config operator parses kubelet.conf reducing usability.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.5.0
Assignee: Urvashi Mohnani
QA Contact: MinLi
URL:
Whiteboard:
Depends On:
Blocks: 1834533 1834534
TreeView+ depends on / blocked
 
Reported: 2020-03-26 10:52 UTC by David Hernández Fernández
Modified: 2020-07-13 17:24 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1834533 1834534 (view as bug list)
Environment:
Last Closed: 2020-07-13 17:23:46 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift machine-config-operator pull 1725 0 None closed Bug 1817430: Parse the kubelet.conf data after an update 2020-12-18 03:01:37 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:24:15 UTC

Description David Hernández Fernández 2020-03-26 10:52:45 UTC
Description of problem: machine-config operator seems to be parsing kubelet.conf. It is still working but usability is dead.

Version-Release number of selected component (if applicable):
$ oc version
Client Version: 4.3.5-202002280657-c25fb9c
Server Version: 4.3.5
Kubernetes Version: v1.16.2


How reproducible: When created CR on Kubelet.

[quicklab@master-2 ~]$  oc label machineconfigpool worker custom-kubelet=logrotation
machineconfigpool.machineconfiguration.openshift.io/worker labeled

[quicklab@master-2 ~]$ cat logrotation.yaml 
apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: cr-logrotation
spec:
  machineConfigPoolSelector:
    matchLabels:
      custom-kubelet: logrotation
  kubeletConfig:
    containerLogMaxFiles: 58
    container-log-max-size: 800Mi
[quicklab@master-2 ~]$ oc create -f logrotation.yaml
[quicklab@master-2 ~]$ oc get kubeletconfig -o yaml
apiVersion: v1
items:
- apiVersion: machineconfiguration.openshift.io/v1
  kind: KubeletConfig
  metadata:
    creationTimestamp: "2020-03-24T11:46:11Z"
    finalizers:
    - 99-worker-75c6f8de-867f-471e-8a09-09d05ee48e0d-kubelet
    generation: 1
    name: cr-logrotation
    resourceVersion: "373981"
    selfLink: /apis/machineconfiguration.openshift.io/v1/kubeletconfigs/cr-logrotation
    uid: e0dc2521-25f5-4982-8592-3ec83a6139c9
  spec:
    kubeletConfig:
      container-log-max-size: 800Mi                  ««««««««««««««««««««««««««« added.
      containerLogMaxFiles: 58                        ««««««««««««««««««««««««««« added.
    machineConfigPoolSelector:
      matchLabels:
        custom-kubelet: logrotation                  ««««««««««««««««««««««««««« added.
  status:
    conditions:
    - lastTransitionTime: "2020-03-24T11:46:11Z"
      message: Success
      status: "True"
      type: Success
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
[quicklab@master-2 ~]$ 
[quicklab@master-2 ~]$ oc get machineconfig | grep -i kubelet
01-master-kubelet                                           d5599de7a6b86ec385e0f9c849e93977fcb4eeb8   2.2.0             22h
01-worker-kubelet                                           d5599de7a6b86ec385e0f9c849e93977fcb4eeb8   2.2.0             22h
99-worker-75c6f8de-867f-471e-8a09-09d05ee48e0d-kubelet      d5599de7a6b86ec385e0f9c849e93977fcb4eeb8   2.2.0             74s

Actual results:
The content is working but parsed:
sh-4.4# cat /etc/kubernetes/kubelet.conf
{"kind":"KubeletConfiguration","apiVersion":"kubelet.config.k8s.io/v1beta1","staticPodPath":"/etc/kubernetes/manifests","syncFrequency":"0s","fileCheckFrequency":"0s","httpCheckFrequency":"0s","rotateCertificates":true,"serverTLSBootstrap":true,"authentication":{"x509":{"clientCAFile":"/etc/kubernetes/kubelet-ca.crt"},"webhook":{"cacheTTL":"0s"},"anonymous":{"enabled":false}},"authorization":{"webhook":{"cacheAuthorizedTTL":"0s","cacheUnauthorizedTTL":"0s"}},"clusterDomain":"cluster.local","clusterDNS":["172.30.0.10"],"streamingConnectionIdleTimeout":"0s","nodeStatusUpdateFrequency":"0s","nodeStatusReportFrequency":"0s","imageMinimumGCAge":"0s","volumeStatsAggPeriod":"0s","cgroupDriver":"systemd","cpuManagerReconcilePeriod":"0s","runtimeRequestTimeout":"0s","maxPods":250,"kubeAPIQPS":50,"kubeAPIBurst":100,"serializeImagePulls":false,"evictionPressureTransitionPeriod":"0s","featureGates":{"LegacyNodeRoleBehavior":false,"NodeDisruptionExclusion":true,"RotateKubeletServerCertificate":true,"ServiceNodeExclusion":true,"SupportPodPidsLimit":true},"containerLogMaxSize":"50Mi","containerLogMaxFiles":58,"systemReserved":{"cpu":"500m","memory":"500Mi"}}

Expected results: Files not being parsed. Files being shown with correct yaml/json indentation.

Temporary workaround:
sh-4.4# cat /etc/kubernetes/kubelet.conf  | jq "."
{
  "kind": "KubeletConfiguration",
....
 

What I did to overrride the config was the following:

sh-4.4# jq "." < /etc/kubernetes/kubelet.conf  > /etc/kubernetes/newkubelet.conf
sh-4.4# cat /etc/kubernetes/kubelet.conf (old)
{"kind":"KubeletConfiguration","apiVersion":"kubelet.config.k8s.io/v1beta1","staticPodPath":"/etc/kubernetes/manifests","syncFrequency":"0s","fileCheckFrequency":"0s","httpCheckFrequency":"0s","rotateCertificates":true,"serverTLSBootstrap":true,"authentication":{"x509":{"clientCAFile":"/etc/kubernetes/kubelet-ca.crt"},"webhook":{"cacheTTL":"0s"},"anonymous":{"enabled":false}},"authorization":{"webhook":{"cacheAuthorizedTTL":"0s","cacheUnauthorizedTTL":"0s"}},"clusterDomain":"cluster.local","clusterDNS":["172.30.0.10"],"streamingConnectionIdleTimeout":"0s","nodeStatusUpdateFrequency":"0s","nodeStatusReportFrequency":"0s","imageMinimumGCAge":"0s","volumeStatsAggPeriod":"0s","cgroupDriver":"systemd","cpuManagerReconcilePeriod":"0s","runtimeRequestTimeout":"0s","maxPods":250,"kubeAPIQPS":50,"kubeAPIBurst":100,"serializeImagePulls":false,"evictionPressureTransitionPeriod":"0s","featureGates":{"LegacyNodeRoleBehavior":false,"NodeDisruptionExclusion":true,"RotateKubeletServerCertificate":true,"ServiceNodeExclusion":true,"SupportPodPidsLimit":true},"containerLogMaxSize":"50Mi","containerLogMaxFiles":58,"systemReserved":{"cpu":"500m","memory":"500Mi"}}
sh-4.4# cat /etc/kubernetes/newkubelet.conf (new corrected)
{
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  "staticPodPath": "/etc/kubernetes/manifests",
 ..........................................
  "systemReserved": {
    "cpu": "500m",
    "memory": "500Mi"
  }
}
sh-4.4# cp -p /etc/kubernetes/kubelet.conf /etc/kubernetes/parsed_backup_kubelet.conf (get a backup of the parsed one, just in case)
sh-4.4# mv /etc/kubernetes/newkubelet.conf /etc/kubernetes/kubelet.conf (replace the new one corrected with the parsed one)
sh-4.4# cat /etc/kubernetes/kubelet.conf (confirm it is working as expected)
{
  "kind": "KubeletConfiguration",
..................................................
  }
}

Comment 1 Antonio Murdaca 2020-03-26 11:42:30 UTC
I guess we just need to json format with indent using golang, moving to Node

Comment 2 David Hernández Fernández 2020-03-27 10:12:16 UTC
That would be sweet, thanks for taking a look Antonio. Note that the file is still usable and readable after being parsed, the issue is just the usability.

Comment 3 Urvashi Mohnani 2020-05-11 21:27:50 UTC
Fix is in https://github.com/openshift/machine-config-operator/pull/1725

Comment 7 Urvashi Mohnani 2020-05-26 13:52:53 UTC
Hi Sunil, your output above looks correct. The JSON is parsed and indented correctly for the kubelet.conf file. This should be moved to verified please.

Comment 8 Sunil Choudhary 2020-05-27 05:17:51 UTC
Hi Urvashi,

Sorry I mean to say the value of containerLogMaxSize 800Mi which I specified in custom kubelet yaml is not reflected in node's kubelet conf.
Yeah, the JSON parsing is correct now.

Comment 9 Urvashi Mohnani 2020-05-27 14:59:25 UTC
Hey Sunil,

In your custom kubelet yaml, you need to use "containerLogMaxSize" instead of "container-log-max-size" and that should work.

Comment 12 errata-xmlrpc 2020-07-13 17:23:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.