+++ This bug was initially created as a clone of Bug #1817430 +++

Description of problem: machine-config operator seems to be parsing kubelet.conf. It is still working but usability is dead.

Version-Release number of selected component (if applicable):
$ oc version
Client Version: 4.3.5-202002280657-c25fb9c
Server Version: 4.3.5
Kubernetes Version: v1.16.2


How reproducible: When created CR on Kubelet.

[quicklab@master-2 ~]$  oc label machineconfigpool worker custom-kubelet=logrotation
machineconfigpool.machineconfiguration.openshift.io/worker labeled

[quicklab@master-2 ~]$ cat logrotation.yaml 
apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: cr-logrotation
spec:
  machineConfigPoolSelector:
    matchLabels:
      custom-kubelet: logrotation
  kubeletConfig:
    containerLogMaxFiles: 58
    container-log-max-size: 800Mi
[quicklab@master-2 ~]$ oc create -f logrotation.yaml
[quicklab@master-2 ~]$ oc get kubeletconfig -o yaml
apiVersion: v1
items:
- apiVersion: machineconfiguration.openshift.io/v1
  kind: KubeletConfig
  metadata:
    creationTimestamp: "2020-03-24T11:46:11Z"
    finalizers:
    - 99-worker-75c6f8de-867f-471e-8a09-09d05ee48e0d-kubelet
    generation: 1
    name: cr-logrotation
    resourceVersion: "373981"
    selfLink: /apis/machineconfiguration.openshift.io/v1/kubeletconfigs/cr-logrotation
    uid: e0dc2521-25f5-4982-8592-3ec83a6139c9
  spec:
    kubeletConfig:
      container-log-max-size: 800Mi                  ««««««««««««««««««««««««««« added.
      containerLogMaxFiles: 58                        ««««««««««««««««««««««««««« added.
    machineConfigPoolSelector:
      matchLabels:
        custom-kubelet: logrotation                  ««««««««««««««««««««««««««« added.
  status:
    conditions:
    - lastTransitionTime: "2020-03-24T11:46:11Z"
      message: Success
      status: "True"
      type: Success
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
[quicklab@master-2 ~]$ 
[quicklab@master-2 ~]$ oc get machineconfig | grep -i kubelet
01-master-kubelet                                           d5599de7a6b86ec385e0f9c849e93977fcb4eeb8   2.2.0             22h
01-worker-kubelet                                           d5599de7a6b86ec385e0f9c849e93977fcb4eeb8   2.2.0             22h
99-worker-75c6f8de-867f-471e-8a09-09d05ee48e0d-kubelet      d5599de7a6b86ec385e0f9c849e93977fcb4eeb8   2.2.0             74s

Actual results:
The content is working but parsed:
sh-4.4# cat /etc/kubernetes/kubelet.conf
{"kind":"KubeletConfiguration","apiVersion":"kubelet.config.k8s.io/v1beta1","staticPodPath":"/etc/kubernetes/manifests","syncFrequency":"0s","fileCheckFrequency":"0s","httpCheckFrequency":"0s","rotateCertificates":true,"serverTLSBootstrap":true,"authentication":{"x509":{"clientCAFile":"/etc/kubernetes/kubelet-ca.crt"},"webhook":{"cacheTTL":"0s"},"anonymous":{"enabled":false}},"authorization":{"webhook":{"cacheAuthorizedTTL":"0s","cacheUnauthorizedTTL":"0s"}},"clusterDomain":"cluster.local","clusterDNS":["172.30.0.10"],"streamingConnectionIdleTimeout":"0s","nodeStatusUpdateFrequency":"0s","nodeStatusReportFrequency":"0s","imageMinimumGCAge":"0s","volumeStatsAggPeriod":"0s","cgroupDriver":"systemd","cpuManagerReconcilePeriod":"0s","runtimeRequestTimeout":"0s","maxPods":250,"kubeAPIQPS":50,"kubeAPIBurst":100,"serializeImagePulls":false,"evictionPressureTransitionPeriod":"0s","featureGates":{"LegacyNodeRoleBehavior":false,"NodeDisruptionExclusion":true,"RotateKubeletServerCertificate":true,"ServiceNodeExclusion":true,"SupportPodPidsLimit":true},"containerLogMaxSize":"50Mi","containerLogMaxFiles":58,"systemReserved":{"cpu":"500m","memory":"500Mi"}}

Expected results: Files not being parsed. Files being shown with correct yaml/json indentation.

Temporary workaround:
sh-4.4# cat /etc/kubernetes/kubelet.conf  | jq "."
{
  "kind": "KubeletConfiguration",
....
 

What I did to overrride the config was the following:

sh-4.4# jq "." < /etc/kubernetes/kubelet.conf  > /etc/kubernetes/newkubelet.conf
sh-4.4# cat /etc/kubernetes/kubelet.conf (old)
{"kind":"KubeletConfiguration","apiVersion":"kubelet.config.k8s.io/v1beta1","staticPodPath":"/etc/kubernetes/manifests","syncFrequency":"0s","fileCheckFrequency":"0s","httpCheckFrequency":"0s","rotateCertificates":true,"serverTLSBootstrap":true,"authentication":{"x509":{"clientCAFile":"/etc/kubernetes/kubelet-ca.crt"},"webhook":{"cacheTTL":"0s"},"anonymous":{"enabled":false}},"authorization":{"webhook":{"cacheAuthorizedTTL":"0s","cacheUnauthorizedTTL":"0s"}},"clusterDomain":"cluster.local","clusterDNS":["172.30.0.10"],"streamingConnectionIdleTimeout":"0s","nodeStatusUpdateFrequency":"0s","nodeStatusReportFrequency":"0s","imageMinimumGCAge":"0s","volumeStatsAggPeriod":"0s","cgroupDriver":"systemd","cpuManagerReconcilePeriod":"0s","runtimeRequestTimeout":"0s","maxPods":250,"kubeAPIQPS":50,"kubeAPIBurst":100,"serializeImagePulls":false,"evictionPressureTransitionPeriod":"0s","featureGates":{"LegacyNodeRoleBehavior":false,"NodeDisruptionExclusion":true,"RotateKubeletServerCertificate":true,"ServiceNodeExclusion":true,"SupportPodPidsLimit":true},"containerLogMaxSize":"50Mi","containerLogMaxFiles":58,"systemReserved":{"cpu":"500m","memory":"500Mi"}}
sh-4.4# cat /etc/kubernetes/newkubelet.conf (new corrected)
{
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  "staticPodPath": "/etc/kubernetes/manifests",
 ..........................................
  "systemReserved": {
    "cpu": "500m",
    "memory": "500Mi"
  }
}
sh-4.4# cp -p /etc/kubernetes/kubelet.conf /etc/kubernetes/parsed_backup_kubelet.conf (get a backup of the parsed one, just in case)
sh-4.4# mv /etc/kubernetes/newkubelet.conf /etc/kubernetes/kubelet.conf (replace the new one corrected with the parsed one)
sh-4.4# cat /etc/kubernetes/kubelet.conf (confirm it is working as expected)
{
  "kind": "KubeletConfiguration",
..................................................
  }
}

--- Additional comment from Antonio Murdaca on 2020-03-26 11:42:30 UTC ---

I guess we just need to json format with indent using golang, moving to Node

--- Additional comment from David Hernández Fernández on 2020-03-27 10:12:16 UTC ---

That would be sweet, thanks for taking a look Antonio. Note that the file is still usable and readable after being parsed, the issue is just the usability.

--- Additional comment from Urvashi Mohnani on 2020-05-11 21:27:50 UTC ---

Fix is in https://github.com/openshift/machine-config-operator/pull/1725