Verified on 4.4.0-0.nightly-2020-03-06-141620 = 1. Label the worker mcp == $ oc label mcp/worker custom-kubelet=small-pods machineconfigpool.machineconfiguration.openshift.io/worker labeled [mnguyen@pet30 4.4]$ oc get mcp/worker -o yaml apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfigPool metadata: creationTimestamp: "2020-03-10T17:16:10Z" generation: 2 labels: custom-kubelet: small-pods machineconfiguration.openshift.io/mco-built-in: "" name: worker resourceVersion: "58322" selfLink: /apis/machineconfiguration.openshift.io/v1/machineconfigpools/worker uid: 7517b173-22ca-4f5e-aac2-f1b4a6da40fe --snip-- = 2. create kubletconfig = $ cat <<EOF>kc.yaml > apiVersion: machineconfiguration.openshift.io/v1 > kind: KubeletConfig > metadata: > name: foobar > spec: > machineConfigPoolSelector: > matchLabels: > custom-kubelet: small-pods > kubeletConfig: > apiVersion: kubelet.config.k8s.io/v1 > kind: kubeletConfig > maxPods: 100 > EOF = 3. Apply the kubeletconfig file = $ oc get mc NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 00-master 961ea024d5fbdd2c3daed6a9a2d4e8069a2849de 2.2.0 148m 00-worker 961ea024d5fbdd2c3daed6a9a2d4e8069a2849de 2.2.0 148m 01-master-container-runtime 961ea024d5fbdd2c3daed6a9a2d4e8069a2849de 2.2.0 148m 01-master-kubelet 961ea024d5fbdd2c3daed6a9a2d4e8069a2849de 2.2.0 148m 01-worker-container-runtime 961ea024d5fbdd2c3daed6a9a2d4e8069a2849de 2.2.0 148m 01-worker-kubelet 961ea024d5fbdd2c3daed6a9a2d4e8069a2849de 2.2.0 148m 99-master-691675c6-709b-4119-9c3d-793f274c88fe-registries 961ea024d5fbdd2c3daed6a9a2d4e8069a2849de 2.2.0 148m 99-master-ssh 2.2.0 149m 99-worker-7517b173-22ca-4f5e-aac2-f1b4a6da40fe-kubelet 961ea024d5fbdd2c3daed6a9a2d4e8069a2849de 2.2.0 18s 99-worker-7517b173-22ca-4f5e-aac2-f1b4a6da40fe-registries 961ea024d5fbdd2c3daed6a9a2d4e8069a2849de 2.2.0 148m 99-worker-ssh 2.2.0 149m rendered-master-f93a612a63a36e8c16a93b14c212a9e4 961ea024d5fbdd2c3daed6a9a2d4e8069a2849de 2.2.0 148m rendered-worker-58a6175e3cfe5016a9d028abb2e3b689 961ea024d5fbdd2c3daed6a9a2d4e8069a2849de 2.2.0 13s rendered-worker-f67898c76cbf120517cfbf15afdbe494 961ea024d5fbdd2c3daed6a9a2d4e8069a2849de 2.2.0 148m $ oc apply -f kc.yaml kubeletconfig.machineconfiguration.openshift.io/foobar created = 3. Wait for kubeletconfig to propagate = $ oc get mcp/worker NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE worker rendered-worker-f67898c76cbf120517cfbf15afdbe494 False True False 3 0 0 0 149m $ watch oc get node $ oc get mcp/worker NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE worker rendered-worker-58a6175e3cfe5016a9d028abb2e3b689 True False False 3 3 3 0 157m = 4. check /etc/kubernetes/kubetletconfig for "maxPods":100 = $ oc get node NAME STATUS ROLES AGE VERSION ip-10-0-128-39.us-west-2.compute.internal Ready master 160m v1.17.1 ip-10-0-132-9.us-west-2.compute.internal Ready worker 150m v1.17.1 ip-10-0-145-124.us-west-2.compute.internal Ready worker 150m v1.17.1 ip-10-0-158-20.us-west-2.compute.internal Ready master 160m v1.17.1 ip-10-0-161-205.us-west-2.compute.internal Ready master 160m v1.17.1 ip-10-0-164-93.us-west-2.compute.internal Ready worker 151m v1.17.1 $ oc debug node/ip-10-0-132-9.us-west-2.compute.internal Starting pod/ip-10-0-132-9us-west-2computeinternal-debug ... To use host binaries, run `chroot /host` If you don't see a command prompt, try pressing enter. sh-4.2# chroot /host sh-4.4# cat /etc/kubernetes/kubelet.conf {"kind":"KubeletConfiguration","apiVersion":"kubelet.config.k8s.io/v1beta1","staticPodPath":"/etc/kubernetes/manifests","syncFrequency":"0s","fileCheckFrequency":"0s","httpCheckFrequency":"0s","rotateCertificates":true,"serverTLSBootstrap":true,"authentication":{"x509":{"clientCAFile":"/etc/kubernetes/kubelet-ca.crt"},"webhook":{"cacheTTL":"0s"},"anonymous":{"enabled":false}},"authorization":{"webhook":{"cacheAuthorizedTTL":"0s","cacheUnauthorizedTTL":"0s"}},"clusterDomain":"cluster.local","clusterDNS":["172.30.0.10"],"streamingConnectionIdleTimeout":"0s","nodeStatusUpdateFrequency":"0s","nodeStatusReportFrequency":"0s","imageMinimumGCAge":"0s","volumeStatsAggPeriod":"0s","cgroupDriver":"systemd","cpuManagerReconcilePeriod":"0s","runtimeRequestTimeout":"0s","maxPods":100,"kubeAPIQPS":50,"kubeAPIBurst":100,"serializeImagePulls":false,"evictionPressureTransitionPeriod":"0s","featureGates":{"IPv6DualStack":true,"LegacyNodeRoleBehavior":false,"NodeDisruptionExclusion":true,"RotateKubeletServerCertificate":true,"SCTPSupport":true,"ServiceNodeExclusion":true,"SupportPodPidsLimit":true},"containerLogMaxSize":"50Mi","systemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}} = 5. I didn't check /etc/kubelernetes/kubelet.conf before so I edited to to see if the changes actually propagated. Changed maxpods to 200 = $ oc edit kubeletconfig foobar kubeletconfig.machineconfiguration.openshift.io/foobar edited $ oc get kubeletconfig foobar -o yaml apiVersion: machineconfiguration.openshift.io/v1 kind: KubeletConfig metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"machineconfiguration.openshift.io/v1","kind":"KubeletConfig","metadata":{"annotations":{},"name":"foobar"},"spec":{"kubeletConfig":{"apiVersion":"kubelet.config.k8s.io/v1","kind":"kubeletConfig","maxPods":100},"machineConfigPoolSelector":{"matchLabels":{"custom-kubelet":"small-pods"}}}} creationTimestamp: "2020-03-10T19:45:06Z" finalizers: - 0c31f5cd-1e77-4441-bb56-1d91a21bf0e4 - 15a3c369-beaf-4d85-a99b-e54aef38e0c2 generation: 2 name: foobar resourceVersion: "69519" selfLink: /apis/machineconfiguration.openshift.io/v1/kubeletconfigs/foobar uid: bbbd32fa-3e7b-417f-97b9-cd90b0d0f67e spec: kubeletConfig: apiVersion: kubelet.config.k8s.io/v1 kind: kubeletConfig maxPods: 200 <== machineConfigPoolSelector: matchLabels: custom-kubelet: small-pods = 6. Wait for kubeletconfig to propagate = $ oc get mc NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 00-master 961ea024d5fbdd2c3daed6a9a2d4e8069a2849de 2.2.0 178m 00-worker 961ea024d5fbdd2c3daed6a9a2d4e8069a2849de 2.2.0 178m 01-master-container-runtime 961ea024d5fbdd2c3daed6a9a2d4e8069a2849de 2.2.0 178m 01-master-kubelet 961ea024d5fbdd2c3daed6a9a2d4e8069a2849de 2.2.0 178m 01-worker-container-runtime 961ea024d5fbdd2c3daed6a9a2d4e8069a2849de 2.2.0 178m 01-worker-kubelet 961ea024d5fbdd2c3daed6a9a2d4e8069a2849de 2.2.0 178m 99-master-691675c6-709b-4119-9c3d-793f274c88fe-registries 961ea024d5fbdd2c3daed6a9a2d4e8069a2849de 2.2.0 178m 99-master-ssh 2.2.0 179m 99-worker-7517b173-22ca-4f5e-aac2-f1b4a6da40fe-kubelet 961ea024d5fbdd2c3daed6a9a2d4e8069a2849de 2.2.0 30m 99-worker-7517b173-22ca-4f5e-aac2-f1b4a6da40fe-registries 961ea024d5fbdd2c3daed6a9a2d4e8069a2849de 2.2.0 178m 99-worker-ssh 2.2.0 179m rendered-master-f93a612a63a36e8c16a93b14c212a9e4 961ea024d5fbdd2c3daed6a9a2d4e8069a2849de 2.2.0 178m rendered-worker-22440c70a99eec97e3c10914b4a05918 961ea024d5fbdd2c3daed6a9a2d4e8069a2849de 2.2.0 2s rendered-worker-58a6175e3cfe5016a9d028abb2e3b689 961ea024d5fbdd2c3daed6a9a2d4e8069a2849de 2.2.0 30m rendered-worker-f67898c76cbf120517cfbf15afdbe494 961ea024d5fbdd2c3daed6a9a2d4e8069a2849de 2.2.0 178m $ oc get mcp/worker NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE worker rendered-worker-58a6175e3cfe5016a9d028abb2e3b689 False True False 3 0 0 0 179m = 7. Check /etc/kubernetes/kubelet.conf has "maxPods":200 = $ oc get node NAME STATUS ROLES AGE VERSION ip-10-0-128-39.us-west-2.compute.internal Ready master 3h8m v1.17.1 ip-10-0-132-9.us-west-2.compute.internal Ready worker 178m v1.17.1 ip-10-0-145-124.us-west-2.compute.internal Ready worker 178m v1.17.1 ip-10-0-158-20.us-west-2.compute.internal Ready master 3h8m v1.17.1 ip-10-0-161-205.us-west-2.compute.internal Ready master 3h8m v1.17.1 ip-10-0-164-93.us-west-2.compute.internal Ready worker 178m v1.17.1 $ oc debug node/ip-10-0-132-9.us-west-2.compute.internal Starting pod/ip-10-0-132-9us-west-2computeinternal-debug ... To use host binaries, run `chroot /host` If you don't see a command prompt, try pressing enter. sh-4.2# chroot /host sh-4.4# cat /etc/kubernetes/kubelet.conf {"kind":"KubeletConfiguration","apiVersion":"kubelet.config.k8s.io/v1beta1","staticPodPath":"/etc/kubernetes/manifests","syncFrequency":"0s","fileCheckFrequency":"0s","httpCheckFrequency":"0s","rotateCertificates":true,"serverTLSBootstrap":true,"authentication":{"x509":{"clientCAFile":"/etc/kubernetes/kubelet-ca.crt"},"webhook":{"cacheTTL":"0s"},"anonymous":{"enabled":false}},"authorization":{"webhook":{"cacheAuthorizedTTL":"0s","cacheUnauthorizedTTL":"0s"}},"clusterDomain":"cluster.local","clusterDNS":["172.30.0.10"],"streamingConnectionIdleTimeout":"0s","nodeStatusUpdateFrequency":"0s","nodeStatusReportFrequency":"0s","imageMinimumGCAge":"0s","volumeStatsAggPeriod":"0s","cgroupDriver":"systemd","cpuManagerReconcilePeriod":"0s","runtimeRequestTimeout":"0s","maxPods":200,"kubeAPIQPS":50,"kubeAPIBurst":100,"serializeImagePulls":false,"evictionPressureTransitionPeriod":"0s","featureGates":{"IPv6DualStack":true,"LegacyNodeRoleBehavior":false,"NodeDisruptionExclusion":true,"RotateKubeletServerCertificate":true,"SCTPSupport":true,"ServiceNodeExclusion":true,"SupportPodPidsLimit":true},"containerLogMaxSize":"50Mi","systemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}}
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581