Hide Forgot
Description of problem: when dynamically update kubelet configurations via MCO, some master/node not take effect. Version-Release number of selected component (if applicable): $ oc version oc v4.0.0-0.150.0 kubernetes v1.12.4+f39ab668d3 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://qe-chuan-api.qe.devcluster.openshift.com:6443 kubernetes v1.12.4+f39ab668d3 $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-01-30-174704 True False 1d Cluster version is 4.0.0-0.nightly-2019-01-30-174704 How reproducible: always Steps to Reproduce: 1.edit machineconfigpool master, add label"custom-kubelet: small-pods" #oc edit machineconfigpool master for example: metadata: creationTimestamp: 2019-01-31T07:10:04Z generation: 3 labels: custom-kubelet: small-pods(add this line) operator.machineconfiguration.openshift.io/required-for-upgrade: "" 2.create a kubeletconfig cr #oc create -f master-kube-config.yaml master-kube-config.yaml is like: apiVersion: machineconfiguration.openshift.io/v1 kind: KubeletConfig metadata: name: set-max-pods spec: machineConfigPoolSelector: matchLabels: custom-kubelet: small-pods kubeletConfig: maxPods: 101 3.check machineconfig #oc get machineconfig NAME GENERATEDBYCONTROLLER IGNITIONVERSION CREATED OSIMAGEURL 00-master 4.0.0-0.150.0.0-dirty 2.2.0 1d 00-master-ssh 4.0.0-0.150.0.0-dirty 1d 00-worker 4.0.0-0.150.0.0-dirty 2.2.0 1d 00-worker-ssh 4.0.0-0.150.0.0-dirty 1d 01-master-kubelet 4.0.0-0.150.0.0-dirty 2.2.0 1d 01-worker-kubelet 4.0.0-0.150.0.0-dirty 2.2.0 1d 99-master-3b8aeac4-2527-11e9-abad-061ed6739886-kubelet 4.0.0-0.150.0.0-dirty 14m master-2b4f50b26bc51feb09185d5eb011b528 4.0.0-0.150.0.0-dirty 2.2.0 14m master-2c4b02ec6ba2c1808e9eebfb21f73053 4.0.0-0.150.0.0-dirty 2.2.0 1d worker-384651b9a54ad8a6bb455e90961e0492 4.0.0-0.150.0.0-dirty 2.2.0 1d #oc get machineconfig 99-master-3b8aeac4-2527-11e9-abad-061ed6739886-kubelet -o yaml apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: annotations: machineconfiguration.openshift.io/generated-by-controller-version: 4.0.0-0.150.0.0-dirty creationTimestamp: 2019-02-01T08:44:30Z generation: 1 labels: machineconfiguration.openshift.io/role: master name: 99-master-3b8aeac4-2527-11e9-abad-061ed6739886-kubelet ownerReferences: - apiVersion: machineconfiguration.openshift.io/v1 kind: KubeletConfig name: set-max-pods 4.login all masters of cluster(in my case it is 3), check /etc/kubernetes/kubelet.conf has the kubelet configuration "maxPods: 101" Actual results: 1.edit machineconfigpool master succ. 2.create kubeletconfig succ. 3. "99-master-3b8aeac4-2527-11e9-abad-061ed6739886-kubelet" correspond to KubeletConfig named "set-max-pods" 4.only 1 master update, and the other 2 not. Expected results: 4.all masters update. Additional info: The phenomenon is also the same when we edit machineconfigpool node.
There is a patch that went in extremely close to when this bug was reported. The nodes (master in this case) would have been marked degraded. If you could try this again on a fresh installation that would be great! [1]. https://github.com/openshift/machine-config-operator/pull/352
verified! version info: [core@ip-10-0-171-251 ~]$ oc version oc v4.0.0-0.170.0 kubernetes v1.12.4+45dbe929fa features: Basic-Auth GSSAPI Kerberos SPNEGO [root@localhost lyman]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-02-12-005016 True False 17h Cluster version is 4.0.0-0.nightly-2019-02-12-005016
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0758