Bug 1671677

Summary: dynamic kubelet update not take effect for some master/worker
Product: OpenShift Container Platform Reporter: MinLi <minmli>
Component: NodeAssignee: Ryan Phillips <rphillips>
Status: CLOSED ERRATA QA Contact: Jianwei Hou <jhou>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.1.0CC: aos-bugs, jokerman, mmccomas
Target Milestone: ---   
Target Release: 4.1.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined
 Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-04 10:42:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description MinLi 2019-02-01 09:25:51 UTC 
Description of problem:
when dynamically update kubelet configurations via MCO, some master/node not take effect.

Version-Release number of selected component (if applicable):
$ oc version
oc v4.0.0-0.150.0
kubernetes v1.12.4+f39ab668d3
features: Basic-Auth GSSAPI Kerberos SPNEGO
Server https://qe-chuan-api.qe.devcluster.openshift.com:6443
kubernetes v1.12.4+f39ab668d3

$ oc get clusterversion 
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-01-30-174704   True        False         1d      Cluster version is 4.0.0-0.nightly-2019-01-30-174704

How reproducible:
always

Steps to Reproduce:
1.edit machineconfigpool master, add label"custom-kubelet: small-pods"
#oc edit machineconfigpool master
for example:
metadata:
  creationTimestamp: 2019-01-31T07:10:04Z
  generation: 3
  labels:
    custom-kubelet: small-pods(add this line)
    operator.machineconfiguration.openshift.io/required-for-upgrade: ""

2.create a kubeletconfig cr
#oc create -f master-kube-config.yaml
master-kube-config.yaml is like:
apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: set-max-pods
spec:
  machineConfigPoolSelector:
    matchLabels:
      custom-kubelet: small-pods
  kubeletConfig:
    maxPods: 101

3.check machineconfig 
#oc get machineconfig
NAME                                                     GENERATEDBYCONTROLLER   IGNITIONVERSION   CREATED   OSIMAGEURL
00-master                                                4.0.0-0.150.0.0-dirty   2.2.0             1d        
00-master-ssh                                            4.0.0-0.150.0.0-dirty                     1d        
00-worker                                                4.0.0-0.150.0.0-dirty   2.2.0             1d        
00-worker-ssh                                            4.0.0-0.150.0.0-dirty                     1d        
01-master-kubelet                                        4.0.0-0.150.0.0-dirty   2.2.0             1d        
01-worker-kubelet                                        4.0.0-0.150.0.0-dirty   2.2.0             1d        
99-master-3b8aeac4-2527-11e9-abad-061ed6739886-kubelet   4.0.0-0.150.0.0-dirty                     14m       
master-2b4f50b26bc51feb09185d5eb011b528                  4.0.0-0.150.0.0-dirty   2.2.0             14m       
master-2c4b02ec6ba2c1808e9eebfb21f73053                  4.0.0-0.150.0.0-dirty   2.2.0             1d        
worker-384651b9a54ad8a6bb455e90961e0492                  4.0.0-0.150.0.0-dirty   2.2.0             1d        

#oc get machineconfig 99-master-3b8aeac4-2527-11e9-abad-061ed6739886-kubelet -o yaml 
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  annotations:
    machineconfiguration.openshift.io/generated-by-controller-version: 4.0.0-0.150.0.0-dirty
  creationTimestamp: 2019-02-01T08:44:30Z
  generation: 1
  labels:
    machineconfiguration.openshift.io/role: master
  name: 99-master-3b8aeac4-2527-11e9-abad-061ed6739886-kubelet
  ownerReferences:
  - apiVersion: machineconfiguration.openshift.io/v1
    kind: KubeletConfig
    name: set-max-pods

4.login all masters of cluster(in my case it is 3), check /etc/kubernetes/kubelet.conf has the kubelet configuration "maxPods: 101"

Actual results:
1.edit machineconfigpool master succ.
2.create kubeletconfig succ.
3. "99-master-3b8aeac4-2527-11e9-abad-061ed6739886-kubelet" correspond to KubeletConfig named "set-max-pods"
4.only 1 master update, and the other 2 not.

Expected results:
4.all masters update.

Additional info:
The phenomenon is also the same when we edit machineconfigpool node.
Comment 1 Ryan Phillips 2019-02-01 16:53:12 UTC 
There is a patch that went in extremely close to when this bug was reported. The nodes (master in this case) would have been marked degraded. If you could try this again on a fresh installation that would be great!

[1]. https://github.com/openshift/machine-config-operator/pull/352
Comment 2 MinLi 2019-02-13 02:57:27 UTC 
verified!

version info:
[core@ip-10-0-171-251 ~]$ oc version 
oc v4.0.0-0.170.0
kubernetes v1.12.4+45dbe929fa
features: Basic-Auth GSSAPI Kerberos SPNEGO

[root@localhost lyman]# oc get clusterversion 
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE     STATUS
version   4.0.0-0.nightly-2019-02-12-005016   True        False         17h       Cluster version is 4.0.0-0.nightly-2019-02-12-005016
Comment 5 errata-xmlrpc 2019-06-04 10:42:30 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758