Bug 1686356

Summary: when delete a kubeletconfig ,return time is too long(nearly 10 minutes)
Product: OpenShift Container Platform Reporter: MinLi <minmli>
Component: NodeAssignee: Ryan Phillips <rphillips>
Status: CLOSED ERRATA QA Contact: Sunil Choudhary <schoudha>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.1.0CC: aos-bugs, jokerman, mmccomas, schoudha, sjenning
Target Milestone: ---   
Target Release: 4.1.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-04 10:45:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description MinLi 2019-03-07 10:18:19 UTC
Description of problem:
when delete a kubeletconfig ,return time is too long(nearly 10 minutes)

Version-Release number of selected component (if applicable):
4.0.0-0.nightly-2019-03-04-234414
oc v4.0.0-0.182.0

How reproducible:
always

Steps to Reproduce:
1.edit machineconfigpool worker, add label"custom-kubelet: max-pods-worker"
#oc edit machineconfigpool worker
for example:
metadata:
  creationTimestamp: 2019-03-07T07:10:04Z
  generation: 1
  labels:
    custom-kubelet: max-pods-worker(add this line)
  ...

2.create a kubeletconfig cr
oc create -f worker-kube-config.yaml
yaml file as:
apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: set-max-pods-worker
spec:
  machineConfigPoolSelector:
    matchLabels:
      custom-kubelet: max-pods-worker
  kubeletConfig:
    maxPods: 249

3.oc get machineconfig
generate a new machineconfig named "99-worker-74b4fa0b-4091-11e9-96ec-067bf345463e-kubelet"

4.login all worker node, check /etc/kubernetes/kubelet.conf has the kubelet configuration "maxPods: 249"

5.oc delete kubeletconfig set-max-pods-worker


Actual results:
5.show message "kubeletconfig.machineconfiguration.openshift.io "set-max-pods-worker" deleted", and return time of this command is nearly 10 minutes.

Expected results:
show message "kubeletconfig.machineconfiguration.openshift.io "set-max-pods-worker" deleted", and return immediately


Additional info:
when take step 5, check machineconfig in another terminal,the 99-worker-74b4fa0b-4091-11e9-96ec-067bf345463e-kubelet disappear immediately.


#oc logs machine-config-controller-8cc654f5d-rlckl -n openshift-machine-config-operator | grep set-max-pods-worker
error log:
I0307 09:45:50.893105       1 kubelet_config_controller.go:412] Applied KubeletConfig set-max-pods-worker on MachineConfigPool worker
I0307 09:45:51.048254       1 kubelet_config_controller.go:245] Error syncing kubeletconfig set-max-pods-worker: Operation cannot be fulfilled on kubeletconfigs.machineconfiguration.openshift.io "set-max-pods-worker": the object has been modified; please apply your changes to the latest version and try again
I0307 09:45:51.698293       1 kubelet_config_controller.go:412] Applied KubeletConfig set-max-pods-worker on MachineConfigPool worker
E0307 09:45:51.859637       1 kubelet_config_controller.go:250] Operation cannot be fulfilled on kubeletconfigs.machineconfiguration.openshift.io "set-max-pods-worker": the object has been modified; please apply your changes to the latest version and try again
I0307 09:45:51.859670       1 kubelet_config_controller.go:251] Dropping kubeletconfig "set-max-pods-worker" out of the queue: Operation cannot be fulfilled on kubeletconfigs.machineconfiguration.openshift.io "set-max-pods-worker": the object has been modified; please apply your changes to the latest version and try again
I0307 09:45:52.500112       1 kubelet_config_controller.go:412] Applied KubeletConfig set-max-pods-worker on MachineConfigPool worker
I0307 09:45:52.648116       1 kubelet_config_controller.go:245] Error syncing kubeletconfig set-max-pods-worker: Operation cannot be fulfilled on kubeletconfigs.machineconfiguration.openshift.io "set-max-pods-worker": the object has been modified; please apply your changes to the latest version and try again
I0307 09:45:53.292696       1 kubelet_config_controller.go:412] Applied KubeletConfig set-max-pods-worker on MachineConfigPool worker
I0307 09:45:53.455527       1 kubelet_config_controller.go:245] Error syncing kubeletconfig set-max-pods-worker: Operation cannot be fulfilled on kubeletconfigs.machineconfiguration.openshift.io "set-max-pods-worker": the object has been modified; please apply your changes to the latest version and try again

Comment 2 Ryan Phillips 2019-03-07 22:58:47 UTC
MinLi: Are you apply this configuration to the master nodes? The master nodes run the Machine Config Controller and would likely take some time to come back and reconcile the deletion.

Comment 3 Ryan Phillips 2019-04-01 14:28:31 UTC
PR merged. https://github.com/openshift/machine-config-operator/pull/536

Comment 7 errata-xmlrpc 2019-06-04 10:45:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758