Bug 1820230

Summary: Changes in node-exporter daemonset are persistent and does not get revert back to default configuration
Product: OpenShift Container Platform Reporter: Aditya Deshpande <adeshpan>
Component: MonitoringAssignee: Simon Pasquier <spasquie>
Status: CLOSED NEXTRELEASE QA Contact: Junqi Zhao <juzhao>
Severity: low Docs Contact:
Priority: unspecified    
Version: 3.11.0CC: alegrand, anpicker, erooth, kakkoyun, lcosic, mloibl, pkrupa, rkshirsa, spasquie, surbania
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-04-10 12:13:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Aditya Deshpande 2020-04-02 14:45:06 UTC
Description of problem:

In OCP 3.11, one of the customer is following the procedure as mentioned below to edit the daemonset of node-exporter in openshift-monitoring project and deleted one cipher-suite value. 

# oc edit ds node-exporter

deleted value from tls-cipher-suites arguent mentioned in kube-rbac-proxy container.
~~~
- --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
~~~

Here, you can see the value TLS_RSA_WITH_AES_128_CBC_SHA256 is deleted.

After deletion and saving the changes, node-exporter pods were rolled out out with that configuration.

As observed, the node-exporter daemonset never changed and reconsiling to the default configuration never happened and the change made to daemonset is persistent which is strange.


Version-Release number of selected component (if applicable):
v3.11.170

Actual results:

Changes done in node-exporter daemonset becomes persistent which should not be the case.

Expected results:
Operator should revert back the default configuration of daemonset and the value which was deleted should reappear.

Additional info:
- Tried deleting the node-exporter daemonset and waited if the daemonset creates automatically but it did not get created automatically and brought up by `# oc create -f ds.yaml` from backup of daemonset yaml file which was taken.
- Attaching cluster-monitoring-operator pod logs while this activity happened.