Bug 2070930

Description of problem:

When a proxy is removed from cluster-wide proxy settings, the corresponding environment variables are not removed from the machine-config-daemon daemonset.

Version-Release number of MCO (Machine Config Operator) (if applicable):

4.8.17

Platform (AWS, VSphere, Metal, etc.): 

Irrelevant (reproducible in many platforms)

Are you certain that the root cause of the issue being reported is the MCO (Machine Config Operator)?
(Y/N/Not sure):

Of course

How reproducible:

Always

Did you catch this issue by running a Jenkins job? If yes, please list:

Reproduced manually

Steps to Reproduce:
1. Install a cluster with proxy or configure a proxy in the cluster
2. Ensure that the proxy settings are applied and machine-config-daemon (the variables are added correctly if needed, that's not part of the problem)
3. Remove the proxy from cluster-wide proxy settings (so that the cluster no longer uses a proxy)
4. Wait until all the components are re-configured, including MCO

Actual results:

Proxy settings still in machine-config-daemon environment variables

Expected results:

machine-config-daemon daemonset properly re-configured

Additional info:

I'll privately attach an example must-gather and audit logs, but you should be able to easily reproduce the issue by following the "steps to reproduce" in any cluster.

Some relevant observations:
- The `proxy:` field at the `controllerconfig` object IS properly set to `null` as one would expect.
- Other MCO components seem to be properly re-configured (or, at least, don't contain the proxy after removed)
- However, audit logs confirm that machine-config-operator posts updates to the machine-config-daemon daemonset object that include the proxy environment variables even after proxy removal and after proxy field on controllerconfig is already set to null

So this looks like a problem in the logic used to merge observed and desired state for the machine-config-daemon daemonset, such that it does add the proxy variables when needed but doesn't remove them when needed.

A valid and tested workaround is to delete the daemonset, because MCO then re-creates it correctly.