Bug 1895360

Summary:	Machine Config Daemon removes a file although its defined in the dropin
Product:	OpenShift Container Platform	Reporter:	Vadim Rutkovsky <vrutkovs>
Component:	Machine Config Operator	Assignee:	Antonio Murdaca <amurdaca>
Status:	CLOSED ERRATA	QA Contact:	Michael Nguyen <mnguyen>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.7
Target Milestone:	---
Target Release:	4.7.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-02-24 15:31:26 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1899735

Description Vadim Rutkovsky 2020-11-06 13:23:29 UTC

Steps To Reproduce:

1. Create systemd dropin via a file
2.  Realize that its suboptimal, convert into a dropin
3.  Ignition will rewrite the file, but MCD's deleteStaleData would remove it file since its no longer in .Storage.Files
4.  Machine would reboot and MCD would complain that file is not found.

This is blocking OKD 4.5 -> 4.6 upgrade, as in 4.5 we placed kubelet MCO dropin for proxy setup via storage.files and in 4.6 its replaced by a unit dropin.

Comment 2 Michael Nguyen 2020-11-24 22:46:07 UTC

Verified on 4.7.0-0.nightly-2020-11-24-113830



$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2020-11-24-113830   True        False         128m    Cluster version is 4.7.0-0.nightly-2020-11-24-113830
$ cat drop-in-file.yaml 
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: drop-in-file
spec:
  config:
    ignition:
      version: 3.1.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf;base64,W1VuaXRdCg==
        filesystem: root
        mode: 0644
        path: /etc/systemd/system/machine-config-daemon-firstboot.service.d/override.conf
$ oc create -f drop-in-file.yaml 
machineconfig.machineconfiguration.openshift.io/drop-in-file created
$ oc get mc
NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
00-worker                                          5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
01-master-container-runtime                        5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
01-master-kubelet                                  5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
01-worker-container-runtime                        5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
01-worker-kubelet                                  5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
99-master-generated-registries                     5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
99-master-ssh                                                                                 3.1.0             161m
99-worker-generated-registries                     5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
99-worker-ssh                                                                                 3.1.0             161m
drop-in-file                                                                                  3.1.0             3s
rendered-master-06fc8e27c85447046c0bf7a773d9ab64   5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
rendered-worker-4e48ec375c1e899ef0d4d8a9b6123ac1   5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             32m
rendered-worker-f28ffee0e4a09287edecc8431995df67   5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
$ oc get mc
NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
00-worker                                          5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
01-master-container-runtime                        5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
01-master-kubelet                                  5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
01-worker-container-runtime                        5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
01-worker-kubelet                                  5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
99-master-generated-registries                     5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
99-master-ssh                                                                                 3.1.0             161m
99-worker-generated-registries                     5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
99-worker-ssh                                                                                 3.1.0             161m
drop-in-file                                                                                  3.1.0             8s
rendered-master-06fc8e27c85447046c0bf7a773d9ab64   5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
rendered-worker-40dfad2479e4cbc293f887a8296183f4   5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             3s
rendered-worker-4e48ec375c1e899ef0d4d8a9b6123ac1   5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             32m
rendered-worker-f28ffee0e4a09287edecc8431995df67   5b694bd01f4482c474a9ec85c1a07d60fd135a34   3.1.0             150m
$ oc get mcp/worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
worker   rendered-worker-40dfad2479e4cbc293f887a8296183f4   True      False      False      3              3                   3                     0                      3h20m
$ oc debug node/ip-10-0-134-48.us-west-2.compute.internal
Starting pod/ip-10-0-134-48us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# cat /etc/systemd/system/machine-config-daemon-firstboot.service.d/override.conf 
[Unit]
sh-4.4# exit
exit
sh-4.2# exit 
exit

Removing debug pod ...
$ oc edit mc/drop-in-file
machineconfig.machineconfiguration.openshift.io/drop-in-file edited
$ oc get mc/drop-in-file -o yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  creationTimestamp: "2020-11-24T21:10:34Z"
  generation: 3
  labels:
    machineconfiguration.openshift.io/role: worker
  managedFields:
  - apiVersion: machineconfiguration.openshift.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .: {}
          f:machineconfiguration.openshift.io/role: {}
      f:spec:
        .: {}
        f:config:
          .: {}
          f:ignition:
            .: {}
            f:version: {}
          f:systemd:
            .: {}
            f:units: {}
    manager: oc
    operation: Update
    time: "2020-11-24T22:32:00Z"
  name: drop-in-file
  resourceVersion: "94477"
  selfLink: /apis/machineconfiguration.openshift.io/v1/machineconfigs/drop-in-file
  uid: c79aa973-f417-48d3-9359-77c1ec0ac54f
spec:
  config:
    ignition:
      version: 3.1.0
    systemd:
      units:
      - dropins:
        - contents: |
            [Unit]
          name: override.conf
        enabled: true
        name: machine-config-daemon-firstboot.service
$ oc get mcp/worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
worker   rendered-worker-f28ffee0e4a09287edecc8431995df67   False     True       False      3              0                   0                     0                      3h53m
$ oc debug node/ip-10-0-134-48.us-west-2.compute.internal
Starting pod/ip-10-0-134-48us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# cat /etc/systemd/system/machine-config-daemon-firstboot.service.d/override.conf 
[Unit]
sh-4.4# exit
exit
sh-4.2#

Comment 5 errata-xmlrpc 2021-02-24 15:31:26 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633