Steps To Reproduce: 1. Create systemd dropin via a file 2. Realize that its suboptimal, convert into a dropin 3. Ignition will rewrite the file, but MCD's deleteStaleData would remove it file since its no longer in .Storage.Files 4. Machine would reboot and MCD would complain that file is not found. This is blocking OKD 4.5 -> 4.6 upgrade, as in 4.5 we placed kubelet MCO dropin for proxy setup via storage.files and in 4.6 its replaced by a unit dropin.
Verified on 4.7.0-0.nightly-2020-11-24-113830 $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.0-0.nightly-2020-11-24-113830 True False 128m Cluster version is 4.7.0-0.nightly-2020-11-24-113830 $ cat drop-in-file.yaml apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker name: drop-in-file spec: config: ignition: version: 3.1.0 storage: files: - contents: source: data:text/plain;charset=utf;base64,W1VuaXRdCg== filesystem: root mode: 0644 path: /etc/systemd/system/machine-config-daemon-firstboot.service.d/override.conf $ oc create -f drop-in-file.yaml machineconfig.machineconfiguration.openshift.io/drop-in-file created $ oc get mc NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 00-master 5b694bd01f4482c474a9ec85c1a07d60fd135a34 3.1.0 150m 00-worker 5b694bd01f4482c474a9ec85c1a07d60fd135a34 3.1.0 150m 01-master-container-runtime 5b694bd01f4482c474a9ec85c1a07d60fd135a34 3.1.0 150m 01-master-kubelet 5b694bd01f4482c474a9ec85c1a07d60fd135a34 3.1.0 150m 01-worker-container-runtime 5b694bd01f4482c474a9ec85c1a07d60fd135a34 3.1.0 150m 01-worker-kubelet 5b694bd01f4482c474a9ec85c1a07d60fd135a34 3.1.0 150m 99-master-generated-registries 5b694bd01f4482c474a9ec85c1a07d60fd135a34 3.1.0 150m 99-master-ssh 3.1.0 161m 99-worker-generated-registries 5b694bd01f4482c474a9ec85c1a07d60fd135a34 3.1.0 150m 99-worker-ssh 3.1.0 161m drop-in-file 3.1.0 3s rendered-master-06fc8e27c85447046c0bf7a773d9ab64 5b694bd01f4482c474a9ec85c1a07d60fd135a34 3.1.0 150m rendered-worker-4e48ec375c1e899ef0d4d8a9b6123ac1 5b694bd01f4482c474a9ec85c1a07d60fd135a34 3.1.0 32m rendered-worker-f28ffee0e4a09287edecc8431995df67 5b694bd01f4482c474a9ec85c1a07d60fd135a34 3.1.0 150m $ oc get mc NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 00-master 5b694bd01f4482c474a9ec85c1a07d60fd135a34 3.1.0 150m 00-worker 5b694bd01f4482c474a9ec85c1a07d60fd135a34 3.1.0 150m 01-master-container-runtime 5b694bd01f4482c474a9ec85c1a07d60fd135a34 3.1.0 150m 01-master-kubelet 5b694bd01f4482c474a9ec85c1a07d60fd135a34 3.1.0 150m 01-worker-container-runtime 5b694bd01f4482c474a9ec85c1a07d60fd135a34 3.1.0 150m 01-worker-kubelet 5b694bd01f4482c474a9ec85c1a07d60fd135a34 3.1.0 150m 99-master-generated-registries 5b694bd01f4482c474a9ec85c1a07d60fd135a34 3.1.0 150m 99-master-ssh 3.1.0 161m 99-worker-generated-registries 5b694bd01f4482c474a9ec85c1a07d60fd135a34 3.1.0 150m 99-worker-ssh 3.1.0 161m drop-in-file 3.1.0 8s rendered-master-06fc8e27c85447046c0bf7a773d9ab64 5b694bd01f4482c474a9ec85c1a07d60fd135a34 3.1.0 150m rendered-worker-40dfad2479e4cbc293f887a8296183f4 5b694bd01f4482c474a9ec85c1a07d60fd135a34 3.1.0 3s rendered-worker-4e48ec375c1e899ef0d4d8a9b6123ac1 5b694bd01f4482c474a9ec85c1a07d60fd135a34 3.1.0 32m rendered-worker-f28ffee0e4a09287edecc8431995df67 5b694bd01f4482c474a9ec85c1a07d60fd135a34 3.1.0 150m $ oc get mcp/worker NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE worker rendered-worker-40dfad2479e4cbc293f887a8296183f4 True False False 3 3 3 0 3h20m $ oc debug node/ip-10-0-134-48.us-west-2.compute.internal Starting pod/ip-10-0-134-48us-west-2computeinternal-debug ... To use host binaries, run `chroot /host` If you don't see a command prompt, try pressing enter. sh-4.2# chroot /host sh-4.4# cat /etc/systemd/system/machine-config-daemon-firstboot.service.d/override.conf [Unit] sh-4.4# exit exit sh-4.2# exit exit Removing debug pod ... $ oc edit mc/drop-in-file machineconfig.machineconfiguration.openshift.io/drop-in-file edited $ oc get mc/drop-in-file -o yaml apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: creationTimestamp: "2020-11-24T21:10:34Z" generation: 3 labels: machineconfiguration.openshift.io/role: worker managedFields: - apiVersion: machineconfiguration.openshift.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:labels: .: {} f:machineconfiguration.openshift.io/role: {} f:spec: .: {} f:config: .: {} f:ignition: .: {} f:version: {} f:systemd: .: {} f:units: {} manager: oc operation: Update time: "2020-11-24T22:32:00Z" name: drop-in-file resourceVersion: "94477" selfLink: /apis/machineconfiguration.openshift.io/v1/machineconfigs/drop-in-file uid: c79aa973-f417-48d3-9359-77c1ec0ac54f spec: config: ignition: version: 3.1.0 systemd: units: - dropins: - contents: | [Unit] name: override.conf enabled: true name: machine-config-daemon-firstboot.service $ oc get mcp/worker NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE worker rendered-worker-f28ffee0e4a09287edecc8431995df67 False True False 3 0 0 0 3h53m $ oc debug node/ip-10-0-134-48.us-west-2.compute.internal Starting pod/ip-10-0-134-48us-west-2computeinternal-debug ... To use host binaries, run `chroot /host` If you don't see a command prompt, try pressing enter. sh-4.2# chroot /host sh-4.4# cat /etc/systemd/system/machine-config-daemon-firstboot.service.d/override.conf [Unit] sh-4.4# exit exit sh-4.2#
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633